Skip to main content

LLM Model Configuration

LLM (Large Language Model) settings control the AI model used for RAG chat responses in the Playground.

Available Providers​

ProviderModel catalogNotes
OpenAIModels available to the configured OpenAI projectIncludes compatible fine-tuned chat models
AnthropicModels available to the configured Anthropic accountIncludes current context and output-token limits when provided
OpenRouterText models your OpenRouter key may use, after your OpenRouter provider and privacy settingsOne key reaches many hosted models; context and output limits come from OpenRouter
Hugging FaceChat models available through Hugging Face Inference ProvidersNeeds a fine-grained token with "Make calls to Inference Providers" permission

Configuration Options​

  1. Provider: Select OpenAI, Anthropic, OpenRouter or Hugging Face
  2. API Key: Enter your provider's API key
  3. Model: Choose the specific LLM model
  4. Temperature: Adjust creativity (0.0 = focused, 2.0 = creative)
  5. Max Tokens: Limit response length

API Key Management​

Each provider requires its own API key. Keys saved here are used only for chat and Enhanced sync; embeddings use the separate key saved on the Embedding Model page, even for the same provider. Saving or removing a key on one page never changes the other.

  • OpenAI: Get your key from platform.openai.com
  • Anthropic: Get your key from console.anthropic.com
  • OpenRouter: Create a key at openrouter.ai/settings/keys (keys start with sk-or-)
  • Hugging Face: Create a fine-grained token at huggingface.co/settings/tokens with the Make calls to Inference Providers permission (tokens start with hf_). Saving checks that the token is valid; if it lacks that permission, chat reports that Hugging Face refused the request and names the missing permission.

Both providers bill model usage to your own account with them. If your OpenRouter credits run out, or a Hugging Face request goes beyond your account's included credits or plan, chat says the provider needs credits or a paid plan. Add credits or upgrade with the provider, then retry. Your key and model selection stay valid.

Where your content goes​

Chat sends your question and the retrieved document text to the provider you choose, under your own key. With OpenRouter, OpenRouter forwards each request to a model host that your OpenRouter account allows; you control those hosts in your OpenRouter privacy and provider settings. With Hugging Face, Hugging Face forwards each request to one of its inference providers for the chosen model. The settings page shows this notice when you select either provider.

The settings page shows a status banner indicating whether your API key is configured:

  • Green checkmark: API key is configured and ready
  • Yellow warning: API key required - enter your key to enable chat

API-key format and provider authentication are checked before saving. If the provider's model service is temporarily unavailable, the key is saved with a warning so you can retry model discovery later. Keys and provider responses are handled on the server and are not returned to the browser.

Model Discovery​

  • After you configure a key, the page loads compatible chat models available to that provider account.
  • Use Refresh after an account or model-access change. Results are cached briefly to avoid unnecessary provider calls.
  • If live discovery is unavailable, the page clearly labels and displays a small built-in fallback catalog.
  • A saved model that is no longer available remains visible with an Unavailable label. Select an available replacement before saving.
  • OpenAI image, audio, realtime, embedding, moderation, deep-research, and other specialist models are excluded because RAG chat uses the Chat Completions contract.
  • Controls adapt to model metadata. Temperature is disabled for model families that manage sampling internally, and response tokens are capped at the provider limit when one is reported.
  • OpenAI does not report token limits in its Models API. Known model families use maintained limits; unknown compatible models use a conservative response limit until their metadata is added.
  • OpenRouter lists only text models your key may use. Temperature is offered only for models that accept it.
  • Hugging Face can route a model to any of several hosts, so the context window shown is the smallest one among them.
  • When a provider returns more than 30 models, a Filter models box above the model list narrows it by name or ID.

Temperature Guide​

TemperatureBehaviorBest For
0.0 - 0.3Focused, deterministicFactual Q&A, technical docs
0.4 - 0.7BalancedGeneral use (default: 0.7)
0.8 - 1.2Creative, variedContent generation
1.3 - 2.0Highly creativeBrainstorming, exploration