AnythingLLM
All-in-one private AI assistant and RAG platform for documents and knowledge bases
Introduction
AnythingLLM is a private AI assistant platform that supports RAG, agents, and multi-user workspaces. Its Generic OpenAI LLM provider lets you connect to any OpenAI-compatible endpoint — including TokenMix — so you can use Claude, GPT-5, or Qwen without leaving your private infrastructure. You'll also configure a Generic OpenAI Embedder so AnythingLLM can index documents for RAG.
Prerequisites
Self-host AnythingLLM with Docker:
docker pull mintplexlabs/anythingllm
docker run -d -p 3001:3001 \
-v anythingllm_storage:/app/server/storage \
--name anythingllm \
mintplexlabs/anythingllm
Access AnythingLLM at http://localhost:3001 and complete the onboarding wizard.
You also need a TokenMix API key. Sign up at tokenmix.ai → Console → API Keys and create a new key.
Setup Steps
Step 1 — Open LLM settings
In AnythingLLM, click the Settings gear icon (bottom-left) → AI Providers → LLM.
Step 2 — Select Generic OpenAI as the LLM provider
In the provider list, select Generic OpenAI.
Step 3 — Fill in the LLM connection details
| Field | Value |
|---|---|
| Base URL | https://api.tokenmix.ai/v1 |
| API Key | Your TokenMix key (sk-tm-…) |
| Model Name | e.g. claude-sonnet-4.6 |
| Token context window | Set per model (e.g. 200000 for Claude Sonnet) |
Click Save changes.
Step 4 — Configure the Embedding provider
Still in Settings → AI Providers, click Embedder.
Select Generic OpenAI as the embedder, then fill in:
| Field | Value |
|---|---|
| Base URL | https://api.tokenmix.ai/v1 |
| API Key | Your TokenMix key (sk-tm-…) |
| Model Name | e.g. text-embedding-3-large |
Click Save changes.
Step 5 — Create a workspace and upload documents
Go back to the main screen, create a new Workspace, and upload your documents (PDF, Word, TXT, etc.). AnythingLLM will embed them using the model you configured in Step 4.
Once embedding is complete, chat with your documents using the LLM you configured in Step 3.
Verify
Send a message in your workspace. A valid response referencing your document content confirms both LLM and embedder are working through TokenMix.
Check Console → Usage on TokenMix to see both chat and embedding requests logged.
FAQ
Q: After saving, AnythingLLM shows "Could not connect to LLM provider".
A: Verify the Base URL is exactly https://api.tokenmix.ai/v1 (no trailing slash) and the API key starts with sk-tm-. Also confirm your TokenMix account has a positive balance.
Q: Which embedding models does TokenMix offer?
A: Visit tokenmix.ai → Models and filter by type Embedding. Copy the short_id (e.g. text-embedding-3-large) into the Embedder's Model Name field. The embedding model must be different from the chat model.
Q: Can I change the LLM model per workspace? A: The global LLM setting applies to all workspaces by default. In AnythingLLM Pro/Cloud, per-workspace model overrides are available; in the open-source version, change the global setting to switch models.