Swap a hosted model API for a local one without touching the client

A lot of AI plumbing is hard-wired to one provider's API shape. Retrieval pipelines call Cohere's /v2/rerank. Browser extensions and SDKs call OpenAI's /v1/audio/speech. If you want to run the model yourself, for cost, privacy or latency, the cheapest migration is not a new client. It is a server that speaks the old API, so you change one base URL and nothing else.

I have built this twice: a Qwen3 reranker behind a Cohere-compatible endpoint, and a Kokoro text-to-speech server behind an OpenAI-compatible one. This post is about the pattern and the details that make it work.

The pattern

before:  your pipeline ──► api.cohere.com/v2/rerank
after:   your pipeline ──► localhost:8888/v2/rerank ──► Qwen3-Reranker-0.6B (GPU)
                             same request, same response

The rules I follow:

1. Copy the contract exactly: path, request fields, response shape, field names, even the metadata clients might read.
2. Accept fields you ignore. If the client sends "model": "rerank-v3.5", accept it. Rejecting it breaks the drop-in promise.
3. Ship tests that exercise the contract, not just the model.

A Cohere-compatible reranker

A reranker takes a query and a list of candidate documents from first-stage retrieval and scores how relevant each one is. It is a cross-encoder: it reads the query and document together, which makes it more accurate than comparing embeddings, at the cost of one forward pass per pair.

The request is the one a Cohere client already sends:

curl -X POST http://localhost:8888/v2/rerank \
  -H "Content-Type: application/json" \
  -d '{
    "model": "rerank-v3.5",
    "query": "What is the capital of the United States?",
    "top_n": 3,
    "documents": [
      "Washington, D.C. is the capital of the United States.",
      "New York City is the largest city in the United States.",
      "Los Angeles is a major city on the west coast."
    ]
  }'

The response mirrors Cohere's, including the meta block:

{
  "results": [
    {"index": 0, "relevance_score": 0.9876},
    {"index": 1, "relevance_score": 0.5432}
  ],
  "id": "uuid-here",
  "meta": {
    "api_version": {"version": "2", "is_experimental": false},
    "billed_units": {"search_units": 1}
  }
}

billed_units means nothing on a local server. It is there because some client code reads it, and a missing key is a crash in someone else's pipeline.

The model is not a classifier

The non-obvious part was how Qwen3-Reranker-0.6B produces a score. It is a causal language model, not a sequence-classification model with a score head. Loading it with AutoModelForSequenceClassification fails with IndexError: too many indices for tensor. The correct approach, following the model card:

1. Format the query and document with a system prompt that asks a yes/no relevance question.
2. Run the model and take the logits at the final position: model(**inputs).logits[:, -1, :].
3. Pull out the logits for the "yes" and "no" tokens.
4. Apply log-softmax over those two and return the probability of "yes" as the relevance score.

logits = model(**inputs).logits[:, -1, :]
pair = torch.stack([logits[:, no_id], logits[:, yes_id]], dim=1)
scores = torch.log_softmax(pair, dim=1)[:, 1].exp()

Padding for batches

Scoring several documents at once needs padding, and the tokenizer ships without a pad token, so batches larger than one fail with "Cannot handle batch sizes > 1 if no padding token is defined". The fix: set the pad token to <|endoftext|> (ID 151643), pass pad_token_id to the model, request attention masks, and pad on the left. Left padding matters for a causal model because you read the logits at the last position. With right padding, the last position for shorter inputs is a pad token.

The service uses CUDA with FP16 when available and a max length of 8192 tokens, and comes with bash and Python test scripts that hit the endpoint the way a client would.

An OpenAI-compatible TTS server

The same idea applies to speech. The Kokoro TTS API exposes /v1/audio/speech, so anything that can call OpenAI's TTS can call it instead:

curl -X POST http://localhost:5000/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{"input": "Hello, world!", "voice": "af_heart"}'

The payoff is clients I did not write. The Read Aloud Chrome extension supports OpenAI TTS. Point it at http://127.0.0.1:5000/v1 and it reads any web page aloud with 28 local voices (20 US English, 8 British English), mapped as OpenAI-style voice entries:

{ "lang": "en-GB", "model": "tts-1", "voice": "bf_emma" }

That is the accessibility goal behind the project: free, local, natural-sounding read-aloud for anyone, including dyslexic users, without a subscription.

Compatibility is the floor, not the ceiling. Alongside the standard endpoint the server adds /v1/audio/speech/robust, with heavier text processing (markdown, Unicode, numbers, abbreviations), /v1/audio/speech/stream for streaming, and playback controls (/play, /stop, /status) for local use. Extra endpoints are fine as long as the standard path behaves exactly as the client expects.

Where compatibility leaks

• Unknown fields. Clients send parameters you do not support. Accept and ignore them unless ignoring them would silently produce wrong results.
• Error shapes. Clients parse error bodies too. Match the status codes and JSON structure, not just the happy path.
• Browser clients need CORS. Both servers enable it, because extensions and web apps call them directly.
• Semantics, not just syntax. A relevance score from a different model is not on the same scale as Cohere's. If downstream code applies a hard threshold like score > 0.5, re-tune it after the swap.

Why bother

A compatible surface turns "migrate to a self-hosted model" from a project into a config change, makes it easy to A/B a local model against the hosted one, and lets tools you did not write use your models. Both projects are small, single-file-ish services, and that is the point: the hard part is the details of the contract and the model, not the amount of code.

Code: github.com/nodeblackbox/qwen3rerankerapicoherestyle and github.com/nodeblackbox/Kokoro-Voice-Api.