Model Routing

OpenRoute provides two model routing options.

Auto Router

The Auto Router is a special model ID that you can use to select between chosen high-quality models based on your prompt.

{
  "model": "openroute/auto",
  ... // other parameters
}

The generated response will set model to the actually used model.

models Parameter

The models parameter allows you to automatically try other models when your primary model's provider is down, rate-limited, or refuses to respond due to content moderation.

{
  "models": ["anthropic/claude-3.5-sonnet", "gryphe/mythomax-l2-13b"],
  ... // other parameters
}

If your chosen model returns an error, OpenRoute will try using the fallback models. If the fallback models are down or return errors, OpenRoute will return that error.

By default, any error may trigger using fallback models, including context length validation errors, content moderation flags from filtered models, rate limits, and downtime.

Requests are priced using the finally used model, which will be returned in the model property of the response body.

Using with OpenAI SDK

To use the models array with the OpenAI SDK, include it in the extra_body parameter. In the example below, gpt-4o will be tried first, then the models array will be tried in order as fallbacks.

import OpenAI from 'openai';

const openrouteClient = new OpenAI({
  baseURL: 'https://www.openroute.cn/api/v1',
  // API key and headers
});

async function main() {
  // @ts-expect-error
  const completion = await openrouteClient.chat.completions.create({
    model: 'openai/gpt-4o',
    models: ['anthropic/claude-3.5-sonnet', 'gryphe/mythomax-l2-13b'],
    messages: [
      {
        role: 'user',
        content: 'What is the meaning of life?',
      },
    ],
  });
  console.log(completion.choices[0].message);
}

main();

from openai import OpenAI

openai_client = OpenAI(
  base_url="https://www.openroute.cn/api/v1",
  api_key="<OPENROUTE_API_KEY>",
)

completion = openai_client.chat.completions.create(
    model="openai/gpt-4o",
    extra_body={
        "models": ["anthropic/claude-3.5-sonnet", "gryphe/mythomax-l2-13b"],
    },
    messages=[
        {
            "role": "user",
            "content": "What is the meaning of life?"
        }
    ]
)

print(completion.choices[0].message.content)