Skip to main content

Model Routing

Conduit's model routing system allows you to define how requests are directed to different LLM providers based on a variety of criteria and strategies.

Model Mappings​

The core of Conduit's routing system is the model mapping table. Each mapping consists of:

  • Virtual Model Name - The name clients use in their requests
  • Provider Model - The actual provider-specific model name
  • Provider - Which LLM provider to use
  • Priority - A numerical value used for routing decisions
  • Weight - Optional value for weighted routing strategies

For example, you might map the virtual model name my-gpt4 to OpenAI's gpt-4 or Anthropic's claude-3-opus-20240229.

Routing Strategies​

Conduit offers several strategies for routing requests:

StrategyDescription
SimpleUses the first available mapping for a requested model
PriorityUses the mapping with the highest priority
Least CostRoutes to the provider with the lowest cost
Round RobinDistributes requests evenly across providers
RandomRandomly selects among available providers
Least UsedFavors providers with fewer recent requests
Least LatencyRoutes to the provider with the lowest recent latency

Configuring Routing​

Routing can be configured through the Web UI:

  1. Navigate to Configuration > Model Mappings to define mappings
  2. Go to Configuration > Routing to set the routing strategy
  3. Configure additional parameters like fallbacks and health checks

Fallback Configuration​

Fallbacks allow you to automatically redirect requests when a provider is unavailable:

  1. Navigate to Configuration > Routing > Fallbacks
  2. Define fallback rules with primary and backup providers
  3. Set conditions like timeout thresholds or error codes

Example fallback rule:

  • Primary: gpt-4 on OpenAI
  • Fallback: claude-3-opus on Anthropic
  • Condition: If OpenAI returns a 429 (rate limit) error

Advanced Routing Features​

Health Checks​

Conduit can perform health checks to detect provider issues:

  1. Navigate to Configuration > Provider Health
  2. Configure how often to check provider availability
  3. Set automatic fallback behavior

Context-Aware Routing​

You can implement custom routing logic using the API:

{
  "model": "my-gpt4",
  "routing": {
    "strategy": "least_cost",
    "fallback_enabled": true
  },
  "messages": [{"role": "user", "content": "Hello!"}]
}

Best Practices​

  • Define clear priorities for different providers
  • Set up fallbacks for critical models
  • Monitor provider usage and adjust routing as needed
  • Use cost-based routing for budget optimization
  • Implement health checks for improved reliability

Next Steps​