Skip to main content
Modulo

Model Backends

Model backends provide a common interface for routing LLM calls across different providers. Each agent in a pipeline selects a model backend by name.

Supported backends

Anthropic

Pipeline config 
{
  "type": "anthropic",
  "config": {
    "api_key": "sk-ant-...",
    "default_model": "claude-sonnet-4-20250514"
  }
}

Supports all Claude models via the Messages API. Recommended for structured pipeline tasks and agent reasoning.

OpenAI

Pipeline config 
{
  "type": "openai",
  "config": {
    "api_key": "sk-proj-...",
    "default_model": "gpt-4o"
  }
}

Supports GPT-4o, GPT-4o-mini, o-series reasoning models. Good for code generation and classification tasks.

Local models

Connect to any OpenAI-compatible local endpoint:

Pipeline config 
{
  "type": "openai",
  "config": {
    "base_url": "http://localhost:11434/v1",
    "api_key": "ollama",
    "default_model": "llama3"
  }
}

Works with Ollama, vLLM, LocalAI, and any OpenAI-compatible server.

Stub (for testing)

A deterministic mock backend that returns configurable preset responses. Use in unit tests and CI only, never in production.

There is no standalone stub provider. The test backend registers under the custom provider and is not listed in the UI’s provider dropdown; create it over the API with "provider": "custom" (see Getting Started).

Health checks

Modulo health-checks each backend on a configurable interval and automatically rotates unhealthy ones out of the pool. The status dashboard shows live health per backend.

Per-node selection

Each agent node in a pipeline can specify a model_backend to override the pipeline default. This lets you route different steps to different providers.

Failover

When a model backend is unhealthy, Modulo automatically fails over to a configured fallback backend. Set fallback_backend_ids on any backend to define the failover order. If the primary and all fallbacks are unhealthy, the run fails with a BackendUnavailableError.

Failover emits a model_failover audit event with the primary and fallback backend IDs for observability.

Next: Agents