Skip to main content
Modulo

Variant Execution (A/B Testing)

A variant group is only useful if runs actually spread across its variants. Variant execution is the mechanism that chooses a variant for each run, applies that variant’s overrides, and reports what each variant did and did not exercise.

  • Variant: a named configuration override applied to a pipeline run, typically differing in prompt, model backend, or parameters
  • Weights: each variant has a weight controlling its selection probability
  • Quota enforcement: concurrent run limits per pipeline apply regardless of variant selection
  • Coverage gaps: detects which eval definitions are not covered by each variant

Variant groups are created, managed, and executed through the Variant Groups API. Results can be compared side by side in the Variant A/B Testing view.

How selection works

Runs go through the variant-group run endpoint. The endpoint picks one variant by weighted random selection and applies that variant’s run_context_overrides to the run. The pipeline definition itself is not changed, so the same pipeline can serve every variant. Because the overrides apply per run, two runs of one pipeline can differ in prompt version, model backend, or parameters without a second pipeline definition.

Coverage gaps

Coverage gaps are reported per variant, so you can see which eval definitions each variant actually exercises. The coverage-gaps endpoint on a variant group lists them; see Variant Groups for the endpoint and Evals for the checks behind it.

Next: Contribution Provenance