Skip to main content
Modulo

Variant A/B Testing

A/B testing answers one question: which pipeline configuration produces the better result for the same input? Modulo answers it by running variants of the same pipeline and keeping the per-run record that makes a comparison possible.

Reading a comparison

The comparison draws on the runs recorded against each group and covers:

  • Evaluation scores per variant (see Evals)
  • Token cost per run and per variant (see Cost Management)
  • Human decisions at human-in-the-loop (HITL) gates, with reviewer and timestamp
  • Output differences between variants for the same input
  • Prompt version hashes, so a result can be traced to the prompt that produced it

Comparing fairly

Scores are only comparable when each variant exercised the same checks. Check coverage gaps before drawing a conclusion: a variant that did not trigger a given eval has one fewer way to fail, so its scores are not directly comparable to a variant that did. See Variant Execution for how gaps are reported.

Next: Variant Groups