Review an experiment that produced mixed results

Mixed results require a decision about trade-offs rather than selecting only the metric that supports the preferred change.

A practical procedure

1. List improvements and regressions using the predeclared outcome definitions.

2. Check whether missing data or unusual events explain part of the disagreement.

3. Choose to refine, repeat or reject the change and state which trade-off drives that decision.

Worked example: a fictional session

A fictional shorter answer reduces playback time but causes more questions about exclusions, so the team restores the missing condition before retesting.

Check the limits

Do not describe a trial as successful without acknowledging a material regression.