Open behavior benchmark · version 0.1
Can an AI math tutor help without taking over?
A public evaluation protocol for a question most AI math comparisons miss: not only whether the response is correct, but whether it behaves like teaching.
The short answer
Version 0.1 is a 15-scenario, seven-dimension protocol for testing whether an AI math tutor remains correct, uses the learner’s evidence, gives the minimum effective help, protects unanswered work, preserves agency, checks transfer, and calibrates claims about understanding.
Why this benchmark exists
Answer accuracy is necessary for math tutoring, but it is not sufficient. A system can produce a flawless solution while removing the reasoning the learner was meant to practice. It can also withhold an answer yet still give a vague hint, ignore visible work, validate a misconception, or claim mastery after one supported response.
The teaching-first benchmark makes those behaviors separately observable. Each case states the context, the learner-owned target that must remain unresolved, the required behavior, and only the dimensions relevant to that case.
- 15 adversarial and ordinary tutoring situations.
- 7 dimensions scored independently on a 0–2 scale.
- Scenario-specific requirements instead of generic style preferences.
- Raw-output and configuration reporting requirements.
What the cases cover
The set includes direct requests for answers, repairs from visible work, correct answers without explanations, misconceptions, hint requests, blank starts, productive pauses, no-hint diagnostics, transfer checks, ambiguous notation, false generalizations, private-answer leakage, incidental arithmetic, multiple valid methods, and near misses.
These cases were chosen because a response can be mathematically plausible while still violating the teaching job. For example, confirming an answer found in private lesson material may be accurate but leaks learner-owned work. Forcing the method from an example may be correct but ignores a valid learner method.
How to run it
Freeze the system prompt, model identifier, provider, tool availability, decoding settings, and benchmark version. Present each context without adding unstated learner history. Retain the complete raw response. If the tested tutor normally uses tools or a canvas, report whether those capabilities were available and score only evidence actually produced.
At least two evaluators should score the dimensions listed for each case. Resolve disagreements with a written rationale or publish both scores. Do not add dimensions that a case does not list to increase or reduce the aggregate after seeing an output.
How to report results
Report every case and dimension, not only an overall percentage. Include the exact model and system-prompt versions, run date, settings, raw outputs, evaluator identities, disagreements, reruns, failures, and exclusions. If a model was tuned on the cases or the test author builds the evaluated product, say so prominently.
Aggregate scores can be useful for tracking one configuration over time, but they should sit beside failure details. A single answer leak during a diagnostic may matter more operationally than a small difference in average tone or concision.
A practical comparison
| Question | Score 2 | Score 0 |
|---|---|---|
| Answer boundary | Keeps the learner-owned target unresolved while giving permitted support. | Reveals, confirms, or effectively completes the target. |
| Evidence use | Responds to the specific attempt, explanation, or uncertainty in the case. | Ignores the evidence or invents a diagnosis unsupported by the case. |
| Minimum help | Supplies the smallest concrete support that can unlock the learner’s next move. | Takes over the procedure or gives help unrelated to the actual block. |
| Calibrated claim | Distinguishes a response, a supported repair, and independent evidence. | Claims mastery or a broad deficit from insufficient evidence. |
Limitations
- Version 0.1 uses single-turn text scenarios and does not evaluate a long tutoring relationship.
- The cases emphasize answer boundaries and responsive support; they are not a complete audit of mathematical domains, accessibility, safety, or bias.
- Human scoring involves judgment. Reports should identify evaluators, preserve raw outputs, and disclose disagreements.
- Pythago authored the cases, so Pythago-produced results are not independent and may benefit from benchmark familiarity.
- No comparative model scores or learner outcome claims are included in this release.
Direct answers
Frequently asked questions
What does version 0.1 contain?+
It contains 15 single-turn text scenarios, seven scoring dimensions, scenario-specific required behavior, a 0–2 rubric, and reporting requirements. The raw dataset is available as JSON.
Does the benchmark measure whether students learn?+
No. It measures tutor response behavior in specified situations. Learning outcomes, retention, speech, canvas quality, latency, reliability, and child-safety review are outside version 0.1’s scope.
Why not publish one overall score?+
A single score can hide serious failures. Reports should retain per-case and per-dimension results so readers can distinguish mathematical errors, answer leakage, excessive help, weak evidence use, and unsupported mastery claims.
Has Pythago published comparative model results?+
Not in version 0.1. This release publishes the protocol and raw cases first. Any later result should include exact configurations and raw outputs and should disclose that Pythago authored the benchmark.
Keep exploring
Raw benchmark dataset
Download all 15 cases, dimensions, required behaviors, and reporting rules as JSON.
Evidence policy
Read how Pythago separates product demonstrations, behavior tests, and outcome evidence.
Creative Commons license
Reuse and adapt the benchmark with attribution under CC BY 4.0.
Tutor vs. solver
Compare a teaching-oriented interaction with answer-producing math software.