Benchmarks
Ox Alpha benchmarks.
How the stealth model performs on an independent run of 10 real-world coding tasks — next to the frontier models it was lined up against.
80%
Ox Alpha mean pass rate
8 of 10 tasks solved
8 of 10 tasks solved
65%
best reference model
Fable 5 (fable-5 [max])
Fable 5 (fable-5 [max])
+15
points ahead of the field
on these 10 tasks
on these 10 tasks
Full results
All 10 tasks, all 5 models.
| Task | fable-5 [max] | glm-5.3 [max] | gpt-5.6-sol [max] | grok-4.6 [xhigh] | ox-alpha |
|---|---|---|---|---|---|
| anko-typed-variable-bindings | 4/4 | 4/4 | 2/4 | 1/4 | ✓ |
| arktype-json-schema-refs | 2/4 | 1/4 | 3/4 | 1/4 | ✓ |
| fastapi-deprecation-headers | 4/4 | 3/4 | 3/4 | 4/4 | ✓ |
| helm-unified-manifest-stream | 4/4 | 4/4 | 4/4 | 4/4 | ✓ |
| igel-persist-feature-schema | 3/4 | 3/4 | 0/4 | 4/4 | ✓ |
| katex-multicolumn-array-spans | 2/4 | 4/4 | 3/4 | 4/4 | ✓ |
| meriyah-explicit-resource-decl | 1/4 | 0/4 | 0/4 | 0/4 | ✓ |
| query-persist-restored-state | 2/4 | 3/4 | 1/4 | 2/4 | ✓ |
| scc-bounded-memory-spilling | 4/4 | 3/4 | 4/4 | 4/4 | ✕ |
| vulture-persistent-analysis-cache | 0/4 | 0/4 | 1/4 | 1/4 | ✕ |
| Mean on these 10 | 65% | 62% | 52% | 62% | 80% |
Independent community benchmark — 10 real-world coding tasks. Reference models: passes out of 4 attempts per task. Ox Alpha: pass/fail. Highlighted row: the task every reference model scored 1/4 or worse on — Ox Alpha solved it. Third-party data, presented as published.
Head to head
One model at a time.
vs Fable 5
fable-5 [max] · 65%
- 5 tasks Ox Alpha had the edge
- 3 both solved cleanly
- 1 Fable 5 had the edge
- 1 neither solved
vs GLM-5
glm-5.3 [max] · 62%
- 5 tasks Ox Alpha had the edge
- 3 both solved cleanly
- 1 GLM-5 had the edge
- 1 neither solved
vs GPT-5
gpt-5.6-sol [max] · 52%
- 7 tasks Ox Alpha had the edge
- 1 both solved cleanly
- 2 GPT-5 had the edge
- 0 neither solved
vs Grok 4
grok-4.6 [xhigh] · 62%
- 4 tasks Ox Alpha had the edge
- 4 both solved cleanly
- 2 Grok 4 had the edge
- 0 neither solved
Highlights
What stands out in the data
- 8 of 10 solved — the highest pass rate in the table, 15 points clear of Fable 5.
- Solved the table's hardest task. On
meriyah-explicit-resource-declno reference model managed better than 1 pass in 4. - Consistent across the easy ones. Every task at least two reference models passed cleanly, Ox Alpha passed too — except
scc-bounded-memory-spilling. - Two misses:
scc-bounded-memory-spilling</code> and <code>vulture-persistent-analysis-cache. The second one stumped almost everyone; the first is Ox Alpha's clearest weak spot here.
Methodology
How to read these numbers
- 110 real-world coding tasks drawn from open-source repositories — each named after the project and the change required.
- 2Reference models got 4 attempts per task; the table shows how many passed. Ox Alpha is recorded as a single pass/fail.
- 3"Mean" is the average pass rate across the 10 tasks (for Ox Alpha: tasks solved ÷ 10).
- 4Third-party data. We didn't run this evaluation ourselves; it's reproduced as published.
- 5Small sample. One task either way moves a mean by 10 points. Treat this as directional and test on your own problems.
The best benchmark is yours.
Paste the bug, the spec, or the whole repo. Ox Alpha is free while it's in stealth.
Try Ox Alpha