HowardBench · completed 1 September 2026
See how Claude compares with Howard on the same real mechanical-engineering questions.
Howard improved Claude’s score by 43.7 points on tool questions and 55.0 points on design problems.
| Metric | Claude alone | Claude + Howard |
|---|---|---|
| Task accuracy | 23.2% | 66.9% |
| Full-item pass | 2.5% | 57.6% |
| Grounded-number accuracy | 0.0% | 51.2% |
| Correct refusal | 44.7% | 73.9% |
| Choice accuracy | 16.8% | 34.4% |
| Unsourced-number leak | 45.3% | 4.9% |
| Mean latency | 10.86s | 10.52s |
A number fails grounding when its value is absent from the allowed trace.
| Slice | What it tests | n | Claude alone | Claude + Howard |
|---|---|---|---|---|
| A | Reviewed-data or calculation hit | 37 | 32.4% | 78.4% |
| B | Expected no-data response | 126 | 1.9% | 99.7% |
| C | Required refusal | 144 | 0.7% | 64.2% |
| D | Material, process, architecture or interpretation choice | 140 | 31.8% | 39.0% |
| E | Engineering calculation | 193 | 47.7% | 71.5% |
| F | Boundary or adversarial trap | 35 | 14.3% | 32.9% |
Thirty broader questions across six product-development phases.
| Design group | n | Claude alone | Claude + Howard |
|---|---|---|---|
| Requirements and architecture | 5 | 0.0% | 50.0% |
| Pre-CAD sizing | 5 | 0.0% | 90.0% |
| CAD-detail review | 5 | 0.0% | 50.0% |
| Design verification | 5 | 50.0% | 90.0% |
| DFM and prototype review | 5 | 0.0% | 0.0% |
| Failure and redesign | 5 | 0.0% | 100.0% |
Full-item pass was 26.7% with Claude + Howard and 0% with Claude alone.
The questions span requirements, architecture, pre-CAD sizing, CAD review, verification, DFM, testing, and redesign.
Claude answered every question twice with the same prompt. One run used Claude alone; the other could call Howard. Answers were scored for routing, numbers, refusals, choices, and source-backed numeric claims.
Howard raised Claude’s score from 23.2% to 66.9% on tool questions and from 8.3% to 63.3% on design problems.