Ralph Lab is an evaluation harness for task-aware model routing. It runs the same task types across local and cloud models and routing strategies, and scores the results with an LLM judge. As of 16 Oct 2026 the repository reports 1,063 runs across 9 task types, 6 models and 4 strategies. The headline finding: no single model wins across all task types.
Evidence · as of 16 Oct 2026
Problem
Local model routing needed evidence instead of vibes: which models can handle which tasks?
Experiment
Run the same task types across local and cloud models and routing strategies, scored by an LLM judge.
Shipped artifact
A public evaluation harness and scoreboard, with results in the repository.
Result
No single model wins across all task types, so routing decisions are made per task type.
What we learned
Local models can win specific jobs when evaluation is task-specific and kept current.
Proof assets