What is it?
NicheBench is our Python CLI for evaluating language models on framework-specific tasks. We started with Drupal, where an answer depends on the framework version, its APIs, and the project around it. You can use the harness to ask questions, generate code, and test bug fixes against a defined task.
The classic evaluation flow pairs a model under test with a judge model. The first produces an answer; the second assesses it using the task's rubric. That gives you a way to inspect how a model handles the kind of work you have in mind.
We publish the harness source and documentation under the MIT licence. The task data lives in private submodules, and raw results and reports are excluded from the public repository. Running an evaluation takes setup and access to suitable task data.
Why?
When you choose a model for framework work, the details matter. A Drupal task can depend on a service definition, a plugin API, or the lifecycle of an entity. An answer can look convincing while using assumptions from a different version. We want the evaluation to include that context.
A generic score gives you one view of a model. Framework-specific tasks let you ask narrower questions. Does it understand this API? Can it produce a change that fits the surrounding code? Which checks does the change pass when the environment runs it?
We would start with tasks that resemble the work you expect to hand to a model. Keep the framework version and environment explicit. Then you can read a failure with the relevant context in front of you and decide what to investigate next.
How it works?
You configure the model under test, the judge, and the task set. In the classic flow, NicheBench runs quiz, code-generation, or bug-fixing tasks and evaluates the answers against their rubrics. The harness separates task execution from judging, so you can examine the answer and its assessment.
There is also an optional runtime mode. It is disabled by default and needs an explicit opt-in. For Drupal tasks, it uses an isolated workspace with DDEV to run the task's checks. The run collects artifacts and a final diff, giving you code changes and execution evidence to inspect.
Runtime task definitions describe the workspace and the checks that count. Scoring uses task-specific rubrics and configurable weights. A critical-check gate can affect the result when a required check fails. An optional LLM judge can contribute an assessment alongside the runtime evidence.
The setup deserves attention - your task definition decides what the evaluation measures. You need a suitable environment, accessible model endpoints, and task data. Read the authoring and scoring documentation before treating a report as evidence for a decision.
What it unblocks / where it leads
We want to make model selection easier to investigate for teams working inside a particular framework. A task-level report gives you something concrete to discuss. You can follow the answer, inspect a diff, and see which checks passed or failed.
That can help you shape the next experiment. You might adjust a task, improve its checks, or compare another model using the same context. The usefulness of the comparison depends on the tasks and evaluation setup you choose. We are sharing the machinery here; model rankings and performance outcomes need their own evidence.
If you want to try it, start with the repository README and work through the runtime documentation when you need executable checks. We would be interested in the framework tasks you want to evaluate.