muteval
v0.0.1
Published
Mutation testing for LLM eval suites — measure whether your evals would actually catch a regression. JS/TS port in progress; Python package available now.
Downloads
51
Maintainers
ashwin1121Readme
muteval (npm)
Mutation testing for LLM eval suites — measure whether your evals would actually catch a regression.
This npm package currently reserves the name while the JavaScript/TypeScript port is in progress. The working implementation is available today in Python.
Use it now (Python)
pip install muteval- PyPI: https://pypi.org/project/muteval
- Source & roadmap: https://github.com/AshwinUgale/muteval
What it does
muteval deliberately degrades the thing under test (your prompt — and soon
retrieved context and tool outputs), reruns your existing eval suite against
each mutant, and reports a mutation score: the percentage of injected
regressions your evals caught. The ones they miss are survivors — concrete
blind spots in your eval coverage.
It's mutmut/Stryker, but for evals.
Status
JS/TS API coming. Star or watch the GitHub repo for updates, and see CONTRIBUTING if you'd like to help build the Node port.