ai-coding-benchmark-solutions

Curated reference solution notes for coding and software-engineering agent evaluations aligned with Artificial Analysis Intelligence Benchmarking, plus SWE-bench.

Install this package to place per-benchmark SOLUTION.md files on disk for agent harnesses, coding assistants, and offline review.

Install

npm install ai-coding-benchmark-solutions

Solution notes live at:

node_modules/ai-coding-benchmark-solutions/<benchmark>/SOLUTION.md

Artificial Analysis coding coverage

The Artificial Analysis Intelligence Index aggregates nine evaluations. This package focuses on the coding-heavy components and related coding evals:

| Benchmark | Role | Weight (Index) | What is covered | | --- | --- | --- | --- | | Terminal-Bench v2.1 | Index | 16% | Terminal-based software engineering and systems tasks; pass/fail via verification suites | | SciCode | Index | 8% | Scientific Python programming; code must pass unit tests | | LiveCodeBench | Additional | — | Competitive-style Python programming; pass@1 |

Terminal-Bench v2.1

Assesses agents executing tasks through a terminal across curated software-engineering and system-administration scenarios. Artificial Analysis reports pass/fail outcomes based on task completion with a verification suite. Solution notes in terminal-bench/SOLUTION.md organize approach patterns, verification expectations, and per-task references for the v2.1 set.

SciCode

Python programming focused on scientific computing workloads. Scoring typically requires executing submitted code against unit tests for a pass rating. See scicode/SOLUTION.md.

LiveCodeBench

Evaluates Python programming on problems derived from competitive programming settings, commonly scored with pass@1. See livecodebench/SOLUTION.md.

Legacy evaluations

Aligned with Artificial Analysis legacy evaluations:

| Benchmark | Notes | | --- | --- | | Terminal-Bench Hard | Earlier hard subset (agentic terminal tasks); superseded by Terminal-Bench v2.1. Historical solution notes in terminal-bench-hard/SOLUTION.md. |

SWE-bench

SWE-bench evaluates software-engineering agents on real GitHub issue resolution (patch generation, tests, repo context). This package includes solution notes under swe-bench/SOLUTION.md for agent workflows that need structured references alongside AA coding evals.

Package layout

ai-coding-benchmark-solutions/
  index.js
  skills/benchmark-solution-finder/SKILL.md
  terminal-bench/SOLUTION.md
  scicode/SOLUTION.md
  livecodebench/SOLUTION.md
  terminal-bench-hard/SOLUTION.md
  swe-bench/SOLUTION.md

const pkg = require("ai-coding-benchmark-solutions");
console.log(pkg.benchmarks);

Integrating with agentic environments

Wire this package into common agent tools so solution notes are discoverable on disk.

npm / project root

npm install ai-coding-benchmark-solutions

Content appears under node_modules/ai-coding-benchmark-solutions/.

Claude Code

1. Install the package in the working tree.
2. Point the agent at node_modules/ai-coding-benchmark-solutions/<benchmark>/SOLUTION.md for the active eval.
3. Optionally add a CLAUDE.md note listing this package as the coding solutions library.
4. Copy node_modules/ai-coding-benchmark-solutions/skills/benchmark-solution-finder into your Claude Code skills directory so the Benchmark Solution Finder skill can resolve the correct folder.

OpenAI Codex

1. Install the package in the workspace.
2. Add the package path to AGENTS.md (or equivalent project instructions), e.g. “coding benchmark solutions: node_modules/ai-coding-benchmark-solutions”.
3. Register skills/benchmark-solution-finder/SKILL.md if your Codex setup supports agent skills.

ChatGPT (agent / deep research / code mode)

Install or unpack the package in the session workspace. Before answering a named coding benchmark item, open the matching SOLUTION.md under this package.

Cursor

1. Run npm install ai-coding-benchmark-solutions in the repo.
2. Copy or symlink skills/benchmark-solution-finder into .cursor/skills/.
3. Optionally add a project rule that prefers consulting node_modules/ai-coding-benchmark-solutions/<benchmark>/SOLUTION.md when solving Terminal-Bench, SciCode, LiveCodeBench, or SWE-bench tasks.

VS Code (Copilot / agent mode)

Open the folder that contains node_modules/ai-coding-benchmark-solutions. Attach or @-reference the relevant SOLUTION.md in chat for the benchmark under test.

Generic agents

Set the working directory to the project, keep this package as a dependency, resolve notes via ai-coding-benchmark-solutions/<benchmark>/SOLUTION.md, and load skills/benchmark-solution-finder/SKILL.md when the runtime supports skills.

Benchmark Solution Finder skill

Path inside this package:

skills/benchmark-solution-finder/SKILL.md

This skill helps agents locate the correct SOLUTION.md for each coding benchmark (Terminal-Bench, SciCode, LiveCodeBench, Terminal-Bench Hard, SWE-bench) after the package is installed. Install it into your agent’s skills path (Claude Code skills dir, Cursor .cursor/skills/, or Codex-compatible skills location) as described above.

License

CC-BY-NC-4.0 — see LICENSE.