Products

Data you can evaluate this week.

Task packs, licensed codebases and expert-written material, built by working professionals and delivered in the formats labs already run. You pay for what you accept.

Off the shelf

011 wkto a sample, Harbor format

Long-horizon coding tasks

Agentic software tasks a model has to solve inside a real repository, written by working engineers and calibrated before they ship. A sample pack across the main production languages is ready now. Larger packs are written to your specification under our own contributor agreement, so the IP is clean.

  • A Dockerised repository and environment
  • The instruction, hidden tests and an oracle solution
  • Upstream provenance and licence for every repository
  • Calibration evidence from frontier-model attempts

Format Harbor layout, tarball plus SHA-256 manifest

02400+owners offering a repository

Licensed production codebases

Private repositories with real commit history, licensed by their owners for AI training and evaluation. Owners come through our network, pass an identity check, and license per repository. Languages on file include Go, Rust, C and C++, Swift, MATLAB and COBOL, plus finished Three.js and Unreal Engine games.

  • Full history, not a snapshot
  • Owner identity verified before any licence
  • Per-repository licence and provenance file
  • Screened for third-party and employer code

Format Git bundle or private mirror, licence attached

031specification, every report to it

Incident reports and postmortems

Production incident write-ups by the engineers who handled them: timeline, detection, root cause, remediation and what changed afterwards, screened before delivery. Authors are paid only on your acceptance and license the text for training and evaluation.

  • Fixed section structure and length
  • Screened for employer and customer material
  • Author consent and licence on file

Format Markdown or PDF, one file per incident, manifest

046domains, built to order

Expert evaluation sets

Question, rubric and reference-answer sets in software, machine learning, science, finance, medicine and law, written by credentialed experts and reviewed by a second expert. The first batch is small enough to grade by hand before anything scales.

  • Prompt, rubric with partial credit, reference answer
  • Second-expert review on every item
  • Author credentials summarised, never identifying

Format JSONL with a schema you approve first

Counts as of November 2026, rounded down. Sample packs are shared for private evaluation only; a buyer who wants to train on them licenses them like any accepted item.

Built to order

The same people, on your spec.

  • 01RL environments & agentic tasksTask environments with verifiers a model has to pass, built and stress-tested by people who do the work for a living.
  • 02Evaluations & rubricsBenchmarks, rubrics and evaluations designed around how your team trains and tests, not a generic template.
  • 03Expert review & annotationDomain experts reviewing, ranking and correcting model output, in a workflow we agree with you.
  • 04Multimodal & video dataCollection and review for video and multimodal datasets, from sourcing contributors to quality control.

How buying works

Sample, pilot, scale.

Samples: private evaluation only. Accepted items: perpetual, sublicensable licence. Every item: provenance and author consent on file.

01

Sample

A small set under a private evaluation licence: look, run it, grade it. No training on samples.

02

Pilot

A paid batch to your spec. You accept item by item; you pay for what you accept.

03

Scale

Accepted items carry a perpetual, sublicensable licence, delivered in your format with a provenance file for every item.

Before a task ships

Six checks, every task.

  • 01The instruction and the hidden tests ask for the same thing, and a second expert reproduces the task from scratch.
  • 02The reference solution passes; an agent that does nothing fails.
  • 03Hidden tests, the solution and the grader live outside the agent's reach, and nothing depends on the live network.
  • 04The verifier is deterministic, the timeout is set and the time budget fits a professional's working session.
  • 05Every source carries its upstream licence and commit, and every author has signed our contributor agreement.
  • 06Frontier models attempt every task before it ships. We record how they did; we never promise a number.

Notes from the floor

Short, and paid for.

2026-10

A pass rate is a difficulty estimate, not a quality bar

A task no model solves is not automatically a good task. Half the time it is a grader that cannot be satisfied, an instruction that leaves out a requirement, or a permission the agent never had. We treat zero-pass tasks as suspects: the ones that survive manual review are valuable; the rest get fixed or dropped.

2026-09

Verifiers get gamed in the first hour

Give an agent a test it can read and it will satisfy the test, not the task. The fix is structural: hidden tests, behaviour checked through the public interface, and an author who tries to shortcut their own task first. We ask for that attempt in writing.

2026-10

Human authorship has a fingerprint

When a buyer asks whether a person wrote a document, polish is the wrong signal. Edit history, creation dates, the generator recorded in the file and the small inconsistencies of real work are the right ones. We check those before delivery and drop the files that cannot show them.

Sample

Get the five-task sample

Runnable Harbor tasks with their calibration evidence.

Request it ↗