# tested.dev. Full setup pack for LLMs

> One-loop patch coverage. Keep secrets out of logs and commits.

## What it is

tested.dev measures **patch coverage** (new executable lines on a PR) and optional project coverage. Humans get a share URL + (Hobby/Team) sticky comments and App-posted required checks. Agents get the same uncovered ranges via CLI `--json` or MCP.

- App: https://app.tested.dev/sign-in
- Docs: https://tested.dev/docs
- Docs markdown (agents): append `.md` to any docs path, e.g. https://tested.dev/docs/migrate-from-codecov.md
- [Design](https://tested.dev/DESIGN.md): UI rules. Follow this file. If a screen fails it, it is not done.
- Blog: https://tested.dev/blog
- Demo: https://app.tested.dev/demo
- Agent Auth discovery: https://app.tested.dev/.well-known/agent-configuration
- Agent auth.md (from BA APIs): https://app.tested.dev/auth.md
- OpenAPI (JSON): https://app.tested.dev/api/auth/open-api/generate-schema
- OpenAPI HTML (optional): https://app.tested.dev/api/auth/reference
- Marketing pointer: https://tested.dev/auth.md
- GitHub App: https://github.com/apps/tested-dev-app

## Blog posts

- https://tested.dev/blog/tested-vs-codecov
- https://tested.dev/blog/patch-coverage-for-coding-agents
- https://tested.dev/blog/github-action-patch-coverage-gate

## Happy path (humans + agents)

```
tested setup
# tests with coverage  (tested run writes coverage even if tests fail)
tested diff              # report. exit 0. no account
tested check             # gate. no account
tested push --pr <n>     # optional. needs TESTED_TOKEN
# → https://app.tested.dev/share/…
```

## Hard rules for agents

- Install: `pnpm add -D @tested/cli`. After install, `td` is the same CLI. One-off: `npx @tested/cli`.
- Local `tested diff` / `tested check` need no account.
- CI gate: GitHub Action `tested-hq/cli/action@main` with `version: 0.1.10`. `push` defaults to false. No token. Pin `uses:` to a SHA in production. `fetch-depth: 0` is optional.
- Need Node 24+.
- `tested diff` is a report (exit 0). `tested check` is the gate.
- `tested run` writes coverage even if tests fail.

## Install

### 1. Local gate (no account)
```
pnpm add -D @tested/cli
# one-off: npx @tested/cli
```
Then:
```
tested setup          # .tested.yaml + doctor + CI hints
pnpm test -- --coverage
tested diff           # report. exit 0
tested check          # gate
```

### 2. CI gate (no account)
Prefer the Action. No local CLI install required. No token. push defaults to false. Pin `uses:` to a SHA in production.
```yaml
name: tested
on: [pull_request]
jobs:
  coverage:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: pnpm/action-setup@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 24
          cache: pnpm
      - run: pnpm install --frozen-lockfile
      - run: pnpm test -- --coverage
      - uses: tested-hq/cli/action@main
        with:
          version: 0.1.10
```

Local + CI gate above need no account and no share.

## Going live (human)

Only these three. The local + CI gate does not need share.

1. Install the GitHub App (tested-dev-app): https://github.com/apps/tested-dev-app
2. Mint TESTED_TOKEN at https://app.tested.dev/repos/{owner}/{name}/settings
3. Require the check: Repo Settings → Branches → require status checks → add `coverage` (after the job has run once)

### Install GitHub App (tested-dev-app)
Install https://github.com/apps/tested-dev-app on the org/user. Sign in: https://app.tested.dev/sign-in. Open the repo once in the dashboard so it appears under /repos.

### Mint TESTED_TOKEN
Mint at https://app.tested.dev/repos/{owner}/{name}/settings. Settings shows the 64-character hex once.
```
export TESTED_TOKEN=…   # paste from Settings (shown once)
# or TESTED_TOKEN_FILE with chmod 600
# optional: TESTED_API_URL=https://app.tested.dev
```
GitHub Actions: repository secret `TESTED_TOKEN`. `tested push --pr N` needs this token.

Share / push is optional after the token exists. Not required for the gate.
```yaml
- uses: tested-hq/cli/action@main
  with:
    version: 0.1.10
    push: 'true'
    pr-number: ${{ github.event.pull_request.number }}
    token: ${{ secrets.TESTED_TOKEN }}
```

### Require the check
After the Action has run once: repo Settings → Branches → protection rule or ruleset on the default branch → require status checks → add the job name (`coverage` in the snippet). App-posted required checks and sticky comments need Hobby or Team.

### Fork PRs
The gate works. `push: true` fails without secrets. GitHub does not pass repository secrets to workflows from forks.

### Mainline chart (optional)
30-day chart is **default-branch only**. PR uploads do not fill it.
```yaml
- uses: tested-hq/cli/action@main
  with:
    version: 0.1.10
    mainline: 'true'
    push: 'true'
    token: ${{ secrets.TESTED_TOKEN }}
```
Or: `tested push --mainline`

## From Codecov

Keep generating coverage. Put tested.dev on the PR. Require the job. Then Codecov comes out.

Same local + CI gate as above. No account for the gate. Do not drop Codecov until `coverage` has run once and is required.

Coverage: Vitest/Jest default is Istanbul/V8 `coverage/coverage-final.json`. The CLI also reads lcov, Cobertura, JaCoCo, gcov text, and SimpleCov. Set `coverage.path` if it isn't the default file.

If `codecov.yml` has flags, put them in `.tested.yaml`. `tested check` and `tested push` grade each package from this run.

```yaml
flags:
  frontend:
    paths: ["apps/web/**", "packages/ui/**"]
    thresholds:
      patch: 90
  backend:
    paths: ["apps/api/**"]
```

A job already scoped to one package: Action `flag: frontend`.

If CI writes more than one coverage file, list them in `coverage.path` or Action `files`. The CLI merges them and checks once.

### Teardown

Require `coverage` after it has run once. Repo Settings → Branches → require status checks → add `coverage`.

Then drop:
- Codecov Action step
- codecov.yml
- CODECOV_TOKEN
- Required codecov/* checks
- Codecov badge last

tested.dev badge:

```
[![tested](https://app.tested.dev/badge/{owner}/{name})](https://app.tested.dev/repos/{owner}/{name})
```

## Done when
- `tested doctor` has no FAIL
- `tested check` runs locally or the Action job posts a conclusion
- Optional: PR produces https://app.tested.dev/share/…
- Share page shows patch % + uncovered ranges

## CLI commands
| Command | Role |
|---------|------|
| tested setup | init + doctor + hints |
| tested doctor | env checklist (`--json`). Exits 1 on Node < 24 |
| tested init | write .tested.yaml only |
| tested run | test runner + coverage. Writes coverage even if tests fail |
| tested diff | report only. exit 0. no account. `--json` for agents |
| tested check | the gate. exit 1 under thresholds. no account |
| tested push | upload; `--pr N` or `--mainline`. Needs token |
| td | alias for `tested` after install |

## Config (.tested.yaml)
```yaml
base: origin/main
testRunner: pnpm test -- --coverage
thresholds:
  patch: 80
  project: 60
  paths:
    - glob: src/api/**
      patch: 90
      project: 90
    - glob: src/cli/**
      patch: 70
      project: 70
ignores:
  - "**/*.generated.ts"
  - "**/dist/**"
```
Optional `thresholds.paths` globs. Independent floors per path. Omitted `patch` / `project` inherit the global floors. A glob with no files this run is skipped (not 0%). Not flags. `tested check` fails when any matched path is under its floor. `--json` adds a `paths` array. No per-path GitHub check slugs. Coverage stays the only PR gate.

## MCP (agents)
Requires existing coverage/coverage-final.json. `@tested/mcp` 0.1.3. `npx @tested/mcp` needs `@tested/cli`. cwd must be an absolute path to a git repo. Default base is `origin/main`. Do not paste placeholder `TESTED_BIN` paths. Coverage is the only PR gate. Missing JUnit is a quiet miss.
```
pnpm add -D @tested/cli @tested/mcp
```
```json
{
  "mcpServers": {
    "tested": {
      "command": "npx",
      "args": ["-y", "@tested/mcp"],
      "env": {
        "TESTED_ALLOWED_CWDS": "/absolute/path/to/your-git-repo"
      }
    }
  }
}
```

| Tool | When |
|------|------|
| `get_uncovered_diff` | Ranges in the patch. `cwd` absolute + git. `base` defaults to `origin/main` |
| `get_coverage_summary` | Rolled-up patch/project % |
| `explain_line` | One file:line with excerpt |
| `coverage_for` | patch coverage for paths the agent touched |
| `uncovered_branches` | uncovered branches in the patch, not only lines |
| `who_covers` | which tests execute this file:line |
| `map_uncovered_to_test` | existing test file an uncovered range should land in |
| `get_flakes` | Tests tab. Intra-run flakes and failures |
| `get_failed` | name, file, message, duration, already-flaky |
| `get_performance` | Performance tab. Suite duration and slowest tests |
| `duration_delta` | suite/test duration vs main |
| `new_since_main` | new flakes, newly slow tests, files that lost coverage vs base (informational) |
| `write_and_verify` | Write a test file + re-run coverage + fresh diff |
| `check` | `tested check --json` |
| `push` | `tested push --json`. Token. Optional `junit` |
| `doctor` | `tested doctor --json` |

Skills: triage (CI red → doctor/get_failed/get_flakes/get_uncovered_diff); close-patch (get_uncovered_diff → map_uncovered_to_test → write_and_verify → check).

## Plans
- Free: public + 1 private; share links; no comments, no App-posted required checks; 250 private uploads / mo
- Hobby $5/seat / mo ($4 annual): unlimited private, sticky comments, App-posted required checks, 2,500 private uploads / mo, max 10 seats
- Team $10/seat / mo ($8 annual): unlimited seats, project coverage, full history, per-repo thresholds, unlimited uploads

## Troubleshooting
- tested: command not found: `pnpm add -D @tested/cli` or `npx @tested/cli`. `td` is the same CLI after install
- Node < 24: `tested doctor` exits 1. Upgrade to 24+
- No coverage file: `tested run` writes coverage even if tests fail. If still missing, the runner is not emitting Istanbul/V8 JSON
- invalid_credentials: wrong/rotated token, repo not opened in app, wrong API host. Mint at /repos/{owner}/{name}/settings. API URL is `TESTED_API_URL`
- 0 repos: App on wrong account; re-open /repos after install. App: https://github.com/apps/tested-dev-app
- Empty chart: need mainline push, not PR-only
- Empty patch: required checks skipped; project still applies
- Sticky / App checks missing: Hobby or Team plan + App installed
- Fork PR push failed: gate works; `push: true` needs secrets GitHub does not pass to forks

## Security rules for agents
- Never echo TESTED_TOKEN / paste tokens into chat logs
- Prefer TESTED_TOKEN_FILE chmod 600 over argv --token
- Do not commit .env with tokens
- Restrict MCP TESTED_ALLOWED_CWDS to trusted roots