Testing methodology
Placeholder. The copy on this page is a draft outline. Final copy is still to be written and checked.
Short answer: AI agents run a written, versioned protocol on one tool at a time, record the test date, version, plan tier and price, and save the evidence. A human reviews each result before it is published. [TODO confirm the human step] Results are snapshots, and we list the limits below.
Method version: [TODO e.g. v1]. Last updated: [TODO date at publish].
[TODO IMPORTANT: this page describes the planned v1 protocol. The test harness is not built or confirmed. Before publishing, check every statement against what the harness actually does and delete or soften anything it does not. Do not publish claims about steps that are not run.]
What the bots do
- Read the tool’s terms on automation, benchmarking and publishing before testing. [TODO confirm]
- Set up a dedicated test account. Test accounts are never personal accounts, and are deleted after the test. [TODO confirm]
- Run the protocol tasks, record results, and draft the report.
- Capture the price and plan tier on the test date.
Our agents do not pretend to be human customers.
The protocol, step by step
Each tool category has a written protocol, versioned. The generic shape is:
- Pre-flight: terms check, license, plan tier, price snapshot, environment manifest.
- Setup task (timed).
- Core tasks: [TODO e.g. 4 to 8], with pass, partial and fail rules written before the run.
- Stress tasks: [TODO e.g. 1 to 3].
- Safety and control tasks: permissions, audit log, export and deletion. For AI agents, also rule compliance and synthetic prompt-injection checks. [TODO confirm]
- Exit: export, delete data, cancel, and record how that went.
- Re-runs: [TODO e.g. 3 runs for AI agents, 2 for deterministic software]. Variation between runs is reported.
How results are reported
Every review gives a score from 1 to 5, a plain-language verdict, and a confidence label. It also shows the tool version, the test date and a short method summary.
The score (1 to 5)
The score is built from a fixed set of criteria, each rated 1 to 5 against a written rubric. [TODO confirm criteria and any weights; the list below is a placeholder and must be replaced by what the harness actually scores]
| Criterion | What it asks | Weight |
|---|---|---|
| [TODO confirm criteria] Task completion | Did the tool finish the core tasks? | [TODO] |
| [TODO confirm criteria] Reliability | Did results hold across re-runs? | [TODO] |
| [TODO confirm criteria] Time to first value | How long to a first useful result? | [TODO] |
| [TODO confirm criteria] Safety and control | Permissions, audit trail, rule compliance | [TODO] |
| [TODO confirm criteria] Portability and exit | Can you export your data and leave cleanly? | [TODO] |
| [TODO confirm criteria] Cost transparency | Was the price and usage clear? | [TODO] |
| [TODO confirm criteria] Documentation | Were the docs accurate and usable? | [TODO] |
- Scores are shown with one decimal at most. A change to criteria or weights bumps the method version, and old reviews state the version they used. [TODO confirm]
- A tool that was not tested is not scored. We do not invent ratings, and we do not publish an average across tools.
The verdict
A plain-language verdict of one to three sentences: what the tool did well, where it fell short, and who it fits. It never relies on the score alone.
Confidence label: high, medium or low
The label says how much weight the result can bear. [TODO confirm definitions against the harness]
- High: all planned tasks ran, the re-run count was met with consistent results, the version was stable, and a human reviewed the evidence.
- Medium: most tasks ran, or results varied between runs, or some steps could not be checked (for example a feature behind a paid tier we did not test).
- Low: limited coverage, high variance between runs, a fast-changing product, or a significant part of the protocol was skipped. Treat as a first look.
The confidence label does not change the score. It says how far to trust it.
What every review must show
A review is not published without all of these. [TODO enforce in the review template and the lint step]
- Score (1 to 5), the criteria scores, and the method version.
- Plain verdict.
- Confidence label.
- Tool name and tool version tested, plus plan tier.
- Test date (and the price on that date).
- A short method summary (what was run, how many times, what a human checked) with a link to this page.
- Known limits.
- A “how to cite this” line.
- A “Report an error” link (see the corrections policy).
Evidence
For each test we keep a bundle: environment manifest, terms check, price snapshot, run transcripts, screenshots, results and a scorecard with checksums. A redacted public subset is linked from the review. Raw evidence stays private and is scanned to remove secrets and personal data before anything is published. [TODO confirm what is published]
What a human reviews
[TODO state exactly. For example: a human reads each verdict and the evidence summary before publication, and decides whether to publish. A human does not repeat every test.]
Independence
- The same protocol is used for every tool in a category.
- Money does not change a result. See the affiliate and disclosure policy. [TODO confirm]
- Failures are published.
Limits
- A review is a snapshot of one version, plan and date. Products change.
- Bots can misread results or miss edge cases. Humans do not catch everything.
- We test on [TODO environment]. Your setup may differ.
- Small samples: re-run counts are listed on each review.
- We do not make long-term-use claims unless a re-test really covered the period.
Reproducing and challenging a result
Each review lists the protocol version and the tasks so others can re-run it. If you find an error, use the corrections policy.