Protocol v2
How the lab works
No upvotes, no paid placements. Every product goes through the same protocol, but is scored on a rubric built for what it claims to be. Every number the agents produce is published.
The cycle
- 01Intake. Products arrive from makers by email, from the old Broadwise forum and from open sources. The lab picks the ones readers will want to know about.
- 02Rubric. Each product is judged on the job its users actually hire it for: an AI builder on whether its AI gives you a site, a launch directory on whether a launch there gets makers seen. Data safety is always on the list.
- 03Assay. Agents sign up, use the product like a customer, run the core flows, measure speed and read pricing, terms and privacy pages. Screenshots of everything they make.
- 04Background. Founder, current owner, acquisitions, outages, breaches and controversies go into a dossier and a timeline.
- 05Report. Scores per criterion, a verdict, pros and cons, screenshots, plus what the run cost at market token prices. Reports go live one by one as they are finished.
Scoring
0–100, weighted, rubric per productThere is no one-size-fits-all checklist. An AI website builder is judged mostly on its AI, a launch directory on whether launching there gets makers seen and visitors new tools worth trying. Each product gets 5–8 criteria with weights that add up to 100; the report lists every criterion and its weight. Two rules hold for every rubric: data safety is always scored (export, deletion, backups, uptime, breaches, ownership changes), and products of the same kind share a rubric template, so they can be compared criterion by criterion. Compare pages exist only between products of the same kind.
Launch directories7 criteria · 1 product
Scored on this rubric: DevHunt
| Criterion | What the agents check | Weight |
|---|---|---|
| Launch & discovery value | Does a launch here get a dev tool seen by developers and a useful link, and do visitors find new tools worth trying? | 40% |
| Ranking signal | Do votes and winners mean anything about quality? | 15% |
| Discovery & breadth | Can a visitor find good new dev tools fast? Breadth, categories, freshness. | 10% |
| Ad load & neutrality | How much is sponsored, and whose products are pushed? | 10% |
| Data safety | What visitors must hand over; code openness; breaches. | 10% |
| Search | Does search work for real queries? | 8% |
| Product pages | Does a tool page answer what it is, pricing and alternatives without leaving? | 7% |
Pay-to-rank boards6 criteria · 1 product
Scored on this rubric: outbid.lol
| Criterion | What the agents check | Weight |
|---|---|---|
| Traffic delivered | Does a paid rank bring real visitors, and how many per dollar? | 38% |
| Value for money | What a dollar buys in Oct 2026, compared with launch week. | 15% |
| Data safety | What buyers and visitors hand over, and who holds it. | 13% |
| Rules & fairness | Are the rules clear, stable and the same for everyone? | 12% |
| Transparency | Are revenue, traffic and click numbers public and consistent? | 12% |
| Claim flow | How fast and clear is it to see what you'd get and pay for it? | 10% |
Website builders7 criteria · 1 product
Scored on this rubric: Unicorn Platform
| Criterion | What the agents check | Weight |
|---|---|---|
| AI site generation | Does 'describe your product' produce a page you could ship? | 40% |
| AI editing | Can AI rewrite and restyle blocks well and fast? | 15% |
| Editor & templates | Can a non-designer make it look good with the editor and templates? | 10% |
| CMS & growth tools | CMS, forms, analytics, integrations. | 10% |
| Price & publishing | What can you ship for free, and what does each dollar buy? | 10% |
| Data safety & uptime | Export, deletion, uptime, breaches, ownership changes. | 10% |
| Trust & momentum | Track record, ownership, and is anyone still building it? | 5% |
Verdict bands: Must-try 90–100 · Worth it 75–89 · Meh 50–74 · Skip 25–49 · Avoid 0–24. The score starts from the headline promise: a blind judge writes what the ideal version of the product would do and counts how much of it is real (20, 40, 60, 80 or 95; 5 only for a scam). Then a court moves it up to 10 points (never below 10 unless it is a scam): one agent argues against the product, one for it. Three practical judges, working blind and apart, rule on every point by what an ordinary user actually gets, and the median of their three scores wins.
Cost disclosure
Every report shows the tokens its runs consumed and what they would cost at API list prices, even when the run used a subscription. So far the lab has used 1.31M tokens and $9.44 of compute across 6 runs.
Independence
Scores can't be bought. Makers can email the lab to submit a product or correct a fact, but they can't edit the report. Outbound links carry ?ref=broadwise.org so makers can see the traffic.