The State of the Evidence

What is being counted

Each published entry gets one evidence grade: the strongest kind of source that supports its core claims — not a grade per figure, and not a judgement of whether the project was any good. A well-run project whose results only its operator has published is official. A failure documented by a national audit office is independent. The grade describes the state of the record, not the state of the city.

The grades are assigned under a written protocol during the source check, and they are AI-assigned like the rest of the atlas — the entries are AI-researched summaries of the sources each page lists, produced under a documented rule set rather than individually re-checked by a human. That is worth stating plainly on a page about evidence quality: this is a consistent reading of the record by one method, not an audit by a third party. The underlying data is open, so the reading can be checked.

🔬 Peer-reviewed

The strongest source behind this entry is an independent peer-reviewed or academic study.

📊 Independent

The strongest source is an independent non-academic evaluation: an audit office, a government or EU review, or an NGO.

🏛️ Official

The strongest source is the operator's or the city's own reporting — an official report, dataset or press release.

📰 Media

The strongest source is press reporting; no primary evaluation was located.

• No source of record

No source of record was located: what backs this entry is vendor material, an encyclopedia or an aggregator rather than the operator, the press or an evaluation.

1 🏛️ Two-thirds of the record is the operator's own word

Across all 1167 published entries, 69.0% rest on figures published by the body that runs the project — the city, the utility, the transport authority, the vendor. Peer-reviewed evaluation exists for 7.7%. Add every independent audit, regulator's review and NGO evaluation to the academic work and the share of the atlas that anyone outside the project has examined is 17.6%.

This is not an accusation of dishonesty. Operators publish the numbers they have, and often they are the only body positioned to collect them. It does mean that when a city compares itself against what other cities report, it is mostly comparing press releases.

2 📊 Where a result is claimed, it is usually the seller's number

728 entries state a concrete outcome figure — a percentage saved, a journey time cut, a leak rate reduced. Those are the numbers that travel: into slide decks, procurement documents and news copy. 64.6% of them come from the operator. 11.3% have been through peer review.

The atlas labels this on every project page rather than smoothing it away, which is why the count is possible at all. A figure with an operator's name on it is reported as an operator's figure, never as a measured fact.

3 🔍 The recorded failure rate depends on who was allowed to measure

Sort the atlas by who examined a project and the failure rate moves sharply. Among entries whose best source is the operator, 6.5% are recorded as failed or scaled back. Among entries examined by an independent auditor, regulator or NGO, 20.0% — roughly four times as many. Where the press is the only source that looked, 15.9%.

Read the peer-reviewed row before drawing the obvious conclusion: at 13.3% it sits below the operator-reported rate, which is the opposite of the pattern. That is a selection effect, and this corpus can show it rather than just claim it — the peer-reviewed entries have been running for a median of 11 years against 7 for the operator-reported ones, and 88.9% of them are still live. Academics study systems that lasted long enough to be worth a multi-year study and a journal cycle; the pilot that folded after eighteen months never becomes a paper. The academic subset is not a random sample of the atlas, it is a sample of survivors, and at 90 entries a handful of cases moves it. So the claim the table supports is narrower than "scrutiny finds failure": among the groups that produced a figure at all, the operator-reported share of failure is the lowest, and independent and press examination both land several times higher. Whether self-reporting omits shortfalls, or outside evaluations simply get commissioned once a project is contested, this data cannot say. Both are ordinary. Both are reasons to grade the source.

4 🕒 The newest projects are the least examined

Evaluation lags deployment, and the gap is wide. Of the entries that started before 2010, 26.8% have been examined by someone at arm's length. For projects launched since 2020, 10.6%.

This is the least surprising finding here and possibly the most consequential. A city choosing technology in 2026 is choosing from the cohort with the thinnest evidence base, and the systems with a decade of independent study behind them are the ones now being replaced. The evidence arrives, on average, after the procurement decision.

5 💸 Four out of five entries have no public price

A cost figure could be traced for 20.2% of published entries — 236 of 1167. The rest are public infrastructure whose price is not in any source we could find. Where money is visible, it is visible unevenly: 38.0% of public-private partnerships carry a cost figure, against 9.5% of privately run systems.

The pattern is not that PPPs are more open by disposition; it is that a contract award has to be published and an internal budget line does not. Publication follows procurement law, not policy on transparency.

6 🗂️ The evidence is thinnest where the spending is heaviest

Independent examination is not spread evenly across the field. Mobility is the largest category in the atlas at 443 entries and one of the least independently examined; energy is the thinnest of all, with 3.8% peer-reviewed.

Environmental projects score best, which is less a compliment to city departments than a side effect of academia: air quality, heat and flooding are established research subjects with people already measuring them for other reasons. Where no research community happens to be watching, nobody is.

How current the record is

Every published entry carries the date its sources were last checked, shown on the entry itself. Across the atlas the median entry was checked 26 days ago, 0 are older than 365 days, and 0 carry no check date at all. The middle figure is the one to watch. A source check describes a moment: portals move, operators are replaced, pilots end without an announcement, and a page that was accurate the day it was written can be quietly wrong two years later. Re-verification is therefore part of the editorial cycle rather than a one-off, and the build refuses to stay silent about entries that drift past the threshold.

What this report is not

• Not a sample of the world's smart-city projects. This atlas is curated. Entries are selected for being documentable, which biases the corpus toward projects that left a public trail — and away from the ones that quietly never happened. Every share here describes this record, not the field.
• Not a quality ranking. The evidence grade says who examined a project, not how well it worked. Some of the best entries in the atlas are graded official because nobody outside the operator has ever looked.
• Failure counts are a floor. An entry is only recorded as failed or scaled back when a named source says so. Quiet abandonment rarely produces one, so the true share is higher than 9.6% — by how much, we cannot say.
• "Ongoing" is a default, not a verdict. 76.3% of entries are recorded as ongoing, which means running with no clear outcome yet. It should not be read as success and does not belong in a success rate.
• The grader is not independent of the graded. The same AI-assisted process that wrote the entries assigned the grades. That is a real limitation, stated here rather than in a footnote — the defence is that the rule set is written down, the sources are on every page, and the data is open enough for anyone to grade it differently.

Check this yourself

Nothing on this page is a private calculation. The whole corpus is published as JSON under CC BY 4.0, and every figure above is a count over two fields — evidence and outcome — plus cost, cat, fundingModel and yearStart. Download the data, group by the field, and you will get these numbers or find a mistake worth telling us about.