How the number is made

We measure what three AI assistants say when a buyer asks a question in a client’s category. The assistants are ChatGPT, Claude and Perplexity. Each question is sent to each of them through their APIs with web search enabled, and every answer is stored.

A separate model reads each answer and records whether the business was named, whether its site was returned as a source, and which other businesses appeared. Nothing is installed on the client’s site and no access is required for the measurement itself.

Answer presence is the business named in the answer text. Citation presence is the business’s own domain among the sources the answer cited, which can be measured only when the engine searched the web. Share of voice is measured from the same answers.

The report has three further pillars that come from the website audit. They stay audit only and report no value. A guessed number would look the same on the page as a measured one.

Every rate we publish is movement on a frozen set of questions, across a stated list of engines. It makes no claim about questions we did not ask, or about the category as a whole. That choice decides how the interval is built, and it is set out under the rate below.

A number with no stated scope gets read as a verdict on everything. Stating the scope is what lets the interval be exact about the thing it describes.

Candidate questions are ranked from what people have been seen asking. Search Console queries, which people typed, carry the most weight. Buyer questions found in the engines’ own stored answers come next, then questions a monitoring vendor suggests, then questions we wrote ourselves. A question we wrote cannot outrank one that was observed.

Each candidate is ranked on demand, on whether it leads to a page that makes money, and on headroom: a question where the site ranks in Google but is not cited in AI answers has the most room to move, and one where it is cited everywhere has none. Questions about discounts and promo codes are cost questions and are kept, because for an affiliate site they are often the ones that convert best.

Our own comparison site’s first frozen set was written from the category, and none of its ten questions matched anything a person had searched for. The query that earned 11 of its 14 organic clicks was on no scoreboard for a month.

Asking an engine (an AI assistant we measure) “is [your brand] good?” guarantees a mention and measures nothing at all. We use the questions a real buyer types, such as “what’s the best X for Y?”, and never name the client in any of them. Then we measure whether you surface unprompted.

Any generated question that names the brand or its domain is discarded before the scan runs. If you show up, you earned it. Questions containing the client’s own name sit in the taxonomy (our classification of question kinds) and are never measured.

An answer to “is this business any good” names them whatever the engines think of them, so it measures the question instead of the business. Every question we ask is one a customer asks without knowing the client exists. See brand-blind.

A question is frozen only when it names its route to revenue, for example an affiliate click to a named merchant from a named page. A route recorded for a page that does not exist yet still counts as money. The route is stored beside the set and kept outside the fingerprint, so older sets still verify.

Freezing is permanent. A question with no route to revenue would be measured forever and would drag the mean for every question that can pay. We once recommended building a comparison page for a category the client held no affiliate in, and it would have won citations worth nothing.

Only a set a person approves is frozen, and never as a side effect of an unattended scan. Profile answers we research on a client’s behalf are drafts. The freeze refuses until the client has confirmed their own intake. A category the client states is used word for word; a category we infer leaves the scan exploratory and unfrozen, because the same site read twice produced two different categories.

Once approved, the set is frozen: stored with a hash, a fingerprint computed from the exact questions, and printed on every report. Every later scan of that domain reuses it. A stored set that no longer matches its hash stops the scan. Two scans with different hashes are never compared, by code.

The before and after is only fair when both scans asked the same questions. Regenerate them and an improvement cannot be told apart from an easier test. Freezing is also what makes each question its own control.

Coverage grows by freezing a second set, never by editing the first. A later set refuses any question already frozen in an earlier one, its measurement window starts the day it is frozen, and /proof publishes the baseline set only. A later set can stand down from daily collection to save spend and resume afterwards, still frozen. The baseline set cannot stand down.

Questions divide into two kinds, and the divide matters. Ask an assistant for a category in a suburb and it answers from Google Maps and review data, so an established business usually wins. Ask it what something costs, how you compare, whether you can help with a specific problem, or who supplies an industry, and it answers from website content and third-party pages, where far fewer businesses have built anything.

A single found rate (the share of answers you appear in) averages those two halves and describes neither. So wherever each half holds at least three questions, the rate is split by kind and each half carries its own band. The classification is applied to the stored questions and kept out of them, which means it re-reads every measurement we have ever taken instead of stranding the old ones.

The same business can read as thriving or invisible depending only on which kinds of question ended up on its scoreboard.

The consumer assistants buyers use search the web before answering a buying question; that is true of ChatGPT, which we measure, and equally of assistants we do not measure. An API call without that search measures the bare model’s memory, a different artefact from the answer a buyer is shown. So every engine in our measurement set is probed with its provider’s own web-search tool: native for Perplexity, and the official search tools for ChatGPT and Claude.

Grounded probing (the engine answering with its web search on) also exposes the sources behind each answer, which is how we can measure citation presence as well as answer presence. Every score records its mode. If the mode ever changes, the system declares a new baseline on the same frozen questions and never compares across the break.

We measure the answer buyers see. The model’s memory is a different test.

AI answers change substantially from run to run. Ask the same question twice and you can get two different vendor lists. A single pass is uninterpretable: you cannot tell a real result from a coin flip.

So every question is asked again on every collection day, on every engine in our stated measurement set. Each run records whether you appeared. The rate across those runs is the measurement.

How much an engine may do per answer is set by an engine profile: how many web searches Claude may run, and how much reasoning ChatGPT’s model uses. The profile is recorded on every daily row and every score. A measurement window that mixes profiles is refused, and a change of profile starts a new baseline.

From 15 to 29 August 2026 our collection setting allowed Claude one web search per answer. The setting was restored to two searches per answer on 12 September 2026, and figures either side of that change are not compared.

An engine allowed fewer searches can find fewer sources. A rate that moved because the setting moved would be credited to the client’s work.

The collector stamps every row with a Melbourne date. A server’s default date is UTC, ten hours behind Melbourne, and a run either side of 10am could split one pass across two dates and count it twice in a window. One pass per day means one pass per day where the buyers live.

Before a pass spends anything, every engine and the grader are called with a time limit on each call. If one is out of credit the day is refused and the operator is told which provider to top up. A graded answer needs a grader, so a grader that runs dry stops the pass.

Each question’s answers are saved as soon as they are graded. A pass that is killed keeps what it collected, and a re-run fills only the missing cells. The same cell cannot be collected twice in one day.

On the first real collection day the grader ran out of credit at question three and every engine’s cell failed with it. Checking the engines without the grader missed the part that failed.

A model asked “did you mention Acme?” will tell you yes whether or not it did. So the model that produces the answer never grades it. A separate analyst model reads every response and extracts which brands were named, in what order, and whether they were recommended or only listed. A check fails the build if any engine shares the analyst’s model.

Leading questions and a model marking its own homework are the two easiest ways to manufacture a good score. Brand-blind questions close the first; a different model doing the grading closes the second.

Today the analyst is a small Anthropic model. It never grades its own answers, but it does read Claude’s, which is one lab reading its own house. That is less independence than the phrase “independent analyst” promises, and our checks warn about it on every run.

Every score records which model graded it. When two scans were graded by different models, the comparison says so, because a different grader can read the same answer differently and part of any movement would be the instrument.

The answers are stored, so when the grader changes we re-grade the stored answers and both ends of a comparison share one reader. A re-grade reads mentions again. Citations were established from the source list the engine returned at collection time, which is not in the answer text, so the stored citation flags are carried through unchanged. A re-grade is saved only on instruction and is refused when more than 10% of rows fail.

A first re-grade that re-derived citations from the answer text turned a real 29% citation rate into 9%. The source list is the evidence, and the text cannot stand in for it.

A probe that failed, from a rate limit or a timeout, leaves the denominator (the count of answers a rate is divided by). It is never counted as a miss.

A rate-limited engine has told us nothing about whether it would name you. Counting its silence against you would report an outage as invisibility.

From 15 to 29 August 2026 our collection setting allowed Claude one web search per answer. 166 of its 261 answers in that window say the search stopped early (“I’ve hit the search limit for this turn”), and each of those still carried the source list from the search that did run.

An answer that reports its search stopping did not use those sources, so counting it would state a rate for evidence the answer never had. Those answers are excluded from every Claude figure we publish, the way a failed probe is, instead of being carried with a caveat. The rows stay in the database exactly as collected; the exclusion is applied wherever a rate is computed. The rule is tested on real sentences, including ones that mention a limit without describing a stopped search.

A truncated answer is not a measurement of what the engine would find.

The named rate (answer presence) counts answers whose text names the business. The cited rate (citation presence) counts answers whose sources include the business’s own domain, read from the source list the engine returned. For a comparison or affiliate site, being the source is often the win, because the brands named in the answer are the products being compared.

On our own comparison site, measured through a retired third-party feed on 8 and 10 August 2026 (87 and 147 answers), the site was cited as a source in 29% and 31% of answers and the brand was named in 1.1% and 0.7%. On our own daily collection from 15 to 29 August 2026, 624 completed graded answers on 38 questions, the site was cited in 136 (22%), across 21 of its pages, and the brand was named in 12.

These are snapshots of one property; we hold no series that shows which rate moves first.

A report that counts only one of the two can call a working source invisible.

Visibility Score is a weighted blend of how often and how prominently the business appears, because “named last, grudgingly” and “recommended first” are different business outcomes, and a mention rate prices them identically. The cases below are asserted by an automated check on every push.

The headline rate is the mean of the per-question rates, one rate for each question on each engine. It is kept apart from a pooled rate (all hits over all answers), which drifts from the mean when probes fail unevenly. A fortnight of daily passes on twenty-five questions across three engines is about 1,050 answers, which supports a 90% band of a few points either side.

The report shows the band only when every question has more than one graded answer on each engine. That is a number worth acting on.

An interval needs more than one sample. A question asked once returns no interval at all. The report checks the data itself before it shows one, separately from whether a plan includes intervals, and both checks stay separate on purpose.

An interval drawn around one answer would present a guess as a range.

A single question on a single engine is only fourteen answers, giving a 90% interval of about 20 points either side. That cannot support a decision. So per-question rates are shown as directional counts (cited 9 of 14) with no band. The code sets 20 answers as the least a per-question band would need, and no page shows one.

The thing we estimate is the frozen set on the stated engines. For a mean over that fixed grid of questions and engines, each cell’s own sampling error is the only error there is. A cluster-robust correction would widen the band to cover questions we might have asked, and we make no claim about those.

A wider band would look more cautious while answering a question the report never asks, and it would hide real movement on the questions it does ask.

A change counts as real on the report only when the same questions, asked again, move by more than their own noise. Each question’s after is compared with its own before, and the 90% interval on the question-by-question difference must exclude zero. Each frozen question is its own control. The rule is fixed before we look.

If a month does not clear that bar, we will tell you it was within noise, and why.

Checking whether the before band and the after band overlap throws away the pairing, and it is stricter than asking whether the difference is non-zero. It would report real movement as “cannot tell” and deny a client credit for work they paid for.

Pairing needs each question’s own rate stored on the score. Older scans are filled in from their stored answers. Where that cannot be done, the comparison falls back to the overlap test and says so in its output.

Engines change on their own. A model update can lift or sink your visibility in a week with nothing you did. If we only measured you, we would take credit for the weather.

So a control is measured on the same frozen questions, in the same answers, at the same moment: nothing extra to run, and no chance of the control being measured on a different day or a different test. The attributable lift is the difference in differences, computed from the stored scores; it is a separate test from the report and is not yet printed on it.

For a brand that needs to be named, the control is the rivals named at scoping. For a comparison or affiliate site, where the brands in an answer are the products being compared, it is every other domain used as a source in the same answers; that is the control committed for our own comparison site on 10 August 2026.

On our own comparison site, the five rival brands named as controls appear in zero of 90 answers. A control built from them would subtract nothing and present the raw change as attributable.

The control is committed in writing, with a timestamp, before the outcome measurement, and it cannot be overwritten. A control committed later is reported as not pre-specified. The test refuses to run across any change of question set, source, web search mode or engine list.

Rival names suggested by a monitoring vendor are treated as signal only. Their mapping from a brand to a website is never used as a control, because it has mapped rival names to a pen manufacturer and a networking company.

A control picked after seeing who moved can be made to show anything.

Each window starts the day after the last saved measurement of the same question set. The first starts on the day the set was frozen, or on a stated start date. A window is saved once, when it holds enough collection days.

An earlier version saved a measurement on every trigger once fourteen days existed, so each window contained the last. An after that contains its before cannot show a change.

A measurement is refused when more than 10% of its rows could not be graded. The same refusal applies to a window that mixes engine profiles.

Failed rows leave the denominator, so a half-graded window would publish a confident rate computed on whichever answers came back before the grader stopped. That is a biased sample presented as a clean one.

Labs ship new models, retraining changes what a model already believes, and retrieval changes which pages get read. A number measured today describes today’s engines. When a lab ships a new model, the ground moves under everyone in your category at once: you, and every competitor, on the same day.

That is the reason a single measurement is a photograph and the series is the evidence. Your competitors are measured in the same answers at the same moment, so when the ground moves, it moves under them too; a one-off report ages with the engines, and the series does not.

A comparison is made only between scores that share every one of these. When a series crosses one, we compare within the newest run of scores that share them all, and state what was excluded and why.

A change you make today cannot appear in an answer today. Six things have to happen first, the engines run on different clocks, and the citation you earn decays whether or not anyone acts on it. All of that is set out with its evidence on how pages become sources. What decides which sources an engine reaches for in the first place differs so much between engines that only 2.7% of cited domains are shared across five of them, which is set out in the research we build on.

We run this exact standard on our own business. At our first measurement the AI engines named us in zero of 149 answers. That run is published on /data; it was an ungrounded run (engines answering without web search), so the grounded measurements that follow, on /proof, start a new baseline beside it, as they move or fail to.

Our own numbers, including the unflattering ones, are on Proof. Our live measurement ↗