# AI Answer Benchmarks by Category: How to Run Your Own

> Run an AI answer benchmark by category: ask ChatGPT, Gemini, Perplexity and Claude buyer prompts, log brands and sources, compare share of voice.

- Author: Sheetal Shakshi, Founder's Office, Nurdd
- Published: 2026-11-09
- Publisher: Noma by Nurdd (https://nomahq.app)
- Original page: https://nomahq.app/blog/ai-answer-benchmark-by-category

An AI answer benchmark by category is a fixed set of buyer questions put to several AI engines on the same day, with every brand named and every source cited logged, so you can see how a whole market is described, not just where your own brand sits. Run it with 20 prompts, four engines and three repeats, then compare share of voice.

## Key takeaways

- Benchmark the category, not one brand: the unit is a set of buyer prompts, and every brand in the answers gets scored on the same terms.
- Ask ChatGPT, Gemini, Perplexity and Claude the same prompts on the same day, three times each, and record the date, the engine and the exact wording.
- Log four things per answer: brands named, order, sources cited and the words used about each brand.
- Report share of voice as a fraction with its numerator and denominator, and report each engine separately before you combine them.
- Treat single answers as samples. SparkToro found repeated runs almost never return the same brand list, so only rates across many runs mean anything.
- Compare your result with up to three rivals using the [free Competitor AI Share of Voice tool](https://nomahq.app/tools/ai-share-of-voice), then repeat monthly on the same prompt set.

## Why benchmark a category instead of ranking one brand?

A brand-level score answers "how are we doing?" with no reference point. A category benchmark answers "who does each engine name when a buyer asks about this market, and why?" That second question is the one that changes decisions: which pages to fix, which publishers to pitch, which rivals to study.

Three reasons the category view is fairer and more useful:

- **It removes the self-selection problem.** When you test only your own brand, you tend to write prompts that name it. A category benchmark uses unbranded prompts, so every brand starts from the same position.
- **It survives the noise.** Single-brand positions swing from run to run. Category-level rates, such as the share of answers naming each brand, are steadier because they pool many answers.
- **It shows the sources.** The cited pages for a category are shared by every brand in it. Finding the review sites, comparison posts and forums that engines lean on tells you where to earn mentions, whoever you are.

The IAB's August 2026 framework, *Measuring Visibility in the AI Era*, makes the case for a shared standard: it says "Only 16% of brands systematically track AI visibility today, in part because there is no common standard" ([IAB, August 2026](https://www.iab.com/wp-content/uploads/2026/07/IAB_Measuring_Visibility_in_the_AI_Era_August_2026-1.pdf)). A benchmark you define and publish internally, with its method and date, is a small version of that standard.

## What does the research say about how stable AI answers are?

Stability decides how you design the test, so read this before you write a prompt.

SparkToro and Gumshoe.ai ran 2,961 prompts across ChatGPT, Claude and Google Search AI Overviews, using 12 brand-recommendation prompts that each ran 60 to 100 times per platform. ChatGPT and AI Overviews each returned the same brand list less than 1% of the time, and the same list in the same order less than 0.1% of the time. The researchers found that visibility percentage across many queries was more consistent than ranking position ([Search Engine Journal, January 2026](https://www.searchenginejournal.com/ai-recommendations-change-with-nearly-every-query-sparktoro/566242/)).

The practical reading: one answer proves nothing, and "position 2 in ChatGPT" is a weak claim. "Named in 11 of 30 answers" is a strong one. Build the benchmark around rates. Our own guide to [AI citation stability](https://nomahq.app/blog/ai-citation-stability) covers the same issue for sources.

### Engines also disagree with each other

Bluefish, as reported by PPC Land in November 2026, found that ChatGPT and Google put a different brand at the top of their AI answers in 35% of 5,810 buying topics. PPC Land notes that Bluefish describes its method in a single sentence and does not name the categories, the Google product queried or the dates, so treat the figure as a direction, not a law ([PPC Land, November 2026](https://ppc.land/chatgpt-and-google-differ-on-top-brand-in-35-of-buying-topics-bluefish-says/)).

The same PPC Land article reports a separate study of UK software by Recrawled, covering 1,608 questions, in which YouTube appeared in 25% of AI Overview answers and in none of 360 ChatGPT answers ([PPC Land, November 2026](https://ppc.land/chatgpt-and-google-differ-on-top-brand-in-35-of-buying-topics-bluefish-says/)). Source mixes differ sharply by engine, so count per engine.

Semrush's AI Visibility Index 2026 points the same way. It analysed 126 million US AI search prompts from January to April 2026 and found only 36 brands kept top-100 visibility across every platform it tracked ([Semrush via Business Wire, June 2026](https://uk.finance.yahoo.com/news/semrush-releases-expanded-2026-ai-130200643.html)). A brand that looks strong in one engine can be absent in another, which is why a benchmark must report engines separately. More on why in [why AI engines disagree](https://nomahq.app/blog/why-engines-disagree).

> "Only 16% of brands systematically track AI visibility today, in part because there is no common standard." IAB, *Measuring Visibility in the AI Era*, August 2026.

## How do I benchmark which brands AI engines recommend in my category?

Follow these seven steps. Each one has an output you can file.

1. **Define the category in one sentence.** Use buyers' words, not your product taxonomy. "Project management software for agencies" beats "work orchestration platform". Output: a category label and a market (country and language).
2. **List the brands you expect.** Write 8 to 15 names, including yours, plus an "other" bucket for names you did not expect. Output: a brand list with alternate spellings.
3. **Write 20 unbranded buyer prompts.** Mix five types: discovery ("best tools for X"), comparison ("A vs B for Y"), use case ("what should a 10-person team use for Z"), problem ("how do I fix W") and trust ("is X worth it"). Keep none of them naming a brand, except the comparison set. The [free AI Prompt Finder](https://nomahq.app/tools/ai-prompt-finder) shows questions buyers ask in a category, tagged by intent, if you need a starting list. [How to choose prompts to track](https://nomahq.app/blog/choose-prompts-to-track) gives a scoring sheet.
4. **Freeze the wording.** Copy each prompt into the template below with an ID. Do not rephrase between engines.
5. **Run all engines on the same day, three times each.** Use a fresh session or a logged-out window for every run, and note whether web search was on. 20 prompts x 4 engines x 3 runs gives 240 answers.
6. **Log each answer.** Brands named in order, sources cited, a short phrase on how each brand is described, and any factual error.
7. **Calculate and read.** Per engine and per prompt type, compute each brand's share of answers, then read the sources and the disagreements.

Manual work at this scale takes several hours. That is fine for a first baseline, and it teaches you how the answers look before you trust any tool's summary. For ChatGPT specifically, [a repeatable method to check whether ChatGPT recommends your business](https://nomahq.app/blog/check-if-chatgpt-recommends-my-business) goes deeper.

## What is a good template for an AI answer benchmark report?

Use two sheets: a raw log with one row per answer, and a summary built from it. The raw log is what makes the benchmark auditable.

### Sheet 1: the answer log

| Column | What to record | Why it matters |
| --- | --- | --- |
| Run date | YYYY-MM-DD, plus time zone | Engines and indexes change; the date makes the result citable |
| Engine and mode | ChatGPT, Gemini, Perplexity or Claude; web search on or off | Answers with and without search draw on different material |
| Prompt ID and text | Exact wording, prompt type, market | Lets anyone repeat the run |
| Run number | 1, 2 or 3 | Separates repeat variance from real differences |
| Brands named | Each brand, in order | Feeds mention rate and order notes |
| Sources cited | Full URLs, then the domain and page type | Shows what the engine relied on |
| Description words | The adjectives and claims next to each brand | Shows portrayal, not only presence |
| Errors | Wrong prices, features, or company facts | Gives you a fix list |

### Sheet 2: the category summary

| Section | Contents |
| --- | --- |
| Header | Category, market, run dates, engines, prompt count, runs per prompt, total answers |
| Brand table | Each brand's answers named in, divided by total answers, per engine and overall |
| Prompt-type split | The same rates for discovery, comparison, use-case, problem and trust prompts |
| Source table | Top 10 cited domains per engine, with page type (review, forum, vendor, news, video) |
| Disagreement list | Prompts where engines name different leading brands |
| Errors and actions | Each wrong claim, its likely source, and the owner |
| Limits | Sample size, repeat variance, what you did not test |

Your template covers three things any visibility measure needs: presence is the mention rate, prominence is the order, and portrayal is the description words. A prompt test cannot show whether an answer led to a click or a sale; for that, see [measuring AI influence beyond referrals](https://nomahq.app/blog/geo-referral-traffic-measurement).

## How do I compare ChatGPT, Gemini and Perplexity answers for the same buyer question?

Put the answers side by side for each prompt and ask four questions in the same order.

| Check | What to compare | What a gap tells you |
| --- | --- | --- |
| Who is named | Brand sets per engine | A brand missing from one engine usually lacks the pages or mentions that engine reads |
| Who leads | First-named brand and order | Different leaders suggest different source pools |
| What is cited | Domains and page types | An engine citing forums where another cites vendor pages points to where to earn mentions |
| How it is described | Adjectives, price claims, fit claims | Mismatched descriptions flag outdated or inconsistent sources |

Judge a difference as real only when it holds across the three repeats. If ChatGPT names a brand in three of three runs and Gemini in none of three, that is a gap worth chasing. If the split is 2 of 3 against 1 of 3, collect more runs first.

Noma by Nurdd's [post on AI engines disagreeing about the top brand](https://nomahq.app/blog/ai-engines-disagree-on-top-brand) shows how to run the engine-by-engine check for a single topic.

## How can I see which sources AI engines cite for my industry?

Collect every URL cited across the 240 answers, then group them. Three groupings pay off:

- **By domain.** Count how many answers cite each domain. The top 10 per engine is your target list.
- **By page type.** Label each URL as vendor page, review site, comparison article, forum thread, news, video or documentation. The mix differs by engine and by category.
- **By owner.** Separate pages you control from pages others control. Owned pages need edits; third-party pages need outreach, corrections or earned coverage.

Do not assume one engine's source mix applies to another. The Recrawled figure reported by PPC Land above, YouTube in 25% of AI Overview answers and none of 360 ChatGPT answers in a UK software study, is a reminder to count per engine. For a worked method on third-party pages, see [how to get listed on pages AI engines cite](https://nomahq.app/blog/best-aeo-tools-lists-ai-engines-cite) and the [Earned Media for AI Answers](https://nomahq.app/resources/earned-media-for-ai) playbook.

A cited source is not always a named brand. A page can be cited while a different brand gets the mention, which is why the log keeps both. See [citation versus brand mention](https://nomahq.app/blog/citation-vs-brand-mention).

## How do I calculate share of voice, with an example?

Share of voice here is the number of answers that name a brand divided by the total number of answers. State both numbers. A rate without its denominator cannot be checked.

Worked example with invented counts, to show the arithmetic only (these are not measured results). Suppose 240 answers were logged for a category. Brand A is named in 54, Brand B in 96 and Brand C in 30.

| Brand | Answers naming it | Total answers | Mention rate |
| --- | --- | --- | --- |
| Brand A | 54 | 240 | 22.5% |
| Brand B | 96 | 240 | 40.0% |
| Brand C | 30 | 240 | 12.5% |

Two ways to express share of voice, and they answer different questions. Mention rate (above) says how often a brand appears. Share of mentions divides a brand's mentions by all brand mentions logged, so the shares add to 100% across the brand list. Pick one, label it, and keep it across reports. For more on the maths and sample sizes, read [how to measure AI share of voice across engines](https://nomahq.app/blog/measure-ai-share-of-voice).

Report per engine before pooling. A pooled 22.5% can hide a brand at 40% in one engine and 5% in another.

## Is there a free tool to compare my brand's AI share of voice with competitors?

Yes. The [Free Competitor AI Share of Voice tool](https://nomahq.app/tools/ai-share-of-voice) from Noma by Nurdd compares how often ChatGPT, Gemini, Claude and Perplexity name you against up to three competitors, and which sources they cite. Use it as a quick cross-check of your manual sheet: enter your brand and three rivals from your category list, then compare its per-engine picture with your own log.

Differences between the tool and your sheet are expected, since prompts differ and answers vary. Compare direction and ranking of brands, not exact percentages. If both show the same brand ahead on the same engine, trust it more; if they conflict, add runs.

## What can a one-off benchmark not tell you?

- **Trends.** One run is a snapshot. Repeat on the same prompts monthly to see movement.
- **Real buyer volume.** AI engines publish no prompt volumes, so your prompt set reflects judgement, not demand. See [AI prompt volume](https://nomahq.app/blog/ai-prompt-volume).
- **Personalisation.** Logged-out tests miss answers shaped by a user's history or location.
- **Revenue.** Mentions are an input. Link them to pipeline with referral data and a "how did you hear about us" field, as covered in [the AI search attribution guide](https://nomahq.app/blog/ai-search-attribution-guide).

HubSpot's November 2026 guide on benchmarking AI search against your industry makes a similar point: because answers vary run to run, benchmark measures are estimates based on a defined prompt set ([HubSpot, November 2026](https://blog.hubspot.com/marketing/ai-search-industry-benchmarks)). State that limit in every report.

## Common mistakes to avoid

- Writing prompts that name your own brand, which inflates your rate.
- Running engines on different days, then blaming the engine for a difference caused by timing.
- Reporting one run per prompt.
- Pooling engines before looking at each.
- Dropping the denominator.
- Changing the prompt set between months, which breaks comparison. Add prompts in a new version instead.

## Where Noma fits

Noma by Nurdd asks 13 AI engines, including ChatGPT, Gemini, Perplexity and Claude, the questions your buyers ask, every day, from a fixed prompt set. It shows where your brand is named, where competitors are named instead, each engine's visibility, position and sentiment, and which pages the answers cite. That turns the manual sheet above into a daily series you can compare over time. [Book a demo](https://nomahq.app/demo) or [start the 3-day free trial](https://platform.nomahq.app/signup) to see it on your own brand.

The trial takes a card at signup so billing can begin on day 3; cancel before then and nothing is charged. Plans and prices are on the [pricing page](https://nomahq.app/pricing).

## How we researched this

We reviewed a published research study on answer repeatability (SparkToro and Gumshoe.ai, as reported by Search Engine Journal), vendor and trade reports on engine disagreement (Bluefish and Recrawled via PPC Land, Semrush), an industry measurement framework (IAB) and a marketing publisher's benchmarking guide (HubSpot). Noma has no in-house category data to publish, so this post includes no Noma-measured figures. The worked example uses invented counts and says so. Facts, dates and links were checked on 2026-11-08.

## What to do next

This week, fix your category sentence, write the 20 prompts and run the first pass by hand. Then cross-check with the [free share of voice tool](https://nomahq.app/tools/ai-share-of-voice). When you want it running daily across engines, [book a demo](https://nomahq.app/demo) or [start the free trial](https://platform.nomahq.app/signup).