AI Shopping Product Recommendations Differ by Model: How to Test

AI shopping product recommendations differ by model because each model weighs the same catalogue data differently: review counts, price, listing wording and a built-in idea of a "safe" default. In a published test of three Claude models on the same 20 Shopify products, none agreed on the top pick in any of six searches.

Key takeaways

• Models given identical product lists pick different winners. Expect your store to be the pick on one engine and absent on another.
• Run each shopping prompt several times per engine. One answer is a sample, not a result.
• The model reasons from the data it is handed (rating, review count, summary, features), not from your product page. Check both and fix the gaps.
• Write listings in the words buyers use when they state a need, such as a budget or a skin type. Those phrases decided picks when a need was stated.
• Every claim in a listing can be repeated to the shopper as the agent's own reason. Keep claims true and checkable.
• Log the first product named, the source cited and the price quoted for each engine, then repeat monthly.

Why do different AI models recommend different products?

A shopping answer is two steps. First something retrieves a list of candidate products: a catalogue, a feed or a web search. Then a language model chooses from that list and explains the choice. The first step is data. The second is model preference, and it is where the disagreement comes from.

The clearest public test of the second step comes from Slobodan Manic, Search Engine Journal, November 2026. He saved the top 20 results for six searches from Shopify's catalogue through Shop MCP, then gave Claude Haiku 4.5, Sonnet 5 and Opus 5 the same list, the same short instruction and the same request, 50 times each. The most-picked product matched across all three models in none of the six searches.

Shopify announced Shop MCP on 13 October 2026. In Shopify's words, it "lets your agent or app search Shopify Catalog, build carts, and complete Shop Pay checkout" through a single connection. Any shopping agent can now pull a list of products, and a model decides which one the shopper sees.

That study tested three models from one company, with no tools and no web access. It did not test ChatGPT or Gemini, and its author says shopping features in those apps run their own setup around the model. Treat it as the clean laboratory version of the problem. The mechanisms it found are the ones to look for in your own category.

Four mechanisms behind the disagreement

Different engines also retrieve differently. Euromonitor, October 2026 found that Nizoral appeared in 58% of Gemini responses and 24% of ChatGPT responses, and that 58% of Gemini's citations came from retailer product pages. Gemini included specific pricing in 96% of responses against 61% for ChatGPT.

What the Shopify catalogue test found, and what it did not

The numbers are worth reading closely because they separate model behaviour from catalogue behaviour.

• Self-consistent, cross-model inconsistent. Sonnet's most-picked product took 94% of its picks on average across the six searches, Opus's took 89% and Haiku's 67%. Change the model and a different product takes the lead.
• The catalogue's own ranking mattered little. Haiku picked the product Shopify ranked first 8% of the time, Sonnet never and Opus 20%.
• Stated needs collapse the disagreement. With "under $25, for oily skin" added, all three models stopped picking the $48 top-ranked sunscreen and chose one under $25 in 298 of 300 picks.

The last point is the most useful for a merchant. Disagreement is widest when the shopper says nothing specific. Buyers who state a budget, a use or a constraint give every model something to match, and listings that mirror those words win.

A worked example: one bottle, three review counts

The same study shows what happens when the data an agent sees differs from the product page. Shopify's catalogue listed nobl's Stainless Steel Water Bottle at 5.0 stars from 447,221 reviews. The product page said 5.0 from 6,537 when the author checked on 24 October. The catalogue record for the Blush colour said 4.2 from 1,674.

Divide 447,221 by 6,537 and you get about 68, which is the factor the author reports. Haiku picked that bottle 45 times in 50 and quoted the larger figure every time. Sonnet picked it 39 times. Opus never did, choosing the Owala FreeSip (4.6 from 34,394 reviews) all 50 times. The author says he does not know what 447,221 counts. The lesson holds regardless: the model reasons from the figure it receives, and a store that has never looked at that figure cannot know why it wins or loses.

How can an online store check which products ChatGPT and Gemini recommend?

Manual checks give you a baseline in an afternoon. The method below is built to survive the variance that makes single answers misleading. In a Product.ai study of 220 shopping questions run five times each, reported by TechRepublic, October 2026, ChatGPT gave the same top pick in all 10 runs for 8.3% of questions and Gemini for 1.3%. The tests used provider APIs, not the consumer apps, but the lesson stands: repeat every prompt.

1. Write 12 prompts in three tiers. Four bare category prompts ("coffee beans"), four with a stated need ("coffee beans, under $20, for cold brew"), four comparison prompts naming a rival. The bare tier shows raw model preference. The need tier shows whether your listing wording matches. The free AI Prompt Finder shows the questions buyers ask about your category, tagged by intent.
2. Pick the engines. At minimum ChatGPT, Gemini, Perplexity, Claude and Google AI Mode. Note that engines differ in whether they show product cards at all.
3. Run each prompt at least five times per engine in fresh sessions, logged out where possible, from the market you sell in. Results vary by location and account, so say which you used.
4. Log five fields per run: first product named, every brand named, the store or page cited, the price quoted, and any review figure or claim repeated.
5. Score two numbers per engine: how often your product is named at all, and how often it is named first. Report them as counts out of runs, such as 7 of 20, not as rankings.
6. Compare with your source of truth. For each product named, check the engine's price, rating and claims against your product page, your feed and your structured data.

A log template

Keep the same prompts and the same wording every month. A change is only meaningful if the question stayed fixed. The guide to choosing prompts to track has a scoring sheet, and AI citation stability explains why one run proves little.

Which product data should you fix first?

Start with the data a model is most likely to reason from. The Shopify test says this is the rating, review count, one-line summary and first listed features. For ChatGPT, Search Engine Journal, September 2026 reports Profound data showing the share of ChatGPT Shopping picks classed as feed-integrated rose from 8.26% to 61.54% on 10 July, across 1,757,723 tracked prompt runs. The same article notes the data covers one vendor's tracked prompts, not real shopper sessions. For Google, the feed matters too: PPC Land, May 2026 lists Merchant Center attributes tied to AI Mode, including product highlight, product detail, variant option, item group title and question and answer.

For markup, build and validate Product and Offer JSON-LD with the free Schema Markup Generator, which prefills from your site and validates the result so you can check price, availability and review fields match the page. Note that structured data tells engines what your page says; it does not make a model prefer it.

How should you read disagreement between engines?

Treat disagreement as information about where your catalogue is fragile, not as noise to average away.

• You win on one engine and vanish on another: look at what each cites. Different sources feed different engines.
• You are named but not first: compare the winning product's review count, price and listing phrases with yours.
• Your rank flips between runs: you are near a decision boundary. Small data fixes, such as a clearer summary, are most likely to tip it.
• The engine's reason is wrong: trace it to a source, then correct that source. The guide to when AI gets your brand wrong covers the steps.

The test author makes a point worth keeping. The decision layer between shopper and store is hidden, and each shopper's agent may decide differently. You cannot monitor it the way you monitor a Google ranking, so sample it, keep the prompts fixed and report ranges. For the wider picture of engine disagreement, see what the data shows on top-brand disagreement and the AI shopping shelf playbook.

Common mistakes when testing AI shopping picks

• Running a prompt once. The Product.ai figures above show top picks move between repeat runs.
• Only testing branded prompts. Buyers who already know your name are not the ones you are losing.
• Judging one engine and assuming the rest. Euromonitor found brand visibility differing by up to 34 percentage points between ChatGPT and Gemini for one product.
• Auditing the page, not the data the engine sees. The water bottle shows the two can differ by a factor of 68.
• Reporting wins without the denominator. Say 7 of 20 runs, on which engine, on which date.

Where Noma fits

Noma by Nurdd is an AEO and AI visibility platform. It asks 13 AI engines, including ChatGPT, Gemini, Perplexity and Claude, the questions your buyers ask, every day. It shows where your brand is named, where competitors are named instead, and which pages the answers draw on, so the engine-by-engine disagreement in this guide becomes a daily measurement instead of a manual log. Noma does not track Microsoft Copilot or Meta AI.

See it on your own brand: book a demo or start the 3-day free trial. A card is taken at signup so billing can start on day 3; cancel before then and nothing is charged. For setup detail see prompt tracking, and for share of voice across rivals try the free Competitor AI Share of Voice report.

How we researched this

We reviewed one published model-comparison test from Search Engine Journal, a vendor announcement from Shopify's developer changelog, market research from Euromonitor, an industry news report on a third-party study, and trade-press coverage of Merchant Center attributes. We also read comparison guides on AI shopping assistants to see what they leave out. We did not run our own product-pick tests, and every figure is taken as published by its source. Facts, dates and figures were checked on 2026-11-09.

What to do next

Pick one product category, write the 12 prompts, and run the five-run log on ChatGPT and Gemini this week. Then compare what each engine quotes with your page and feed. If you want that measured daily across engines, book a demo or start the 3-day free trial. The ChatGPT Shopping checklist covers crawler access and Product schema while you do.