Case study / synthetic_data.txt
The vendor data was synthetic, so I built the real thing
I built the audit, methodology, prompts, automation, and code. I used Claude to help with the coding.
The beta was the starting point. I continued developing it afterward.The dashboard was confident.
The raw responses told another story.
The audit changed the question from “What does the dashboard say?” to “What actually happened when the model answered?”
Problem. Commercial AI-visibility tools charge enterprise money to report how LLMs answer your category’s questions. The numbers looked confident. Something smelled off.
Audit. I pulled the raw API responses instead of the dashboard. The field indicating live retrieval was false on 100% of 4,132+ audited answers. The “per-country” data correlated at 0.998 — meaning no real geography. And the persona prompts contained the brand name, inflating persona mention rates to 62–83% against a 25% clean baseline. The retrieval was simulated.
Build. I replaced it: a real-retrieval pipeline against live model APIs — 46 prompts × 5 country/language views × 7 clean personas, 1,610 answers per run — logging every background search the model actually runs, every citation, every brand, and sentiment. Multilingual by design. Automated weekly.
Why it matters. If the measurement is fiction, every strategy built on it is too. Trust, but pull the raw responses.
Validate the data.
Understand the mechanics.
Save budget.
I wanted to validate the data I was seeing in the original tool, learn more about AEO by understanding what was happening underneath, and save budget.
As I got further into the build, I realized I could collect additional actionable data I wasn’t getting from the vendor. That made it possible to build a better tool for the company’s specific needs.
The searches behind the answer.
We built a query fan-out flow to understand how the models’ behavior was evolving. It let us examine the searches generated behind an answer, rather than looking only at the final response.
We noticed those searches were bringing in brand names and even using the site: operator. That gave us a more useful view of how brands and specific websites entered the retrieval process.
We used the patterns to form a working view of which answers seemed more dependent on stored model knowledge, where we expected results to be harder to change, and which still leaned heavily on live search.
That distinction was a working hypothesis about answer behavior, not a direct inspection of the models’ training data.
More time to build.
Less compromise afterward.
Buying off the shelf would have saved build time. Building took roughly two weeks to reach a usable beta, with continued development afterward.
In return, we got a product built specifically for the company, access to more useful data, and a lower cost. The tradeoff was time; the payoff was a better fit.
The answer was only
half the story.
queries
queries
[domain]
How a brand enters the answer.
Illustrative query patterns, not a captured run. Actual searches vary by prompt and model.
From a question.
To evidence you can inspect.
A simplified view of the documented workflow. Each run logs the model’s searches and the evidence behind its answers.