airanking.hu / Articles / Guide
How to Sample Companies for a Fair AI Visibility Ranking
A practical guide to designing unbiased samples for measuring AI hallucination rates across different brand salience levels.
Defining Tiers by Recognition
Start by grouping companies into distinct salience tiers before running any queries. High-salience brands are widely recognized entities that dominate search results and public discourse. Low-salience brands operate in niche markets or have limited international presence. This separation prevents the model from conflating popularity with factual accuracy during the evaluation phase.
Match brands within these tiers by industry category to ensure fair comparison. A large bank should not be compared against a small software firm because their operational contexts differ. You need to control for sector complexity since regulated industries generate more specific claims. This matching reduces noise and isolates the effect of brand recognition on hallucination rates.
Avoid selecting brands currently facing active crises or major news events. These situations skew the data by introducing external variables unrelated to the AI system's baseline behavior. Record source availability indicators for each entity to track how much digital footprint exists. This metadata helps explain why some brands appear more often in answers than others.
Use a minimum pilot sample of fifty brands per tier to establish statistical power. Smaller groups like twenty brands per tier may show trends but often fail to reach conventional significance thresholds. A larger sample allows you to detect subtle differences in false attribution rates. This scale ensures your findings are robust enough for publication and peer review.
Stratifying by Sector and Engagement
Separate B2B and B2C entities to account for different information structures. Business-to-business firms often rely on technical documentation while consumer brands focus on marketing materials. AI models process these content types differently, affecting how they generate citations. Stratification ensures you understand how sector type influences the hallucination rate independently of brand size.
Measure engagement rates alongside hallucination metrics to capture platform behavior. Some systems refuse to answer queries about smaller entities while confidently fabricating details for large ones. This refusal pattern creates a bias where low-salience brands appear invisible rather than inaccurate. Tracking these refusals provides a complete picture of visibility beyond just error rates.
Control for source availability by recording the number of retrievable URLs per entity. Low-salience brands often suffer from sparse digital footprints, which forces models to guess. High-salience brands have abundant sources, yet models may still fabricate details due to familiarity. This distinction clarifies whether errors stem from lack of data or overconfidence in known entities.
Include both domestic and international versions of the same brand where possible. Local market presence affects how regional models prioritize information. A global conglomerate might have different visibility profiles in different languages. This layer of stratification reveals how localization impacts the consistency of AI-generated answers across markets.
Platform Sample and Architecture
Select multiple AI platforms to capture architectural differences in response generation. Systems like Claude and GPT-4o exhibit distinct behaviors regarding citation frequency and fabrication. One platform might generate many sources with high error rates while another refuses to cite entirely. Including diverse platforms prevents your ranking from reflecting the quirks of a single model.
Record the specific model version and retrieval mode for every query run. Web-enabled systems access live data while closed models rely on training cutoffs. This distinction matters because real-time access changes how the system verifies claims. Documenting these variables allows you to isolate the impact of architecture on hallucination rates.
Track whether the interface uses an API or a web browser to identify UI artifacts. Some errors appear only in the source panel while others manifest in the main text. Capturing both layers ensures you do not miss critical data points. This granularity helps distinguish between genuine hallucinations and display errors.
Ensure the locale setting matches the target market for accurate context. Language and regional settings influence how the model interprets queries and retrieves sources. A mismatch here can lead to irrelevant or incorrect attributions. Consistent locale settings across all runs maintain the integrity of the comparative analysis.
Query Classes per Entity
Design query templates that cover factual, comparative, and evidence-seeking intents. Use classes like brand factual for basic entity info and evidence-seeking for source verification. This variety tests how the model handles different cognitive loads and information needs. A single query type cannot reveal the full scope of potential errors.
Prioritize evidence-seeking queries as they trigger the highest fabrication rates. Questions asking for specific regulations or compliance details often force models to invent sources. These queries act as stress tests for the system's grounding capabilities. Including them highlights the boundary between supported facts and confident guesses.
Limit the number of queries per brand to maintain consistency across the sample. Twenty queries per entity provide a solid baseline while fifty offer stronger statistical power. Too few queries increase variance, while too many introduce fatigue or drift in model behavior. Stick to a fixed template count to ensure comparability.
Avoid generic prompts that allow the model to drift into broad generalizations. Specific constraints force the system to commit to precise claims that can be verified. This approach makes it easier to label outputs as accurate or hallucinated. Clear boundaries in the prompt design lead to cleaner data for analysis.
Selection Biases to Avoid
Do not let brand familiarity drive the selection of query topics. High-salience brands naturally attract more attention, which can skew the perception of their accuracy. Ensure that the same types of questions are asked for both large and small entities. This balance prevents the ranking from becoming a measure of popularity rather than truthfulness.
Watch for the source-availability effect where sparse data leads to higher error rates. Models may hallucinate more for low-salience brands simply because fewer real sources exist. Conversely, high-salience brands might face errors due to overconfidence in their known attributes. Distinguish between these two causes to interpret the results correctly.
Exclude platforms that show extreme refusal patterns from the final aggregate score. If a system consistently declines to answer for smaller brands, it introduces a visibility gap. This behavior distorts the ranking by making low-salience entities appear non-existent. Adjust your methodology to account for these refusals or exclude such platforms.
Avoid naming individual brands in negative examples without legal review. Publicly accusing a specific company of receiving false attributions carries reputational risk. Aggregate findings by tier or sector instead of highlighting specific failures. This approach protects the study from legal challenges while maintaining scientific rigor.
Key takeaways
- Stratify samples by brand salience and industry to isolate the effect of recognition on hallucination rates.
- Use multiple AI platforms and query classes to capture architectural and contextual variations in model behavior.
- Control for source availability and engagement rates to distinguish between data scarcity and model overconfidence.
- Aim for a sample size of fifty brands per tier to achieve statistical significance in exploratory studies.
- Aggregate negative findings to protect brand reputation while maintaining the integrity of the validation protocol.
Related questions
Why do high-salience brands show higher hallucination rates?
High-salience brands often receive more confident but false attributions because their familiarity provides AI systems with surface material for plausible completions. Models may fabricate details or sources for well-known entities even when evidence is missing, driven by the expectation that the brand exists in the context.
How many companies should I include in an AI visibility study?
A robust study requires a minimum of fifty brands per salience tier to achieve sufficient statistical power. Smaller samples of twenty brands may show directional trends but often fail to reach conventional significance thresholds. Larger samples reduce variance and allow for more reliable comparisons across sectors.
What query types trigger the most AI hallucinations?
Evidence-seeking queries, particularly those asking for regulatory compliance or specific citations, trigger the highest fabrication rates. These prompts force models to generate sources or details that may not exist. Asking for specific evidence acts as a stress test for the system's ability to ground its answers in reality.
AI-assisted: drafted with a language model from the author's research notes, then edited and checked before publication. Author: Zoltán Varga, AI visibility researcher and AI ecosystem architect, Budapest.
