A study by researchers at the National University of Singapore and Xi'an Jiaotong University found that OpenAI's API and Google search results together matched 48.7% of the sites cited by ChatGPT, but only 26.2% of the exact pages. The study tested four versions of ChatGPT, including GPT-5.5 Instant, using 80 questions in English and Chinese.
TL;DR
- OpenAI's API and Google search matched 48.7% of sites and 26.2% of pages cited by ChatGPT in a study by researchers at the National University of Singapore and Xi'an Jiaotong University.
- The study highlights the limitations of using OpenAI's API and Google search as substitutes for ChatGPT's citations, as they matched sites more often than exact pages.
- The study suggests that monitoring strategies for ChatGPT's citations should consider both webpage and site levels and account for ChatGPT's prompt rewriting behavior.
What happened
The study, posted on Oct. 11, 2026, tested whether OpenAI's API and Google search results could stand in for ChatGPT's citations. The researchers found that the two combined matched 26.2% of the pages and 48.7% of the sites cited by ChatGPT's GPT-5.5 Instant setting. The study included four versions of ChatGPT, with the majority of data collected from March 29 through July 29, 2026.
The researchers used 80 questions from Search Arena, a public database of user interactions with search-enabled chatbots, with 40 questions in English and 40 in Chinese. The questions covered four types of requests: recommendation, fact lookup, information synthesis, and analysis. Each question was run 20 times on GPT-5.3 Instant and GPT-5.5 Instant, while GPT-5.2 Instant and GPT-5.4 Thinking had fewer questions due to being decommissioned before data collection was completed.
Why it matters
The study highlights the limitations of using OpenAI's API and Google search as substitutes for ChatGPT's citations, as they matched sites more often than exact pages. This suggests that monitoring strategies for ChatGPT's citations should consider both webpage and site levels.
The study also found that ChatGPT rewrote questions and changed sources between runs, which could impact the accuracy of monitoring strategies. The authors suggest running the same prompt multiple times and taking into account ChatGPT's prompt rewriting behavior.
The study did not test any commercial tracking tools, leaving open questions about the effectiveness of such tools in monitoring ChatGPT's citations. Additionally, the study predates the release of GPT-5.6 and GPT-6, which could have different citation behaviors.
Key facts
- OpenAI's API and Google search matched 26.2% of pages and 48.7% of sites cited by ChatGPT's GPT-5.5 Instant setting.
- The study included 80 questions from Search Arena, with 40 in English and 40 in Chinese.
- Google search results alone accounted for 6.4% of cited pages and 22.8% of cited sites.
- The OpenAI API's entire source list accounted for 22.5% of all cited pages.
- ChatGPT rewrote questions and changed sources between runs, with 44.5% of distinct pages returned for a given question shown in only one of the 20 GPT-5.5 Instant runs.
- The study predates the release of GPT-5.6 and GPT-6, which could have different citation behaviors.
- The study did not test any commercial tracking tools.
- The study suggests that monitoring strategies for ChatGPT's citations should consider both webpage and site levels and account for ChatGPT's prompt rewriting behavior.
Context
This study adds to the ongoing discussion about the accuracy and reliability of AI-generated citations. As AI models become more integrated into search and information retrieval, understanding their citation behavior is crucial for developers, startups, and investors in the AI industry.
The study also highlights the challenges of monitoring AI-generated content, as models like ChatGPT may rewrite prompts and change sources between runs. This could impact the effectiveness of tracking tools and monitoring strategies.
As OpenAI continues to release new models, such as GPT-5.6 and GPT-6, it will be important to repeat these studies to understand how citation behaviors may change with new models.
