Increasing Volume of AI-Generated Data Threatens Future Large Language Model Reliability
By 2028, 50% of organizations will implement a zero-trust posture for data governance due to the proliferation of unverified AI-generated data, according to Gartner, Inc., a business and technology insights company.
“Organizations can no longer implicitly trust data or assume it was human generated,” said Wan Fui Chan, Managing VP at Gartner. “As AI-generated data becomes pervasive and indistinguishable from human-created data, a zero-trust posture establishing authentication and verification measures, is essential to safeguard business and financial outcomes.”
AI-Generated Data Heightens Model Collapse Risks and Compliance Demands
Large language models (LLMs) are typically trained on “web-scraped” data and a variety of sources, including books, code repositories and research papers. Some of these sources already contain AI-generated content, and if current trends continue, nearly all will eventually be populated with AI-generated data.
According to the 2026 Gartner CIO and Technology Executive Survey, 84% of respondents expect their enterprise to increase funding for GenAI in 2026. As organizations accelerate both adoption and investment in AI initiatives, the volume of AI-generated data will continue to rise. This means future generations of LLMs will increasingly be trained on outputs from previous models, heightening the risk of “model collapse,” where AI tools’ responses may no longer accurately reflect reality.
“As AI-generated content becomes more prevalent, regulatory requirements for verifying ‘AI-free’ data are expected to intensify in certain regions,” said Chan. “However, these requirements may differ significantly across geographies, with some jurisdictions seeking to enforce stricter controls on AI-generated content, while others may adopt a more flexible approach.
“In this evolving regulatory environment, all organizations will need the ability to identify and tag AI-generated data. Success will depend on having the right tools and a workforce skilled in information and knowledge management, as well as metadata management solutions that are essential for data cataloging.”
Active metadata management practices will become a key differentiator, enabling organizations to analyze, alert and automate decision making across their data assets.
Managing Risks Associated with Rising Unverified AI-Generated Data
Organizations should consider several strategic actions to manage the risks of unverified data:
- Appoint an AI Governance Leader: Establish a dedicated role responsible for AI governance, including zero-trust policies, AI risk management and compliance operations. This leader should work closely with data and analytics (D&A) teams to ensure both AI-ready data and systems capable of handling AI-generated content.
- Foster Cross-Functional Collaboration: Form cross-functional teams that include cybersecurity, D&A and other relevant stakeholders to conduct comprehensive data risk assessments to identify business risks related to AI-generated data and determine which are addressed by existing data security policies and which need new strategies.
- Leverage Existing Governance Policies: Build on current D&A governance frameworks and focus on updating security, metadata management and ethics related policies to address new risks from AI-generated data.
- Adopt Active Metadata Practices: This enables real-time alerts when data is stale or requires recertification, helping organizations quickly identify when business-critical systems may become exposed to inaccurate or biased data.
