Why AI Projects Fail

Summary

RAND’s James Ryseff discusses why AI projects often fail and how organizations can improve their chances of success. The presentation draws on interviews with experienced data scientists and machine-learning engineers from multiple private-sector industries. It focuses broadly on machine-learning projects—including recommendation systems, computer vision, chatbots, and large language models—not just generative AI.

Ryseff cites an estimate that as many as 80% of AI projects fail, while noting that the figure comes from an external article.

Four Common Sources of Failure

1. Leadership and business alignment

   • Leaders may set vague goals without explaining the business process, priorities, or what success means. Technical teams can make the wrong assumptions and discover the mismatch months later.
   • Some organizations mandate AI even when simpler rules or statistical methods would solve the problem more efficiently.
   • Expectations may be unrealistic: AI is treated as a quick add-on or assumed to automate an entire process, despite data, reliability, and implementation constraints.
   • Shifting priorities and insufficient time prevent teams from understanding the data, developing models, and delivering a complete solution.

2. Data problems

   • Data engineering and pipeline maintenance are often undervalued, even though poor-quality, inconsistent, or unreliable data can undermine model performance.
   • Data scientists may lack the business-domain knowledge needed to interpret data. Subject-matter experts may also be reluctant to help if they fear AI will replace their roles.
   • Data collected for recordkeeping may omit the context and behavior needed for useful predictions.
   • Imbalanced datasets can make rare but important cases—such as disease detection—difficult to identify.

3. Technology and use-case problems

   • Teams may prioritize fashionable or prestigious tools over a meaningful business problem.
   • Some tasks are poor fits for AI or exceed current capabilities. Models that perform well on clean academic benchmarks may struggle with messy real-world inputs.
   • Organizations need to recognize where a model’s capabilities end and consider human–AI collaboration rather than insisting on full automation.

4. Mismatch with software-development processes

   • AI work involves experimentation and uncertainty: teams may need to explore data before they know what can be built, and improvements in model accuracy can be difficult to predict.
   • Short, fixed software sprints may not align with data exploration and model refinement. Interviewees suggested longer checkpoints may be more appropriate. Ryseff mentioned two-to-three-month cycles as a possible fit, while emphasizing that there is no settled best practice.

Recommendations

Ryseff offers six broad recommendations:

• Keep business and technical teams in regular conversation about the problem, its purpose, and the relevant domain context.
• Select important, enduring problems and give teams enough time to develop, iterate on, and maintain solutions.
• Focus on solving the problem rather than adopting a particular technology.
• Invest in supporting infrastructure, including dependable data pipelines and tools for deploying, monitoring, and maintaining models.
• Understand AI’s limits and allow technical staff to advise when a simpler approach—or no AI—is more appropriate.
• Give teams time to explore data and test hypotheses, while maintaining reasonable deadlines and checkpoints.

Additional Findings and Q&A

• Compute: Among the private-sector interviewees, compute was generally not a major constraint when organizations could use public cloud. It was more limiting for organizations unable to move sensitive data to the cloud. Ryseff noted that this may differ for government users.
• Talent: Hiring challenges included distinguishing practical ability from credentials, candidates’ limited experience with messy real-world data, mismatches between the talent companies seek and the roles candidates want, and inconsistent definitions of “data scientist.”
• More promising use cases: Applications with measurable objectives, frequent user interactions, and rapid feedback—such as e-commerce recommendations and digital advertising—tended to be more successful. In less digital settings, human–AI teaming may be more suitable than replacing people.
• Potentially difficult tasks: Problems relying heavily on tacit knowledge held by employees can be challenging when that knowledge is not captured in the data. Leaders should let technical teams assess the data before deciding whether AI is necessary.
• Government challenges: Ryseff’s observations were not based on this study’s private-sector interviews, but he identified contracting and acquisition processes, limited cloud adoption, and siloed government datasets as likely obstacles.
• Scope limitations: The study did not identify a consistent pattern of unexpectedly poor application categories. Agentic AI was too new during the research to assess; Ryseff suggested a future study could examine its failure modes.

Main Speakers and Sources

• James Ryseff: RAND senior technical policy analyst and presenter.
• Anun Ninan: RAND National Security Research Division associate director, who introduced the webinar and moderated the Q&A.
• Research sources: Anonymous interviews with experienced private-sector data scientists and machine-learning engineers.