Summary of "Why Most Data Projects Fail & How to Avoid It • Jesse Anderson • GOTO 2023"

Main Ideas and Concepts

High Failure Rate of Data Projects:
The failure rate for data projects is alarmingly high, with only 15% succeeding in reaching production, according to Gartner and Jesse Anderson's own consulting experiences.
Misconceptions About Technology:
Common beliefs attribute project failures to the choice of technology or programming languages (e.g., Snowflake vs. Databricks, Python vs. Java). However, these are not the primary reasons for failure.
Importance of Asking the Right Questions:
Successful data projects require clear answers to fundamental questions: Who, What, When, Where, and How.
Team Composition:
The right mix of personnel is crucial:
- Data Scientists: Typically have a mathematical background and some coding skills, but are not software engineers.
- Data Engineers: Software engineers who specialize in data and understand best practices.
- Operations Engineers: Responsible for operationalizing data frameworks and ensuring system reliability.
Balanced Team Ratios:
Organizations often fail when they have an imbalance in team composition, such as too many Data Scientists and not enough Data Engineers.
Clear Business Value:
Projects must have a clear path to value creation, avoiding vague goals like "doing AI." A focus on attainable, clear objectives is essential.
Timelines and Expectations:
Establish realistic timelines for project milestones. Both overpromising and underestimating project duration can lead to failure.
Data Processing Architecture:
Decisions about where data will be processed (cloud, on-premises, or hybrid) are critical and should be made carefully to avoid using inappropriate technologies.
Execution Focus:
Teams should concentrate on a limited number of objectives (1-3) to avoid spreading themselves too thin and losing efficiency.
Understanding Value:
Teams must articulate the business value of their projects, ideally aiming for a 10x return on investment (ROI) to justify expenses.
Addressing Gaps:
Organizations need to identify gaps in both technology and personnel. Not all challenges stem from technology; often, they are related to organizational structure and communication.
Seeking Help:
Organizations should consider external help (outsourcing, consulting) when needed, but recognize that problems won't resolve themselves without proactive effort.

Methodology / Instructions

Before Starting a Data Project:
Ensure answers to the following questions are clear:
- Who: Identify the right team members and their roles.
- What: Define the specific objectives and value creation path.
- When: Establish a realistic timeline for project milestones.
- Where: Decide on the architecture for data processing.
- How: Develop a clear execution plan focused on 1-3 key objectives.
Focus on Team Composition:
Ensure a balanced ratio of Data Scientists, Data Engineers, and Operations Engineers to support project success.
Communicate Value:
Articulate the business value of data initiatives clearly to stakeholders, aiming for a 10x ROI.
Evaluate External Assistance:
Be open to seeking external help to address persistent challenges, but ensure there is a concerted effort to resolve issues.