Lessons from Human Data Analysts to Improve the AI Variety

If AI is positioned to unlock self-serve analytics, it needs to evolve beyond simple text-to-SQL tools and learn from the real-world expertise of human data analysts.

Over the past few weeks, I’ve been exploring the landscape of self-serve analytics and examining why they often fall short of achieving true “data democratization.” I’ve touched on some common UX pitfalls and the technical limitations caused by evolving data models. Now, it’s time to turn our attention to AI—specifically the AI Data Analyst.

The Rise of the AI Data Analyst

In the context of self-serve analytics, the “AI Data Analyst” often takes the form of a text-to-SQL chatbot. The idea is simple: a business user types in a data question, and in return, they get a plot and some explanatory text to help them interpret it. Amazing, right? Problem solved. Data teams are freed from disruptive ad-hoc requests, and business teams get instant answers to their pressing questions. We’ve finally reached the promised land of data democratization.

🦗

If only it were that easy.

After a decade working on and with data teams, I know what it takes to be successful as a data analyst - and SQL is just one piece of the puzzle. What can we learn from the best human data analysts that can inform the design of their AI counterparts?

The AI Analyst's Critical Blind Spot

Over the years, I’ve interviewed hundreds of data analysts and data scientists. I’ve honed my process, and I can usually tell whether or not someone will thrive in the role. Unfortunately, the AI Data Analyst wouldn’t pass.

Take this example from a recent interview: The candidate had a solid technical background, and I could see she was eager to prove herself.

I gave her this example scenario:

“Imagine I’m the VP of Growth, excited to have you on the team, and I want you to focus on customer segmentation as your first analysis. What information would you need? How would you go about it?”

Without missing a beat, she started listing off data sources, outlining SQL queries, talking about metrics, JOINS, and statistics. But she missed a crucial step—she didn’t ask me anything about what I was really trying to understand or the broader business context. Was I trying to segment current customers or prospective ones? Would the segmentation inform our personalization strategy or retention efforts? Without this context, the analysis will most certainly miss the mark.

I’ve seen this pattern play out again and again, particularly with junior analysts: they jump straight into solving the problem without fully understanding it. It’s a strong signal that they might struggle in the role, even if they excel technically. And it’s concerning that AI data analysts are programmed to behave in the same way.

What About More Specificity?

A customer segmentation is a broad ask, so it’s tempting to think the solution requires better scoping, but this approach has its own pitfalls

• Scenario: Business user is very explicit with their request.

• Risk: The request is defined too narrowly, and because the business user doesn’t have awareness of the data - they don’t know if the result will be valuable.

Take this revised request:

“Give me a list of unique customers (cust_id) who have placed an order within the past year. Include a column for the gender of the customer and a column to tell me if they’ve spent more or less than $50 on all of their purchases.”

Simple, right? Well, not quite. The person defining the request may not realize that the “gender” column is sparsely populated, with values for only 2% of customers. This level of specificity leaves the data analyst with little room to exercise creative freedom or suggest better approaches, leading to results that likely don’t meet the underlying need.

I’ve seen this happen often with offshore/external teams that are disconnected from the business context. They rely heavily on pre-scoping, but the outcomes rarely align with what the stakeholder actually needs.

What Great Data Analysts Do Differently

Skilled analysts approach problems in a completely different way. They don’t just ask follow-up questions to clarify what was initially requested; they dig deeper to uncover the underlying business goal. They ask, “What are you really trying to accomplish?”

Take the segmentation request as an example. A seasoned data analyst would pause to ask, “What’s the business goal here? Are we trying to tailor marketing strategies, or are we looking to understand long-term value differences?” After knowing that we’re interested in retaining high value customers by sending them a free “thank you” gift, we can tailor the data request. Instead of using the gender column to personalize the gift, we could use their purchase history to inform the gift that best suits their taste.

This approach transforms the role of the analyst:

From Executor to Thought Partner: A senior analyst doesn’t just run the numbers; they guide the stakeholder in framing the problem correctly. They become a thought partner, helping to shape the analysis before any SQL is written. This leads to better outcomes and reduces frustration for everyone involved.

The Role of a Thought Partner

A good data analyst isn’t just a query builder; they’re a thought partner. Instead of simply clarifying questions, they explore the underlying business objectives. They bring their understanding of the data to the table and work to align their analysis with the stakeholder’s goals.

This approach isn’t just about getting the right answer; it’s about asking the right question in the first place.

Using AI to Close the Context Gap

AI has immense potential in data analysis, but to truly leverage it, we need to rethink how it’s used. Right now, AI often falls into the trap of executing poorly defined queries. What we need is for AI to help users craft better requirements from the outset.

Imagine an AI that doesn’t just follow orders but understands the bigger picture. An AI that can guide you in framing the right questions, based on a deep understanding of both your data and your business context. This is the future of AI in data analysis—a future where AI evolves from being just another tool to becoming a strategic partner that drives better outcomes and more meaningful insights.

Rethinking AI and Data Analysis

As AI continues to advance, it’s important to remember that flawless execution isn’t everything. The real value lies in understanding and contextualizing the problem at hand. By fostering better collaboration between business teams and data analysts—whether human or AI—we can unlock more meaningful results.

If we’re relying solely on AI to churn out answers, it’s time to rethink our approach. Focus on using AI to bridge the context gap, and we’ll start to see a real difference in the quality of our insights.