📊 Data Driven or Data Distracted? Why the Right Question Beats More Data 🎯

✨ Last night was the final #CCX event of the year! (Can’t believe I am saying that! 😱) and we finished with a bang! A super relevant talk with four speakers, with four different angles, one recurring theme: Who is telling the story?….. Are you telling real data stories…. Or is it a story with embellished numbers? 

🎙️ The speakers:

⚜️ Thomas Shillingford — UX Lead at Ausgrid

⚜️ Olga Pasikova — Senior Analytics Director at Razorfish

⚜️ Rob Tot — UX and Service Design Consultant

⚜️ David Berigny — AI/ML Engineer and CX Strategist

🎤 Our Facilitator: 

⚜️ Sophie Greenfield – Senior AI Product Designer at 15Ventures (CBA)


🎖️ Speaker 1: Thomas Shillingford — Are we data driven or data distracted?

Thomas opened with Robert McNamara, US Secretary of Defense during the Vietnam War. Body counts were the measure of progress, and the numbers said America was winning. McNamara later admitted the measure was wrong.

💡 McNamara wasn’t unfocused. He was extremely data driven, just driven by the wrong measure. That’s what being data distracted looks like.

Thomas then brought it back to a digital transformation project for a NSW regulator, which involved joining up licensing with compliance and complaints across roughly 28 agencies. Having the data wasn’t the same as being allowed to use it. Each agency had its own privacy and security agreements, so this was a governance problem as much as a technical one.

He shared four tests that helped him through the ambiguity:

🔸 Start with the question, not the data. With 28 agencies, they couldn’t research everyone, so they chose two (SafeWork and Fair Trading) that represented around 90% of volume.

🔸 Let the data go either way. Tablets for recording evidence tested well in the workshop room, then failed in the field. Investigators already had their own tools, and legally their notes had to be written on the spot.

🔸 Look for what’s missing. Thomas pointed to the WWII story of reinforcing planes where the bullet holes were, when the real insight was the parts without them. Officers writing notes by hand and entering them later was a sign of what the system wasn’t capturing.

🔸 Measure the right things. Complaint-handling speed looked like a win, but what mattered was resolving complaints faster with fewer misclassified or wrongly escalated ones. That meant fixing the definitions first.


🔍 Speaker 2: Olga — From data dump to next steps

Olga started her career in campaign reporting before moving into A/B testing, where she learned to build proper hypotheses and ask why something is happening and who it’s happening to.

The pattern she sees often: a product or design team asks a vague question like “is this flow working?”, and an analyst responds with a mountain of data. And this can lead to questions not being answered or data not answering the right questions – leading to things getting lost of translation and no clear steps on how to move forward. 

Without business context, that doesn’t lead to good decisions. She’s been that analyst, and described getting stuck in analysis paralysis.

🖼️ Olga has built frameworks to help her navigate ambiguity: 

Her approach when she is working with product and distilling the requirements:

1️⃣ Start with the question – what data exists in the business? What is the current knowledge base about the problem? 

2️⃣Quantitative analysis – What and where? 

3️⃣Qualitative analysis – why? 

4️⃣ Test & learn – experimentation 

💡Don’t brief the analytics team with a metric, brief them with a question you’re trying to answer.

Another framework she uses to check over a brief: 

🔸What? – Business context, trends, benchmarks 

🔸Who? Audiences and intent

🔸Why? Behaviours by segment

🔸Friction: Where do customers drop off?

🔸Impact and next steps. 

🔸 Check back in with the product team along the way, not only at the end

💡 Never hand over a snapshot. Always finish with what to do next.

🍔A fast-food example: overall conversion looked fine, but segmenting the audience revealed a small, high-value group placing catering orders over around $100. They were converting poorly. Session replays showed the site was sending catering searches to the wrong landing page. It was so visually obvious that the business didn’t need convincing, and the team went straight to fixing and testing.

Olga also shared a great example of an engagement matrix with us, that was broken down into stages of the customer lifecycle and their relevant metrics. This was super helpful in helping people navigating and measuring the customer funnel experience, rather than just jumping from brand awareness to conversion – like a lot of people do.



🎭 Speaker 3: Rob Tot — Calling out the distractions

Rob’s definition of distraction: attention being pulled onto something other than the thing that actually matters. 

It can come from outside (competing signals, noise, secondary sources), from inside (internal stress, feeling inadequate), or from the data itself (a dashboard that doesn’t support a decision).

👀 He named three distractions to watch for:

🔸 Vanity metrics. Numbers that look impressive but don’t reflect what matters, or that mostly exist to make people look good or to create competition between teams.

🔸 Data theatre. A scripted, performative meeting about data where nobody knows what to do next with it. He was candid that he’s been guilty of this too.

🔸 Performative dashboards. Dashboards that get walked through without an understanding of its purpose and why it exists.

🔧 His fixes:

  • Keep your attention on the objective!  He writes it at the top of the whiteboard and opens presentations with it.
  • Get people to compete on metrics that matter.
  • Insert more relevant data into existing rituals.
  • Find useful information and uses of a dashboard… or built a better one!

🤖 Speaker 4: David Berigny — The two armed problem

David closed the talks by looking at how generative AI “understands” information. He described two models:

🔸 Similarity-based (what AI does): like affinity-mapping sticky notes at huge scale, great at finding things that are probably related without an explicit reason why

🔸 Structured (folders, taxonomies, ontologies): this belongs here, with close to 100% certainty, because it’s explicitly defined

⚠️ The risk is that AI is probabilistic. It can sound confident without being correct, which is dangerous in areas like government policy or legislation, where a wrong answer has legal and reputational consequences.

He took us through a tool that he built to show an experiment that pairs AI inference with a traceable, verifiable structure, so you can follow an answer back to the source data rather than just trusting it.


🤔 The Panel Q&A

❓ How do analysts make data more palatable for product designers?
Align on the objective or North Star metric first, then slice the data to tell a story: what’s happening, why, and what could change. Data and product teams should check in at every step of the funnel, not just at the end.

❓ Whose job is it to ask the right questions?
By the time data is being discussed, the rationale and ownership should already be agreed. If it’s unclear who owns the decision, the upfront conversation probably didn’t happen.

❓ Who holds the power in practice?
In large corporates and government, accountability runs up the hierarchy. In faster-moving environments, it’s often whoever controls the budget. And in organisations with legacy systems, the one person who understands the old platform can become an unofficial power centre.

❓ Are we outsourcing sense-making to AI?
To a real extent, yes. People with broad, cross-functional understanding are best placed to sanity-check outputs. The risk is someone who doesn’t know the subject well enough to question AI feeding it information and trusting whatever comes back.

❓ Can gut feel legitimately override data?
Yes, but decide up front how you’ll judge whether the bold call worked. Rob also talked about giving less experienced team members “top cover” to try new approaches. Either way, you generate data about what’s actually happening.

❓ How do you avoid being led the wrong way by data?
Define your success measures at the requirements or wireframe stage, before you build. Decide what each possible outcome would mean for your next decision. Rob called it a muscle that gets stronger with repetition.


💫 The Thread That Connected It All

Four different talks, but they kept circling the same idea.

🔸 Thomas showed what happens when you’re driven by the wrong measure.

🔸 Olga showed what happens when you hand over data without context or a next step.

🔸 Rob showed how attention can drift to vanity metrics, data theatre and dashboards with no decisions being made.

🔸 David showed how AI can serve confident answers without being able to trace them to certain facts. 

The problem is rarely a lack of data. It’s the absence of a clear objective to anchor it to, and the discipline to keep coming back to that objective.



🙏 A huge thanks to the team at #CCX (Kevin Wilkins, Lydia Bradshaw, Sylvia Luk, and James Earnshaw) for curating another brilliant evening and for all of the events you’ve hosted this year!

💙 Also sending thanks to The Onset for hosting. And Amplitude (Simon Rumble) for sponsoring! 

#CCX #DataDriven #ProductDesign #UXDesign #Analytics #CX #AI #ProductManagement

Rebecca Haar Profile photo

Rebecca Haar

Some content Heading

A block of text conveying a message / cTA we want our blog readers to see as they read our articles.

Recent Articles

📊 Data Driven or Data Distracted? Why the Right Question Beats More Data 🎯

🛠️ Makeathon: Making Things with Figma Make 🌿

🌱 Lead Change That Matters: Navigating complexity, building  momentum and making the transition 🔄

🧠 Systems Designing Back, and Judgement Going Quiet: Two Very Different Perspectives About AI 🎭

🚀 Rapid Prototyping with Google 🏆

🤹 The Hidden Performance Skills of Great Designers 🎭