🧠 The Human in the Loop β€” Keeping the Customer at the Core of AI πŸ€–

✨ Nothing like throwing an elevated talk on your 11th birthday! Happy birthday to Sydney Design Thinking 🎈!! Last night, I sat in a room full of people who are all figuring out the same thing… AI β€” and last night… it hit different!

πŸƒπŸ»β€β™€οΈ I feel like everyone is rushing to optimise something, build something, make something using AI β€” and I am forever feeling behind. Does anyone else feel that? AI is evolving so fast, it’s hard to keep up. But we do. We iterate, learn, and churn. But are we doing it for the right reasons?

πŸ¦‰The speakers last night are all deeply experienced in the AI world β€” and yet, they’re still figuring things out too. What they brought was a reflective perspective that can only come from time and hard-won experience. They challenged me to think beyond optimisation, and to understand the real ramifications when AI is deployed without a clearly defined problem to solve.

πŸŽ™οΈ The speakers:
⚜️ Riley Coleman β€” AI Design Leader at AI Flywheel
⚜️ Kristin de la Fuente β€” Design Lead, AI Products at SEEK
⚜️ Alex Zinoviev β€” Head of Product at Neo Intelligence
⚜️ Tarra van Amerongen β€” SVP of Design at Xero (Panel Facilitator)


πŸš€ Speaker 1: Riley Coleman β€” ‘The Pause Is Where the Value Is’

πŸ”Έ Riley Coleman from AI Flywheel opened with a reframe that hit immediately: the goal of AI isn’t to remove humans from the loop β€” it’s to protect the parts of the loop that matter most.

πŸ”Έ Human autonomy and oversight aren’t just compliance checkboxes. They’re about preserving the decisions and tasks we actually enjoy doing β€” the ones that require creativity, critical thinking, and judgement. The pause before a decision? That’s not inefficiency. That’s where we do our best work.

βž–βž–βž–βž–βž–βž–

πŸ’‘ Case Study 1 β€” COMPAS (Criminal Reoffending Risk Assessment)

To make this tangible, Riley walked us through four case studies that ranged from sobering to genuinely thought-provoking.

A tool designed to give judges an AI-generated risk score to inform sentencing. In theory, it was meant to be one input alongside a judge’s own considerations. In practice?

πŸ”Έ Judges couldn’t understand how the score was rationalised
πŸ”Έ The model was systematically skewing results against those with prior convictions
πŸ”Έ Automation bias crept in β€” and judicial judgements began to reflect the AI’s, not the judge’s

The expert was technically in the loop. But their judgement wasn’t. That’s a critical distinction.

βž–βž–βž–βž–βž–βž–

πŸ’‘ Case Study 2 β€” Insurance Claims Assessor

Should high-risk, high-impact decisions be handed to AI? Riley’s answer: absolutely not. But here’s where it gets smart.

πŸ”Έ The AI was only ever allowed to say yes β€” handling automatic approvals, clear-cut cases, the low-hanging fruit
πŸ”Έ Anything vague or uncertain was escalated to a human assessor
πŸ”Έ Crucially, humans made their decision first β€” then were shown the AI’s judgement
πŸ”Έ They had to document their reasoning β€” keeping expertise genuinely in the loop, not just nominally

The result? Claim times dropped significantly. And human judgement stayed sharp.

βž–βž–βž–βž–βž–βž–

πŸ’‘ Case Study 3 β€” AI-Assisted Research

You’ve asked AI to review themes and surface insights from qualitative research. Looks great. But β€” did it tag everything correctly? Did you check the original source?

πŸ”Έ AI can only work with what’s in the transcript
πŸ”Έ It can’t read the room β€” the non-verbal cues, the hesitation, the body language
πŸ”Έ The question Riley posed: should humans tag first, then bring AI in?

It’s a small workflow change. But it’s the difference between AI augmenting your expertise and quietly replacing it.

βž–βž–βž–βž–βž–βž–

πŸ’‘ Case Study 4 β€” Creativity, Cognitive Friction, and a Very Personal Wake-Up Call

This one was personal. Riley noticed they were experiencing a loss in creativity and memory β€” and went and got tested for early-stage dementia. Thankfully, that wasn’t the cause. It was a case of not using it and losing it β€” literally.

πŸ”Έ If we over-automate, we remove the cognitive friction that keeps our thinking sharp
πŸ”Έ The craft matters β€” not just the output

Riley’s challenge: think carefully about how AI is changing your ways of working. Are you building it in thoughtfully? Or slowly outsourcing the parts of your brain you can’t afford to lose?

βž–βž–βž–βž–βž–βž–

πŸ€” Riley’s Four Conditions for Meaningful Oversight

Before handing a decision to AI β€” or rubber-stamping one β€” ask yourself:

❓ Information β€” Does the human have all the relevant info, and do they understand the AI’s rationalisation?
❓ Time β€” Do they have enough time to process, question, and push back?
❓ Competence β€” Do they have the expertise to actually evaluate what they’re approving?
❓ Authority β€” Can they genuinely say no?

If any of these four are missing, the oversight is theatre β€” not protection.

βž–βž–βž–βž–βž–βž–

πŸ’‘ Riley’s Personal Tips β€” Keeping the Human in the Loop

πŸ”Έ The 24-hour rule β€” Wait a day between AI-generated writing and editing. Return as a creator, not a passenger
πŸ”Έ Paper first β€” Before screens. Different neural pathways, different thinking
πŸ”Έ Human first β€” Ten minutes alone before AI enters any brief. Your ideas first, always
πŸ”Έ Boundaries β€” AI only before 4pm. Protect your unassisted thinking time



πŸš€ Speaker 2: Kristin de la Fuente β€” Trust Me, I’m an Agent

πŸ”Έ Kristin opened with a provocation that reframed the whole AI conversation: most AI design failures aren’t technical. They’re human. They’re psychological. And she had the receipts.

πŸ’‘ The Delegation Paradox

Despite working deeply in the AI space, Kristen describes herself as still a sceptic. So when LinkedIn’s AI efficiency hype got too loud, she ran an experiment β€” built herself a team of agents to do exactly what everyone was promising. And then promptly didn’t let them help.

πŸ”Έ She didn’t want to learn something new
πŸ”Έ Reviewing the agents’ work felt harder than just doing it herself
πŸ”Έ The agents kept asking for permission, pinging her constantly
πŸ”Έ She became the very thing she didn’t want to be β€” a micromanager

She wanted speed. She wanted to delegate. But she couldn’t let go of control. Sound familiar?

The agents were optimised for speed. Kristen was optimised for not losing oversight. Nobody had aligned on expectations and needs. It was a two-way street β€” and both sides were going in different directions.

βž–βž–βž–βž–βž–βž–

πŸ’‘ Applying This to SEEK β€” Where Trust Really Gets Tested

Kristen works in hiring and automation β€” a space where the stakes of misplaced trust are high on both sides.

πŸ”Έ Hirers need to find the right people faster β€” without losing control of the decision
πŸ”Έ Candidates need fair opportunities β€” without being filtered out by a system they can’t see or challenge

Her question for product teams: what tasks are hirers comfortable delegating β€” scheduling, filtering, admin β€” and what requires stronger human control β€” judgement calls, trade-offs, high-stakes and ambiguous decisions?

🌟 Meaningful control isn’t about automating everything. It’s about knowing what not to automate.

βž–βž–βž–βž–βž–βž–

πŸ’‘ Turning Trust Principles into Product Mechanics

This is where it got practical. Kristen’s team translates high-level responsible AI principles into actual design decisions. One example: earn trust through transparency, visibility, and control. Not as a value statement β€” as a product mechanic.

πŸ”Έ What do hirers want to do, not do, and genuinely enjoy?
πŸ”Έ Design around that. Protect it. Don’t automate it away.

And the broader provocation for the room: we don’t need more people to care about responsible AI. We need people to act on it.

βž–βž–βž–βž–βž–βž–

πŸ’‘ From Interface Design to Relationship Design

The most thought-provoking shift Kristen offered was this: stop thinking about features and start thinking about relationships. What kind of working relationship are you designing between a human and an AI system?

She mapped three stages from her own personal team of agents:

πŸ”Έ Observer β€” AI provides context summaries and drafts replies. Low trust required, high value immediately
πŸ”Έ Collaborator β€” Agents playing different roles, always asking for confirmation before acting
πŸ”Έ Delegator β€” The long game. You’ve invested the time to train the system, calibrate it, and genuinely trust it to act

Each stage has a different trust journey. And trust, as Kristen put it simply β€” grows as confidence is earned.

🌟 The most important design work isn’t happening on the screen. It’s in the relationships, the models, the logic, the processes, and the guardrails underneath.



πŸ€” The Room Weighs In β€” Live Polls & Panel Discussion

The evening didn’t stop at the talks. Riley, Kristin, and Alex Zinoviev took to a panel facilitated by Tarra van Amerongen β€” and the audience had plenty to say.

πŸ“Š Poll 1 β€” The Junior Designer Pairing

❓ A junior designer is provided with an AI agent to write UI copy and their role is to review the AI-generated work. Is this the right pairing?

71% said: it depends. 22% said: no. 7% said: yes.

The audience cut straight to it β€” the real risk here isn’t the AI, it’s the reviewer.

πŸ”Έ A junior may not have enough context to know what good looks like β€” which makes reviewing AI output genuinely hard
πŸ”Έ Without that foundation, you get rubber stamping β€” approving work you can’t actually evaluate
πŸ”Έ And the bigger question the room kept circling back to: how will juniors build expertise if AI is doing the work first?

Low stakes? Maybe. High stakes? The room said no.

βž–βž–βž–βž–βž–βž–

πŸ“Š Poll 2 β€” The Health Insurance Dilemma

❓ A health insurer can offer lower premiums to all customers with a 95%-accurate AI claims system β€” but there’s a bias against 5% of customers with complex medical needs, resulting in 20% lower claim approvals. Should the insurer use it?

59% said: no. 30% said: it depends. 11% said: yes.

πŸ”Έ The consensus β€” AI should only ever be used for net good decisions, not net bad ones
πŸ”Έ Decisions with this kind of trade-off shouldn’t sit with an automated system
πŸ”Έ The value might be real for the majority β€” but the edge cases are where the harm lands hardest
πŸ”Έ And the harder question: how do you evolve a system like this over time to actually make it work?

βž–βž–βž–βž–βž–βž–

πŸ“Š Poll 3 β€” AI-Simulated User Testing

❓ Would you use an AI tool trained on billions of data points, offering 90% accuracy in simulating your target audience, to test the usability of a new digital application?

60% said: yes. 31% said: it depends. 9% said: no.

πŸ”Έ Yes β€” but only if it’s not the only method of testing
πŸ”Έ The sceptics raised a sharp point: you’re using past data to test something new. You might lose the why entirely
πŸ”Έ The nuance the room landed on: useful as a starting point, dangerous as the finish line



πŸ’¬ Panel Highlights β€” The Questions That Mattered

πŸ’‘ On letting people go for efficiency gains that haven’t materialised yet:

πŸ”Έ Alex β€” there’s a cost to building, and a cost to figuring out. Are you solving the actual problem, or just moving fast? πŸ”Έ Kristen β€” when you optimise for speed over understanding, you lose sight of how humans actually want to work. Intent matters

βž–βž–βž–βž–βž–βž–

πŸ’‘ Fast forward three years β€” what AI trends won’t matter?

πŸ”Έ Riley β€” stop chasing tools. Understand the problem first, then find what helps. Build sustainable practices and micro-learning moments β€” the muscle to respond and collaborate in real time, not just react to the next shiny thing

βž–βž–βž–βž–βž–βž–

πŸ’‘ On designers moving from pixel-pushers to system orchestrators:

πŸ”Έ Riley β€” pixels and words are the medium, not the skill. Stay focused on problem solving πŸ”Έ Alex β€” used to spend two days preparing a presentation, half the time on production. Now he uses Claude and has the deck done in ten minutes. What that time buys him isn’t speed β€” it’s higher-leverage thinking. More time on impact, less on execution

βž–βž–βž–βž–βž–βž–

πŸ’‘ What will you never delegate to AI?

πŸ”Έ Kristen β€” understanding the problem when it comes to hiring. You need expertise and judgement before you automate. There’s a meaningful difference between done and valuable

βž–βž–βž–βž–βž–βž–

πŸ’‘ On governance versus reality:

πŸ”Έ Riley β€” studied AI ethics in London. Philosophical. Largely useless. Governance policies don’t reach the people actually on the tools. The real question: how do you surface your own governance in the design work itself? And honestly β€” we often want a human in the loop so there’s someone to blame. AI can’t be held responsible

πŸ”Έ The room acknowledged it: AI governance is the new sustainability β€” everyone agrees it matters, fewer people know what to actually do about it

βž–βž–βž–βž–βž–βž–

πŸ’‘ If every product team in Sydney changed one thing tomorrow:

πŸ”Έ Tarra β€” solve the problem yourself first. Then get AI to do it. Compare. Contrast. Keep testing as the tools evolve
πŸ”Έ Alex β€” write down the name of the person accountable when something goes wrong. If you can’t find that person, don’t ship the feature. Create ownership
πŸ”Έ Riley β€” run a pre-mortem. A creative workshop that imagines what’s already failed, finds the root cause, and tests how robust your thinking really is. And get AI literate β€” not as a nice-to-have, but as a baseline. Advocate for yourself and your organisation
πŸ”Έ Kristen β€” the capability gap in organisations is real. People on the tools are learning. But are senior leaders making informed decisions too? Great decisions happen when everyone’s working from the same level of understanding



πŸ’« The Bigger Picture

What a belter of a talk. Honestly β€” expansive, reflective, and a timely reminder not to let AI dilute our superpowers. Our taste. Our touch. Our craft.

The question isn’t whether AI will change the way we work β€” it already has. The question is: are you building it in with intention? Or slowly outsourcing the parts of yourself you can’t get back?

Use it. Don’t lose it.



πŸ™ Massive thanks to the Sydney Design Thinking team β€” Ben Pecotich, Lucas Mara, Kate Linton, and Kylie Mageropoulos β€” for 11 incredible years. What a community you’ve built.

πŸ’Ÿ And thanks to Sh8peshifters (Alan and Diana), GenAI lab, Dynamic4 and Thoughtworks for hosting β€” love the new office!

Also, how amazing is this scribe!!



#SydneyDesignThinking #HumanInTheLoop #AIDesign #ResponsibleAI #DesignThinking #AIEthics #ProductDesign #UXDesign #AIGovernance #HumanCentredAI #Sydney

Rebecca Haar Profile photo

Rebecca Haar

Some content Heading

A block of text conveying a message / cTA we want our blog readers to see as they read our articles.

Recent Articles

πŸ“Š Data Driven or Data Distracted? Why the Right Question Beats More Data 🎯

πŸ› οΈ Makeathon: Making Things with Figma Make 🌿

🌱 Lead Change That Matters: Navigating complexity, buildingΒ  momentum and making the transition πŸ”„

🧠 Systems Designing Back, and Judgement Going Quiet: Two Very Different Perspectives About AI 🎭

πŸš€ Rapid Prototyping with Google πŸ†

🀹 The Hidden Performance Skills of Great Designers 🎭Β