β¨ Nothing like throwing an elevated talk on your 11th birthday! Happy birthday to Sydney Design Thinking π!! Last night, I sat in a room full of people who are all figuring out the same thing… AI β and last night… it hit different!
ππ»ββοΈ I feel like everyone is rushing to optimise something, build something, make something using AI β and I am forever feeling behind. Does anyone else feel that? AI is evolving so fast, it’s hard to keep up. But we do. We iterate, learn, and churn. But are we doing it for the right reasons?
π¦The speakers last night are all deeply experienced in the AI world β and yet, they’re still figuring things out too. What they brought was a reflective perspective that can only come from time and hard-won experience. They challenged me to think beyond optimisation, and to understand the real ramifications when AI is deployed without a clearly defined problem to solve.
ποΈ The speakers:
βοΈ Riley Coleman β AI Design Leader at AI Flywheel
βοΈ Kristin de la Fuente β Design Lead, AI Products at SEEK
βοΈ Alex Zinoviev β Head of Product at Neo Intelligence
βοΈ Tarra van Amerongen β SVP of Design at Xero (Panel Facilitator)
π Speaker 1: Riley Coleman β ‘The Pause Is Where the Value Is’

πΈ Riley Coleman from AI Flywheel opened with a reframe that hit immediately: the goal of AI isn’t to remove humans from the loop β it’s to protect the parts of the loop that matter most.
πΈ Human autonomy and oversight aren’t just compliance checkboxes. They’re about preserving the decisions and tasks we actually enjoy doing β the ones that require creativity, critical thinking, and judgement. The pause before a decision? That’s not inefficiency. That’s where we do our best work.
ββββββ
π‘ Case Study 1 β COMPAS (Criminal Reoffending Risk Assessment)
To make this tangible, Riley walked us through four case studies that ranged from sobering to genuinely thought-provoking.
A tool designed to give judges an AI-generated risk score to inform sentencing. In theory, it was meant to be one input alongside a judge’s own considerations. In practice?
πΈ Judges couldn’t understand how the score was rationalised
πΈ The model was systematically skewing results against those with prior convictions
πΈ Automation bias crept in β and judicial judgements began to reflect the AI’s, not the judge’s
The expert was technically in the loop. But their judgement wasn’t. That’s a critical distinction.
ββββββ
π‘ Case Study 2 β Insurance Claims Assessor
Should high-risk, high-impact decisions be handed to AI? Riley’s answer: absolutely not. But here’s where it gets smart.
πΈ The AI was only ever allowed to say yes β handling automatic approvals, clear-cut cases, the low-hanging fruit
πΈ Anything vague or uncertain was escalated to a human assessor
πΈ Crucially, humans made their decision first β then were shown the AI’s judgement
πΈ They had to document their reasoning β keeping expertise genuinely in the loop, not just nominally
The result? Claim times dropped significantly. And human judgement stayed sharp.
ββββββ
π‘ Case Study 3 β AI-Assisted Research
You’ve asked AI to review themes and surface insights from qualitative research. Looks great. But β did it tag everything correctly? Did you check the original source?
πΈ AI can only work with what’s in the transcript
πΈ It can’t read the room β the non-verbal cues, the hesitation, the body language
πΈ The question Riley posed: should humans tag first, then bring AI in?
It’s a small workflow change. But it’s the difference between AI augmenting your expertise and quietly replacing it.
ββββββ
π‘ Case Study 4 β Creativity, Cognitive Friction, and a Very Personal Wake-Up Call
This one was personal. Riley noticed they were experiencing a loss in creativity and memory β and went and got tested for early-stage dementia. Thankfully, that wasn’t the cause. It was a case of not using it and losing it β literally.
πΈ If we over-automate, we remove the cognitive friction that keeps our thinking sharp
πΈ The craft matters β not just the output
Riley’s challenge: think carefully about how AI is changing your ways of working. Are you building it in thoughtfully? Or slowly outsourcing the parts of your brain you can’t afford to lose?
ββββββ
π€ Riley’s Four Conditions for Meaningful Oversight

Before handing a decision to AI β or rubber-stamping one β ask yourself:
β Information β Does the human have all the relevant info, and do they understand the AI’s rationalisation?
β Time β Do they have enough time to process, question, and push back?
β Competence β Do they have the expertise to actually evaluate what they’re approving?
β Authority β Can they genuinely say no?
If any of these four are missing, the oversight is theatre β not protection.
ββββββ
π‘ Riley’s Personal Tips β Keeping the Human in the Loop
πΈ The 24-hour rule β Wait a day between AI-generated writing and editing. Return as a creator, not a passenger
πΈ Paper first β Before screens. Different neural pathways, different thinking
πΈ Human first β Ten minutes alone before AI enters any brief. Your ideas first, always
πΈ Boundaries β AI only before 4pm. Protect your unassisted thinking time
π Speaker 2: Kristin de la Fuente β Trust Me, I’m an Agent

πΈ Kristin opened with a provocation that reframed the whole AI conversation: most AI design failures aren’t technical. They’re human. They’re psychological. And she had the receipts.
π‘ The Delegation Paradox
Despite working deeply in the AI space, Kristen describes herself as still a sceptic. So when LinkedIn’s AI efficiency hype got too loud, she ran an experiment β built herself a team of agents to do exactly what everyone was promising. And then promptly didn’t let them help.

πΈ She didn’t want to learn something new
πΈ Reviewing the agents’ work felt harder than just doing it herself
πΈ The agents kept asking for permission, pinging her constantly
πΈ She became the very thing she didn’t want to be β a micromanager
She wanted speed. She wanted to delegate. But she couldn’t let go of control. Sound familiar?
The agents were optimised for speed. Kristen was optimised for not losing oversight. Nobody had aligned on expectations and needs. It was a two-way street β and both sides were going in different directions.

ββββββ
π‘ Applying This to SEEK β Where Trust Really Gets Tested
Kristen works in hiring and automation β a space where the stakes of misplaced trust are high on both sides.
πΈ Hirers need to find the right people faster β without losing control of the decision
πΈ Candidates need fair opportunities β without being filtered out by a system they can’t see or challenge
Her question for product teams: what tasks are hirers comfortable delegating β scheduling, filtering, admin β and what requires stronger human control β judgement calls, trade-offs, high-stakes and ambiguous decisions?
π Meaningful control isn’t about automating everything. It’s about knowing what not to automate.
ββββββ
π‘ Turning Trust Principles into Product Mechanics

This is where it got practical. Kristen’s team translates high-level responsible AI principles into actual design decisions. One example: earn trust through transparency, visibility, and control. Not as a value statement β as a product mechanic.
πΈ What do hirers want to do, not do, and genuinely enjoy?
πΈ Design around that. Protect it. Don’t automate it away.
And the broader provocation for the room: we don’t need more people to care about responsible AI. We need people to act on it.
ββββββ
π‘ From Interface Design to Relationship Design
The most thought-provoking shift Kristen offered was this: stop thinking about features and start thinking about relationships. What kind of working relationship are you designing between a human and an AI system?
She mapped three stages from her own personal team of agents:

πΈ Observer β AI provides context summaries and drafts replies. Low trust required, high value immediately
πΈ Collaborator β Agents playing different roles, always asking for confirmation before acting
πΈ Delegator β The long game. You’ve invested the time to train the system, calibrate it, and genuinely trust it to act
Each stage has a different trust journey. And trust, as Kristen put it simply β grows as confidence is earned.

π The most important design work isn’t happening on the screen. It’s in the relationships, the models, the logic, the processes, and the guardrails underneath.

π€ The Room Weighs In β Live Polls & Panel Discussion

The evening didn’t stop at the talks. Riley, Kristin, and Alex Zinoviev took to a panel facilitated by Tarra van Amerongen β and the audience had plenty to say.
π Poll 1 β The Junior Designer Pairing
β A junior designer is provided with an AI agent to write UI copy and their role is to review the AI-generated work. Is this the right pairing?
71% said: it depends. 22% said: no. 7% said: yes.
The audience cut straight to it β the real risk here isn’t the AI, it’s the reviewer.
πΈ A junior may not have enough context to know what good looks like β which makes reviewing AI output genuinely hard
πΈ Without that foundation, you get rubber stamping β approving work you can’t actually evaluate
πΈ And the bigger question the room kept circling back to: how will juniors build expertise if AI is doing the work first?
Low stakes? Maybe. High stakes? The room said no.
ββββββ
π Poll 2 β The Health Insurance Dilemma
β A health insurer can offer lower premiums to all customers with a 95%-accurate AI claims system β but there’s a bias against 5% of customers with complex medical needs, resulting in 20% lower claim approvals. Should the insurer use it?
59% said: no. 30% said: it depends. 11% said: yes.
πΈ The consensus β AI should only ever be used for net good decisions, not net bad ones
πΈ Decisions with this kind of trade-off shouldn’t sit with an automated system
πΈ The value might be real for the majority β but the edge cases are where the harm lands hardest
πΈ And the harder question: how do you evolve a system like this over time to actually make it work?
ββββββ
π Poll 3 β AI-Simulated User Testing
β Would you use an AI tool trained on billions of data points, offering 90% accuracy in simulating your target audience, to test the usability of a new digital application?
60% said: yes. 31% said: it depends. 9% said: no.
πΈ Yes β but only if it’s not the only method of testing
πΈ The sceptics raised a sharp point: you’re using past data to test something new. You might lose the why entirely
πΈ The nuance the room landed on: useful as a starting point, dangerous as the finish line
π¬ Panel Highlights β The Questions That Mattered
π‘ On letting people go for efficiency gains that haven’t materialised yet:
πΈ Alex β there’s a cost to building, and a cost to figuring out. Are you solving the actual problem, or just moving fast? πΈ Kristen β when you optimise for speed over understanding, you lose sight of how humans actually want to work. Intent matters
ββββββ
π‘ Fast forward three years β what AI trends won’t matter?
πΈ Riley β stop chasing tools. Understand the problem first, then find what helps. Build sustainable practices and micro-learning moments β the muscle to respond and collaborate in real time, not just react to the next shiny thing
ββββββ
π‘ On designers moving from pixel-pushers to system orchestrators:
πΈ Riley β pixels and words are the medium, not the skill. Stay focused on problem solving πΈ Alex β used to spend two days preparing a presentation, half the time on production. Now he uses Claude and has the deck done in ten minutes. What that time buys him isn’t speed β it’s higher-leverage thinking. More time on impact, less on execution
ββββββ
π‘ What will you never delegate to AI?
πΈ Kristen β understanding the problem when it comes to hiring. You need expertise and judgement before you automate. There’s a meaningful difference between done and valuable
ββββββ
π‘ On governance versus reality:
πΈ Riley β studied AI ethics in London. Philosophical. Largely useless. Governance policies don’t reach the people actually on the tools. The real question: how do you surface your own governance in the design work itself? And honestly β we often want a human in the loop so there’s someone to blame. AI can’t be held responsible
πΈ The room acknowledged it: AI governance is the new sustainability β everyone agrees it matters, fewer people know what to actually do about it
ββββββ
π‘ If every product team in Sydney changed one thing tomorrow:
πΈ Tarra β solve the problem yourself first. Then get AI to do it. Compare. Contrast. Keep testing as the tools evolve
πΈ Alex β write down the name of the person accountable when something goes wrong. If you can’t find that person, don’t ship the feature. Create ownership
πΈ Riley β run a pre-mortem. A creative workshop that imagines what’s already failed, finds the root cause, and tests how robust your thinking really is. And get AI literate β not as a nice-to-have, but as a baseline. Advocate for yourself and your organisation
πΈ Kristen β the capability gap in organisations is real. People on the tools are learning. But are senior leaders making informed decisions too? Great decisions happen when everyone’s working from the same level of understanding
π« The Bigger Picture
What a belter of a talk. Honestly β expansive, reflective, and a timely reminder not to let AI dilute our superpowers. Our taste. Our touch. Our craft.
The question isn’t whether AI will change the way we work β it already has. The question is: are you building it in with intention? Or slowly outsourcing the parts of yourself you can’t get back?
Use it. Don’t lose it.
π Massive thanks to the Sydney Design Thinking team β Ben Pecotich, Lucas Mara, Kate Linton, and Kylie Mageropoulos β for 11 incredible years. What a community you’ve built.
π And thanks to Sh8peshifters (Alan and Diana), GenAI lab, Dynamic4 and Thoughtworks for hosting β love the new office!
Also, how amazing is this scribe!!


#SydneyDesignThinking #HumanInTheLoop #AIDesign #ResponsibleAI #DesignThinking #AIEthics #ProductDesign #UXDesign #AIGovernance #HumanCentredAI #Sydney