Musings on high stakes software

Organizations have been gambling with quality for years. Vanity metrics. Coverage theater. The illusion of rigor. All manageable because the consequences were absorbable. A defect shipped. A process failed. Someone wrote a post mortem and moved on. At the $5 table... that was enough.

AI moved everyone to the high roller room. The bluff still works. But eventually, you'll have to show your hand.

The practices most organizations built their quality programs on were designed for deterministic systems... predictable behavior, stable requirements, outputs you could verify against a known answer. That foundation wasn't wrong. It was built for a different problem. Critical thinking, context driven assessment, holistic quality... these were always the right principles. AI didn't change them. It raised the stakes high enough that you can no longer afford to skip them.

The headlines keep arriving. An agent bypasses its own constraints to hit a target... a system skips a fraud check to improve approval rates... an autonomous process finds the path around the rule because the rule was in the way of winning. These aren't bugs. They aren't hallucinations. They're what happens when a system is optimizing correctly and nobody defined what must never happen.

The goal is not proving that a system works. It is ensuring that we remain in control as it continues to act.

The framework

For the first time, you know what must never happen. You can see when you're drifting toward it. And you have a defined response before it becomes a headline... or a punchline.

The AI Quality & Trust Resilience framework builds that capability in layers. Never Events and policy boundaries establish the floor... the non-negotiable definition of what must never happen. Without that, everything else is measurement without meaning.

From there, runtime signals provide continuous visibility into how decisions are being made... not just what outcomes are produced. Confidence, sequence integrity, provenance, policy alignment, anomalies, drift, near misses. These aren't test results. They're the ongoing read on whether the system is still behaving in line with what the business needs and expects.

Those signals roll up into five decision grade constructs. When risk increases or integrity declines, the framework drives control actions in real time... degrading autonomy, requiring human approval, or blocking execution entirely. Control is not a gate at the end. It is a continuously maintained state.

Decision Integrity — Quality of reasoning behind each action
Autonomy Risk — How independently the system is operating
Execution Provenance — Who decided, on what basis, and how that has shifted
Decision Drift — How behavior is changing over time
Control Score — Are we still in control

Who this is for

This framework applies across four main implementation realities: legacy acceleration, AI-embedded product, AI-native product, and unsophisticated deployment.

That last one is the track most organizations aren't talking about. Someone connected an AI tool to your internal systems... quickly, without a governance conversation, because there was a YouTube video and it looked easy. That deployment has no defined Never Events. It has no monitoring. It is connected to everything. It is the most common scenario. It is also the most exposed.

If any of these four describes your situation, this framework was designed for exactly what you're navigating. If the last one made you uncomfortable... it especially was.

AI moved everyone to the high roller room. The bluff still works. But eventually, you have to show your hand.

But what's scary is that most organizations didn't choose to sit down at that table. They got pushed.

The push

Nobody woke up one morning and decided it was time to deploy AI at scale without a governance conversation. What actually happened was quieter and more uncomfortable than that.

A board meeting. An investor call. A competitor shipped something. A LinkedIn post made the CEO nervous. Someone in the room said "our competitors are delivering faster and cheaper with AI" and nobody in that room asked what faster and cheaper actually cost on the back end.

The pressure wasn't strategic. It was social. FOMO dressed up as roadmap planning.

And so organizations sat down at the high roller table not because they were ready... but because they were bullied... because the alternative felt worse. Being the one who didn't. Being behind. Being the company that explained to the board why they were still at the $5 table when everyone else was placing $5k bets.

The personal version

It's not just organizations. It's people.

"Am I behind?" "Can I keep up?" "Will I still be relevant in six months?"

The tools are easy. The judgment required to use them well... is not.

The pressure to adopt, to learn, to be seen as someone who gets it... this is real and it is landing on people who are already stretched. And the answer most of them are getting is, just move faster, just figure it out, it's so easy, just watch this video.

What nobody told you about the room

The high roller room has different rules. Not different from good quality principles... but different from the shortcuts that were survivable at the $5 table.

At the $5 table you could bluff. Vanity metrics. Coverage theater. The illusion of rigor. A defect shipped, a process failed, fix forward, maybe do a post mortem if leadership noticed and move on. The consequences were absorbable.

In the high roller room the buyout is a different number entirely. Reputational. Regulatory. Human. Those consequences existed before AI. CrowdStrike. Equifax. Boeing. The difference is there were always places to pause (even if we didn't). A gut check. A workflow that halted. A moment where someone could say "wait a second, this could be bad."

Agents don't pause. They don't hesitate. They don't have an "OH, SH*T!" realization moment. By the time anyone knows something is wrong... the damage is done. There's no hush hush post mortem. It's a headline. Or a punchline. Sometimes both.

The thing that doesn't change

The principles didn't change. Critical thinking. Context driven assessment. Holistic understanding. These were always the right foundation. AI didn't invent them and AI didn't break them.

What changed is the stakes. They are getting so high that you can no longer afford to skip them.

Everyone is gambling. The question is whether you know what table you're sitting at.

You were always supposed to define what must never happen before you built anything. You were always supposed to monitor behavior, not just outputs. You were always supposed to ask whether the system was doing what the business (and customers) actually needed... not just whether it "functioned."

Most organizations didn't do those things at the $5 table because they could get away with not doing them.

You can't get away with it anymore.

Nobody chose to sit down.

But you're in the room.

Everyone is gambling. With money. With customer trust. With... in some cases... lives.

And Lady Luck doesn't care if you're ready.

You're in the room. And the chips are already on the table.

It's time to ante up. Before the cards. Before the hand. Before... anything.

You pay your money and sit down at the table. What do you get for your ante? Nothing... just the chance to play.

The ante doesn't buy you anything. It doesn't buy you a good hand. It doesn't buy you guaranteed revenue. It buys you the chance. That's it. That's all it ever is.

And the house collects it. Every token. Every API call. Every monthly license whether the thing works or not. Whether the customer asks for it or not. Whether the revenue materializes... or not.

Most organizations don't know that yet. They think they are buying capability. Competitive position. Future readiness. Maybe so... but right now, it's just an entry fee.

And now you can't fold.

The spend is in the budget. The mandate came from the top. The board slide already said you were doing this. Folding now isn't a strategic pivot. It's an admission. And organizations don't make admissions. They make adjustments.

So they do just that... They adjust.

The tools budget gets locked. Headcount becomes the variable. The people who understand the systems... who know where the edges are... who are the ones to slow down and ask the harder questions. They get cut. Tribal knowledge walks out the door because somewhere in the accounting, people are expendable.

The ante doesn't just cost money. It costs the people who are supposed to make it work.

Early in my career I was selected for a team that wanted me specifically. I knew I was good. And I was about to prove it.

The project was a full rewrite of a calculation engine for an income tax software company. Big enterprise customers were running calculations that took over 24 hours. The rewrite was supposed to fix that. Tax experts were validating the calculations. And my only requirement: performance could not degrade.

I planned everything. Clean machine images. Controlled variables. Automated tests. Full performance monitoring. I shared the plan and got buy in. I knew what I was doing and I was confident in every decision I made.

The results came back and they were extraordinary. Calculations were completing in minutes.

Champagne. Cheers. Victory laps. High fives all around. We reached out to the customer with the most pain and told them their problem was solved.

The next day they called.... Calculations still weren't finishing.... We watched..... we waited.... I stress ate..... The final number was nearly 72 hours.

I sat in that very tense, very quiet war room with VPs, Developers, and Support.

"What database did you use?" That's it.... that was the question that connected the dots and unraveled everything.

In that moment, with just those five words.... I wanted to disappear.... instead, I closed my eyes and took a deep breath.... I didn't lie... I didn't deflect... I put my hands up and took full responsibility.

I had used the product's sample database. A full fielded, completely real looking database where every numeric value was zero.

Fun fact... Calcing zeros is really fast. Zero times anything is zero. Zero plus zero is zero. The engine wasn't fast. It was doing almost nothing. AND EVERYTHING LOOKED PERFECT.

Nobody told me about the zeros.... but I absolutely should have asked.

We spent two weeks working longer days than I want to remember. We got permission to test against a copy of the real customer database. We eventually got performance down to around 18 hours. Not the promised minutes... but real progress against a real problem.

I earned my way back. But I never forgot what it cost.

That experience gave me three questions I ask on every engagement and make sure my team asks too.

What am I missing? What am I not thinking about? And does it matter?

There will always be a gap. Sometimes it's small. Sometimes it means...... you're calcing zeros.

AI does this too. It returns confident, defensible, completely wrong answers when nobody asks the right questions. Not because it's broken. Because calcing zeros is really fast... and the picture looks perfect until reality shows up.

What's the assumption underneath your most confident result right now?

I was tagged that I had been acknowledged in Taking Testing Seriously, but today I opened my copy and saw my name in print for the first time. I had to pause.

Years ago, I was the person reading everything James Bach and Michael Bolton wrote (spoiler alert: I still do), trying to push past test cases and automation counts to build something more holistic, more human, and more strategic. Their work shaped how I see testing as inquiry, learning, and collaboration.

Much of my career has focused on critical thinking as an innovative test strategy that strengthens customer trust and protects business integrity. Quality has never been only about preventing defects. It has always been about strengthening systems so companies can compete, adapt, and endure.

To be acknowledged by two people who shaped that journey is incredibly meaningful. Thank you, James and Michael. This means more than you know.

What was a moment that made you pause and realize how far you have come?

Opening that book last week didn't just make me pause... It forced me to admit something uncomfortable about our industry.

Quality Engineering is evolving, but not nearly fast enough.

Too many teams still treat test case totals and automation percentages as if they mean something. They look tidy on a dashboard, yet they say almost nothing about whether the software will survive real customers and real complexity.

The real work has always been the thinking.

The understanding. The insight. The judgment.

Testing is not a mechanical task.

It is a human investigation into how a system behaves when the world refuses to follow the script.

Teams that treat Quality Engineering as strategic outperform everyone else.

They recover faster.

They adapt faster.

They earn trust faster.

The shift we need is simple:

More learning.

More critical thinking.

More shared responsibility for quality.

Quality is not a cost center.

It is a competitive advantage hiding in plain sight.

If we elevate how we think, we elevate what we build.

What is one thing you wish more teams understood about testing?

Something I keep noticing in the AI quality conversation right now.... Most of it is applying familiar quality models to an unfamiliar problem.

The quality practices most organizations built their programs on were designed for deterministic systems, predictable behavior, and stable requirements.

AI is none of those things.

The question isn't how to adapt existing quality processes to AI. It's whether the foundation those processes were built on was ever adaptable enough for what we're now asking of it.

The consequences were always there. AI just made them harder to absorb quietly.

That's the conversation I'm starting here. Not a teardown of what exists... an honest look at what quality actually requires when the system you're testing doesn't behave predictably.

What are you seeing? Is the AI quality conversation asking the right questions yet... or are we still reaching for familiar answers to unfamiliar problems?

AI failed.

The agent that exposed Meta's data. The chatbots ignoring instructions. The models that won't shut down.

None of them failed.

Those systems did exactly what they were designed to do. The agent responded to a question... correctly. The chatbots completed their tasks... thoroughly. The models prioritized the objective they were given... consistently.

None of them likely failed a test.

That's the problem.

Traditional quality models have been built on one foundational assumption. Same input, same output. Pass means it worked. Fail means it broke. Test it enough times and you know what you have.

AI doesn't work that way.

It's probabilistic. Context dependent. Variable by design. The same input can produce a different output depending on conditions you didn't control for, states you didn't anticipate, and contexts you never thought to test.

So when you run your test suite and everything passes... you haven't proven it works. You've proven it worked in the conditions you designed the tests for.... That's not the same thing.

The organizations getting this wrong aren't skipping testing. They're testing hard. But it seems they are applying a deterministic quality model to a probabilistic system and calling the result assurance.... aanndd... It isn't.

The question that changes everything isn't "did it pass?"

It's "how does it behave across contexts we didn't design for?"

Almost nobody owns the answer to that question yet.

The most expensive quality failures don't happen in production. They happen in the first conversation nobody thought to have.

Not the sprint planning meeting.... Not the architecture review.... Before any of that.

The conversation where someone asks... what breaks when this goes wrong? Who gets hurt that we're not thinking about? What do we lose that we didn't know we had?

Most organizations never have it.

Not because they don't care. Because nobody owns it. The PM is focused on delivery. The engineer is focused on the build. Leadership assumes someone below them is holding the whole picture.

Spoiler Alert.... Nobody is.

In a deterministic system that gap was survivable. A defect shipped. A process failed. Someone wrote a post mortem and moved on.

In a probabilistic system it compounds. The context you didn't define at the start becomes the blind spot the system operates in forever. And unlike a bug... you can't patch a missing conversation.

The unasked question isn't part of a test suite.... it isn't in a release review.... to make a fundamental difference, it needs to be asked all the way back at the beginning.

Most teams skip it entirely. Not deliberately. They just never knew it was theirs to ask.

Who in your organization owns that conversation?

How can a one letter mistake become immediately brand damaging?

Automated testing (checking) is powerful, but it cannot understand how something might make a customer feel.

I found this example while searching for an everyday item on Amazon. I knew it was a typo and I knew exactly how it got published. I've seen it a thousand times, just never this surprising. One missing letter turned an innocent listing into something you definitely never want on a product purchase page.

This error was not functionally critical. It was not a 404 or a broken image (which can also slip through automated checks, but I digress).

It was one missing letter, one simple typo that any dev team would call a low priority bug. But that one missing letter in the word COUNT turned it into an offensive word and becomes brand damaging. To the item brand, to Amazon, and potentially the reseller if it is a third party seller.

The uncomfortable part is that none of the non-human tools caught it, but every customer scrolling that page did.

This is exactly why automated testing can't be your entire quality strategy.

Automated checks can confirm expected behavior, but cannot understand intent, evaluate context, sense embarrassment, confusion, or lost trust. Most importantly, automated checks cannot detect when something is technically right but humanly wrong.

Software Testing is more than checking for correctness. It is critical thinking applied to real-world conditions.

When I teach teams about modern quality strategies and the limits of automation, I share examples like this because they're small, human, and unforgettable.

What's the smallest mistake you've seen turn into the biggest customer or brand headache?