Let’s Talk About AI

A model review starts the same way most of the time. Someone puts a number on the screen. “The model is 92% accurate.” A pause follows. The data scientist is already thinking about the denominator. The engineer is wondering which test set it came from. The exec heard a grade, the kind you got in school, and is doing the natural thing, assuming 92% is good, and moving on.

You’ll hear this whole exercise called “evals” in a lot of rooms now, industry shorthand for evaluating how well a model actually performs. Same idea, shorter word, and it comes with the exact same problem. Whatever you call the exercise, nobody in that room is wrong about what the number means to them. They’re just not in the same conversation yet, which is fixable, for everyone in that room, starting with a handful of words we all already use, just not always the same way.

The words already have jobs

Accuracy, precision, recall, specificity. These words had full lives in the English language long before anyone attached a formula to them. Everyone in that room already has a working definition, and it’s not the statistical one. That’s what makes this hard to fix with a glossary. You’re not teaching new words. You’re asking someone to override a meaning they’ve used correctly, in every other context, their entire life. Nobody asks what the term means, because nobody thinks there’s anything to ask. The confusion doesn’t announce itself. It just sits quietly under the decision that gets made next.

I used to think the fix was calibration. Get everyone up to speed on the real definitions, and the room would sync up. That’s backwards. You can’t calibrate vocabulary that people don’t know they’re missing. The better fix is to stop asking the room to unlearn a word overnight, and instead build a bridge from the meaning they already carry to the one the term actually needs here.

An instinct I had to sharpen

For a long time, my instinct was to match the level of technical depth to whoever was in the room, more for the people closest to the model, less for everyone reviewing it from the outside. That’s not wrong. It’s usually right, for a spec review, a written deep-dive, anything where someone has the time and reason to sit with the detail.

A live decision meeting isn’t that room. I once prepared a fairly technical readout for a senior stakeholder with a strong technical background, assuming that background meant appetite for the math in that particular meeting. I was gently corrected, and stripped the detail out in favor of being distinctly clear about the results and my choice of language (in other words, the message needed to be clean and understandable). It turned out to be the better call, and it taught me something more specific than “keep it simple.” The audience that matters in a decision meeting isn’t defined by someone’s degree or title. It’s defined by what they’re there to do, which is decide, right now, with whatever’s in front of them. That’s a different axis than technical versus non-technical, and it’s the one that actually determines whether the detail helps or gets in the way.

What actually determines the right language isn’t who someone is. It’s what the room is there to do. A live decision meeting has one purpose, deciding right now with whatever’s in front of it. Everything I say in that room should serve that purpose first. Someone can be fully capable of the math and still be better served without it, in a thirty-minute meeting where the job is to decide, not to review.

Translating the metric, not simplifying it

Bringing these two elements together. The AI terms listed earlier are terms that can be vulnerable to assumption and misunderstanding. This is troublesome when in a room that is aimed at alignment and producing a decision.

This understanding has led to the following approach when discussing AI model evaluation performance in a room that needs to decide on how those models are performing.

Here’s the version that’s held up for me, built off a two-by-two I first ran into in an executive program built for exactly this problem, translating AI for people who have to lead it. Keep the real term. Just give it a quick mental swap you can run in your head the moment you hear it, so the gap it’s asking you to cross closes fast.

Put those side by side and two failure modes fall right out of them, even though neither one has its own row. Weak precision means too many false alarms. Weak recall or specificity means too many misses. Every one of these four numbers is really just a different angle on the same two ways a system can be wrong, crying wolf, or missing the wolf entirely.

None of this is an argument for retiring the real terms. The room still needs precision and recall when deciding on model performance, said plainly, because that’s the shared language that lets the people who built the model and everyone else in the room point at the same problem. The goal isn’t a permanent substitute. It’s enough shared footing that when someone finally does say “precision” in that room, everyone means the same thing by it, instead of three different things at once.

Remember the exec from the opening, the one who heard “a grade” and moved on. They weren’t wrong to reach for that word. Accuracy really does behave like a score, one number standing in for everything the model did. That’s exactly what makes it dangerous to stop at. A grade tells you how you did. It doesn’t tell you what you missed, and accuracy is the metric people reach for first for that same reason, it sounds the most complete, and it’s often the least useful one, especially anytime the thing you’re trying to catch is rare. A model can be 99% accurate and still miss almost everything that matters, if what matters barely shows up in the data to begin with. That’s worth saying out loud in plain terms, every time, because it’s the sentence that keeps a room from mistaking a number for a verdict.

You can’t max out both at once

The tradeoff that comes up most often, in my experience, is precision against recall. It’s worth covering on its own, because it’s the one that turns a metrics conversation into an actual decision, the kind that ends in a go or a no-go, not just a nod around the table.

Tune a system to catch more of the real thing, and recall goes up. But it also starts flagging more things that turn out to be fine along the way, so precision goes down. Tune it to only flag what it’s confident about, and precision looks great, but plenty of the real thing now gets missed, because recall dropped to get there.

You cannot turn both dials up at the same time. Tightening one loosens the other. That’s not a flaw in the model. It’s the shape of the problem itself, and no amount of better engineering makes the tradeoff disappear.

Here’s the part worth saying plainly, because it’s the part that actually eases a room, choosing to prioritize one of these over the other is not a compromise you should feel bad about. It’s the job. Every model built to catch something has to decide, on purpose, which mistake it would rather make. That decision belongs to the people in the room, not the model, and there is no version of this where you get to avoid making it.

Which means the real conversation was never “which number is higher.” It’s “which mistake can we live with.” A false positive and a false negative are rarely the same size of problem. Sometimes missing the real thing is far more costly than occasionally flagging something fine. Sometimes it’s the reverse, and a wrongly flagged good actor costs more in trust than a few bad actors slipping through cost in damage. That’s a business question, not a modeling question, and it’s the one that actually needs answering before anyone can confidently say go.

What the job actually is

Somebody in the room usually ends up doing this work, whether or not it was ever written into their title, and more often than not these days, that job lands on a product manager. None of this is really about vocabulary, not entirely. It’s about whether people actually walked away aligned, not because everyone nodded along, but because they could turn around and explain it to someone who wasn’t there. That’s the real test, no matter where these words come up. Not what got said, but what someone could actually repeat afterward, in their own words, without hesitating.

Accuracy, precision, recall, specificity. Four words that used to slow things down while everyone quietly guessed at what the others meant. They don’t have to anymore. Anyone can ask about any one of them now, with confidence, without waiting for someone else to translate first. Good data still needs better judgment, and this is what it looks like when the jargon finally stops standing between a good idea and a good decision.