Interview Question That Separates

Let me save you 40 hours of interview prep.

If you're studying PM interview questions from books written before 2023, you're preparing for a job that's disappearing. "How would you prioritize these features?" "Walk me through a product you'd build for X." "Tell me about a time you influenced without authority."

Those questions still show up. But they're not what separates the $250K offer from the $500K one.

There's one question — or a version of it — that I ask every AI PM candidate. And it filters out 90% of them instantly.

"How would you build an eval suite for [product]?"

That's it. And the responses I get tell me everything I need to know.

---

Why This Question Matters

Traditional software is deterministic. You click a button, something happens, it's either right or wrong. You can spec it, QA it, ship it.

AI is non-deterministic. You give the same model the same input twice and you might get different outputs. Both might be "good." Or one might be subtly wrong in a way that takes domain expertise to catch.

This breaks the entire traditional product development model. You can't write acceptance criteria like "when the user clicks Submit, the form saves." Instead you need: "When given a customer profile with these characteristics, the AI-generated recommendation must satisfy these 12 quality criteria with a pass rate above 95% across 500 test cases."

That's an eval. And building one is the single most important skill an AI PM can have.

When I ask candidates this question, I'm not testing whether they know the word "eval." I'm testing whether they understand why AI product development is fundamentally different — and whether they have the mental model to operate in that world.

---

What Bad Answers Look Like

I hear these constantly. They're not wrong, exactly. They're just incomplete in a way that tells me the candidate hasn't actually shipped AI.

The Metrics Answer: "I'd track user engagement, retention, and NPS scores to evaluate if the AI feature is working."

This is a lagging indicator strategy. By the time your NPS drops, you've already shipped garbage to thousands of users. Evals happen before production, not after. You need leading indicators.

The A/B Test Answer: "I'd run an A/B test comparing the AI feature to the existing experience and measure conversion."

A/B tests tell you whether the AI feature is better overall. They don't tell you why it fails in specific cases. They don't catch the 5% of outputs that are embarrassingly wrong. They don't help you improve the system — they just give you a thumbs up or thumbs down.

The Vibes Answer: "I'd review a sample of outputs with the team and flag anything that looks off."

This is manual QA. It doesn't scale. It's not repeatable. It depends entirely on who's reviewing and what mood they're in. I've seen teams "review" 20 outputs, declare victory, and ship a feature that fails catastrophically on edge cases they never tested.

The Delegation Answer: "I'd work with the ML team to define quality metrics."

This tells me you don't know what the metrics should be. You're hoping someone else does. In an AI PM role, you're the person who defines quality. You can't outsource that.

---

What Great Answers Look Like

The best candidates I've hired all did some version of this:

They started with the output, not the system. "First, I'd define what a good output looks like for this product. I'd create a rubric — what are the dimensions of quality? Accuracy, relevance, tone, completeness, safety. Each one gets a clear definition and a scoring criteria."

They thought about coverage. "Then I'd build a test set. Not 20 examples — hundreds. Covering the common cases, the edge cases, the adversarial cases. Different user segments, different input types, different contexts. I want the test set to represent the full distribution of real-world usage."

They distinguished between automated and human evals. "Some criteria can be checked automatically — factual accuracy against a known source, format compliance, length constraints. Others need human judgment — was the tone appropriate? Was the recommendation actually useful? I'd build both pipelines."

They talked about regression. "Every model update, every prompt change, every new feature gets run against the eval suite before it ships. If quality drops below threshold on any dimension, we don't ship. The eval suite is our quality gate."

They connected evals to product decisions. "The eval results don't just tell me pass/fail. They tell me where the system struggles, which informs the roadmap. If accuracy is great but tone is off for enterprise users, that's a targeted prompt engineering fix. If the model handles English well but struggles with other languages, that's a model selection conversation."

That's what an AI PM sounds like. Notice how none of that is about stakeholder management or sprint planning. It's about understanding what quality means, how to measure it, and how to use those measurements to build better product.

---

The EVAL Framework

Here's a framework you can use to answer any AI PM eval question in an interview. I've taught this to candidates I've coached, and it works.

E — Establish quality dimensions

What does "good" mean for this product? Break it into measurable dimensions. Accuracy. Relevance. Safety. Tone. Completeness. Latency. Cost. Not every dimension matters equally — rank them for your specific use case.

V — Validate with test cases

Build a test set that covers your full input distribution. Common cases (80%), edge cases (15%), adversarial cases (5%). Each test case has an input, expected output characteristics (not an exact expected output — AI is non-deterministic), and scoring criteria.

A — Automate where possible

Human evals are expensive and slow. Automate everything you can. Use model-graded evals (have a frontier model score outputs against your rubric). Use rule-based checks for format and safety. Reserve human evaluation for the subjective dimensions that resist automation.

L — Loop into development

Evals aren't a one-time gate. They're a continuous loop. Every code change runs against the eval suite. Results are tracked over time. Regressions trigger investigation. Improvements get measured quantitatively. The eval suite is your product quality system.

Memorize this. Practice it. Use it. It works for any AI product — chatbots, recommendation systems, content generation, search, agents, whatever.

---

The Real Reason This Question Filters

Here's the thing most candidates miss: this question isn't really about evals. It's about how you think.

When I ask "How would you build an eval suite?", I'm really asking:

• Do you understand that AI outputs are non-deterministic?
• Can you define quality rigorously, not just intuitively?
• Do you think in systems, not just features?
• Can you bridge the gap between product goals and technical implementation?
• Have you actually shipped AI, or have you just read about it?

The vocabulary is the tell. Candidates who've actually built AI products use words like "test set," "rubric," "regression," "threshold," "human-in-the-loop," "model-graded eval" naturally. They don't fumble for terminology. They don't say "we'd check if it's good."

I once interviewed a candidate who, when asked this question, pulled up an eval spec they'd written at their previous company. Redacted, anonymized, but real. They walked me through their rubric, their test set design philosophy, their automated scoring pipeline.

I made an offer that afternoon. They're now one of the strongest PMs on my team.

---

The Salary Gap Is the Signal

Let's talk numbers for a second.

A strong traditional PM at a top SaaS company — $200K to $300K total comp. Solid career. Nothing wrong with it.

An AI PM who can do everything that traditional PM does plus build evals, prototype with code, and make model selection decisions — $350K to $500K+ total comp. At the same companies.

That's a $200K+ gap for the same level of seniority. The market is telling you exactly what it values.

And the gap is widening. Every company is racing to ship AI features. The supply of PMs who can actually build them is tiny. Demand is exploding. Basic economics.

The fastest way to cross that gap? Learn to speak about evals the way engineers speak about architecture. Fluently. With depth. With real examples.

---

Try This Week

Take a product you know well — ideally one with AI features. Write a one-page eval spec for it.

Define 5 quality dimensions. Write 10 test cases. For each test case, specify the input and what "good" looks like on each dimension.

Now go back and stress-test it. What edge cases did you miss? What about adversarial inputs? What about different user segments?

Do this once and you'll understand evals better than 90% of PM candidates. Do this five times and you'll be able to answer the question cold in any interview.

---

The vocabulary is the filter. Learn to speak AI-native or get filtered out.

Every interview loop I run, the eval question is the dividing line. Not because it's a trick question — it's completely fair and completely learnable. But because it reveals whether you've made the mental shift from traditional product management to AI product engineering.

The good news? You can make that shift. It starts with understanding that AI quality isn't a vibes check. It's a system you build.

Build the system. Get the job. Close the gap.