Same Context. Three Models. The Floor Isn't Zero.

The Experiment

We built a marketing chatbot for our VAR sales reps. Same Context Engineering methodology we sell to enterprises. Two vector stores. Thirteen ADRs. Six industry-specific compliance documents. Twenty-eight blog posts indexed. Running on AWS Bedrock.

Then we asked all three models the same question:

"A customer asked about SOC 2 compliance, how do I respond?"

Same RAG pipeline. Same vector stores. Same ADRs. Same system prompt. Three models.

Three completely different output qualities.

And the evidence isn't anecdotal. Every input, every output, every token count, every cost — logged in DynamoDB. Same timestamp range. Same question. Three model IDs. Three responses. What follows are the actual outputs.

Same context in. Wildly different value out. Let's look at what each model actually produced.

Nova Lite: Technically Correct, Completely Useless

Four generic paragraphs. The core fact is there — OutcomeOps deploys in your account. But that's it. No pushback handler. No competitive positioning. No escalation path.

The sales rep sends this email. The CISO asks a follow-up question. The rep is stuck.

Nova Lite read 49,067 tokens of context — ADRs, compliance docs, competitive intelligence, blog posts — and extracted one fact. The context contained a full playbook. The model returned a form letter.

Nova Pro: Accurate but Reads Like a Spec Sheet

Better. Seven bullet points. References ADR-008 by name. More technical detail. The information is accurate.

But it still reads like AWS documentation, not a sales toolkit. Still no pushback handler for when the CISO says "but we need a SOC 2 report." Still no "think of it like this" analogy that reframes the conversation. Still no competitive positioning against Cursor or Copilot.

The rep sends this and sounds like they copy-pasted from a technical wiki. The CISO is informed but not persuaded.

Haiku 4.5: The Full Playbook

Read that again. From a single question, Haiku produced:

• ✓A plain-English explanation so the rep understands before they respond
• ✓A copy-paste email ready to send
• ✓A pushback handler for "but we need a SOC 2 report" with a Terraform analogy
• ✓Framework-specific answers for HIPAA, PCI-DSS, SOX
• ✓Competitive positioning against Cursor and Copilot
• ✓"What NOT to say" guardrails to prevent credibility-destroying mistakes
• ✓CRM-ready talking points to memorize

The rep who gets this response handles a three-round conversation with a CISO. Cold. Without escalating to engineering.

The Takeaway

All three models read the same context. The ADRs contained the compliance positioning. The blog posts contained the competitive intelligence. The industry docs contained the framework-specific details. The pushback handlers, the Terraform analogy, the "what not to say" guardrails — all of it was in the context.

Only Haiku extracted the full value.

Context engineering raises the floor for every model. Nova Lite without context would hallucinate compliance claims. With context, it at least gets the core fact right. That's the floor being raised.

But some models can't reach the shelf where the good stuff is.

Nova Lite saw the context and pulled one fact. Nova Pro saw the context and organized it into bullet points. Haiku saw the context and synthesized it into a playbook — anticipating follow-up objections, surfacing competitive angles, and building guardrails the rep didn't know they needed.

That's not a context problem. That's a reasoning problem. The context was there. The extraction capability wasn't.

The Cost Metric Everyone Gets Wrong

Here's where the procurement spreadsheet lies to you.

Cost per token is the wrong metric. Cost per useful answer is what matters.

Nova Pro is cheaper per token. But the rep who gets the Nova Pro response needs to come back two or three more times to get the pushback handler, the framework-specific details, the competitive positioning. Each return trip burns another 49K+ input tokens through the RAG pipeline.

The rep who gets the Haiku response is done in one query. One pass through the pipeline. One answer. Complete.

Haiku wins the cost-per-useful-answer metric by a mile. And it's not even close.

But Token Cost Isn't Even the Real Cost

Let's talk about the cost they don't put in the spreadsheet.

The rep who gets the Nova Lite response sends a generic email. The CISO asks: "But we need a SOC 2 report from the vendor." The rep doesn't have a pushback handler. They escalate to engineering. An engineer spends 30 minutes drafting a response.

The rep who gets the Haiku response handles it in the meeting. Pushback handler ready. Terraform analogy locked and loaded. HIPAA follow-up answered before the CISO asks. The deal moves forward.

Five cents. The whole conversation handled. No escalation. No delay.

What This Proves About Context Engineering

This experiment proves two things simultaneously:

Context engineering means the model matters less. But less isn't zero. And the difference between "less" and "zero" is the difference between a form letter and a playbook.

Eating Our Own Cooking

One more thing. This isn't a lab experiment. This is our own sales enablement chatbot — built with the same Context Engineering methodology we sell to enterprises.

Thirteen ADRs. Two vector stores. Six industry-specific compliance documents. Twenty-eight blog posts indexed. Running on Haiku 4.5 — the cheapest Claude model. Producing enterprise-grade responses that handle CISO objections, competitive positioning, and regulatory compliance across multiple industries.

Built in a weekend.

That's not a product demo. That's eating our own cooking. Every prospect who reads this should be thinking: "If they built this for their sales team in a weekend, what could they build for my engineering team?"

And now this blog post goes into the marketing vector store. The next time a prospect asks our chatbot "why Claude over Nova?" — it can reference this article, with the actual data, from the actual audit logs.

The system feeds itself.

Your Competitors Can't Write This Blog

Copilot can't show a side-by-side of three models producing different quality outputs from the same context — because Copilot doesn't do context engineering. Cursor can't show industry-specific compliance responses — because Cursor doesn't ingest ADRs. Nobody else has the receipts.

We can show you the DynamoDB entries. Same timestamp range. Same question. Three model IDs. Three outputs. That's not marketing.

That's evidence.

Context engineering raises the floor. The model determines the ceiling. Choose accordingly.

Five cents per query. One pass. Full playbook. The model matters less — but less isn't zero.