162 green tests, and clicking the button still didn't work

Last June I shipped house ads into Condensr, the video summarizer I built. Not network ads. First-party promos that show a free user what they would get on a paid plan, with a button that says See plans.

The pull request was #96. It merged on June 13 with 521 lines added and 6 removed across 9 files. The test suite came back 162 of 162 green. The build was green. I wrote it up and moved on. (I was pleased with myself.)

Then I read it back, and asked a question the test output could not answer.

Does clicking See plans actually work?

I had 162 passing tests and no answer.

What the tests actually proved

That pull request wrote fifteen tests for the ad, spread across three files. Ten of them sat in the ad component's own file, and those mocked the router. That is normal practice and it is not wrong on its own. A mocked router lets you assert that a click handler fires and calls navigate('/account') with the right argument. Mine did.

Read that assertion closely. It proves the code called a function. It says nothing about what happens after the function is called.

The other five tests are the ones that taught me something. They ran a real router, with no mock anywhere near them. They missed it too. Every one of them asked whether the ad was on the page. Not one of them clicked it.

So the mock was never really the problem. What I actually shipped was a sentence a user would say out loud: as a free user, I see an ad, I click it, and I land somewhere I can upgrade. Fifteen tests covered the first clause of that sentence. Nobody had walked the rest of it.

What the live click found

So I clicked it. Real browser, real backend, a throwaway free-tier account.

The ad rendered. The click routed to the account page. The account page had a View Plans control. The control opened the plans modal. The path worked.

And the prices were missing, because they come from Stripe configuration my local environment did not have.

That last part is the part that matters. The gap was not in the code I wrote. It was in the code my code handed off to. That page already existed, I had not touched it, and no test of mine had any reason to visit it. The ad was mine. The destination was not. The user story crossed both, and only a person walking the whole sentence was ever going to find the seam.

The rule that came out of it

Two obligations now attach to any change that adds or alters a user-facing flow.

The journey gets clicked live before anyone says done. Not "the component renders." The full sentence, every hop, including the hops I did not write. Real data states per audience - signed out, each tier, empty, populated. If my local environment cannot reach the end of the journey, I say so out loud and name what covered the rest.

The journey lands in the automated suite. A flow verified once by hand decays quietly. If a change introduces a new user story, an end-to-end spec goes into the deploy-gated suite so the story is re-proven on every deploy. The spec asserts the outcome a user would describe, not the implementation underneath. Ad visible for the free tier, absent for the paid one, click lands on plans.

Why I am telling you this

A green suite is a strong signal about the thing it covers, and it is silent about everything else. The silence reads exactly like success. That is what makes it expensive.

Mocks prove calls. They do not prove journeys. If nobody has walked the sentence a user would say, the sentence is unverified, whatever the number at the bottom of the test run says.

162 of 162, and the question was still open.