What a hundred passing tests didn't catch

We were partway through Classline's first sprint — teacher login, a class roster, tap a student, award a point. The Go side was done: handlers wired, sessions working, the append-only awards log recording correctly. Over a hundred test functions passed, cleanly, across every package. Then we ran the seed script, opened a browser, logged in, and tapped a student.

Nothing happened.

The bug

class.html — the roster page — loaded a script tag:

<script src="/static/htmx.min.js"></script>

There was no such file. There was no such route. /static/htmx.min.js 404'd, htmx never loaded, and every hx-get/hx-post attribute on the page was inert markup — a button that looks clickable and does nothing, because the library that would have wired it up never arrived.

The fix, once found, was small: vendor htmx into the repo, embed it, serve it from a route that sits outside session auth (a static asset can't require a login, or the login page itself couldn't load its own styling). Twenty-some lines, one commit.

Why a hundred tests missed it

This is the part worth sitting with, because none of those tests were badly written. They did exactly what they were asked to do.

The handler tests rendered class.html through Go's html/template and asserted things about the string that came out: does it contain the right student names, the right point totals, a POST form pointed at /logout and not a bare link. That's a real and useful thing to check. None of them asked a second question: does everything that string references actually exist?

A <script src="..."> is a promise the HTML makes to the browser, not a promise the HTML fulfills by itself. html/template will happily emit src="/static/htmx.min.js" whether or not that path resolves to anything. The Go compiler has nothing to say about it — it's a string literal inside another string literal. Our handler tests, being handler tests, never sent an HTTP request for that path; they never got the chance to notice it wasn't there. And the browser was the only thing in the loop that treats a missing script as load-bearing failure rather than four extra bytes of markup.

That's the general shape of the miss, and it isn't specific to <script> tags. A unit test — even a good one, even a hundred of them — verifies the things it was written to check. It cannot verify a promise made inside the artifact under test to something outside the process the test runs in. A template that references an asset is making exactly that kind of promise. So is a link to another route, a form pointed at an endpoint that got renamed, a Content-Type header a client will actually parse. None of that shows up in a diff of expected-string versus actual-string. It shows up when something tries to fetch it.

What we changed

Two tests, both still in the suite:

The first is the narrow one — TestStaticHtmxAssetServed sends a real GET /static/htmx.min.js through the actual router and checks for a 200, a non-empty body, and a JavaScript content type. It would have caught exactly this bug, and it exists mostly so this specific regression can never come back silently.

The second is the one that matters more. TestEveryTemplateAssetReferenceResolves doesn't know what htmx is. It scans every template file for a literal src= or href= pointed at a local path, skips the dynamic ones (anything with {{ in it — those are application routes, already covered by their own handler tests) and skips external links, and then fetches every remaining path through the real router. It would have caught the htmx bug without anyone having decided in advance that htmx specifically was the risk. It will catch the next missing asset too, whatever it turns out to be called, because it isn't checking for this bug — it's checking the property that made this bug possible.

We verified both tests actually catch what they claim to: temporarily removing the vendored file breaks the build at the go:embed directive (a different, earlier failure — also useful), and temporarily pointing a template at a path that doesn't exist fails TestEveryTemplateAssetReferenceResolves cleanly. Then we put the real files back and reran the whole suite.

The general lesson

Unit tests describe a boundary — this input, this output — and they are extremely good at holding that boundary steady over time. What they cannot do, structurally, is notice something true about the world outside the boundary they were told to check. "This HTML string contains the substring I expected" is a fact about a string. "This HTML string, loaded in a browser, actually works" is a fact about a browser, a network request, and a file that either exists or doesn't — and no amount of assertions about the string will ever stand in for asking that question directly.

The fix wasn't writing more tests of the kind we already had. It was writing one test that crossed the boundary the others couldn't see past — and then, because we'd rather not rediscover that boundary by hand every time, writing a second one that keeps checking it automatically. A green test suite is evidence about the properties you told it to watch. It was never evidence about the ones you didn't think to.