The Gate I Built

I’ve spent a working life just outside the rooms where work gets anointed. Not entirely outside — I wrote tech books that went out under Sams and O’Reilly imprints , among others, and built apps that shipped under other companies’ logos, CNN and Xfinity among them. I was in those rooms often enough. I just paid a toll at the door every time, and the thing that vouched for the work was never me. It was the imprint on the spine, the logo on the app. That’s most of the truth about me. Here’s the part I’m least comfortable with: for about four years, I helped build one of those gates myself. And the thing making it look ridiculous now is a different machine — one that does for free the exact trick my gate was built to measure.

In May of 2000 I signed a one-page contract with a company called ReviewNet and became, in their words, a “Technology Review Team Member.” Five hundred dollars a month, work-for-hire, everything I wrote theirs forever. What I wrote was tests. Not school tests — hiring tests. ReviewNet sold employers a bank of technical screening questions, and a candidate who wanted a programming job would sit down and answer mine before a single human at the company ever gave them a chance. I’d started a couple of years earlier with a fifty-question Java exam. By 2000 I was writing and vetting questions across the whole stack a web shop hired for, Java fundamentals up through J2EE and JSP and the scripting glue that held it together. I was, in the most literal sense the language allows, a gatekeeper. I built the turnstile and never had to stand at it.

Why the gate existed

Before I either defend or indict the thing, you have to see the hole it was filling, because the hole was real and it still is. Software development has no working credential. A computer science degree certifies that you can reason about algorithms, which is not the same as being able to ship a functioning piece of software with real features, and everyone who has ever made a hire knows it. Some of the best developers I’ve worked with never finished college. Some of the worst held advanced degrees. There’s no bar exam, no medical board, no license to practice. Microsoft had been certifying people since 1993, and the certs tested narrow product trivia that often had little to do with whether you could actually build anything. The industry knew it, too: by the late nineties it had a name for the graduates — paper MCSEs , certified on the exam and lost in a server room — and a joke about what the letters really stood for, Minesweeper Consultant and Solitaire Expert. The one credential the field had, the field didn’t believe. So a hiring manager looking at four hundred résumés had, functionally, nothing — no accreditation to lean on, no signal to trust, and a real job to fill.

And the obvious alternative — sitting your best engineers down to interview every plausible candidate — cost the one thing a software shop can least spare: those same engineers’ hours at the keyboard. An automated question bank scaled. It promised to vet people without pulling the people you’d hire them to work alongside off their own work. That promise is most of why the thing existed at all.

ReviewNet sold that manager a signal. A crude one, but a signal. That’s the honest origin of the thing. Not a scam dreamed up to charge tolls, but a primitive first swing at a genuinely hard, genuinely unsolved problem: how do you vet competence in a field that refuses to be credentialed? Call it version 0.1. I wrote a great deal of version 0.1.

From the wrong side of it

I know that vacuum from the receiving end, which is the part that makes the rest of this hard to write. Long before ReviewNet, I came out of an electrical engineering program with something close to a 2.9 GPA. Engineering school was absolute survival for me. I kinda knew at year three that I really should’ve chosen computer science but I was determined to see it through, and I did…with that 2.9. My wife went through the same kind of program with a 4.0 and walked straight into a job at Motorola; when she hand-carried my resume in, they wouldn’t take the application. Not “no thanks” — they wouldn’t let me in the door far enough to be turned down on the merits. The filter ran on a number, my number was below the line, and no human was ever going to see the rest of me.

What the number couldn’t see was that I’d spent college actually building things — shipping working software to real, paying customers, the messy applied work that barely moves a GPA. By the one measure that turns out to matter on the job, I was further along than my transcript said. The software world was full of people like that: self-taught, half-credentialed, dropouts and B-students who could nonetheless build the thing, diamonds in the rough if you knew how to spot us, invisible if you didn’t. A system built around accreditation had no way to see us at all. Mostly still doesn’t.

Then I grew up and built the filter to filter out future me’s. Ouch.

What it actually measured

The trouble with the version 0.1 hiring filter was the trouble with any test made of questions. It couldn’t measure whether you could do the job; it could only measure whether you could answer questions about what the job requires, and those are different skills. Anyone who has crammed for a test and forgotten it by Monday knows they’re different.

Fifty Percent of What? made the case that we keep reaching for a number because the things we actually fear are hard to measure and a number at least looks countable. A proxy. The screening test is that same reflex aimed at hiring. Can this person do the work is expensive to find out; you have to hand them real work and watch. So we sold a cheaper thing in its place — fifty questions standing in for an interview standing in for the job. The bet was that someone who could answer them off the top of his head had probably done the reps, and someone who’d done the reps could probably do the work. Probably. The whole instrument ran on that “probably.”

And here’s the deeper flaw, the one only hindsight makes obvious: rote recall of syntax and API trivia turns out to have almost nothing to do with whether a person can actually build software. The best developers I know don’t carry encyclopedic API details in their heads at all; they have the opposite skill — reading an unfamiliar interface fast, recognizing the pattern under it from the hundred they’ve seen before, sensing where the gotchas are buried. It’s what Tom and Ray Magliozzi did on Car Talk for thirty-five years: diagnose a car they couldn’t see, on the other end of a phone line, by asking the right handful of questions. Neither of them was reciting part numbers. They had the manuals for that.

The man who can fix it is on the other end of that line. He has never seen this engine and never will. He'll find it anyway.

Same in software. The competence was never the memorized fact. It was knowing how to find and absorb the fact you didn’t have, which in 2000 meant knowing your way around the documentation in the context of creative problem solving. Which is the joke at my expense, and the whole essay in one line: ReviewNet hired me to write these tests partly because I’d written the reference books on the subjects — and then we built a test that pretended the reference didn’t exist.

Where the machine actually changes things

Here I have to be careful, because the easy takeaway from this is wrong. The easy version says a language model answers all fifty hiring questions in seconds, so the test is dead. But plenty of companies didn’t use these as take-home screens; they sat the candidate at a machine in the office and watched — proctored, live, the answers coming out of his own head or not at all. In that room the model on your phone does you no good.

What AI changed is quieter and worse. The thing my questions rewarded was unaided recall, and that is now exactly the part of the job a machine does for free. On any real workday the developer I was screening will have the model right at his elbow, the way he’ll have the docs and the search bar and his own old code. The way the Magliozzis had Chilton’s automotive reference manuals . So even a proctored test that runs perfectly is measuring the candidate in the one condition he will never actually work in: alone, from memory. You can run it flawlessly and still learn the wrong thing, because you tuned it for a skill the job stopped requiring the day the machine arrived at every desk.

The one I was proud of

There was one piece of it I still kinda like, because it was the only part trying to be a forge instead of a tollbooth , and because the instinct behind it was right. Call it version 0.2. In 2002 I built what we called a scenario: a small Java program presented as a running thing instead of a multiple-choice list, a slideshow walking the candidate through what it did, with questions that asked not what does this keyword mean but what would you do here, and why. The idea was to quit testing recall and start testing judgment: watch someone reason through a real-ish situation instead of reciting facts.

That instinct didn’t die. It won. It’s essentially what the famously challenging interviews at places like Google do now — put a person in front of a problem and watch them think, live, out loud. My version was a dial-up cartoon of a thing the whole industry eventually agreed was the right idea, so I don’t get to be smug about how crude the fifty questions were; the better answer was already visible in 2002, and serious companies spent the next two decades building it out.

And version 0.2 is beatable too, on the same terms — if you let the candidate use a model. Which exposes the real question hiding under the whole enterprise: are you measuring what a person can do alone, or what they can do with the tools they’ll actually have on the job? That’s a question about taste rather than memory, and it’s the one every test now has to answer for itself. This one was about the gate, and the gate I built measured none of it.

The part that’s on me

What I’m least proud of isn’t the crudeness of the thing. It’s the overclaim. Everyone in that business knew the test was a “probably,” and the business needed it sold as a “definitely,” and the gap between those two words is where both the money and the unfairness lived. Good people got filtered out by my questions for the crime of not having memorized something they could have looked up in twenty seconds. I knew exactly what that felt like. I’d been filtered out myself, by a different number, and I built the machine anyway. For five hundred dollars a month.

What’s left

So when people ask whether AI is going to wreck technical hiring, I think they’re braced for the wrong impact. The screening gate was never a sturdy thing the machine knocked over. It was a version-0.1 patch over a hole the industry has never fixed — no reliable way to certify a developer — and AI didn’t make the hole. It just flooded the desk with the exact capability the tests were built to ration, until the patch looked as flimsy as it always was. And now we’re seeing tech hiring devolve into machines reviewing documents created by machines…hardly an improvement.

But pull off the recall test and the scenario test and every version that comes after, and you reach the thing none of them could touch — the thing my four years were a way of not admitting. You can’t really know a person from the outside. Technical skill was always just the foot in the door, the cheapest and most measurable part, which is exactly why we measured it. Everything that actually decides whether a hire works is intangible: judgment, what someone does when it breaks at two in the morning, whether they finish what they start, whether they write horrific spaghetti code, whether you’d want them in the room on the worst day. No question bank reaches that. No scenario reaches it. The best interview at the best company is still, under the suit it’s wearing, a guess.

The only thing that ever actually worked for me was the slow way. A family member once suggested I take up golf, for the networking, for business. It struck me as the worst reason anyone has ever picked up an activity. I skateboarded instead, because I loved it, and some of the longest working relationships of my life came out of that. The job I have right now traces back to a friendship built on a board when I was a teenager. Stephen, who I still work with, once got asked how he finds good people, and he answered — joking, but not really — “well, it’s sometimes people I skateboarded with at my grandmother’s house.” Nobody was networking. We were just there, for years, doing a hard thing with each other and taking our lumps along the way, which turns out to be the most you can ever learn about a person.

That’s the joke at the bottom of the whole business, mine included. Hiring is a crapshoot we dress up as a process, because the honest alternative — that you can’t know a stranger until you’ve worked beside them — doesn’t scale and doesn’t sell. Years of someone’s public record can show you a great deal; a résumé caught at the gate, on a Tuesday, shows you almost nothing. The test lives at the gate, and the gate is the worst possible place to try to know anyone. A number that couldn’t see what I’d built kept me out of a job I could already do; I spent four years building numbers with the same blind spot. And the machine that finally exposed the tests can’t patch what’s underneath, because what’s underneath was never a measurement problem. It’s the oldest one there is: another person, and how little of them you can know before you decide to trust them. The honest move was always to hand someone something real and find out together.