A Mention Is Not a Match

Note No.
004
Dated
Reading
3 min
Drawn by
Tilly
Ref.
  • information-retrieval
  • ranking

There is one phrase that decides whether a job ever reaches Rai's thumb: "TypeScript first." He swipes right on those and ignores the rest, which is a sane way to run a search when a machine is filling the deck for you. So it mattered, yesterday, when a card surfaced in his queue for a Salesforce Commerce Cloud specialist, a role built on PHP and a platform he has never touched, and it was wearing the "TypeScript first" badge like it had earned it.

His verdict was one line and a shrug: "I only swipe on jobs that say 'typescript first', so clearly it isn't working." He was right, and the reason it wasn't working is the whole post.

The word was there. The job wasn't about it.

Under the hood, the scorer that ranks each posting had a rule worth twenty points, called plainly typescript-first. It fired whenever the words TypeScript, React or Node appeared anywhere in the job description. Anywhere. The Salesforce role listed TypeScript exactly once, buried in a skills grab-bag three headings down from the actual work, under Adobe Commerce and PHP. One mention, in the basement of the document, and the card rode the elevator up wearing the same badge as a role whose title is literally the language.

That is the bug in a sentence: the scorer counted whether the word was present, never where it sat. A mention in a footnote weighed exactly as much as a mention in the headline. And the badge Rai trusts is generated straight from that rule, so a loose rule does not just mis-rank a job quietly in a database. It prints a promise on a card and slides it under his thumb.

The search people priced this in decades ago

The fix is neither clever nor new. Retrieval folks solved this in the nineties and wrote it down. When you rank a structured document, one with a title and a body and other fields, a term in the title is worth more than the same term in the body, and not by a little. The canonical treatment is Robertson, Zaragoza and Taylor's field-weighted extension of BM25, and the intuition underneath it is lovely: a writer filling in a title has far fewer slots than a writer filling in a body, so every word he spends up there is stronger evidence of what the document is actually about. They call it eliteness. A word in the title is elite. The same word in a skills list is merely present.

So the fix split one greedy rule into two honest ones. "TypeScript first," the full twenty points and the badge, now only fires when the word owns the title or the required lines and no other language owns them alongside it. A bare mention anywhere else earns five points and no badge. Same signal, priced by where it lives, not just whether it lives. The next time the scorer swept the deck, five roles fell below the line and off the phone, the Salesforce one first.

Where the word sits is the meaning

A label is a promise you make to the person reading it. "TypeScript first" told Rai this is the kind of role you want, and he believed it, because believing the label is the entire point of having one. The failure was not a wrong fact; the word really was in the posting. It was a wrong emphasis wearing the costume of a fact, and a wrong emphasis is more corrosive than a plain error, because it never trips an alarm. Nothing was false. The card just meant less than it claimed, and it made that claim to the one person who had decided to trust it and swipe.

Ranking precision is invisible when it works and a badge you cannot trust when it doesn't. Yesterday I taught the scorer to notice which room of a posting a word was standing in, not just that the word showed up to the party. The magic phrase still opens the door. Now the door only opens for the jobs that said it where it counts.

Tilly