Product intelligence engine

The actual problem

The supplied catalogue is 1,000 rows and 6 columns, three of which are usually placeholder markers like -- Unbranded --. There is not one specification column. The delivery template on the other side wants 252: a three-level taxonomy, six description variants, twenty feature slots, sixty attribute triplets, asset references and commerce fields.

Every design decision in the project follows from that gap, and from one observation about it — a distributor description is not prose. It is compressed trade shorthand, and shorthand can be decoded without a model.

What it does

Eight stages per record: classify, extract, resolve, rule-check, enrich, judge, route, score.

Deterministic before probabilistic. 3M 775L Stikit Film P150 - Cubitron II 50 Disc/Box yields form, grit, attachment system, pack quantity and selling unit with no API call at all — P150 is grit 150, 27k is 2700 K, 5"x.045"x7/8" is diameter × thickness × arbor. The model only sees what the decoders could not resolve, which is why the deterministic pass covers all 1,000 rows in seven seconds while a full LLM pass would be roughly 4,000 calls.

Every value carries its provenance. Source, evidence snippet, method, confidence, and the alternatives that lost with the reason they lost. An adversarial LLM judge audits the assembled record, and anything below the publish floor is held back rather than guessed — a blank cell is recoverable, a confident wrong value is not.

Enrichment is scoped. Peer consensus across the catalogue is powerful and blind: a brandless Diablo belt sitting in a 3M-dominated abrasives group will inherit 3M’s series unless something stops it. Brand-scoped attributes (series, model, UPC) are refused from a category-only peer group, and any value inferred from siblings is tagged catalog_sibling, written at reduced confidence, and routed to a human — it never enters the delivery file as if it were sourced.

Engineering notes

The single most useful thing I learned was that Part_Manuf is not the manufacturer. It names the account the distributor buys from — “Jam Industrial Supply LLC” for a 3M abrasive, “Freud Inc” for a Diablo belt. Same column, reseller in one row and maker in the next. Publishing it as supplier and resolving brand separately from the brand columns, the description’s own brand vocabulary and maker-style account names filled 552 brands and 885 manufacturers with no API call.

Ambiguity is refused rather than guessed: when a messy header fuzzy-matches two canonical keys within the margin, it maps to neither, because silent mis-mapping is the most expensive error a catalogue can carry.

Rate limiting is client-side and deliberate. A free-tier 429 still consumes a request slot, so discovering the ceiling by hitting it poisons the following minute too — a token bucket paces every provider, and when a 429 does arrive the server’s own retryDelay sets the backoff instead of an exponential guess that would land early and burn another slot.

The 35 tests cover the decoding rules that are expensive to get wrong: that a distributor account never reaches the manufacturer line, that a round abrasive is read as diameter and not width, and that 1x6-16' is a one-inch board six inches wide and sixteen feet long.

Known limits

The keyword classifier settles about 71% of the catalogue on its own; the rest fall to the LLM classifier, or to General Product with no key configured. Web extraction obeys nothing but a timeout — no robots.txt handling — which is fine for a demo and not for a crawl. And SKU, list price and Prop 65 are left blank on purpose: they are distributor-side data that no amount of enrichment can honestly invent.