Reading: Semantic IDs - Product Understanding at Scale

Instacart on compressing product embeddings into short discrete codes (“semantic IDs”) so recommendation and generative-retrieval models can reason about products that share no category, using catalog taxonomy as a free training signal instead of engagement data.

Source: Instacart Tech Blog on Medium

Link: Semantic IDs: Product Understanding at Scale

---

The problem: taxonomy is a tree, shopping isn’t

Instacart’s catalog is a strict tree: Dairy > Cheese > Parmesan. A customer building a cheese board wants Parmigiano Reggiano plus Castelvetrano olives (Pantry > Condiments > Olives) plus olive tapenade (Deli > Olives Dips and Spreads) — three branches with no shared ancestor below “Food.” A tree can’t express that these belong together. That blind spot hits hardest for new products (no purchase history to lean on), tail categories (too little interaction data), and mislabeled items (a protein bar filed under “Candy”).

What a semantic ID is

A short sequence of integers, e.g. 6_19_32_63, produced by compressing a product’s embedding through a residual vector quantizer (RQ-VAE). Products that mean similar things share a prefix; they only split apart at the level where they actually differ.

6_19        → the "cheese board" neighborhood (Italian cheese, olives, tapenade, deli trays)
6_19_32     → Italian cheeses specifically
6_19_32_4   → Fresh Mozzarella
6_19_32_16  → Crumbled Gorgonzola
6_19_32_63  → Hard Italian cheeses (Parmigiano, Pecorino, Asiago)

Nobody hand-wrote a rule linking Pecorino to olive tapenade — the model learned it by compressing their embeddings into codes that agree on the first two integers.

From embedding to code: how RQ-VAE actually quantizes

Start from a normal continuous embedding (Instacart’s comes from their ESCI search-relevance model). A residual quantizer turns it into integers (here ) using small codebooks, one per level:

For level , with running residual :

Each level asks “which codeword best explains what’s left over from the previous levels” — that’s why it’s residual: level 2 doesn’t re-approximate the whole embedding, only the error level 1 couldn’t capture. Reconstruction is just summing the chosen codewords back up:

Toy version, runnable

import numpy as np

codebook_L1 = np.array([
    [1.0, 1.0, 0.0],    # code 0: "cheese" region
    [-1.0, 0.0, 1.0],   # code 1: "olive" region
])
codebook_L2 = np.array([
    [0.05, -0.05, 0.0],   # code 0: nudge A
    [-0.05, 0.05, 0.05],  # code 1: nudge B
])

def quantize(embedding, codebooks):
    residual = embedding.copy()
    codes = []
    for codebook in codebooks:
        dists = np.linalg.norm(codebook - residual, axis=1)  # distance to each codeword
        code = int(np.argmin(dists))                          # nearest codeword
        codes.append(code)
        residual = residual - codebook[code]                   # what's left to explain
    return codes, residual

products = {
    "parmigiano": np.array([1.02, 0.90, 0.02]),
    "pecorino":   np.array([0.96, 0.80, -0.03]),   # close to parmigiano
    "olives":     np.array([-0.90, 0.10, 1.20]),   # different region
}

for name, emb in products.items():
    codes, residual = quantize(emb, [codebook_L1, codebook_L2])
    print(name, "->", "_".join(map(str, codes)), f"(err={np.linalg.norm(residual):.3f})")

parmigiano -> 0_0 (err=0.062)
pecorino   -> 0_0 (err=0.178)
olives     -> 1_1 (err=0.218)

Parmigiano and pecorino — genuinely similar embeddings — collapse onto the same code; olives, a different region entirely, gets a different one. (These toy codebooks were hand-placed so the demo is legible; in a real system every codeword is learned by gradient descent over the whole catalog, which is the next section.)

Why quantize at all instead of just using the embedding?

Millions of raw float vectors are expensive to store, expensive to nearest-neighbor search, and unusable as input/output tokens for an LLM-style generative model. A 4-level code with ~2,000 total codewords turns every product into 4 small integers — cheap to store, cheap to index, and directly usable as vocabulary tokens for a sequence model (more on that below).

The catch: a vanilla RQ-VAE doesn’t know what “similar” means

Trained on reconstruction alone, the quantizer only tries to compress back to faithfully — it has no notion that two marinara sauces should end up in the same bucket. Two failure modes follow: fragmentation (near-duplicate products split into different branches) and error propagation (a poorly-described product gets embedded badly and quantized into the wrong neighborhood entirely).

Teaching it with catalog structure, not engagement data

Instacart doesn’t have purchase/click history for brand-new products (that’s the cold-start problem this whole system exists to solve), so they can’t use behavioral co-purchase signal the way YouTube’s PLUM does. Instead they use something they always have: the taxonomy tree itself, turned into a graded similarity target.

A simple way to turn that into a loss — push cosine similarity of the embeddings toward the taxonomy-implied target:

$$
\mathcal{L}{\text{contrastive}} = \frac{1}{|\text{pairs}|}\sum{(i,j)} \big(\text{sim}(z_i, z_j) - \text{target}(i,j)\big)^2
$$

Two products with no shared ancestor get pushed apart (target 0); two exact-substitute products get pulled together (target 1); siblings sit in between. No one hand-labels pairs — the label is just read off the tree.

Hierarchical batch sampling — why random batches don’t work

Random sampling over millions of SKUs means almost every batch is entirely unrelated products — no positive pairs to learn from, so the contrastive term above has nothing to push together. Fix: construct batches on purpose.

import random

def build_batch(taxonomy, batch_size=64):
    parent = random.choice(taxonomy.parent_categories())        # e.g. "Pasta & Pizza Sauces"
    children = taxonomy.children(parent)                        # marinara, alfredo, pesto, ...

    batch = []
    for leaf in children:                                       # ~half the batch: related products
        batch += taxonomy.sample_products(leaf, k=batch_size // (2 * len(children)))

    unrelated = taxonomy.random_unrelated_categories(k=5)         # the other half: hard negatives
    for leaf in unrelated:
        batch += taxonomy.sample_products(leaf, k=batch_size // (2 * len(unrelated)))

    return batch

Sampling several products within the same leaf (multiple marinara brands) gives strong positives “for free”; sampling several sibling leaves under the same parent gives moderate positives; sampling unrelated parents gives hard negatives — all without ever writing an explicit pair label, the same trick as idealo’s implicit in-batch negatives from the previous post, just driven by tree structure instead of batch position.

The full training objective

$$
\mathcal{L}{\text{total}} = \mathcal{L}{\text{reconstruction}} + \mathcal{L}{\text{rq}} + \lambda \cdot \mathcal{L}{\text{contrastive}}, \qquad \lambda = 0.01
$$

• — standard “did the codes add back up to the original embedding” term.
• — the usual VQ-VAE codebook/commitment loss, pulling each codeword toward the residuals assigned to it and vice versa, so the codebooks themselves stay well-formed.
• — the taxonomy-guided term from above, kept small (0.01) so it nudges the code space toward catalog structure without overpowering reconstruction fidelity. Coarser levels (L1, L2) get weighted more, so broad groupings (“is this even food”) get priority over fine ones (“which brand of hard cheese”).

Two embedding flavors feeding the same quantizer

	ESCI (precision)	ESCI+Gemma (discovery)
input	raw product text through the search-relevance model	Gemini Flash extracts clean attributes (type, ingredients, diet tags) → embedded with Gemma
optimized for	“is this the exact same thing?”	broader lifestyle/usage similarity
example cluster	interchangeable whole-bean coffees, any brand	products that fit the same meal/occasion
used for	substitution, search, reordering	homepage feeds, cross-sell, exploration

Same RQ-VAE mechanism, two different embedding inputs — the quantizer just compresses whatever structure the embedding space already has, so the embedding choice is the product decision.

Checking it actually worked

Rather than trust downstream metrics alone, they measure the codes directly:

• Similarity–depth correlation: correlation between raw cosine similarity and how many code levels two products share, 0.69–0.84. Among pairs with cosine similarity ≥ 0.9, 98–99% agree on level 1, dropping to 18–37% agreement by level 4 (expected — L4 is supposed to split near-duplicates apart).
• LLM-as-judge on clusters: prompt an LLM to score each leaf cluster on functional coherence, purchase-together likelihood, and shopping-context relevance — a scalable stand-in for human review across thousands of clusters.
• Taxonomy alignment, used backwards: when a product’s code disagrees with its taxonomy label — a “Protein Bar” filed under “Candy” clustering with other protein bars — that’s usually the taxonomy label that’s wrong, not the code. This turns the whole system into an automated catalog-audit signal.

From codes to a generation task

With only ~2,000 codeword tokens total, a product’s semantic ID becomes something a sequence model can generate, one level at a time, instead of something you have to retrieve by nearest-neighbor search:

Exactly the chain rule an LLM uses for next-token prediction — except the “vocabulary” is ~2,000 product codewords instead of English subwords, and the output is the next relevant product rather than the next word. This generative-retrieval approach, tried first on product carousels, delivered +34% add-to-carts and surfaced products from 2.7x more emerging brands — biggest gains in exactly the tail categories that had no behavioral signal to lean on before.

Takeaways

1. A semantic ID = embedding compressed level-by-level by residual vector quantization; each level explains what the previous level’s codeword missed.
2. Reconstruction-only RQ-VAE doesn’t know “similar” — catalog taxonomy distance is used as a free, always-available supervision signal instead of engagement data.
3. Batches need deliberate structure (parent + siblings + unrelated categories) or the contrastive term has nothing to learn from.
4. Loss = reconstruction + codebook + small taxonomy-contrastive term (), coarse levels weighted heavier.
5. Embedding choice determines what the codes capture — precision (ESCI) vs. discovery (ESCI+Gemma) aren’t competing, they’re for different surfaces.
6. Where codes and taxonomy disagree, the code is often right — mislabeled catalog entries surface for free.
7. A small, discrete codeword vocabulary turns “find the right product” into “generate the right tokens” — same math as next-token prediction.