Instacart uses LLMs at the query understanding layer, with offline RAG for head queries and a real-time fine-tuned Llama-3-8B for tail queries, while retrieval downstream is still traditional.

Instacart splits its approach across context engineering via RAG, post-processing guardrails, and fine-tuning Llama-3-8B. Head queries use an offline cached RAG pipeline, while tail queries hit a real-time fine-tuned model that keeps latency under 300ms. ✦ AI generated

The article author · ByteByteGo Newsletter · 2026-07-28 · original ↗

Instacart's strategy is layered across three approaches: Context engineering: Retrieval-augmented generation pulls Instacart-specific context (top-converted categories, historical conversion data, catalog details) into the prompt before the LLM sees a query. Post-processing guardrails: Semantic similarity filters drop LLM outputs that drift away from the original query. Fine-tuning: For the most advanced tasks, the team fine-tunes Llama-3-8B on Instacart's proprietary data, so domain knowledge gets baked directly into the model weights... The serving architecture splits along the head-versus-tail distribution. Head queries hit an offline RAG-and-cache pipeline that is latency-tolerant and deeply context-engineered, while tail queries hit a real-time fine-tuned Llama-3-8B model that keeps latency under 300ms through adapter merging, H100 GPUs, and autoscaling... After this solution, query rewrite coverage jumped from 50% to over 95%, with 90%+ precision across substitutes, broader rewrites, and synonyms. The real-time fine-tuned model improved search quality for the bottom 2% of queries (the cold-start tail), cutting scroll depth by 6% and complaints about poor tail-query results by half. The takeaway is that Instacart's LLMs live at the query understanding layer, with some offline-cached and some online and fine-tuned, while retrieval and ranking downstream are still done by traditional machine learning and information retrieval systems.

Read full article ↗excerpt · fair-use quotation