Technology Overview

Not every retail problem should be solved with public API-based LLMs. Several scenarios demand different approaches.

Highly proprietary data: Data that represents competitive advantage, such as pricing algorithms, supplier relationships, or customer lifetime value models, should remain within corporate infrastructure whenever possible. Even with contractual safeguards, API providers could theoretically access data. In these cases, self-hosted Llama models or purpose-built machine learning solutions are more appropriate.

Ultra-low latency requirements: Use cases requiring sub-50 millisecond responses cannot rely on API calls due to unavoidable network overhead. High-frequency bidding systems, real-time inventory allocation, and instant recommendation updates require on-device inference or specialized low-latency models. Even the fastest LLM APIs introduce 80 to 200 milliseconds of latency, acceptable for chatbots but not for real-time systems.

Edge and offline environments: Scenarios with unreliable connectivity or limited bandwidth, such as mobile apps in emerging markets, in-store kiosks, or warehouse robotics, require local inference. Smaller Mistral models or quantized Llama variants can operate on-device, though frontier-level capabilities were still limited at the edge as of 2024.

Very high-volume workloads: At massive scale, API costs become prohibitive. Processing tens or hundreds of millions of interactions daily with per-call pricing can quickly reach unsustainable levels. In these situations, retailers rely on self-hosted models, precomputed outputs for frequent queries, or traditional machine learning techniques.

The broader lesson is that LLMs are powerful but not universal solutions. By 2025, the most effective retail implementations applied them selectively, balancing their strengths against technical and economic constraints. Choosing the right model matters, but equally important is knowing when not to use an LLM at all.