Technology Overview

Private and embedded LLMs solve problems that public APIs can’t. The use cases cluster around four themes: offline operation, privacy, latency, and compliance.

Offline mode for retail kiosks and mobile apps is the obvious case. A store kiosk in a rural location with unreliable internet still needs to help customers find products. A warehouse associate with a handheld scanner shouldn’t lose AI-powered inventory assistance when they walk into a metal shipping container that blocks cellular signals. Cruise ships, airplanes, and remote retail locations all represent environments where connectivity is expensive, unreliable, or unavailable. Embedded LLMs turn these from AI-hostile environments into AI-capable ones.

Privacy-sensitive customer data creates a different imperative. Customer purchase histories, return patterns, support ticket content, and payment information all sit behind strict privacy controls. Sending this data to a third-party API, even one with strong security guarantees, introduces risk and complexity. Vendor security questionnaires, data processing agreements, third-party audits, and breach notification obligations all multiply when data leaves your infrastructure. Running LLMs privately means your security perimeter doesn’t expand. The data stays in systems you already monitor and protect.

Extremely low latency matters more than most retailers realize. A 200-millisecond API call becomes 400 milliseconds round trip once you account for network overhead. Add TLS handshakes, load balancer routing, and queue wait times, and you’re often looking at 500-800 milliseconds. For interactive experiences, this latency is noticeable. Users perceive delays over 100 milliseconds. A store associate asking a handheld device for product information expects instant results, not the pause that comes with cloud round trips. Local inference can deliver responses in 50-100 milliseconds, fast enough to feel instant.

Multi-tenant marketplace tasks benefit from isolation. A marketplace platform serving hundreds of merchants needs to ensure that data from one merchant never leaks to another. Running separate model instances for each tenant is expensive when using APIs. Running a single private model with proper data isolation is more practical. Each merchant’s queries get processed through the same infrastructure but with strict boundaries ensuring their data—product catalog, customer information, pricing—never mingles with competitors’.

Compliance and regulated environments often mandate private deployment. GDPR in Europe, CCPA in California, and sector-specific regulations like HIPAA for healthcare all impose constraints on where data can be processed and how it can be shared. Defense and government contractors face even stricter requirements. When working with classified data or some NIST 800-171 compliance requirements, sending data to any external API is simply not permitted. Private LLMs become the only way to bring modern AI capabilities into these environments.

The pattern that connects these use cases: Private LLMs shine when the constraints of public APIs, connectivity requirements, latency, data residency, or cost structure, become the limiting factor. They’re not always better. They’re often slower to deploy and harder to maintain. But when the constraints matter, they’re the only viable option.