Technology Overview

The economics of private LLMs don’t follow intuition. Running your own model costs more than API calls at small scale, less at large scale, and the crossover point depends on factors that vary wildly by use case.

Training from scratch is off the table for most retailers. Training a competitive foundation model requires millions in compute, specialized expertise, and months of iteration. Even Meta’s Llama 3 405B model, released in July 2024, required training on 16,000 NVIDIA H100 GPUs. That’s roughly $400 million in hardware alone, not counting the engineering team or the power consumption. No retailer should do this. The economics don’t work unless you’re planning to sell model access as a product.

Fine-tuning is different. This means taking an existing open-source model and continuing training on your specific data to adapt it to your domain. For a retailer, that might mean training on your product catalog, customer service transcripts, or internal documentation. The compute requirements drop dramatically. Fine-tuning Llama 2 7B on a focused dataset might require a single NVIDIA A100 GPU for a few hours, costing hundreds rather than millions. Mistral AI introduced easy fine-tuning for its 7B model in September 2023, demonstrating that small retailers could adapt powerful models to their specific needs with modest compute budgets.

LoRA (Low-Rank Adaptation) and similar parameter-efficient fine-tuning techniques reduce costs further. Instead of updating all model parameters, LoRA trains small adapter layers that modify the model’s behavior while keeping the base model frozen. Microsoft and researchers at the University of Washington published “LoRA: Low-Rank Adaptation of Large Language Models” in June 2021, showing that adapters representing less than 1% of model parameters could achieve 95% of full fine-tuning performance. For retailers, this means you can maintain multiple specialized versions of the same base model, each adapted for a different task, without multiplying your storage and memory requirements.

Inference at scale is where the economics get interesting. API providers typically charge per token, with pricing that ranges from $0.50 to $15 per million tokens depending on model size and speed. A retail customer service system handling 10,000 conversations daily, averaging 1,000 tokens per conversation, would generate 10 million tokens daily, roughly 300 million monthly. At $3 per million tokens (GPT-4 class pricing in 2024), that’s $900 monthly. Running Llama 3 70B on a single AWS P5 instance costs roughly $55 per hour, about $40,000 monthly at full utilization. The API is cheaper.

But change the assumptions. Double the volume to 20,000 conversations daily. The API cost doubles to $1,800 monthly. The private infrastructure cost stays the same: You’re just using more of the instance’s capacity. At 100,000 daily conversations, API costs hit $9,000 monthly while infrastructure costs remain around $70,000. Add a second use case: same infrastructure, marginal cost near zero. The economics flip. This is why large retailers increasingly run hybrid architectures, using public APIs for variable loads and spiky traffic, while running private models for high-volume, predictable workloads.

Hybrid orchestration represents the practical middle ground. Route simple queries to smaller, cheaper models. Send complex reasoning tasks to larger models. Keep sensitive data on private infrastructure while using public APIs for general tasks. LangChain and LlamaIndex, two popular orchestration frameworks that emerged in 2023, made this pattern practical by providing abstractions that work across different model providers and deployment types. A retailer might use a locally hosted Mistral 7B model for product classification, GPT-4 via API for generating marketing copy, and a fine-tuned Llama model on private infrastructure for customer service conversations that include purchase history.