Technology Overview

In 2020, OpenAI’s researchers discovered something that would reshape the AI industry. They had been training progressively larger versions of their GPT language model when they found that model performance improved predictably with scale. But more importantly, entirely new capabilities emerged at specific size thresholds, abilities that simply didn’t exist in smaller models.

Jared Kaplan, along with colleagues at OpenAI and Johns Hopkins, published “Scaling Laws for Neural Language Models” in January 2020. They trained models ranging from 768 to 1.5 billion parameters and discovered that performance improved as a power law. Double the model size, get a predictable improvement. Double it again, get the same relative improvement. This relationship held across six orders of magnitude.

The scaling laws also revealed a crucial balance: Model size and training data needed to grow together. OpenAI found the optimal ratio: For every doubling of model size, you needed roughly 1.7 times more training data. Too much model without enough data led to memorization. Too much data without enough model hit a performance ceiling.

The fundamental difference between small and large models wasn’t just performance but the nature of what they learned. Smaller models captured surface patterns: “customers who bought X also bought Y.” Larger models learned abstract concepts: what “buying” meant across different contexts, currencies, and cultures.

Anthropic demonstrated this empirically in 2022. Its team trained models from 70 million to 52 billion parameters on identical data. The differences were stark. Models below 7 billion parameters couldn’t handle negation (“Find me shoes that are NOT black”). Models below 13 billion couldn’t understand hypotheticals (“If I were shopping for a wedding…”). Models below 52 billion couldn’t maintain consistency across long conversations.

This wasn’t just about getting better at the same tasks. It was about gaining fundamentally different capabilities. A 1-billion-parameter model could match “red dress” to products tagged as red dresses. A 100-billion-parameter model understood that a customer searching for “something bold but professional for my promotion announcement” wanted confidence-projecting clothing in standout colors appropriate for office wear. The smaller model matched keywords. The larger model understood intent, context, and nuance.

Foundation models earned their name because they could serve as the foundation for countless applications. But this required training on incredibly diverse data. Google’s 2022 PaLM model consumed 780 billion tokens spanning 100 languages, code repositories, scientific papers, books, and web pages.

This diversity paid off in unexpected ways. Online home furnishings retailer Wayfair discovered its foundation model could answer questions it was never explicitly taught. When customers asked, “Will this couch fit through a standard doorway?” the model provided accurate answers despite never being programmed with doorway dimensions. It had absorbed that standard doorways were roughly 32 inches wide from thousands of indirect references across its training data.

The diversity also solved ecommerce’s long tail problem. Traditional models trained on popular products failed for niche items. Foundation models could reason about products they’d never encountered. REI’s model could recommend gear for “ultralight thru-hikers attempting a winter FKT” by combining knowledge about backpacking, winter conditions, and speed records from completely different sources.

The transition from pattern matching to genuine reasoning happened only at scale. This wasn’t incremental improvement but a phase change, like ice becoming water. At certain thresholds, models suddenly gained the ability to combine concepts they’d never seen together.

Neiman Marcus demonstrated this with its 20-billion-parameter shopping assistant in 2022. When asked for “something similar to what Blake Lively wore at the Met Gala but appropriate for a corporate fundraiser,” the model had to understand fashion references, celebrity culture, event dress codes, and corporate appropriateness, then synthesize all of this to recommend specific products. No amount of rule-writing could have achieved this flexibility.

Best Buy’s warranty support system showcased another aspect of scaled reasoning. When customers asked complex questions like, “I bought a TV 18 months ago with a 2-year warranty, then purchased extended coverage 6 months later. My remote just broke. What’s covered?” the model would decompose the problem into steps: purchase timeline, warranty terms, extension validity, and component coverage. This multi-step reasoning emerged naturally from scale, not programming.

The most profound aspect of foundation models was emergence: capabilities that appeared spontaneously at certain scales without being explicitly programmed. Chain-of-thought reasoning emerged around 60 billion parameters. Theory of mind appeared around 100 billion. Mathematical reasoning manifested around 175 billion.

GPT-2, with 1.5 billion parameters, could complete sentences and answer simple questions. GPT-3, with 175 billion parameters, could write poetry, solve algebra problems, and generate working code. It wasn’t trained to do any of these specific tasks. These abilities emerged from the model’s scale and the patterns it discovered in its training data.

The economics were staggering. Training GPT-3 cost approximately $4.6 million in compute alone. By 2023, a state-of-the-art foundation model cost over $100 million to train. Only a handful of companies could afford this investment. Target spent $30 million attempting to train its own model before abandoning the effort and licensing technology from Anthropic instead. The message was clear: Foundation models would be built by a few and used by many.