Randomized controlled study finds GitHub Copilot enables developers to complete a coding task 55.8% faster

Background

Researchers sought to quantify the productivity impact of AI pair programming tools under controlled conditions. Prior to this study, claims about Copilot's speed benefits were largely anecdotal or based on self-reported satisfaction surveys. The RCT design was chosen to enable causal inference rather than correlation.

What Was Implemented

• Randomized controlled experiment: 45 treatment (Copilot) vs. 50 control (no Copilot)
• Task: implement an HTTP server in JavaScript; pass a 12-test automated suite as quickly as possible
• Completion measured by objective GitHub Classroom timestamps
• Exit survey captured satisfaction, willingness to pay, and background characteristics
• Ethics approval: Microsoft Research Ethics Review Board
• Participants recruited via Upwork (freelance professionals); 70 of 95 completed the full study

Results

Treatment group completed the task in 71 minutes on average vs. 161 minutes for the control group — a 55.8% speed improvement (p = 0.0017; 95% CI: [21%, 89%]). Copilot users were willing to pay a median of $27.25/month for access. Task completion rates were higher in the treatment group by 7 percentage points (not statistically significant).

Lessons

• For bounded, well-defined coding tasks, AI pair programmers can roughly halve the time to completion
• The effect size has wide confidence intervals ([21%, 89%]) — real-world gains may be toward the lower bound
• Upwork-recruited freelancers on a standardized task may not represent enterprise developer productivity; field studies at Microsoft/Accenture found smaller effects (12–22%)
• RCT design provides stronger causal evidence than the self-reported satisfaction surveys commonly used to market AI coding tools
• Code quality (security, performance) was not measured in this study — an important gap for production use cases