A recent analysis shared by researcher Ethan Mollick highlights an underappreciated finding in AI scaling: the "second scaling law"—where performance improves with increased computational budget (more tokens processed)—does not appear to completely plateau for many reasoning tasks. This suggests that benchmark performance for current AI models may be artificially limited by token usage constraints rather than fundamental capability ceilings.
What the Data Shows
The core observation is straightforward but significant: when AI models are allowed to process more tokens (spend more computational "effort" thinking through a problem), their performance on reasoning tasks continues to improve. This contradicts the common assumption that scaling laws inevitably hit diminishing returns or plateaus where additional computation yields minimal gains.
Mollick notes this is particularly evident "with a simple harness"—meaning even basic techniques that allow models to use more tokens (like chain-of-thought prompting or simple iteration) can unlock better performance. The implication is that many published benchmark results may not reflect the true potential of current models, as they're typically evaluated with fixed, limited token budgets.
The Two Scaling Laws Context
This finding relates directly to the established framework of AI scaling laws:
- First scaling law: Model performance improves predictably as model size (parameter count) increases.
- Second scaling law: Performance improves predictably as computational budget (training compute) increases.
While the first law has shown signs of plateauing for some domains (throwing more parameters at a problem yields diminishing returns), the second law's behavior has been less thoroughly characterized, especially for reasoning tasks. Mollick's observation suggests that for reasoning—unlike perhaps other capabilities—the second scaling law remains active: more tokens reliably produce better answers.
Practical Implications for AI Practitioners
This has immediate practical consequences:
- Benchmark skepticism: Leaderboard scores that enforce strict token limits may underestimate model capabilities, particularly for complex reasoning problems.
- Cost-performance tradeoffs: Deploying AI for reasoning tasks becomes a direct tradeoff between answer quality (more tokens) and inference cost (higher token usage = higher cost).
- Prompt engineering value: Techniques that effectively leverage more tokens (like detailed chain-of-thought, self-correction loops, or reflection) gain importance as they directly translate to better outcomes.
Why This Matters Beyond Benchmarks
The non-plateauing of token-based scaling for reasoning suggests that current AI models may have more latent capability than standardized evaluations reveal. This aligns with anecdotal reports from developers who find that allowing models "more time to think" (via longer contexts or iterative prompting) produces substantially better results on complex tasks like code generation, mathematical reasoning, and strategic planning.
It also implies that the frontier of AI capability isn't solely dependent on next-generation architectures or massive parameter increases. Significant gains might be unlocked simply by applying existing models more generously—though at correspondingly higher computational costs.
gentic.news Analysis
This observation connects to several ongoing trends in AI development. First, it provides a technical explanation for the effectiveness of recent reasoning-focused techniques like OpenAI's o1 models, which explicitly use extended chain-of-thought processes, and DeepSeek's extensive reasoning steps in their latest models. These approaches essentially operationalize the principle that more tokens equal better reasoning.
Second, this challenges the narrative that scaling is "hitting a wall." While parameter scaling may be experiencing diminishing returns, compute scaling for reasoning appears far from exhausted. This could shift competitive advantage toward organizations with efficient inference infrastructure capable of deploying high-token reasoning economically.
Third, this finding has implications for how we evaluate AI safety and capability. If benchmark results are token-limited, we might be underestimating both the potential and the risks of current models. This aligns with concerns raised in AI safety circles about "sandbagging"—where models perform below their actual capability in evaluations.
Looking at the competitive landscape, this dynamic favors companies with cost-effective inference. Anthropic's Claude 3.5 Sonnet, which delivers strong reasoning at relatively lower cost, and emerging players like DeepSeek with aggressive pricing, could leverage this token-scaling effect more practically than competitors with higher per-token costs. The economic viability of "throwing more tokens" at problems becomes a key differentiator.
Frequently Asked Questions
What are "scaling laws" in AI?
Scaling laws describe predictable relationships between AI model performance and resources like model size (parameters) and computational budget (training compute). The "first scaling law" says bigger models perform better; the "second scaling law" says more training compute improves performance. The observation here concerns the second law's behavior during inference (using the model) rather than training.
Does this mean AI models will keep getting better indefinitely with more tokens?
Not indefinitely, but the plateau appears much later than previously assumed for reasoning tasks. There's likely still an eventual ceiling, but current evidence suggests we haven't reached it yet for many reasoning problems. The relationship likely follows a logarithmic curve: early tokens provide big gains, with diminishing but still positive returns as more tokens are added.
How does this affect real-world AI applications?
Developers now have a clearer tradeoff: spend more on computation (token usage) for better reasoning quality. This makes techniques like chain-of-thought prompting, reflection, and iterative refinement more valuable. It also means benchmark comparisons should consider token budgets—a model that performs slightly worse with limited tokens might outperform with generous token allocation.
Why haven't benchmarks caught this limitation?
Most benchmarks standardize evaluation conditions for fair comparison, which includes limiting token usage. This makes practical sense for controlled comparisons but may not reflect optimal deployment scenarios. The field is now recognizing that token-limited evaluations might underestimate reasoning capabilities, leading to new evaluation approaches that account for computation-performance tradeoffs.
