Measurements for understanding the pace of AI development inside frontier labs
By Gammatek ISPL, Industrial Systems & Compliance Analyst at Gammatek ISPL
Last updated: September 2026 | 14 min read
Author note : This piece synthesizes publicly available research from Anthropic's Institute, Epoch AI, Stanford HAI's AI Index, and peer-reviewed arXiv research, plus original analysis of what these trends mean for manufacturing and industrial technology adoption. It is not a summary of any single source — see citations throughout for direct references.
Why This Matters to You Right Now
"AI is moving fast" has become such a background hum that it's easy to stop actually asking: fast compared to what, measured how? That question matters more than it sounds, because the answer determines how quickly the tools showing up in your industry — compliance monitoring, predictive maintenance, safety auditing — will actually change, and how much runway you have before this year's "state of the art" software feels dated. Frontier AI labs have started building real internal measurement systems to answer this precisely, and the numbers they're finding are more specific, and in some cases more surprising, than the general "AI is advancing exponentially" narrative suggests.
The Problem With "AI Is Improving Fast" as a Statement
Every year for the past several years, someone has said AI progress is accelerating. That claim is both true and nearly useless without a specific measurement attached to it — accelerating by what metric, over what time period, in what domain? A model getting better at writing poetry says nothing about whether it's getting better at running a chemical plant's compliance audit. Researchers inside and outside frontier labs have spent the past year building more rigorous ways to answer the question, and at least five distinct measurement approaches have emerged as genuinely useful.
1. Compute Growth — The Oldest and Most Consistent Signal
The most established measurement is also the most mechanical: how much computing power goes into training the largest models. Research on AI governance has found that training compute for the most capable models has historically doubled roughly every five to six months between 2010 and 2024 — a pace dramatically faster than Moore's Law's transistor-doubling rate of roughly every two years. That compute growth has been sustained less by chips getting more efficient and more by companies simply building and buying far larger clusters, funded by rapidly increasing investment.
Epoch AI's more recent analysis adds an important nuance for 2026: the very largest labs, Anthropic and OpenAI among them, have grown their compute faster than the overall AI compute supply, but there's a structural ceiling coming. AI capital expenditure is now approaching roughly $1 trillion annually — nearly 1% of global economic output — and at that scale, further acceleration in frontier compute increasingly depends on the entire semiconductor and cloud industry's ability to keep building, not just on any one company's budget.
Implementation consideration: if your organization is planning a multi-year technology roadmap around AI-powered tools, compute-driven capability jumps are not guaranteed to continue at the same pace indefinitely — planning for a "step function" slowdown as a real possibility, not just continuous exponential improvement, is a more defensible planning assumption than extrapolating the last five years in a straight line.
2. Benchmark Saturation — When "State of the Art" Stops Meaning Much
The second measurement approach tracks performance on standardized tests. Stanford HAI's 2026 AI Index reports that on SWE-bench Verified, a widely used coding benchmark, model performance rose from roughly 60% to near 100% within a single year. That's a genuinely fast improvement — but it also means the benchmark itself is now close to "solved," which tells researchers less about future progress than it used to. This pattern — rapid improvement followed by saturation — recurs across benchmark after benchmark, which is why researchers increasingly treat any single benchmark's headline number with some skepticism once it starts approaching its ceiling.
3. The "Automation Index" — A New, More Direct Measurement
The most novel approach, and the one that gives this article its real news hook, comes from inside Anthropic itself. In a recent Institute publication, Anthropic described building what it calls an Automation Index — a direct measurement of how much of the actual day-to-day work of AI research and development, inside the company, is now being done by AI systems rather than humans.
The methodology is worth understanding because it's unusually rigorous for this kind of internal metric: researchers built a bottom-up map of roughly 15,000 granular AI R&D tasks by sampling 20% of relevant staff each week through July 2026 and using an AI research agent to review each person's work records — Slack messages and internal documentation — to catalog what they actually worked on that week. As of August 2026, the company reported approximately 30,000 AI agents actively doing research and engineering work at any given time on its most-used internal platform.
This matters well beyond one company's internal metrics. It's one of the first attempts to move from "AI seems to be automating research work" to an actual measured percentage, weighted by task importance, tracked over time — and it sets a template other labs and independent researchers are likely to adapt or attempt to replicate.
Implementation consideration: this is also a useful model for any organization, not just an AI lab, thinking about automation adoption internally — rather than asking "are we using AI?" as a yes/no question, building even a rough task-level inventory of where automation has and hasn't actually changed daily work gives a far more honest picture than adoption announcements or software licensing counts.
4. Research Automation Benchmarks — Testing AI on the Work of Building AI
A related and increasingly important measurement category tests AI systems specifically on the tasks of AI research itself, rather than general coding or knowledge tasks. Benchmarks like RE-Bench compare AI agents against human experts on machine learning research engineering tasks, while PaperBench tests whether an AI agent can replicate a peer-reviewed research paper's results from scratch. These are meaningfully different from general capability benchmarks because they attempt to answer a more pointed question: can AI accelerate the creation of the next generation of AI, and if so, how much and how fast?
Researchers studying this question are divided on how much it matters. Some argue that automating AI research could create a feedback loop that accelerates progress further; others argue that automating research alone, without corresponding growth in compute and real-world experimentation capacity, likely won't accelerate overall progress by much on its own. Safety frameworks at companies including Anthropic have started defining explicit thresholds for this kind of automation — for example, treating a certain level of AI R&D automation as equivalent to compressing multiple years of historical progress into a single year, and triggering additional safety review at that threshold.
5. Economic and Adoption Indicators — Measuring Impact, Not Just Capability
The final measurement category steps back from lab-internal metrics entirely and looks at real-world economic adoption. Stanford HAI's 2026 index reports that organizational AI adoption reached 88%, and that generative AI reached 53% of the population faster than the PC or the internet did — though adoption varies significantly by country and correlates strongly with GDP per capita. The same report estimates the value of generative AI tools to U.S. consumers reached roughly $172 billion annually by early 2026.
This category matters most for a practical reason: capability metrics tell you what AI can technically do in a lab setting, but adoption and economic metrics tell you how quickly that capability is actually reaching real organizations, in real workflows, with real accountability requirements — which is a meaningfully slower and messier process than benchmark improvement alone would suggest.
A Comparison: What Each Measurement Actually Tells You
Measurement | What it captures | What it misses |
Compute growth | Raw scale of training investment | Doesn't capture efficiency or real-world usefulness |
Benchmark scores | Capability on specific defined tasks | Saturates quickly; doesn't generalize well |
Automation Index (Anthropic-style) | Actual internal task automation, weighted by importance | Currently company-specific; hard to compare across labs |
Research automation benchmarks (RE-Bench, PaperBench) | AI's ability to accelerate AI research itself | Narrow domain; contested whether it predicts broader progress |
Economic/adoption indicators | Real-world deployment and impact | Lags behind capability by months or years; affected by non-technical factors (regulation, trust, infrastructure) |
No single measurement gives a complete picture — which is itself the most important takeaway for anyone trying to plan around "how fast AI is moving."
What This Means If You're Evaluating AI-Powered Tools for Your Own Operations
This is where the abstract research question becomes a practical one. If you're a plant manager or compliance lead evaluating AI-powered monitoring, compliance, or maintenance software, the multi-metric picture above suggests a few concrete things:
Don't evaluate a vendor's AI claims by benchmark scores alone. A model that tops a coding or reasoning leaderboard tells you little about how reliably it will perform on your specific compliance documentation or maintenance-prediction task — benchmark performance and real-world task performance in a narrow industrial domain are measured completely differently, and the gap between them is often larger than vendors imply.
Expect capability jumps to keep coming, but plan for adoption to lag behind them. The gap between "a lab announces a new capability" and "that capability is reliably usable in a regulated industrial environment, with proper audit trails" is often 12–24 months, based on the adoption-speed data above — build your technology roadmap around that realistic lag rather than the headline capability announcement.
Ask vendors for their own version of a task-level automation breakdown, similar in spirit to the Automation Index approach — not "do you use AI," but specifically which tasks in your compliance or maintenance workflow are actually being handled by AI today, versus which are still fundamentally human-reviewed. This is a far more honest diagnostic question than a marketing claim about "AI-powered" features.
Where This Is Headed
The clearest signal across all five measurement approaches is that AI progress is not one smooth curve — it's several different curves, moving at different speeds, some already flattening (certain benchmarks), some still accelerating (compute-driven capability, in the near term), and one entirely new metric category (task-level automation tracking) that barely existed as a rigorous internal practice a year ago. Anyone making planning decisions based on a single "AI is moving at X pace" headline is working with an incomplete picture — the organizations building this technology are themselves moving toward more granular, multi-metric internal tracking, and that's the more useful model to borrow, whether you're running a frontier AI lab or a manufacturing plant deciding when to adopt AI-powered compliance tools.
How Gammatek Approaches This
As AI-powered features increasingly show up in compliance and maintenance software marketing, we've taken the same lesson to heart internally: capability claims get evaluated task-by-task, against real audit and safety requirements, not against a vendor's benchmark slide. If you're trying to cut through AI marketing claims to understand what will actually hold up in a regulated industrial environment, that's exactly the conversation we have with clients.
See how Gammatek evaluates AI-powered compliance and maintenance tools for industrial use → https://www.gammateksolutions.com/post/the-best-worst-and-strangest-ways-ai-is-really-being-used-at-workhttps://www.gammateksolutions.com/post/fortinet-cyber-security-pricing-2026-firewall-cost-guide https://www.gammateksolutions.com/post/ai-hasn-t-gone-rough-its-worst-than-that
Comments