Would even an AI disaster on the scale of Hiroshima be enough to make humankind protect itself? I fear not
- Gammatek ISPL
- 1 day ago
- 5 min read
By Gammatek ISPL, Industrial Systems & Compliance Analyst at Gammatek ISPL
Last updated: August 2026 | 11 min read
Author credibility note: This analysis draws on Gammatek ISPL's direct work auditing safety and compliance systems across manufacturing, chemical, and pharmaceutical facilities, combined with publicly available incident data and current regulatory guidance as of August 2026. Gammatek is not a reseller of any AI platform named below, and this article contains no sponsored content.
Manufacturing plants are adopting AI-driven monitoring, predictive maintenance, and autonomous decision-making systems faster than most safety regulations can keep up with. That gap matters, because the plants most eager to deploy AI — for cost savings, uptime, and competitive pressure — are often the same ones where a single undetected failure can mean a chemical release, a fire, or a line worker getting hurt. This isn't a hypothetical thought experiment. It's a real, current gap between how fast AI capability is moving and how slowly safety frameworks are catching up, and it's worth understanding clearly before it becomes the headline nobody wants to write.
Why This Gap Exists
AI systems in industrial settings typically fall into three categories: predictive maintenance (flagging equipment likely to fail), process optimization (adjusting parameters in real time for efficiency), and increasingly, autonomous or semi-autonomous control (systems that can take action, not just recommend one). The first category is relatively low-risk — a missed maintenance prediction usually means unplanned downtime, not a safety incident. The third category is where the real exposure sits.
Most industrial safety frameworks — IEC 62443 for OT security, ISO 45001 for occupational safety, and sector-specific rules for chemical and pharma plants — were written before AI-driven autonomous control was common on plant floors. They govern human decision points, documented procedures, and hardware failure modes extensively. They say very little about what happens when a machine learning model, trained on historical data, encounters a scenario it hasn't seen before and makes a confident but wrong call — with no human in the loop fast enough to catch it.
This is the actual mechanism of concern: not AI "going rogue" in a dramatic sense, but silent confidence in edge cases. A predictive model trained on five years of sensor data doesn't know what it doesn't know. If a new failure pattern emerges — a novel combination of temperature, pressure, and vibration that wasn't in its training data — many current systems will either miss it entirely or, worse, actively suppress an alert because the pattern doesn't match anything it's been taught to flag as dangerous.
What This Looks Like in Practice
In Gammatek's compliance audit work across manufacturing and chemical facilities, one recurring pattern stands out: plants that adopted AI-driven monitoring often reduced or reassigned the human oversight roles that used to catch anomalies the software wasn't watching for. The logic makes sense on paper — if the system is 95% accurate, staffing down manual checks looks like a reasonable efficiency gain. But that remaining 5% isn't evenly distributed. It clusters around exactly the kind of novel, high-severity edge cases that matter most, because those are, by definition, the events the model has seen the least of.
A useful comparison: this is structurally similar to the aviation industry's early experience with autopilot systems in the 1980s and 90s. Pilots who over-relied on automation were sometimes slower to recognize and correctly respond to failures precisely because the automation had handled routine flight so well that manual skills atrophied. Aviation solved this over decades through mandatory manual-flying hours and strict human-in-the-loop requirements for critical decisions. Industrial AI adoption is roughly where aviation automation was in its early years — high capability, immature guardrails.
The Regulatory Lag, Concretely
Most industrial safety standards are revised on 3-7 year cycles. IEC 62443's core parts, for example, have seen incremental updates but nothing that comprehensively addresses autonomous AI decision-making in OT environments as of 2026. Meanwhile, the AI monitoring and control platforms plants are deploying today are often on quarterly or even monthly update cycles. That mismatch means a plant can be fully "compliant" on paper — passing every audit against current standards — while running AI systems that no existing standard was written to evaluate.
This is not an argument against AI adoption in industrial settings. Predictive maintenance genuinely does reduce downtime and can catch mechanical failures earlier than manual inspection. The argument is narrower and more specific: the compliance and safety documentation layer needs to explicitly account for AI decision points, not just hardware and human procedures, and right now most plants' compliance frameworks don't.
A Practical Framework for Closing the Gap
Based on the audit patterns we've seen, three concrete steps matter most:
1. Map every AI decision point in your safety-critical systems. Not just "we use AI for maintenance" — specifically, which decisions does the AI make autonomously versus recommend to a human? Which of those decisions, if wrong, could cause a safety incident rather than just downtime? This mapping exercise alone often surfaces gaps plants didn't know they had.
2. Maintain human-in-the-loop requirements for high-severity decision points, even after AI proves reliable. The aviation lesson applies directly: reliability in normal conditions is not the same as reliability in edge cases. Keep manual oversight specifically on the decisions where a wrong call has the highest consequence, regardless of how good the model's track record looks.
3. Build AI-specific incident logging into your compliance documentation. When an AI system misses something or flags a false negative, that needs to be logged and reviewed the same way a human procedural failure would be — not quietly patched in the next model update with no audit trail. Regulators are moving toward requiring this kind of AI decision auditability; plants that build it now won't be scrambling when it becomes mandatory.
What Good Practice Looks Like Today
Practice | Minimal / Exposed | Mature / Managed |
Human oversight on high-severity AI decisions | Reduced after AI proves reliable | Maintained regardless of AI track record |
AI decision logging | Not distinct from general system logs | Dedicated audit trail for AI-flagged and AI-missed events |
Edge case review | Ad hoc, after incidents only | Scheduled review of near-misses and low-confidence AI calls |
Compliance documentation | Covers hardware/human procedures only | Explicitly maps AI decision points against safety standards |
This Isn't Hypothetical — It's a Documentation and Process Gap You Can Close Now
The honest framing here isn't "AI is dangerous" — it's that the gap between AI capability and safety documentation is a real, current, closeable gap, and plants that treat it as a compliance and process question now will be in a much stronger position than those waiting for a regulation to force the issue after an incident makes it unavoidable. Waiting for regulators to catch up isn't a strategy; building the audit trail and human-oversight structure now is.
This is precisely the layer where a dedicated compliance and safety platform earns its place — not replacing your AI monitoring tools, but sitting alongside them to document decision points, maintain audit trails, and keep human oversight requirements enforceable rather than optional. If you're evaluating how your plant's AI adoption maps against current safety and compliance requirements, Gammatek's compliance platform is built specifically to close this kind of documentation gap — see how it works for manufacturing plants, or explore related reading on OT security fundamentals, predictive maintenance implementation, and choosing a network security stack for industrial environments.
Comments