Governance

Dario Amodei, CEO of Anthropic, has published an unusually direct argument about the future of AI development: the companies building the most capable models need to slow down. He is not calling for them to stop, or to give up the benefits AI may bring. He is asking them to leave enough time to understand, evaluate, and control the systems they are making more powerful. What makes this worth reading is who is saying it. Amodei runs one of the companies competing to build those models. He faces the same pressure to move faster as everyone else, and he is arguing that the pace itself has become part of the problem. ...

Anthropic published a new piece last week, The AI-Native SDLC Playbook. It is long, detailed, and worth the time. It also lines up closely with what I have been writing here for the better part of a year. The argument at its center takes one sentence: code is no longer the bottleneck. Everything in the document follows from it. Once agents write most of the implementation, the constraint moves to the stages on either side, the ones still running at human speed. Approval queues build. Controls sized for human output stop matching what arrives at them. Governance that meets monthly cannot govern work that ships hourly. ...

I ended the last post with a question: the next challenge is not building the Judgment Log. It is whether anyone writes in it once the deadline is two hours away. That question only has a useful answer if the artifact is light enough to actually use. So instead of arguing for it further, I want to show it. Take a fictional but familiar scenario. A checkout flow. A promotional window. A promo code validation feature built using AI-assisted development. It shipped. Three weeks later, it broke when the campaign introduced expired codes. In the post-mortem, nobody could answer the three questions that mattered: what did the PM cut and why, what did the designer choose between, and what did the engineer override. ...

In April, Meta employees burned through 73.7 trillion tokens in roughly thirty days. The company found out not because spending crossed some alarming threshold, but because an internal leaderboard, nicknamed Claudeonomics, had turned token consumption into a competition. Employees and teams were ranked by how much they used. The system did exactly what it was built to do: usage went up. What it could never show anyone was whether any of that usage produced something worth the cost. Meta is now dismantling the leaderboard in favor of a centralized monitoring platform called AI Gateway, built to track spending in real time and flag unusual spikes. ...

In a post I wrote last December, I discussed the production triangle: the constraint that governs every production system, including software. You can optimize for two of three dimensions, time, quantity, and quality, but never all three. Push velocity while adding features, and quality absorbs the cost. Every time, without exception. This constraint is not new. Kahneman mapped the cognitive mechanism in Thinking, Fast and Slow: under time pressure, we disengage System 2, the slow, deliberate reasoning that catches the things that will hurt us later, and default to fast, intuitive pattern-matching. Kuutila et al.’s 2020 systematic review of the software engineering literature confirmed the outcome: increased throughput, decreased quality, and a root cause that almost always traces back to the pressure itself being a product of bad estimation. The triangle is not a theory. It is physics. ...

Gartner published a prediction this week: by 2027, 40% of enterprises will demote or decommission autonomous AI agents due to governance gaps that are only identified after production incidents. The root cause they name at Level 3 of their autonomy framework is approval fatigue. Agents execute actions, writing data, sending communications, and modifying configurations, but only after explicit human approval. Under time pressure, that approval becomes reflexive. The control degrades, and the risk compounds in silence. ...

I recently argued on the JFrog blog that trusted AI requires more than model quality. It requires visibility, provenance, governance, and a real system of control around the things models consume, build, and ship. That is the foundation. This post is about what you build on top of it. Because visibility is necessary. Without it, you cannot govern anything. If you cannot see which models are running, where they came from, how they behave, and what they touch, you do not have a governance posture. You have hope dressed up as architecture. ...