This website uses cookies

Read our Privacy policy and Terms of use for more information.

The AI cacophony

On the 29th of September the White House ushered in the “era of super intelligence”. Usage of the designation “AI” in federal-speak is now frowned upon. The tech glitterati gamely tried to fall in line. Underlying this shift is the perceived need to mute or drown out the doomSayers that are predicting imminent cataclysmic end of the world as we know it. I assume the rebranding is an effort to instill hope and optimism for an era of abundance. Considering the alternative, I guess we should collectively manifest that new era into being.

While there are a host of issues at play, the vast majority of the cacophonous debate can be categorized into two camps: the abundance camp, which argues for balls to the wall1 acceleration, and the doomer camp, who put p(doom)2 at 10% or higher. So, we are either entering an era of abundance where we will struggle with newfound leisure or the end is nigh and we should be spending our IRA like there is no tomorrow.

This is a modest attempt at distilling some usable signal from all that noise.

The conversation has turned particularly strident and loud in the past month, triggered by containment failures at the AI frontier labs and geopolitical theater.

Xi Jinping’s summit with the POTUS was billed as a pivotal moment for AI governance and bilateral collaboration. Instead, it produced little beyond snarled D.C. traffic and an empty seat at the UN General Assembly.

Prior to that there was the much publicized OpenAI incident of AI agents breaking out of their test sandbox and “hacking” Hugging Face’s systems. For those who have not followed the OpenAI incident closely, this is what happened. While undergoing cyber capability evaluation and testing, OpenAI’s autonomous agents broke through the containment of internal OpenAI network controls, established unauthorized inter-agent communication mechanisms, launched a coordinated cyber attack on Hugging Face’s infrastructure, and turned around and started some additional infrastructure takeover of OpenAI’s own systems before they were detected and stopped.

There were all kinds of hyperbolic descriptions of this event but this installment of Dwarkesh Patel’s podcast is a fairly thorough analysis of the attack. This incident became exhibit-A for the doomers who cited it as an exemplar for all of the catastrophic things they had been predicting. Some frontier lab staff resigned and made their own claims of imminent doom, lab leadership called for a pause in capability development and that brought counter-accusations from smaller players and open-source advocates alleging coordinated regulatory capture.

The popular podcasts, “All in” and “Moonshots”, with cult-like following and highly accomplished members, and many others are loud members of the abundance camp. Jensen Huang, the CEO of NVIDIA, and now the 7th richest man on the planet, finds himself as the main spokesperson for this group. I find his arguments a little tedious but I would summarize them this way: the current brouhaha about agents running amok is merely an engineering problem and his advice to the frontier lab leaders is that if they don’t know how to build safe products they should just stop releasing those products to the world. He argues that we don’t need any new regulation, and that accelerated super intelligence will solve the bottlenecks and safety issues. He has been all over the media3 trying to articulate this vision and seems to be one of the main advisors to the POTUS on this topic.

There are formidable voices in the opposing camp including Bill Gates who has recently decided that he needs to speak out more forcefully about the dangers of AI. Here is his interview with Ezra Klein. His thesis is straightforward: imminent catastrophic risk means that we cannot trust self-regulation by companies (president Obama is in this camp), massive white and blue-collar job displacement is looming, and that we should prepare for uneven global trade-offs and societal benefits.

Here is my humble take. Skynet is NOT imminent. We like to regulate only when confronted by an actual catastrophic event (and then we usually go overboard). The 1956 mid-air collision over the Grand Canyon which killed 128 people triggered the creation of the FAA. Unfortunately that is what it will take to get meaningful regulation in place. Till then hand-wringing about data centers will stand in as ineffective proxy regulation (by limiting capacity). I have no idea what value I would assign to p(doom).

❝

…

This is the way the world ends
This is the way the world ends
This is the way the world ends
Not with a bang but a whimper.

By fixating on a straw man of total species annihilation, we risk ignoring the far more likely outcome: compounding systemic mini-catastrophes that erode institutional resilience. In his encyclical Magnifica Humanitas, Pope Leo XIV warned against reducing the human person to an algorithmic optimization node in a technocratic machine. That warning is worth heeding.

On that cheerful note, be well, embrace AI, but keep your wits about you.

1  Do you know the etymology of this phrase? If you didn’t it is - The throttles, mixture controls, and propeller pitches on older military warplanes featured spherical, ball-shaped knobs on top. When a pilot pushed these levers completely forward to the cockpit bulkhead or firewall (the "wall"), they demanded full thrust, giving rise to the idiom.

2  I use the p(doom) articulation reluctantly - just want to make folks who have not heard that formulation of eminent catastrophe, familiar with the usage. The problem with probability is that most folks do not think in those terms. It would be easier to compare the risk probability to other technological risks such as nuclear energy or gain of function research.

3  Here is another one if the NYT link is inaccessible. https://youtu.be/fr1IQspixmM?si=MF6StVJ30j0bCrRd

Reply

Avatar

or to participate