I participated, as a superforecaster, in the Forecasting Research Institute (FRI) Forecasting the Economic Effects of AI survey. They’ve published their results in this 224 page paper.
My prior experience in the Existential Risk Persuasion Tournament led me to expect that the average participant would predict less AI impact than I predicted, but I was still shocked by the extent of the disagreement.
Continue ReadingWhat a weekend. Two new wars in Asia don’t qualify as top news.
My first reaction to Hegseth’s conflict with Anthropic was along the lines of: I expected an attempt at quasi-nationalization of AI, but not this soon. And I expected it to look like it was managed by national security professionals. Hegseth doesn’t look like he’s trying to avoid the role of cartoon villain.
On closer inspection, it doesn’t look very much like nationalization. A significant part of what’s going on is bribery. OpenAI’s president donated $25 million to a Trump PAC. Dario supported Harris in 2024, and hasn’t shown signs of shifting his support. The speed with which the Department of War started negotiating with OpenAI suggests that rewarding OpenAI was one of their motivations. If Hegseth wanted to avoid the appearance of corruption, he’d have waited a bit, and pretended to shop around. But bribery seems to be currently legal, and advertising the benefits is likely to be good for business.
On the other hand, his attempts to look like he’s punishing Anthropic look sufficiently clumsy that I’m confused as to whether he wants them to be effective. He advertised Anthropic as both having the best AI and as having the most integrity. I’m pretty sure that’s good for Anthropic’s business.
The breadth of Hegseth’s proposed supply chain risk order is well in excess of what he can plausibly enforce. Polymarket predicts almost no net harm to Anthropic. I’m confused as to what Hegseth expects, and what will happen when his expectations bump up against reality.
Continue ReadingTL;DR: Anthropic has made important progress at setting good goals for AIs. More work is still needed.
Anthropic has introduced a constitution that has a modest chance of becoming as important as the US constitution (Summary and discussion here).
It’s a large improvement over how AI companies were training ethics into AIs a few years ago. It feels like Anthropic has switched from treating Claude like a child to treating it as an adult.
The constitution looks good for AIs of 2026, so I will focus here on longer-term concerns.
Continue ReadingEpistemic status: speculation with a mix of medium confidence and low confidence conclusions.
I argue that corrigibility is all we need in order to make an AI permanently aligned to a principal.
Continue ReadingBook review: The Thinking Machine: Jensen Huang, Nvidia, and the World’s Most Coveted Microchip, by Stephen Witt.
This is a well-written book about the rise of deep learning, and the man who is the most responsible for building the hardware that it needs.
Continue ReadingI’m analyzing what happens to the US economy in the short-term aftermath of the typical job being replaced by AIs and robots. Will there be a financial crisis? Short answer: yes.
This is partly inspired by my dissatisfaction with Tomas Pueyo’s analysis in If I Were King, How Would I Prepare for AI?.
Let’s say 50% of workers lose their jobs at the same time (around 2030), and they’re expected to be permanently unemployed. (I know this isn’t fully realistic. I’m starting with simple models and will add more realism later.)
Continue ReadingThere’s a stock market bubble currently in progress. The key question is: which stocks are unsustainably high?
Continue ReadingWho benefits if the US develops artificial superintelligence (ASI) faster than China?
One possible answer is that AI kills us all regardless of which country develops it first. People who base their policy on that concern already agree with the conclusions of this post, so I won’t focus on that concern here.
This post aims to convince other people, especially people who focus on democracy versus authoritarianism, to be less concerned about which country develops ASI first. I will assume that AIs will be fully aligned with at least one human, and that the effects of AI will be roughly as important as the industrial revolution, or a bit more important.
Continue ReadingBook review: Red Heart, by Max Harms.
Red Heart resembles in important ways some of the early James Bond movies, but it’s more intellectually sophisticated than that.
It’s both more interesting and more realistic than Crystal Society (the only prior book of Harms’ that I’ve read). It pays careful attention to issues involving AI that are likely to affect the world soon, but mostly prioritizes a good story over serious analysis.
I was expecting to think of Red Heart as science fiction. It turned out to be borderline between science fiction and historical fiction. It’s set in an alternate timeline, but with only small changes from what the world looks like in 2025. The publicly available AIs are probably almost the same as what we’re using today. So it’s hard to tell whether there’s anything meaningfully fictional about this world.
Continue ReadingThis is a continuation of my review of IABIED. It’s intended for audiences who already know a lot about AI risk debates. Please at least glance at my main layman-oriented review before reading this.
Eliezer and Nate used to argue about AI risk using a paradigm that involved a pretty sudden foom, and which viewed values through a utility function lens. I’ll call that the MIRI paradigm (note: I don’t have a comprehensive description of the paradigm). In IABIED, they’ve tried to adopt a much broader paradigm that’s somewhat closer to that of more mainstream AI researchers. Yet they keep sounding to me like they’re still thinking within the MIRI paradigm.
Continue Reading