All posts by Peter

Eric Drexler has published a book-length paper on AI risk, describing an approach that he calls Comprehensive AI Services (CAIS).

His primary goal seems to be reframing AI risk discussions to use a rather different paradigm than the one that Nick Bostrom and Eliezer Yudkowsky have been promoting. (There isn’t yet any paradigm that’s widely accepted, so this isn’t a Kuhnian paradigm shift; it’s better characterized as an amorphous field that is struggling to establish its first paradigm). Dueling paradigms seems to be the best that the AI safety field can manage to achieve for now.

I’ll start by mentioning some important claims that Drexler doesn’t dispute:

  • an intelligence explosion might happen somewhat suddenly, in the fairly near future;
  • it’s hard to reliably align an AI’s values with human values;
  • recursive self-improvement, as imagined by Bostrom / Yudkowsky, would pose significant dangers.

Drexler likely disagrees about some of the claims made by Bostrom / Yudkowsky on those points, but he shares enough of their concerns about them that those disagreements don’t explain why Drexler approaches AI safety differently. (Drexler is more cautious than most writers about making any predictions concerning these three claims).

CAIS isn’t a full solution to AI risks. Instead, it’s better thought of as an attempt to reduce the risk of world conquest by the first AGI that reaches some threshold, preserve existing corrigibility somewhat past human-level AI, and postpone need for a permanent solution until we have more intelligence.

Continue Reading

The point of this blog post feels almost too obvious to be worth saying, yet I doubt that it’s widely followed.

People often avoid doing projects that have a low probability of success, even when the expected value is high. To counter this bias, I recommend that you mentally combine many such projects into a strategy of trying new things, and evaluate the strategy’s probability of success.


Eliezer says in On Doing the Improbable:

I’ve noticed that, by my standards and on an Eliezeromorphic metric, most people seem to require catastrophically high levels of faith in what they’re doing in order to stick to it. By this I mean that they would not have stuck to writing the Sequences or HPMOR or working on AGI alignment past the first few months of real difficulty, without assigning odds in the vicinity of 10x what I started out assigning that the project would work. … But you can’t get numbers in the range of what I estimate to be something like 70% as the required threshold before people will carry on through bad times. “It might not work” is enough to force them to make a great effort to continue past that 30% failure probability. It’s not good decision theory but it seems to be how people actually work on group projects where they are not personally madly driven to accomplish the thing.

I expect this reluctance to work on projects with a large chance of failure is a widespread problem for individual self-improvement experiments.


One piece of advice I got from my CFAR workshop was to try lots of things. Their reasoning involved the expectation that we’d repeat the things that worked, and forget the things that didn’t work.

I’ve been hesitant to apply this advice to things that feel unlikely to work, and I expect other people have similar reluctance.

The relevant kind of “things” are experiments that cost maybe 10 to 100 hours to try, which don’t risk much other than wasting time, and for which I should expect on the order of a 10% chance of noticeable long-term benefits.

Here are some examples of the kind of experiments I have in mind:

  • gratitude journal
  • morning pages
  • meditation
  • vitamin D supplements
  • folate supplements
  • a low carb diet
  • the Plant Paradox diet
  • an anti-anxiety drug
  • ashwaghanda
  • whole fruit coffee extract
  • piracetam
  • phenibut
  • modafinil
  • a circling workshop
  • Auditory Integration Training
  • various self-help books
  • yoga
  • sensory deprivation chamber

I’ve cheated slightly, by being more likely to add something to this list if it worked for me than if it was a failure that I’d rather forget. So my success rate with these was around 50%.

The simple practice of forgetting about the failures and mostly repeating the successes is almost enough to cause the net value of these experiments to be positive. More importantly, I kept the costs of these experiments low, so the benefits of the top few outweighed the costs of the failures by a large factor.


I face a similar situation when I’m investing.

The probability that I’ll make any profit on a given investment is close to 50%, and the probability of beating the market on a given investment is lower. I don’t calculate actual numbers for that, because doing so would be more likely to bias me than to help me.

I would find it rather discouraging to evaluate each investment separately. Doing so would focus my attention on the fact that any individual result is indistinguishable from luck.

Instead, I focus my evaluations much more on bundles of hundreds of trades, often associated with a particular strategy. Aggregating evidence in that manner smooths out the good and bad luck to make my skill (or lack thereof) more conspicuous. I’m focusing in this post not on the logical interpretation of evidence, but on how the subconscious parts of my mind react. This mental bundling of tasks is particularly important for my subconscious impressions of whether I’m being productive.

I believe this is a well-known insight (possibly from poker?), but I can’t figure out where I’ve seen it described.

I’ve partly applied this approach to self-improvement tasks (not quite as explicitly as I ought to), and it has probably helped.

Book review: Time Biases: A Theory of Rational Planning and Personal Persistence, by Meghan Sullivan.

I was very unsure about whether this book would be worth reading, as it could easily have been focused on complaints about behavior that experts have long known are mistaken.

I was pleasantly surprised when it quickly got to some of the really hard questions, and was thoughtful about what questions deserved attention. I disagree with enough of Sullivan’s premises that I have significant disagreements with her conclusions. Yet her reasoning is usually good enough that I’m unsure what to make of our disagreements – they’re typically due to differences of intuition that she admits are controversial.

I had hoped for some discussion of ethics (e.g. what discount rate to use in evaluating climate change), whereas the book focuses purely on prudential rationality (i.e. what’s rational for a self-interested person). Still, the discussion of prudential rationality covers most of the issues that make the ethical choices hard.

Personal identity

A key issue is the nature of personal identity – does one’s identity change over time?

Continue Reading

Descriptions of AI-relevant ontological crises typically choose examples where it seems moderately obvious how humans would want to resolve the crises. I describe here a scenario where I don’t know how I would want to resolve the crisis.

I will incidentally ridicule express distate for some philosophical beliefs.

Suppose a powerful AI is programmed to have an ethical system with a version of the person-affecting view. A version which says only persons who exist are morally relevant, and “exist” only refers to the present time. [Note that the most sophisticated advocates of the person-affecting view are willing to treat future people as real, and only object to comparing those people to other possible futures where those people don’t exist.]

Suppose also that it is programmed by someone who thinks in Newtonian models. Then something happens which prevents the programmer from correcting any flaws in the AI. (For simplicity, I’ll say programmer dies, and the AI was programmed to only accept changes to its ethical system from the programmer).

What happens when the AI tries to make ethical decisions about people in distant galaxies (hereinafter “distant people”) using a model of the universe that works like relativity?

Continue Reading

I wrote this post to try to clarify my thoughts about donating to the Longevity Research Institute (LRI).

Much of that thought involves asking: is there a better approach to cures for aging? Will a better aging-related charity be created soon?

I started to turn this post into an overview of all approaches to curing aging, but I saw that would sidetrack me into doing too much research, so I’ve ended up skimping on some.

I’ve ordered the various approaches that I mention from most directly focused on the underlying causes of aging, to most focused on mitigating the symptoms.

I’ve been less careful than usual to distinguish my intuitions from solid research. I’m mainly trying here to summarize lots of information that I’ve accumulated over the years, and I’m not trying to do new research.
Continue Reading

Book review: Artificial Intelligence Safety and Security, by Roman V. Yampolskiy.

This is a collection of papers, with highly varying topics, quality, and importance.

Many of the papers focus on risks that are specific to superintelligence, some assuming that a single AI will take over the world, and some assuming that there will be many AIs of roughly equal power. Others focus on problems that are associated with current AI programs.

I’ve tried to arrange my comments on individual papers in roughly descending order of how important the papers look for addressing the largest AI-related risks, while also sometimes putting similar topics in one group. The result feels a little more organized than the book, but I worry that the papers are too dissimilar to be usefully grouped. I’ve ignored some of the less important papers.

The book’s attempt at organizing the papers consists of dividing them into “Concerns of Luminaries” and “Responses of Scholars”. Alas, I see few signs that many of the authors are even aware of what the other authors have written, much less that the later papers are attempts at responding to the earlier papers. It looks like the papers are mainly arranged in order of when they were written. There’s a modest cluster of authors who agree enough with Bostrom to constitute a single scientific paradigm, but half the papers demonstrate about as much of a consensus on what topic they’re discussing as I would expect to get from asking medieval peasants about airplane safety.

Continue Reading

I tried the Plant Paradox diet, and didn’t notice any significant effects on my health. I was mainly hoping for improvements in my homocysteine and LDL-P, and I saw little change there.

My thyroid levels declined during that time – I expected some decline due to reasons relating to the inflammation which increased my thyroid levels earlier in the year, but the actual decline was larger than I expected. Most likely that was unrelated to the diet, and may have confounded my attempt to evaluate the diet.

Robert’s walnut allergy seemed to be eliminated or substantially reduced (although he doesn’t test that often enough to provide strong evidence that the Plant Paradox diet was what made the difference), and maybe some improvement in his acne (but probably less so than with the SCD diet). He experienced some discomfort when he resumed eating commercial burritos (due to gluten?), but less than after quitting the SCD diet.

I have concluded that this diet has small benefits compared to other reasonable attempts at a paleo diet, and those benefits vary a good deal from person to person. Since it’s a bit less convenient than a typical paleo diet, I’ve abandoned it.

Book(?) review: The Great Stagnation: How America Ate All The Low-Hanging Fruit of Modern History, Got Sick, and Will (Eventually) Feel Better, by Tyler Cowen.

Tyler Cowen wrote what looks like a couple of blog posts, and published them in book form.

The problem: US economic growth slowed in the early 1970s, and hasn’t recovered much. Median family income would be 50% higher if the growth of 1945-1970 had continued.

Continue Reading

Book review: Where Is My Flying Car? A Memoir of Future Past, by J. Storrs Hall (aka Josh).

If you only read the first 3 chapters, you might imagine that this is the history of just one industry (or the mysterious lack of an industry).

But this book attributes the absence of that industry to a broad set of problems that are keeping us poor. He looks at the post-1970 slowdown in innovation that Cowen describes in The Great Stagnation[1]. The two books agree on many symptoms, but describe the causes differently: where Cowen says we ate the low hanging fruit, Josh says it’s due to someone “spraying paraquat on the low-hanging fruit”.

The book is full of mostly good insights. It significantly changed my opinion of the Great Stagnation.

The book jumps back and forth between polemics about the Great Strangulation (with a bit too much outrage porn), and nerdy descriptions of engineering and piloting problems. I found those large shifts in tone to be somewhat disorienting – it’s like the author can’t decide whether he’s an autistic youth who is eagerly describing his latest obsession, or an angry old man complaining about how the world is going to hell (I’ve met the author at Foresight conferences, and got similar but milder impressions there).

Josh’s main explanation for the Great Strangulation is the rise of Green fundamentalism[2], but he also describes other cultural / political factors that seem related. But before looking at those, I’ll look in some depth at three industries that exemplify the Great Strangulation.

Continue Reading

[Warning: this post contains lots of guesses based on weak evidence. I’d be surprised if I got more than 80% of it right.]

I’ve long acted as if a good diet is fairly important, and I’ve gathered lots of relevant evidence. But until recently I classified that evidence into many small topics related to specific nutrients and health problems, and never organized those ideas into an overall assessment of how important a good diet is.

Comments by Jim Babcock prompted me to investigate a broader overview.

This post will mainly focus on evaluating the importance of nutrition for adults in wealthy nations, then will summarize my guesses about how to achieve a good diet.

Continue Reading