这是用户在 2024-11-27 13:58 为 https://darioamodei.com/machines-of-loving-grace#basic-assumptions-and-framework 保存的双语快照页面，由 沉浸式翻译 提供双语支持。了解如何保存？

Dario Amodei 达里奥·阿莫迪埃

Machines of
Loving Grace1
Machines of Loving Grace

How AI Could Transform the World for the Better
人工智能如何让世界变得更好

I think and talk a lot about the risks of powerful AI. The company I’m the CEO of, Anthropic, does a lot of research on how to reduce these risks. Because of this, people sometimes draw the conclusion that I’m a pessimist or “doomer” who thinks AI will be mostly bad or dangerous. I don’t think that at all. In fact, one of my main reasons for focusing on risks is that they’re the only thing standing between us and what I see as a fundamentally positive future. I think that most people are underestimating just how radical the upside of AI could be, just as I think most people are underestimating how bad the risks could be.
我认为自己谈论了很多关于强大人工智能的风险。我担任首席执行官的公司 Anthropic 在如何降低这些风险方面做了很多研究。正因为如此，人们有时会得出结论，认为我是一个悲观主义者或“末日论者”，认为人工智能将主要是不好的或危险的。我根本不这么认为。事实上，我专注于风险的主要原因之一是它们是我们和我认为的根本积极未来之间的唯一障碍。我认为大多数人低估了人工智能积极面的激进程度，就像我认为大多数人低估了风险的严重性一样。

In this essay I try to sketch out what that upside might look like—what a world with powerful AI might look like if everything goes right. Of course no one can know the future with any certainty or precision, and the effects of powerful AI are likely to be even more unpredictable than past technological changes, so all of this is unavoidably going to consist of guesses. But I am aiming for at least educated and useful guesses, which capture the flavor of what will happen even if most details end up being wrong. I’m including lots of details mainly because I think a concrete vision does more to advance discussion than a highly hedged and abstract one.
在这篇文章中，我试图勾勒出这种可能性可能是什么样子——如果一切顺利，一个拥有强大人工智能的世界可能是什么样子。当然，没有人能够以任何确定性或精确性预知未来，强大人工智能的影响可能比过去的技术变革更加不可预测，因此所有这些都不可避免地会包含猜测。但我至少希望这些猜测是经过教育和有用的，即使大多数细节最终是错误的，也能捕捉到将要发生的事情的精髓。我之所以包含许多细节，主要是因为我认为一个具体的愿景比一个高度谨慎和抽象的愿景更有助于推进讨论。

First, however, I wanted to briefly explain why I and Anthropic haven’t talked that much about powerful AI’s upsides, and why we’ll probably continue, overall, to talk a lot about risks. In particular, I’ve made this choice out of a desire to:
首先，然而，我想简要解释为什么我和 Anthropic 没有过多谈论强大人工智能的积极面，以及为什么我们可能总体上会继续大量讨论风险。特别是，我做出这个选择是出于以下愿望：

Yet despite all of the concerns above, I really do think it’s important to discuss what a good world with powerful AI could look like, while doing our best to avoid the above pitfalls. In fact I think it is critical to have a genuinely inspiring vision of the future, and not just a plan to fight fires. Many of the implications of powerful AI are adversarial or dangerous, but at the end of it all, there has to be something we’re fighting for, some positive-sum outcome where everyone is better off, something to rally people to rise above their squabbles and confront the challenges ahead. Fear is one kind of motivator, but it’s not enough: we need hope as well.
尽管上述担忧重重，但我确实认为讨论一个强大 AI 可能带来的美好世界至关重要，同时我们应尽力避免上述陷阱。事实上，我认为拥有一个真正鼓舞人心的未来愿景至关重要，而不仅仅是应对火灾的计划。强大 AI 的许多影响都是对抗性的或危险的，但最终，我们必须有所追求，必须有一个正和的结果，让每个人都过得更好，让人们团结起来，超越他们的争执，面对未来的挑战。恐惧是一种动力，但不足以支撑：我们还需要希望。

The list of positive applications of powerful AI is extremely long (and includes robotics, manufacturing, energy, and much more), but I’m going to focus on a small number of areas that seem to me to have the greatest potential to directly improve the quality of human life. The five categories I am most excited about are:
人工智能的积极应用列表非常长（包括机器人技术、制造业、能源等等），但我将重点关注我认为最有潜力直接改善人类生活质量的少数几个领域。我最兴奋的五个类别是：

  1. Biology and physical health
    生物学与身体健康
  2. Neuroscience and mental health
    神经科学和心理健康
  3. Economic development and poverty
    经济发展与贫困
  4. Peace and governance 和平与治理
  5. Work and meaning 工作和意义

My predictions are going to be radical as judged by most standards (other than sci-fi “singularity” visions2), but I mean them earnestly and sincerely. Everything I’m saying could very easily be wrong (to repeat my point from above), but I’ve at least attempted to ground my views in a semi-analytical assessment of how much progress in various fields might speed up and what that might mean in practice. I am fortunate to have professional experience in both biology and neuroscience, and I am an informed amateur in the field of economic development, but I am sure I will get plenty of things wrong. One thing writing this essay has made me realize is that it would be valuable to bring together a group of domain experts (in biology, economics, international relations, and other areas) to write a much better and more informed version of what I’ve produced here. It’s probably best to view my efforts here as a starting prompt for that group.
我的预测将非常激进，按照大多数标准（除了科幻“奇点”愿景之外），但我真诚地看待它们。我所说的每一件事都很容易出错（重复我上面的观点），但我至少尝试将我的观点建立在半分析性的评估上，即各个领域的进步可能会加快，这在实践中可能意味着什么。我很幸运在生物学和神经科学领域都有专业经验，在经济发展的领域我是一个有知识的业余爱好者，但我确信我会犯很多错误。写这篇论文让我意识到，将一群领域专家（在生物学、经济学、国际关系和其他领域）聚集在一起，写出一个更好、更全面的作品是非常有价值的。也许最好将我的努力视为那个小组的起点。

Basic assumptions and framework
基本假设和框架

To make this whole essay more precise and grounded, it’s helpful to specify clearly what we mean by powerful AI (i.e. the threshold at which the 5-10 year clock starts counting), as well as laying out a framework for thinking about the effects of such AI once it’s present.
为了使这篇论文更加精确和有根据，明确我们所说的“强大的人工智能”（即 5-10 年时钟开始计时的阈值）是很有帮助的，同时还要制定一个框架来思考这种人工智能一旦出现后的影响。

What powerful AI (I dislike the term AGI)3 will look like, and when (or if) it will arrive, is a huge topic in itself. It’s one I’ve discussed publicly and could write a completely separate essay on (I probably will at some point). Obviously, many people are skeptical that powerful AI will be built soon and some are skeptical that it will ever be built at all. I think it could come as early as 2026, though there are also ways it could take much longer. But for the purposes of this essay, I’d like to put these issues aside, assume it will come reasonably soon, and focus on what happens in the 5-10 years after that. I also want to assume a definition of what such a system will look like, what its capabilities are and how it interacts, even though there is room for disagreement on this.
人工智能（我不喜欢“通用人工智能”这个术语） 3 将是什么样子，以及它何时（或是否）会出现，本身就是一个巨大的话题。这是我已经公开讨论过的话题，也可以写一篇完全独立的文章（我可能在某个时候会这么做）。显然，许多人怀疑强大的人工智能将很快被构建，有些人甚至怀疑它是否会被构建。我认为它可能最早在 2026 年出现，尽管它也可能需要更长的时间。但为了这篇论文的目的，我想把这些问题放在一边，假设它将相对较快地到来，并专注于在那之后的 5-10 年内会发生什么。我还想假设一个关于这样一个系统将是什么样子，它的能力是什么以及它是如何互动的定义，尽管在这方面存在分歧的空间。

By powerful AI, I have in mind an AI model—likely similar to today’s LLM’s in form, though it might be based on a different architecture, might involve several interacting models, and might be trained differently—with the following properties:
通过强大的 AI，我心中有一个 AI 模型——可能在形式上类似于今天的LLM，尽管它可能基于不同的架构，可能涉及几个相互作用的模型，并且可能以不同的方式训练——具有以下特性：

We could summarize this as a “country of geniuses in a datacenter”.

Clearly such an entity would be capable of solving very difficult problems, very fast, but it is not trivial to figure out how fast. Two “extreme” positions both seem false to me. First, you might think that the world would be instantly transformed on the scale of seconds or days (“the Singularity”), as superior intelligence builds on itself and solves every possible scientific, engineering, and operational task almost immediately. The problem with this is that there are real physical and practical limits, for example around building hardware or conducting biological experiments. Even a new country of geniuses would hit up against these limits. Intelligence may be very powerful, but it isn’t magic fairy dust.

Second, and conversely, you might believe that technological progress is saturated or rate-limited by real world data or by social factors, and that better-than-human intelligence will add very little6. This seems equally implausible to me—I can think of hundreds of scientific or even social problems where a large group of really smart people would drastically speed up progress, especially if they aren’t limited to analysis and can make things happen in the real world (which our postulated country of geniuses can, including by directing or assisting teams of humans).

I think the truth is likely to be some messy admixture of these two extreme pictures, something that varies by task and field and is very subtle in its details. I believe we need new frameworks to think about these details in a productive way.

Economists often talk about “factors of production”: things like labor, land, and capital. The phrase “marginal returns to labor/land/capital” captures the idea that in a given situation, a given factor may or may not be the limiting one – for example, an air force needs both planes and pilots, and hiring more pilots doesn’t help much if you’re out of planes. I believe that in the AI age, we should be talking about the marginal returns to intelligence7, and trying to figure out what the other factors are that are complementary to intelligence and that become limiting factors when intelligence is very high. We are not used to thinking in this way—to asking “how much does being smarter help with this task, and on what timescale?”—but it seems like the right way to conceptualize a world with very powerful AI.

My guess at a list of factors that limit or are complementary to intelligence includes:

There is a further distinction based on timescales. Things that are hard constraints in the short run may become more malleable to intelligence in the long run. For example, intelligence might be used to develop a new experimental paradigm that allows us to learn in vitro what used to require live animal experiments, or to build the tools needed to collect new data (e.g. the bigger particle accelerator), or to (within ethical limits) find ways around human-based constraints (e.g. helping to improve the clinical trial system, helping to create new jurisdictions where clinical trials have less bureaucracy, or improving the science itself to make human clinical trials less necessary or cheaper).

Thus, we should imagine a picture where intelligence is initially heavily bottlenecked by the other factors of production, but over time intelligence itself increasingly routes around the other factors, even if they never fully dissolve (and some things like physical laws are absolute)10. The key question is how fast it all happens and in what order.

With the above framework in mind, I’ll try to answer that question for the five areas mentioned in the introduction.

1. Biology and health

Biology is probably the area where scientific progress has the greatest potential to directly and unambiguously improve the quality of human life. In the last century some of the most ancient human afflictions (such as smallpox) have finally been vanquished, but many more still remain, and defeating them would be an enormous humanitarian accomplishment. Beyond even curing disease, biological science can in principle improve the baseline quality of human health, by extending the healthy human lifespan, increasing control and freedom over our own biological processes, and addressing everyday problems that we currently think of as immutable parts of the human condition.

In the “limiting factors” language of the previous section, the main challenges with directly applying intelligence to biology are data, the speed of the physical world, and intrinsic complexity (in fact, all three are related to each other). Human constraints also play a role at a later stage, when clinical trials are involved. Let’s take these one by one.

Experiments on cells, animals, and even chemical processes are limited by the speed of the physical world: many biological protocols involve culturing bacteria or other cells, or simply waiting for chemical reactions to occur, and this can sometimes take days or even weeks, with no obvious way to speed it up. Animal experiments can take months (or more) and human experiments often take years (or even decades for long-term outcome studies). Somewhat related to this, data is often lacking—not so much in quantity, but quality: there is always a dearth of clear, unambiguous data that isolates a biological effect of interest from the other 10,000 confounding things that are going on, or that intervenes causally in a given process, or that directly measures some effect (as opposed to inferring its consequences in some indirect or noisy way). Even massive, quantitative molecular data, like the proteomics data that I collected while working on mass spectrometry techniques, is noisy and misses a lot (which types of cells were these proteins in? Which part of the cell? At what phase in the cell cycle?).

In part responsible for these problems with data is intrinsic complexity: if you’ve ever seen a diagram showing the biochemistry of human metabolism, you’ll know that it’s very hard to isolate the effect of any part of this complex system, and even harder to intervene on the system in a precise or predictable way. And finally, beyond just the intrinsic time that it takes to run an experiment on humans, actual clinical trials involve a lot of bureaucracy and regulatory requirements that (in the opinion of many people, including me) add unnecessary additional time and delay progress.

Given all this, many biologists have long been skeptical of the value of AI and “big data” more generally in biology. Historically, mathematicians, computer scientists, and physicists who have applied their skills to biology over the last 30 years have been quite successful, but have not had the truly transformative impact initially hoped for. Some of the skepticism has been reduced by major and revolutionary breakthroughs like AlphaFold (which has just deservedly won its creators the Nobel Prize in Chemistry) and AlphaProteo11, but there’s still a perception that AI is (and will continue to be) useful in only a limited set of circumstances. A common formulation is “AI can do a better job analyzing your data, but it can’t produce more data or improve the quality of the data. Garbage in, garbage out”.

But I think that pessimistic perspective is thinking about AI in the wrong way. If our core hypothesis about AI progress is correct, then the right way to think of AI is not as a method of data analysis, but as a virtual biologist who performs all the tasks biologists do, including designing and running experiments in the real world (by controlling lab robots or simply telling humans which experiments to run – as a Principal Investigator would to their graduate students), inventing new biological methods or measurement techniques, and so on. It is by speeding up the whole research process that AI can truly accelerate biology. I want to repeat this because it’s the most common misconception that comes up when I talk about AI’s ability to transform biology: I am not talking about AI as merely a tool to analyze data. In line with the definition of powerful AI at the beginning of this essay, I’m talking about using AI to perform, direct, and improve upon nearly everything biologists do.

To get more specific on where I think acceleration is likely to come from, a surprisingly large fraction of the progress in biology has come from a truly tiny number of discoveries, often related to broad measurement tools or techniques12 that allow precise but generalized or programmable intervention in biological systems. There’s perhaps ~1 of these major discoveries per year and collectively they arguably drive >50% of progress in biology. These discoveries are so powerful precisely because they cut through intrinsic complexity and data limitations, directly increasing our understanding and control over biological processes. A few discoveries per decade have enabled both the bulk of our basic scientific understanding of biology, and have driven many of the most powerful medical treatments.

Some examples include:

I’m going to the trouble of listing all these technologies because I want to make a crucial claim about them: I think their rate of discovery could be increased by 10x or more if there were a lot more talented, creative researchers. Or, put another way, I think the returns to intelligence are high for these discoveries, and that everything else in biology and medicine mostly follows from them.

Why do I think this? Because of the answers to some questions that we should get in the habit of asking when we’re trying to determine “returns to intelligence”. First, these discoveries are generally made by a tiny number of researchers, often the same people repeatedly, suggesting skill and not random search (the latter might suggest lengthy experiments are the limiting factor). Second, they often “could have been made” years earlier than they were: for example, CRISPR was a naturally occurring component of the immune system in bacteria that’s been known since the 80’s, but it took another 25 years for people to realize it could be repurposed for general gene editing. They also are often delayed many years by lack of support from the scientific community for promising directions (see this profile on the inventor of mRNA vaccines; similar stories abound). Third, successful projects are often scrappy or were afterthoughts that people didn’t initially think were promising, rather than massively funded efforts. This suggests that it’s not just massive resource concentration that drives discoveries, but ingenuity.
为什么我认为是这样？因为当我们试图确定“智力回报”时，我们应该养成询问一些问题的答案。首先，这些发现通常是由少数研究人员做出的，通常是同一些人反复进行，这表明这是技能而非随机搜索（后者可能表明长时间实验是限制因素）。其次，它们通常比实际发生的时间早几年：“CRISPR”是细菌免疫系统中自然存在的成分，自 80 年代以来就已为人所知，但人们花了另外 25 年才意识到它可以被重新用于通用基因编辑。它们还常常因为科学界对有希望的方向缺乏支持而推迟许多年（参见 mRNA 疫苗发明者的简介；类似的故事比比皆是）。第三，成功的项目往往是粗略的或者是被忽视的，人们最初并没有认为它们有前途，而不是大规模资助的努力。这表明，推动发现的不仅仅是资源的大规模集中，还有独创性。

Finally, although some of these discoveries have “serial dependence” (you need to make discovery A first in order to have the tools or knowledge to make discovery B)—which again might create experimental delays—many, perhaps most, are independent, meaning many at once can be worked on in parallel. Both these facts, and my general experience as a biologist, strongly suggest to me that there are hundreds of these discoveries waiting to be made if scientists were smarter and better at making connections between the vast amount of biological knowledge humanity possesses (again consider the CRISPR example). The success of AlphaFold/AlphaProteo at solving important problems much more effectively than humans, despite decades of carefully designed physics modeling, provides a proof of principle (albeit with a narrow tool in a narrow domain) that should point the way forward.
最终，尽管其中一些发现具有“序列依赖性”（你需要先做出发现 A，才能拥有进行发现 B 的工具或知识）——这又可能造成实验延迟——但许多，也许大多数是独立的，这意味着许多可以同时并行工作。这两个事实，以及我作为一名生物学家的总体经验，强烈地暗示我，如果科学家们更聪明、更擅长在人类拥有的庞大生物知识之间建立联系（再次考虑 CRISPR 的例子），那么还有数百个这样的发现等待被发现。AlphaFold/AlphaProteo 在解决重要问题方面比人类更有效，尽管经过了数十年的精心设计的物理建模，这为原理提供了证明（尽管是在狭窄领域使用狭窄的工具），这应该指明前进的方向。

Thus, it’s my guess that powerful AI could at least 10x the rate of these discoveries, giving us the next 50-100 years of biological progress in 5-10 years.14 Why not 100x? Perhaps it is possible, but here both serial dependence and experiment times become important: getting 100 years of progress in 1 year requires a lot of things to go right the first time, including animal experiments and things like designing microscopes or expensive lab facilities. I’m actually open to the (perhaps absurd-sounding) idea that we could get 1000 years of progress in 5-10 years, but very skeptical that we can get 100 years in 1 year. Another way to put it is I think there’s an unavoidable constant delay: experiments and hardware design have a certain “latency” and need to be iterated upon a certain “irreducible” number of times in order to learn things that can’t be deduced logically. But massive parallelism may be possible on top of that15.
因此，我认为强大的 AI 至少可以将这些发现的速率提高 10 倍，将未来 50-100 年的生物进步在 5-10 年内实现。 14 为什么不是 100 倍？也许这是可能的，但在这里，连续依赖和实验时间变得很重要：在 1 年内取得 100 年的进步需要很多事情一开始就要做对，包括动物实验以及设计显微镜或昂贵的实验室设施等。实际上，我对（可能听起来荒谬的）我们能在 5-10 年内取得 1000 年进步的想法持开放态度，但非常怀疑我们能否在 1 年内取得 100 年的进步。另一种说法是，我认为存在一个不可避免的常数延迟：实验和硬件设计有一定的“延迟”，需要迭代一定的“不可减少”次数，才能学习到无法通过逻辑推理得出的事物。但在此之上，可能实现大规模并行化 15 。

What about clinical trials? Although there is a lot of bureaucracy and slowdown associated with them, the truth is that a lot (though by no means all!) of their slowness ultimately derives from the need to rigorously evaluate drugs that barely work or ambiguously work. This is sadly true of most therapies today: the average cancer drug increases survival by a few months while having significant side effects that need to be carefully measured (there’s a similar story for Alzheimer’s drugs). This leads to huge studies (in order to achieve statistical power) and difficult tradeoffs which regulatory agencies generally aren’t great at making, again because of bureaucracy and the complexity of competing interests.
关于临床试验？尽管与它们相关的官僚主义和拖延很多，但事实是，他们的缓慢很大程度上源于对几乎不起作用或作用不明确的药物进行严格评估的需要。这在今天的大多数疗法中都是令人悲哀的事实：平均而言，抗癌药物只能延长几个月的生存期，同时具有需要仔细测量的显著副作用（阿尔茨海默病药物也有类似的故事）。这导致了巨大的研究（为了达到统计效力）和困难的权衡，监管机构通常不擅长做出这些权衡，这再次是因为官僚主义和利益冲突的复杂性。

When something works really well, it goes much faster: there’s an accelerated approval track and the ease of approval is much greater when effect sizes are larger. mRNA vaccines for COVID were approved in 9 months—much faster than the usual pace. That said, even under these conditions clinical trials are still too slow—mRNA vaccines arguably should have been approved in ~2 months. But these kinds of delays (~1 year end-to-end for a drug) combined with massive parallelization and the need for some but not too much iteration (“a few tries”) are very compatible with radical transformation in 5-10 years. Even more optimistically, it is possible that AI-enabled biological science will reduce the need for iteration in clinical trials by developing better animal and cell experimental models (or even simulations) that are more accurate in predicting what will happen in humans. This will be particularly important in developing drugs against the aging process, which plays out over decades and where we need a faster iteration loop.
当某件事做得非常好时，它会变得更快：有加速的审批流程，当效果量更大时，审批的便利性也更大。COVID 的 mRNA 疫苗在 9 个月内获得批准——比通常的进度快得多。尽管如此，即使在这些条件下，临床试验仍然太慢——mRNA 疫苗可能应该在约 2 个月内获得批准。但是，这类延迟（药物从端到端大约 1 年）与 5-10 年内的激进转型非常兼容，结合了大规模并行化和一些但不是太多的迭代（“几次尝试”）。更加乐观的是，AI 赋能的生物科学有可能通过开发更准确的动物和细胞实验模型（甚至模拟）来减少临床试验中的迭代需求，这些模型能更准确地预测人体会发生什么。这将在开发针对衰老过程的药物中尤为重要，衰老过程持续数十年，我们需要更快的迭代循环。

Finally, on the topic of clinical trials and societal barriers, it is worth pointing out explicitly that in some ways biomedical innovations have an unusually strong track record of being successfully deployed, in contrast to some other technologies16. As mentioned in the introduction, many technologies are hampered by societal factors despite working well technically. This might suggest a pessimistic perspective on what AI can accomplish. But biomedicine is unique in that although the process of developing drugs is overly cumbersome, once developed they generally are successfully deployed and used.
最后，关于临床试验和社会障碍的话题，值得明确指出的是，在某些方面，生物医学创新在成功部署方面有着异常强劲的记录，与某些其他技术形成鲜明对比 16 。正如引言中提到的，许多技术虽然在技术上运行良好，但受到社会因素的阻碍。这可能会暗示对人工智能能够实现的事情持悲观态度。但生物医学是独特的，尽管药物开发过程过于繁琐，一旦开发出来，它们通常都能成功部署和使用。

To summarize the above, my basic prediction is that AI-enabled biology and medicine will allow us to compress the progress that human biologists would have achieved over the next 50-100 years into 5-10 years. I’ll refer to this as the “compressed 21st century”: the idea that after powerful AI is developed, we will in a few years make all the progress in biology and medicine that we would have made in the whole 21st century.
总结上述内容，我的基本预测是，AI 赋能的生物学和医学将使我们能够在 5-10 年内压缩人类生物学家在接下来的 50-100 年内可能取得的进步。我将把这称为“压缩的 21 世纪”：即在强大的 AI 开发出来后，我们将在几年内取得整个 21 世纪在生物学和医学上可能取得的进步。

Although predicting what powerful AI can do in a few years remains inherently difficult and speculative, there is some concreteness to asking “what could humans do unaided in the next 100 years?”. Simply looking at what we’ve accomplished in the 20th century, or extrapolating from the first 2 decades of the 21st, or asking what “10 CRISPR’s and 50 CAR-T’s” would get us, all offer practical, grounded ways to estimate the general level of progress we might expect from powerful AI.
尽管预测几年后强大的人工智能能做什么本质上仍然很困难且具有推测性，但询问“在未来 100 年里，人类在没有帮助的情况下能做什么？”这个问题还是有一些具体性的。仅仅看看我们在 20 世纪取得的成就，或者从 21 世纪的前 20 年进行外推，或者询问“10 个 CRISPR 和 50 个 CAR-T”能带给我们什么，所有这些都能提供实际、接地气的方式来估计我们可能从强大的人工智能中期望的一般进步水平。

Below I try to make a list of what we might expect. This is not based on any rigorous methodology, and will almost certainly prove wrong in the details, but it’s trying to get across the general level of radicalism we should expect:
以下我试图列出我们可能期望的内容。这并非基于任何严谨的方法，几乎肯定在细节上会证明是错误的，但它试图传达我们应该期望的激进主义的一般水平：

It is worth looking at this list and reflecting on how different the world will be if all of it is achieved 7-12 years from now (which would be in line with an aggressive AI timeline). It goes without saying that it would be an unimaginable humanitarian triumph, the elimination all at once of most of the scourges that have haunted humanity for millennia. Many of my friends and colleagues are raising children, and when those children grow up, I hope that any mention of disease will sound to them the way scurvy, smallpox, or bubonic plague sounds to us. That generation will also benefit from increased biological freedom and self-expression, and with luck may also be able to live as long as they want.
这份清单值得一看，并思考如果这一切在 7-12 年后实现，世界将会有多么不同（这将符合激进的人工智能时间表）。不用说，这将是一场难以想象的 humanitarian triumph，一次性消除困扰人类数千年的大多数灾难。我的许多朋友和同事都在抚养孩子，当这些孩子长大后，我希望他们提到疾病的方式就像我们提到坏血病、天花或黑死病一样。那一代人也将从增加的生物自由和自我表达中受益，并且幸运的话，他们也可能活到他们想要的年龄。

It’s hard to overestimate how surprising these changes will be to everyone except the small community of people who expected powerful AI. For example, thousands of economists and policy experts in the US currently debate how to keep Social Security and Medicare solvent, and more broadly how to keep down the cost of healthcare (which is mostly consumed by those over 70 and especially those with terminal illnesses such as cancer). The situation for these programs is likely to be radically improved if all this comes to pass20, as the ratio of working age to retired population will change drastically. No doubt these challenges will be replaced with others, such as how to ensure widespread access to the new technologies, but it is worth reflecting on how much the world will change even if biology is the only area to be successfully accelerated by AI.
很难高估这些变化对除了那些期待强大人工智能的小众群体之外的所有人来说会有多令人惊讶。例如，目前美国有成千上万的经济学家和政策专家正在争论如何保持社会保障和医疗保险的偿付能力，以及更广泛地如何降低医疗保健成本（这些成本主要是由 70 岁以上的人以及患有癌症等绝症的人承担）。如果这一切都成为现实，这些项目的状况可能会发生根本性的改善，因为劳动年龄人口与退休人口的比例将发生剧烈变化。毫无疑问，这些挑战将被其他挑战所取代，例如如何确保广泛地获取新技术，但值得反思的是，即使生物学是唯一一个被人工智能成功加速的领域，世界也会发生多大的变化。

2. Neuroscience and mind
2. 神经科学和心智

In the previous section I focused on physical diseases and biology in general, and didn’t cover neuroscience or mental health. But neuroscience is a subdiscipline of biology and mental health is just as important as physical health. In fact, if anything, mental health affects human well-being even more directly than physical health. Hundreds of millions of people have very low quality of life due to problems like addiction, depression, schizophrenia, low-functioning autism, PTSD, psychopathy21, or intellectual disabilities. Billions more struggle with everyday problems that can often be interpreted as much milder versions of one of these severe clinical disorders. And as with general biology, it may be possible to go beyond addressing problems to improving the baseline quality of human experience.
在前一节中，我专注于身体疾病和生物学，没有涉及神经科学或心理健康。但神经科学是生物学的子学科，心理健康与身体健康同样重要。事实上，如果有什么不同的话，心理健康甚至比身体健康更直接地影响人类福祉。数亿人由于成瘾、抑郁症、精神分裂症、低功能自闭症、PTSD、心理病态 21 或智力障碍等问题，生活质量非常低。还有数十亿人挣扎于日常问题，这些问题通常可以解释为这些严重临床疾病的一种较轻版本。就像一般生物学一样，我们可能不仅能够解决这些问题，还能提高人类经验的基准质量。

The basic framework that I laid out for biology applies equally to neuroscience. The field is propelled forward by a small number of discoveries often related to tools for measurement or precise intervention – in the list of those above, optogenetics was a neuroscience discovery, and more recently CLARITY and expansion microscopy are advances in the same vein, in addition to many of the general cell biology methods directly carrying over to neuroscience. I think the rate of these advances will be similarly accelerated by AI and therefore that the framework of “100 years of progress in 5-10 years” applies to neuroscience in the same way it does to biology and for the same reasons. As in biology, the progress in 20th century neuroscience was enormous – for example we didn’t even understand how or why neurons fired until the 1950’s. Thus, it seems reasonable to expect AI-accelerated neuroscience to produce rapid progress over a few years.
我提出的生物学基本框架同样适用于神经科学。该领域的发展由少数与测量工具或精确干预相关的发现推动——在上面的列表中，光遗传学是神经科学的一个发现，而最近 CLARITY 和扩展显微镜也是同一领域的进步，此外，许多细胞生物学的一般方法也直接应用于神经科学。我认为这些进步的速度将由人工智能加速，因此“5-10 年内取得 100 年的进步”的框架同样适用于神经科学，就像它适用于生物学一样，并且出于相同的原因。就像生物学一样，20 世纪神经科学的发展是巨大的——例如，我们直到 20 世纪 50 年代才弄清楚神经元是如何或为什么放电的。因此，合理地预期人工智能加速的神经科学将在几年内取得快速进展。

There is one thing we should add to this basic picture, which is that some of the things we’ve learned (or are learning) about AI itself in the last few years are likely to help advance neuroscience, even if it continues to be done only by humans. Interpretability is an obvious example: although biological neurons superficially operate in a completely different manner from artificial neurons (they communicate via spikes and often spike rates, so there is a time element not present in artificial neurons, and a bunch of details relating to cell physiology and neurotransmitters modifies their operation substantially), the basic question of “how do distributed, trained networks of simple units that perform combined linear/non-linear operations work together to perform important computations” is the same, and I strongly suspect the details of individual neuron communication will be abstracted away in most of the interesting questions about computation and circuits22. As just one example of this, a computational mechanism discovered by interpretability researchers in AI systems was recently rediscovered in the brains of mice.
我们应该给这幅基本图增加一件事，那就是在过去的几年里，我们关于人工智能本身学到的一些东西（或者正在学习）很可能会帮助推进神经科学，即使它仍然只由人类来完成。可解释性是一个明显的例子：尽管生物神经元在表面上与人工神经元以完全不同的方式运作（它们通过尖峰和尖峰率进行通信，因此存在人工神经元中不存在的时间元素，以及与细胞生理学和神经递质相关的一堆细节会极大地改变它们的运作），但“如何通过执行组合线性/非线性操作的简单单元的分布式、训练网络共同完成重要计算”的基本问题是一样的，我强烈怀疑单个神经元通信的细节将在大多数关于计算和电路的有趣问题中被抽象化。作为一个例子，可解释性研究人员在人工智能系统中发现的一种计算机制最近在老鼠的大脑中被重新发现。

It is much easier to do experiments on artificial neural networks than on real ones (the latter often requires cutting into animal brains), so interpretability may well become a tool for improving our understanding of neuroscience. Furthermore, powerful AI’s will themselves probably be able to develop and apply this tool better than humans can.
人工神经网络比真实神经网络更容易进行实验（后者通常需要切割动物大脑），因此可解释性可能成为提高我们对神经科学理解的一种工具。此外，强大的 AI 自身可能比人类更能开发和应用这一工具。

Beyond just interpretability though, what we have learned from AI about how intelligent systems are trained should (though I am not sure it has yet) cause a revolution in neuroscience. When I was working in neuroscience, a lot of people focused on what I would now consider the wrong questions about learning, because the concept of the scaling hypothesis / bitter lesson didn’t exist yet. The idea that a simple objective function plus a lot of data can drive incredibly complex behaviors makes it more interesting to understand the objective functions and architectural biases and less interesting to understand the details of the emergent computations. I have not followed the field closely in recent years, but I have a vague sense that computational neuroscientists have still not fully absorbed the lesson. My attitude to the scaling hypothesis has always been “aha – this is an explanation, at a high level, of how intelligence works and how it so easily evolved”, but I don’t think that’s the average neuroscientist’s view, in part because the scaling hypothesis as “the secret to intelligence” isn’t fully accepted even within AI.
然而，除了可解释性之外，我们从人工智能中关于智能系统训练所学到的东西，应该（尽管我不确定它已经做到了）在神经科学领域引起一场革命。当我从事神经科学研究时，很多人关注我现在认为关于学习错误的问题，因为当时还没有出现规模假设/苦涩教训的概念。简单目标函数加上大量数据可以驱动极其复杂的行为，这使得理解目标函数和架构偏差更有趣，而对涌现计算细节的理解则不那么有趣。近年来，我没有密切关注该领域，但我有一种模糊的感觉，计算神经科学家们还没有完全吸收这个教训。我对规模假设的态度一直是“啊哈——这是一个关于智能如何运作以及它如何轻易演化的高层次解释”，但我不认为这是平均神经科学家的观点，部分原因是因为作为“智能的秘密”的规模假设在人工智能领域内也还没有被完全接受。

I think that neuroscientists should be trying to combine this basic insight with the particularities of the human brain (biophysical limitations, evolutionary history, topology, details of motor and sensory inputs/outputs) to try to figure out some of neuroscience’s key puzzles. Some likely are, but I suspect it’s not enough yet, and that AI neuroscientists will be able to more effectively leverage this angle to accelerate progress.
我认为神经科学家应该尝试将这一基本见解与人类大脑的特定性（生物物理限制、进化历史、拓扑结构、运动和感官输入/输出的细节）相结合，以试图解决神经科学的一些关键难题。其中一些可能是，但我怀疑这还不够，AI 神经科学家将能够更有效地利用这个角度来加速进步。

I expect AI to accelerate neuroscientific progress along four distinct routes, all of which can hopefully work together to cure mental illness and improve function:
我期望人工智能通过四个不同的途径加速神经科学进步，所有这些途径都希望能够协同工作，以治愈精神疾病并改善功能：

It’s my guess that these four routes of progress working together would, as with physical disease, be on track to lead to the cure or prevention of most mental illness in the next 100 years even if AI was not involved – and thus might reasonably be completed in 5-10 AI-accelerated years. Concretely my guess at what will happen is something like:
我的猜测是，这四条进步路线共同作用，就像身体疾病一样，将有望在未来的 100 年内治愈或预防大多数精神疾病，即使没有人工智能的参与——因此，在 5-10 年内，通过人工智能加速，这可能是合理的。具体来说，我对将要发生的事情的猜测是这样的：

One topic that often comes up in sci-fi depictions of AI, but that I intentionally haven’t discussed here, is “mind uploading”, the idea of capturing the pattern and dynamics of a human brain and instantiating them in software. This topic could be the subject of an essay all by itself, but suffice it to say that while I think uploading is almost certainly possible in principle, in practice it faces significant technological and societal challenges, even with powerful AI, that likely put it outside the 5-10 year window we are discussing.
一个经常出现在科幻对人工智能描绘中的话题，但我故意在这里没有讨论，就是“心灵上传”，即捕捉人类大脑的模式和动态并在软件中实现它们。这个话题本身就可以成为一篇论文的主题，但 suffice to say that while I think uploading is almost certainly possible in principle, in practice it faces significant technological and societal challenges, even with powerful AI, that likely put it outside the 5-10 year window we are discussing. 心灵上传，捕捉人类大脑的模式和动态并在软件中实现它们，是科幻对人工智能描绘中经常出现的一个话题，但我故意在这里没有讨论。这个话题本身就可以成为一篇论文的主题，但简而言之，虽然我认为在原则上上传几乎肯定可行，但在实践中，即使拥有强大的 AI，它也面临着重大的技术和社会挑战，这很可能使它超出了我们正在讨论的 5-10 年窗口。

In summary, AI-accelerated neuroscience is likely to vastly improve treatments for, or even cure, most mental illness as well as greatly expand “cognitive and mental freedom” and human cognitive and emotional abilities. It will be every bit as radical as the improvements in physical health described in the previous section. Perhaps the world will not be visibly different on the outside, but the world as experienced by humans will be a much better and more humane place, as well as a place that offers greater opportunities for self-actualization. I also suspect that improved mental health will ameliorate a lot of other societal problems, including ones that seem political or economic.
总的来说，人工智能加速的神经科学可能会极大地改善大多数精神疾病的治疗，甚至可能治愈它们，同时极大地扩展“认知和心理健康自由”以及人类的认知和情感能力。它将和前一部分描述的身体健康改善一样激进。也许从外表上看，世界不会有明显的变化，但人类体验到的世界将是一个更加美好和人性化的地方，同时也是一个提供更多自我实现机会的地方。我还怀疑，改善的心理健康将缓解许多其他社会问题，包括那些看似政治或经济的问题。

3. Economic development and poverty
3. 经济发展与贫困

The previous two sections are about developing new technologies that cure disease and improve the quality of human life. However an obvious question, from a humanitarian perspective, is: “will everyone have access to these technologies?”
前两节是关于开发治疗疾病和提高人类生活质量的新的技术。然而，从人道主义的角度来看，一个明显的问题就是：“每个人都能获得这些技术吗？”

It is one thing to develop a cure for a disease, it is another thing to eradicate the disease from the world. More broadly, many existing health interventions have not yet been applied everywhere in the world, and for that matter the same is true of (non-health) technological improvements in general. Another way to say this is that living standards in many parts of the world are still desperately poor: GDP per capita is ~$2,000 in Sub-Saharan Africa as compared to ~$75,000 in the United States. If AI further increases economic growth and quality of life in the developed world, while doing little to help the developing world, we should view that as a terrible moral failure and a blemish on the genuine humanitarian victories in the previous two sections. Ideally, powerful AI should help the developing world catch up to the developed world, even as it revolutionizes the latter.
发展一种疾病的疗法是一回事，而从世界上根除这种疾病则是另一回事。更广泛地说，许多现有的健康干预措施尚未在世界各地得到应用，同样，这一点也适用于（非健康）技术改进的普遍情况。另一种说法是，世界上许多地区的生活水平仍然极其贫困：撒哈拉以南非洲的人均 GDP 约为 2,000 美元，而美国约为 75,000 美元。如果人工智能进一步提高了发达世界的经济增长和生活质量，而对发展中国家帮助甚微，那么我们应该将其视为严重的道德败坏，并玷污了前两节中真正的慈善胜利。理想情况下，强大的 AI 应该帮助发展中国家赶上发达国家，即使它正在改变后者。

I am not as confident that AI can address inequality and economic growth as I am that it can invent fundamental technologies, because technology has such obvious high returns to intelligence (including the ability to route around complexities and lack of data) whereas the economy involves a lot of constraints from humans, as well as a large dose of intrinsic complexity. I am somewhat skeptical that an AI could solve the famous “socialist calculation problem”23 and I don’t think governments will (or should) turn over their economic policy to such an entity, even if it could do so. There are also problems like how to convince people to take treatments that are effective but that they may be suspicious of.
我并不像相信 AI 能够发明基础技术那样相信它能够解决不平等和经济增长问题，因为技术对智能（包括绕过复杂性和数据不足的能力）有如此明显的高回报，而经济则涉及到许多来自人类的限制，以及大量的内在复杂性。我对 AI 能够解决著名的“社会主义计算问题”有些怀疑，并且我认为政府（或应该）不会将经济政策交给这样一个实体，即使它能够做到。还存在像如何说服人们接受有效但可能让他们怀疑的治疗等问题。

The challenges facing the developing world are made even more complicated by pervasive corruption in both private and public sectors. Corruption creates a vicious cycle: it exacerbates poverty, and poverty in turn breeds more corruption. AI-driven plans for economic development need to reckon with corruption, weak institutions, and other very human challenges.
发展中国家面临的挑战因私营和公共部门普遍存在的腐败而变得更加复杂。腐败形成了一个恶性循环：它加剧了贫困，而贫困反过来又滋生更多的腐败。由人工智能驱动的经济发展计划需要考虑腐败、薄弱的机构以及其他非常人性化的挑战。

Nevertheless, I do see significant reasons for optimism. Diseases have been eradicated and many countries have gone from poor to rich, and it is clear that the decisions involved in these tasks exhibit high returns to intelligence (despite human constraints and complexity). Therefore, AI can likely do them better than they are currently being done. There may also be targeted interventions that get around the human constraints and that AI could focus on. More importantly though, we have to try. Both AI companies and developed world policymakers will need to do their part to ensure that the developing world is not left out; the moral imperative is too great. So in this section, I’ll continue to make the optimistic case, but keep in mind everywhere that success is not guaranteed and depends on our collective efforts.
尽管如此，我确实看到了乐观的充分理由。疾病已被根除，许多国家从贫穷走向富裕，很明显，这些任务中所涉及的决策展现了高智商的回报（尽管存在人类限制和复杂性）。因此，AI 可能比目前做得更好。也可能存在针对人类限制的针对性干预措施，AI 可以专注于这些措施。更重要的是，我们必须尝试。AI 公司和发达国家政策制定者都需要尽自己的一份力，确保发展中国家不被落下；道德责任太重大了。所以在这个部分，我将继续提出乐观的观点，但请记住，成功并非板上钉钉，它取决于我们的共同努力。

Below I make some guesses about how I think things may go in the developing world over the 5-10 years after powerful AI is developed:
以下是我对在强大人工智能开发后的 5-10 年内发展中国家可能的发展趋势的一些猜测：

Overall, I am optimistic about quickly bringing AI’s biological advances to people in the developing world. I am hopeful, though not confident, that AI can also enable unprecedented economic growth rates and allow the developing world to at least surpass where the developed world is now. I am concerned about the “opt out” problem in both the developed and developing world, but suspect that it will peter out over time and that AI can help accelerate this process. It won’t be a perfect world, and those who are behind won’t fully catch up, at least not in the first few years. But with strong efforts on our part, we may be able to get things moving in the right direction—and fast. If we do, we can make at least a downpayment on the promises of dignity and equality that we owe to every human being on earth.
总体而言，我对将人工智能的生物进步迅速带给发展中国家的人们持乐观态度。我抱有希望，但并不自信，认为人工智能也能实现前所未有的经济增长率，并使发展中国家至少超越发达国家现在的水平。我对发达国家和发展中国家都存在的“退出”问题表示担忧，但怀疑这个问题会随着时间的推移而逐渐消失，而人工智能可以帮助加速这一过程。这不会是一个完美的世界，那些落后的人至少在最初的几年内无法完全赶上。但只要我们付出强有力的努力，我们可能能够迅速将事情引向正确的方向。如果我们做到了，我们至少可以为地球上每一个人的尊严和平等做出承诺。

4. Peace and governance
4. 和平与治理

Suppose that everything in the first three sections goes well: disease, poverty, and inequality are significantly reduced and the baseline of human experience is raised substantially. It does not follow that all major causes of human suffering are solved. Humans are still a threat to each other. Although there is a trend of technological improvement and economic development leading to democracy and peace, it is a very loose trend, with frequent (and recent) backsliding. At the dawn of the 20th Century, people thought they had put war behind them; then came the two world wars. Thirty years ago Francis Fukuyama wrote about “the End of History” and a final triumph of liberal democracy; that hasn’t happened yet. Twenty years ago US policymakers believed that free trade with China would cause it to liberalize as it became richer; that very much didn’t happen, and we now seem headed for a second cold war with a resurgent authoritarian bloc. And plausible theories suggest that internet technology may actually advantage authoritarianism, not democracy as initially believed (e.g. in the “Arab Spring” period). It seems important to try to understand how powerful AI will intersect with these issues of peace, democracy, and freedom.
假设前三个部分一切顺利：疾病、贫困和不平等都得到了显著减少，人类经验的基准得到了大幅提升。但这并不意味着人类苦难的所有主要原因都得到了解决。人类仍然相互威胁。尽管有技术进步和经济发展的趋势，导致民主和和平，但这是一种非常松散的趋势，经常（并且最近）出现倒退。20 世纪初，人们认为他们已经把战争抛在了身后；然后发生了两次世界大战。三十年前，弗朗西斯·福山（Francis Fukuyama）写下了“历史的终结”和自由民主的最终胜利；这还没有发生。二十年前，美国政策制定者认为，与中国的自由贸易将随着其变得更富裕而使其自由化；这并没有发生，现在我们似乎正走向与重新崛起的威权集团的第二场冷战。并且有合理的理论表明，互联网技术实际上可能有利于威权主义，而不是最初认为的民主（例如在“阿拉伯之春”期间）。 似乎了解人工智能如何与这些问题（和平、民主和自由）相交是很重要的。

Unfortunately, I see no strong reason to believe AI will preferentially or structurally advance democracy and peace, in the same way that I think it will structurally advance human health and alleviate poverty. Human conflict is adversarial and AI can in principle help both the “good guys” and the “bad guys”. If anything, some structural factors seem worrying: AI seems likely to enable much better propaganda and surveillance, both major tools in the autocrat’s toolkit. It’s therefore up to us as individual actors to tilt things in the right direction: if we want AI to favor democracy and individual rights, we are going to have to fight for that outcome. I feel even more strongly about this than I do about international inequality: the triumph of liberal democracy and political stability is not guaranteed, perhaps not even likely, and will require great sacrifice and commitment on all of our parts, as it often has in the past.
不幸的是，我看不出有什么强有力的理由相信人工智能会优先或结构性地推进民主与和平，就像我认为它将结构性地推进人类健康和缓解贫困一样。人类冲突是敌对的，人工智能原则上可以帮助“好人”和“坏人”。如果有什么的话，一些结构性因素似乎令人担忧：人工智能似乎很可能使宣传和监控变得更好，这两者都是独裁者工具箱中的主要工具。因此，作为个体行动者，我们必须把事情引向正确的方向：如果我们希望人工智能青睐民主和个人权利，我们就必须为这一结果而奋斗。我对这一点比我对国际不平等的看法更为强烈：自由民主的胜利和政治稳定并不保证，甚至可能不太可能，这需要我们所有人的巨大牺牲和承诺，就像过去经常发生的那样。

I think of the issue as having two parts: international conflict, and the internal structure of nations. On the international side, it seems very important that democracies have the upper hand on the world stage when powerful AI is created. AI-powered authoritarianism seems too terrible to contemplate, so democracies need to be able to set the terms by which powerful AI is brought into the world, both to avoid being overpowered by authoritarians and to prevent human rights abuses within authoritarian countries.
我认为这个问题有两个部分：国际冲突和国家的内部结构。在国际方面，当强大的 AI 被创造出来时，似乎非常重要的是民主国家在世界舞台上占据上风。AI 驱动的威权主义似乎太过可怕，以至于无法想象，因此民主国家需要能够设定将强大 AI 带入世界的条件，既为了避免被威权主义者压倒，也为了防止威权国家内的人权侵犯。

My current guess at the best way to do this is via an “entente strategy”26, in which a coalition of democracies seeks to gain a clear advantage (even just a temporary one) on powerful AI by securing its supply chain, scaling quickly, and blocking or delaying adversaries’ access to key resources like chips and semiconductor equipment. This coalition would on one hand use AI to achieve robust military superiority (the stick) while at the same time offering to distribute the benefits of powerful AI (the carrot) to a wider and wider group of countries in exchange for supporting the coalition’s strategy to promote democracy (this would be a bit analogous to “Atoms for Peace”). The coalition would aim to gain the support of more and more of the world, isolating our worst adversaries and eventually putting them in a position where they are better off taking the same bargain as the rest of the world: give up competing with democracies in order to receive all the benefits and not fight a superior foe.
我目前认为最好的方法是通过“协议战略”来实现，即由民主国家联盟寻求通过确保其供应链、快速扩展和阻止或延迟对手获取关键资源（如芯片和半导体设备）来获得对强大人工智能的明显优势（即使只是暂时的）。这个联盟一方面将利用人工智能实现强大的军事优势（即“大棒”），同时向越来越多的国家提供强大人工智能的好处（即“胡萝卜”），以换取支持联盟促进民主的战略（这有点类似于“和平原子”）。该联盟旨在获得越来越多世界的支持，孤立我们最糟糕的对手，并最终使他们处于一个更好的位置：放弃与民主国家的竞争，以获得所有好处，而不是与一个更强大的对手作战。

If we can do all this, we will have a world in which democracies lead on the world stage and have the economic and military strength to avoid being undermined, conquered, or sabotaged by autocracies, and may be able to parlay their AI superiority into a durable advantage. This could optimistically lead to an “eternal 1991”—a world where democracies have the upper hand and Fukuyama’s dreams are realized. Again, this will be very difficult to achieve, and will in particular require close cooperation between private AI companies and democratic governments, as well as extraordinarily wise decisions about the balance between carrot and stick.
如果我们能做所有这些，我们将拥有一个民主国家在世界舞台上引领，并拥有经济和军事实力以避免被独裁者破坏、征服或破坏的世界，并且可能能够将他们的 AI 优势转化为持久的优势。这可能乐观地导致一个“永恒的 1991”——一个民主国家占据上风，福山梦想实现的世界。再次强调，这将非常困难实现，特别是需要私营 AI 公司与民主政府之间的紧密合作，以及关于胡萝卜和大棒之间平衡的极其明智的决策。

Even if all that goes well, it leaves the question of the fight between democracy and autocracy within each country. It is obviously hard to predict what will happen here, but I do have some optimism that given a global environment in which democracies control the most powerful AI, then AI may actually structurally favor democracy everywhere. In particular, in this environment democratic governments can use their superior AI to win the information war: they can counter influence and propaganda operations by autocracies and may even be able to create a globally free information environment by providing channels of information and AI services in a way that autocracies lack the technical ability to block or monitor. It probably isn’t necessary to deliver propaganda, only to counter malicious attacks and unblock the free flow of information. Although not immediate, a level playing field like this stands a good chance of gradually tilting global governance towards democracy, for several reasons.
即使一切顺利，也留下了每个国家民主与威权之间的斗争问题。显然很难预测这里会发生什么，但我确实有一些乐观，考虑到民主国家控制着最强大的 AI 的全球环境，AI 实际上可能在全球范围内结构性地有利于民主。特别是，在这种环境中，民主政府可以利用其优越的 AI 赢得信息战：他们可以对抗威权主义的宣传和影响行动，甚至可能通过提供信息渠道和 AI 服务的方式，使威权主义缺乏技术能力来阻止或监控，从而创造一个全球自由的信息环境。可能不需要传播宣传，只需对抗恶意攻击和解除信息的自由流动。尽管不是立即的，但这种公平的竞争环境有很大的可能性逐渐将全球治理转向民主，原因有几个。

First, the increases in quality of life in Sections 1-3 should, all things equal, promote democracy: historically they have, to at least some extent. In particular I expect improvements in mental health, well-being, and education to increase democracy, as all three are negatively correlated with support for authoritarian leaders. In general people want more self-expression when their other needs are met, and democracy is among other things a form of self-expression. Conversely, authoritarianism thrives on fear and resentment.
首先，第 1-3 节生活质量提高应能促进民主：从历史角度看，至少在一定程度上是这样的。特别是，我预计心理健康、福祉和教育方面的改善将增加对民主的支持，因为这三者都与对独裁领导者的支持呈负相关。一般来说，当其他需求得到满足时，人们希望有更多的自我表达，而民主就是自我表达的一种形式。相反，威权主义依赖于恐惧和怨恨。

Second, there is a good chance free information really does undermine authoritarianism, as long as the authoritarians can’t censor it. And uncensored AI can also bring individuals powerful tools for undermining repressive governments. Repressive governments survive by denying people a certain kind of common knowledge, keeping them from realizing that “the emperor has no clothes”. For example Srđa Popović, who helped to topple the Milošević government in Serbia, has written extensively about techniques for psychologically robbing authoritarians of their power, for breaking the spell and rallying support against a dictator. A superhumanly effective AI version of Popović (whose skills seem like they have high returns to intelligence) in everyone’s pocket, one that dictators are powerless to block or censor, could create a wind at the backs of dissidents and reformers across the world. To say it again, this will be a long and protracted fight, one where victory is not assured, but if we design and build AI in the right way, it may at least be a fight where the advocates of freedom everywhere have an advantage.
其次，免费信息确实有可能削弱威权主义，只要威权主义者无法对其进行审查。未经审查的人工智能还可以为个人提供强大的工具，以破坏威权政府。威权政府通过否认人们某种共同知识来维持生存，使他们无法意识到“皇帝没有衣服”。例如，Srđa Popović曾帮助推翻塞尔维亚的米洛舍维奇政府，他广泛地撰写了关于从心理上剥夺威权主义者权力、打破魔咒并动员支持反对独裁者的技巧。一个超级有效的人工智能版本的 Popović（其技能似乎具有高智力回报）在每个人的口袋里，一个独裁者无法阻止或审查的版本，可以为世界各地的异见者和改革者带来一股风。再次说，这将是一场漫长而持久的斗争，胜利并不确定，但如果我们以正确的方式设计和构建人工智能，至少这将是一场自由倡导者具有优势的斗争。

As with neuroscience and biology, we can also ask how things could be “better than normal”—not just how to avoid autocracy, but how to make democracies better than they are today. Even within democracies, injustices happen all the time. Rule-of-law societies make a promise to their citizens that everyone will be equal under the law and everyone is entitled to basic human rights, but obviously people do not always receive those rights in practice. That this promise is even partially fulfilled makes it something to be proud of, but can AI help us do better?
与神经科学和生物学一样，我们也可以问事物如何“优于正常”——不仅仅是如何避免独裁，而是如何使民主比今天更好。即使在民主国家，不公正的事情也时有发生。法治社会向其公民承诺，每个人在法律面前都是平等的，每个人都有权享有基本人权，但显然人们并不总是能在实践中获得这些权利。即使这个承诺部分得到实现，这也是值得骄傲的，但人工智能能帮助我们做得更好吗？

For example, could AI improve our legal and judicial system by making decisions and processes more impartial? Today people mostly worry in legal or judicial contexts that AI systems will be a cause of discrimination, and these worries are important and need to be defended against. At the same time, the vitality of democracy depends on harnessing new technologies to improve democratic institutions, not just responding to risks. A truly mature and successful implementation of AI has the potential to reduce bias and be fairer for everyone.
例如，人工智能能否通过使决策和程序更加公正来改善我们的法律和司法体系？如今，人们在法律或司法环境中普遍担心人工智能系统将成为歧视的源头，这些担忧很重要，需要加以捍卫。同时，民主的活力取决于利用新技术来改善民主机构，而不仅仅是应对风险。真正成熟和成功的人工智能实施有可能减少偏见，使每个人都更加公平。

For centuries, legal systems have faced the dilemma that the law aims to be impartial, but is inherently subjective and thus must be interpreted by biased humans. Trying to make the law fully mechanical hasn’t worked because the real world is messy and can’t always be captured in mathematical formulas. Instead legal systems rely on notoriously imprecise criteria like “cruel and unusual punishment” or “utterly without redeeming social importance”, which humans then interpret—and often do so in a manner that displays bias, favoritism, or arbitrariness. “Smart contracts” in cryptocurrencies haven’t revolutionized law because ordinary code isn’t smart enough to adjudicate all that much of interest. But AI might be smart enough for this: it is the first technology capable of making broad, fuzzy judgements in a repeatable and mechanical way.
几个世纪以来，法律体系面临着这样的困境：法律旨在保持公正，但本质上具有主观性，因此必须由有偏见的个人来解释。试图使法律完全机械化并没有成功，因为现实世界是混乱的，并不能总是被数学公式所捕捉。相反，法律体系依赖于诸如“残酷和不寻常的惩罚”或“完全缺乏 redeeming 社会重要性”之类的臭名昭著的不精确标准，然后由人类来解释——而且往往是以显示偏见、偏袒或任意性的方式来解释。加密货币中的“智能合约”并没有彻底改变法律，因为普通的代码不够智能，无法裁决许多有趣的事情。但 AI 可能足够智能：它是第一种能够以可重复和机械的方式做出广泛、模糊判断的技术。

I am not suggesting that we literally replace judges with AI systems, but the combination of impartiality with the ability to understand and process messy, real world situations feels like it should have some serious positive applications to law and justice. At the very least, such systems could work alongside humans as an aid to decision-making. Transparency would be important in any such system, and a mature science of AI could conceivably provide it: the training process for such systems could be extensively studied, and advanced interpretability techniques could be used to see inside the final model and assess it for hidden biases, in a way that is simply not possible with humans. Such AI tools could also be used to monitor for violations of fundamental rights in a judicial or police context, making constitutions more self-enforcing.
我并不是建议我们字面地用 AI 系统取代法官，但将公正性与理解和处理混乱、真实世界情况的能力结合起来，似乎应该对法律和司法有一些严肃的积极应用。至少，这样的系统可以作为人类决策的辅助工具。在任何这样的系统中，透明度都将是重要的，成熟的 AI 科学可能提供这种透明度：可以广泛研究这类系统的训练过程，并使用高级可解释技术来深入了解最终模型并评估其隐藏的偏见，这是人类无法做到的。这样的 AI 工具还可以用于监控司法或警察背景下的基本权利违规行为，使宪法更具自我执行能力。

In a similar vein, AI could be used to both aggregate opinions and drive consensus among citizens, resolving conflict, finding common ground, and seeking compromise. Some early ideas in this direction have been undertaken by the computational democracy project, including collaborations with Anthropic. A more informed and thoughtful citizenry would obviously strengthen democratic institutions.
在类似的方向上，人工智能可以用来汇总意见并在公民中推动共识，解决冲突，寻找共同点，并寻求妥协。在这方面的一些早期想法已被计算民主项目所采纳，包括与 Anthropic 的合作。显然，一个更加知情和深思熟虑的公民群体将加强民主制度。

There is also a clear opportunity for AI to be used to help provision government services—such as health benefits or social services—that are in principle available to everyone but in practice often severely lacking, and worse in some places than others. This includes health services, the DMV, taxes, social security, building code enforcement, and so on. Having a very thoughtful and informed AI whose job is to give you everything you’re legally entitled to by the government in a way you can understand—and who also helps you comply with often confusing government rules—would be a big deal. Increasing state capacity both helps to deliver on the promise of equality under the law, and strengthens respect for democratic governance. Poorly implemented services are currently a major driver of cynicism about government27.
也存在一个明显的机遇，可以利用人工智能来帮助提供政府服务——例如健康福利或社会服务，这些服务在原则上对每个人都是可用的，但在实践中往往严重不足，在某些地方甚至比其他地方更糟。这包括医疗服务、机动车管理局、税收、社会保障、建筑规范执行等等。拥有一个深思熟虑且信息丰富的 AI，其任务是向您提供您依法有权从政府那里获得的一切，并以您能够理解的方式提供——同时还能帮助您遵守常常令人困惑的政府规则——这将是一件大事。提高国家能力既有助于实现法律平等承诺，也加强了人们对民主治理的尊重。目前，服务实施不当是人们对政府产生怀疑的主要驱动力。

All of these are somewhat vague ideas, and as I said at the beginning of this section, I am not nearly as confident in their feasibility as I am in the advances in biology, neuroscience, and poverty alleviation. They may be unrealistically utopian. But the important thing is to have an ambitious vision, to be willing to dream big and try things out. The vision of AI as a guarantor of liberty, individual rights, and equality under the law is too powerful a vision not to fight for. A 21st century, AI-enabled polity could be both a stronger protector of individual freedom, and a beacon of hope that helps make liberal democracy the form of government that the whole world wants to adopt.
所有这些想法都有些模糊，正如我在本节开头所说，我对它们的可行性远不如对生物学、神经科学和扶贫的进步有信心。它们可能是不切实际的乌托邦。但重要的是要有远大的愿景，愿意大胆梦想并尝试新事物。将人工智能视为自由、个人权利和法律平等保障者的愿景是如此强大，不容不为之奋斗。21 世纪，人工智能赋能的政体可以既是个人自由的更强保护者，也是帮助将自由民主制变成全世界都希望采用的政府形式的希望灯塔。

5. Work and meaning
5. 工作与意义

Even if everything in the preceding four sections goes well—not only do we alleviate disease, poverty, and inequality, but liberal democracy becomes the dominant form of government, and existing liberal democracies become better versions of themselves—at least one important question still remains. “It’s great we live in such a technologically advanced world as well as a fair and decent one”, someone might object, “but with AI’s doing everything, how will humans have meaning? For that matter, how will they survive economically?”.
即使前四个部分一切顺利——我们不仅缓解了疾病、贫困和不平等，而且自由民主成为政府的主导形式，现有的自由民主国家也成为了更好的版本——至少还有一个重要的问题仍然存在。“我们生活在一个如此技术先进、公平和公正的世界真是太好了”，有人可能会反对，“但是随着人工智能做所有的事情，人类如何找到意义？更不用说，他们如何在经济上生存？”

I think this question is more difficult than the others. I don’t mean that I am necessarily more pessimistic about it than I am about the other questions (although I do see challenges). I mean that it is fuzzier and harder to predict in advance, because it relates to macroscopic questions about how society is organized that tend to resolve themselves only over time and in a decentralized manner. For example, historical hunter-gatherer societies might have imagined that life is meaningless without hunting and various kinds of hunting-related religious rituals, and would have imagined that our well-fed technological society is devoid of purpose. They might also have not understood how our economy can provide for everyone, or what function people can usefully service in a mechanized society.
我认为这个问题比其他问题更难。我并不是说我一定比其他问题更悲观（尽管我确实看到了挑战）。我的意思是，它更模糊，更难以提前预测，因为它涉及到关于社会如何组织的大规模问题，这些问题往往只有在时间流逝和去中心化的方式下才能解决。例如，历史上的狩猎采集社会可能会想象没有狩猎和各种与狩猎相关的宗教仪式，生活就没有意义，他们可能会想象我们的富足科技社会缺乏目的。他们也可能不理解我们的经济如何为每个人提供保障，或者人们在机械化社会中能发挥什么有用的作用。

Nevertheless, it’s worth saying at least a few words, while keeping in mind that the brevity of this section is not at all to be taken as a sign that I don’t take these issues seriously—on the contrary, it is a sign of a lack of clear answers.
尽管如此，至少说几句也是值得的，同时要记住，本节的简短并不代表我不重视这些问题——相反，这表明缺乏明确的答案。

On the question of meaning, I think it is very likely a mistake to believe that tasks you undertake are meaningless simply because an AI could do them better. Most people are not the best in the world at anything, and it doesn’t seem to bother them particularly much. Of course today they can still contribute through comparative advantage, and may derive meaning from the economic value they produce, but people also greatly enjoy activities that produce no economic value. I spend plenty of time playing video games, swimming, walking around outside, and talking to friends, all of which generates zero economic value. I might spend a day trying to get better at a video game, or faster at biking up a mountain, and it doesn’t really matter to me that someone somewhere is much better at those things. In any case I think meaning comes mostly from human relationships and connection, not from economic labor. People do want a sense of accomplishment, even a sense of competition, and in a post-AI world it will be perfectly possible to spend years attempting some very difficult task with a complex strategy, similar to what people do today when they embark on research projects, try to become Hollywood actors, or found companies28. The facts that (a) an AI somewhere could in principle do this task better, and (b) this task is no longer an economically rewarded element of a global economy, don’t seem to me to matter very much.
关于意义的问题，我认为认为你承担的任务没有意义仅仅因为 AI 能做得更好，这很可能是一个错误。大多数人不是世界上任何事情的最好，这似乎并没有特别困扰他们。当然，今天他们仍然可以通过比较优势做出贡献，并可能从他们创造的经济价值中获得意义，但人们也非常享受那些不产生经济价值的活动。我花了很多时间玩电子游戏、游泳、在外面散步和和朋友聊天，所有这些都产生了零经济价值。我可能会花一天时间试图在电子游戏中变得更好，或者更快地骑上山顶，但这对我来说并不重要，有人在某个地方在这方面做得更好。无论如何，我认为意义主要来自人际关系和联系，而不是经济劳动。 人们确实想要成就感，甚至竞争感，在人工智能时代之后，花几年时间尝试一些非常困难的任务，采用复杂的策略，类似于人们今天开始研究项目、试图成为好莱坞演员或创立公司的情况，这将是完全可能的。在我看来，以下事实并不很重要：(a)某个地方的 AI 原则上可以更好地完成这项任务，以及(b)这项任务不再是全球经济中经济回报的元素。

The economic piece actually seems more difficult to me than the meaning piece. By “economic” in this section I mean the possible problem that most or all humans may not be able to contribute meaningfully to a sufficiently advanced AI-driven economy. This is a more macro problem than the separate problem of inequality, especially inequality in access to the new technologies, which I discussed in Section 3.
经济部分实际上在我看来比意义部分更难。在本节中，“经济”一词指的是大多数或所有人可能无法对足够先进的 AI 驱动经济做出有意义的贡献的潜在问题。这是一个比我在第 3 节中讨论的，特别是在获取新技术方面的不平等问题更为宏观的问题。

First of all, in the short term I agree with arguments that comparative advantage will continue to keep humans relevant and in fact increase their productivity, and may even in some ways level the playing field between humans. As long as AI is only better at 90% of a given job, the other 10% will cause humans to become highly leveraged, increasing compensation and in fact creating a bunch of new human jobs complementing and amplifying what AI is good at, such that the “10%” expands to continue to employ almost everyone. In fact, even if AI can do 100% of things better than humans, but it remains inefficient or expensive at some tasks, or if the resource inputs to humans and AI’s are meaningfully different, then the logic of comparative advantage continues to apply. One area humans are likely to maintain a relative (or even absolute) advantage for a significant time is the physical world. Thus, I think that the human economy may continue to make sense even a little past the point where we reach “a country of geniuses in a datacenter”.
首先，在短期内，我同意比较优势将继续使人类保持相关性，并实际上提高他们的生产力，甚至在某些方面使人类之间的竞争更加公平。只要 AI 在特定工作的 90%方面优于人类，剩下的 10%将使人类变得高度杠杆化，增加薪酬，实际上创造出一批新的与 AI 互补和增强其优势的人类工作，从而使“10%”扩展到继续雇佣几乎所有人。事实上，即使 AI 在 100%的任务上比人类做得更好，但它在某些任务上仍然效率低下或成本高昂，或者人类和 AI 的资源投入存在实质性差异，比较优势的逻辑仍然适用。人类可能在物理世界保持相对（甚至绝对）优势的一个领域是相当长的时间。因此，我认为即使在我们达到“数据中心中的天才国家”之后，人类经济可能仍然有意义。

However, I do think in the long run AI will become so broadly effective and so cheap that this will no longer apply. At that point our current economic setup will no longer make sense, and there will be a need for a broader societal conversation about how the economy should be organized.
然而，我认为从长远来看，人工智能将变得非常有效且便宜，这一点将不再适用。到那时，我们当前的经济体系将不再有意义，将需要更广泛的社会对话来讨论经济应该如何组织。

While that might sound crazy, the fact is that civilization has successfully navigated major economic shifts in the past: from hunter-gathering to farming, farming to feudalism, and feudalism to industrialism. I suspect that some new and stranger thing will be needed, and that it’s something no one today has done a good job of envisioning. It could be as simple as a large universal basic income for everyone, although I suspect that will only be a small part of a solution. It could be a capitalist economy of AI systems, which then give out resources (huge amounts of them, since the overall economic pie will be gigantic) to humans based on some secondary economy of what the AI systems think makes sense to reward in humans (based on some judgment ultimately derived from human values). Perhaps the economy runs on Whuffie points. Or perhaps humans will continue to be economically valuable after all, in some way not anticipated by the usual economic models. All of these solutions have tons of possible problems, and it’s not possible to know whether they will make sense without lots of iteration and experimentation. And as with some of the other challenges, we will likely have to fight to get a good outcome here: exploitative or dystopian directions are clearly also possible and have to be prevented. Much more could be written about these questions and I hope to do so at some later time.
虽然这听起来可能很疯狂，但事实是，文明在过去成功地 navigated 过重大的经济转变：从狩猎采集到农业，从农业到封建主义，再到工业主义。我怀疑我们需要一些新的、更奇怪的东西，而这可能是今天没有人能很好地想象出来的东西。这可能只是一个简单的、面向所有人的大规模基本收入，尽管我怀疑这只会是解决方案的一小部分。这可能是一个基于人工智能系统的资本主义经济，然后根据人工智能系统认为对人类有意义的某种次要经济（基于从人类价值观中最终得出的某种判断）向人类分配资源（由于整体经济蛋糕将非常巨大，因此资源量也将巨大）。也许经济是以 Whuffie 点为基础运行的。或者也许人类最终在经济上仍然有价值，以一种通常的经济模型无法预见的方式。所有这些解决方案都有许多可能的问题，而且没有大量的迭代和实验，无法知道它们是否有意义。 并且，与其他一些挑战一样，我们可能不得不为此而战斗以获得良好的结果：剥削性或反乌托邦的方向显然也是可能的，必须防止。关于这些问题可以写很多，我希望在将来某个时候这样做。

Taking stock 审视现状

Through the varied topics above, I’ve tried to lay out a vision of a world that is both plausible if everything goes right with AI, and much better than the world today. I don’t know if this world is realistic, and even if it is, it will not be achieved without a huge amount of effort and struggle by many brave and dedicated people. Everyone (including AI companies!) will need to do their part both to prevent risks and to fully realize the benefits.
通过上述各种主题，我试图描绘出一个既在人工智能一切顺利的情况下是可行的，又比现在的世界更好的世界图景。我不知道这个世界是否现实，即使它是现实的，没有许多勇敢和奉献的人的巨大努力和斗争，它也无法实现。每个人（包括人工智能公司！）都需要尽自己的一份力，既要预防风险，也要充分实现好处。

But it is a world worth fighting for. If all of this really does happen over 5 to 10 years—the defeat of most diseases, the growth in biological and cognitive freedom, the lifting of billions of people out of poverty to share in the new technologies, a renaissance of liberal democracy and human rights—I suspect everyone watching it will be surprised by the effect it has on them. I don’t mean the experience of personally benefiting from all the new technologies, although that will certainly be amazing. I mean the experience of watching a long-held set of ideals materialize in front of us all at once. I think many will be literally moved to tears by it.
但是这是一个值得为之奋斗的世界。如果所有这一切真的在 5 到 10 年内发生——大多数疾病的战胜、生物和认知自由的增长、数十亿人摆脱贫困，共享新技术，自由民主和人权复兴——我相信所有观看的人都会对其产生的影响感到惊讶。我并不是指从所有新技术中个人受益的经历，尽管这肯定会令人惊叹。我指的是看到我们长久以来秉持的一套理想突然在我们面前实现。我认为许多人会为此感动得热泪盈眶。

Throughout writing this essay I noticed an interesting tension. In one sense the vision laid out here is extremely radical: it is not what almost anyone expects to happen in the next decade, and will likely strike many as an absurd fantasy. Some may not even consider it desirable; it embodies values and political choices that not everyone will agree with. But at the same time there is something blindingly obvious—something overdetermined—about it, as if many different attempts to envision a good world inevitably lead roughly here.
在撰写这篇论文的过程中，我注意到了一种有趣的张力。从某种意义上说，这里阐述的愿景极其激进：它并不是几乎每个人都期望在下一个十年发生的事情，可能会让许多人觉得这是一种荒谬的幻想。有些人甚至可能不认为这是可取的；它体现了并非每个人都认同的价值和政治选择。但与此同时，它又有着一种令人眼花缭乱的明显性——一种过度决定性——仿佛许多不同尝试去构想一个美好的世界不可避免地会大致走向这里。

In Iain M. Banks’ The Player of Games29, the protagonist—a member of a society called the Culture, which is based on principles not unlike those I’ve laid out here—travels to a repressive, militaristic empire in which leadership is determined by competition in an intricate battle game. The game, however, is complex enough that a player’s strategy within it tends to reflect their own political and philosophical outlook. The protagonist manages to defeat the emperor in the game, showing that his values (the Culture’s values) represent a winning strategy even in a game designed by a society based on ruthless competition and survival of the fittest. A well-known post by Scott Alexander has the same thesis—that competition is self-defeating and tends to lead to a society based on compassion and cooperation. The “arc of the moral universe” is another similar concept.
在 Iain M. Banks 的《游戏玩家》中，主角——来自一个名为“文化”的社会的成员，这个社会基于的原则与我在此处阐述的类似——前往一个压抑、军事化的帝国，在这个帝国中，领导权由复杂的战斗游戏中的竞争决定。然而，这个游戏足够复杂，以至于玩家在其中的策略往往反映了他们自己的政治和哲学观点。主角设法在游戏中击败了皇帝，这表明他的价值观（文化的价值观）代表了一种即使在由基于残酷竞争和适者生存的社会设计的游戏中也能获胜的策略。Scott Alexander 的一篇著名文章提出了相同的论点——竞争是自我毁灭的，往往会导致一个基于同情与合作的社会。道德宇宙的“弧线”是另一个类似的概念。

I think the Culture’s values are a winning strategy because they’re the sum of a million small decisions that have clear moral force and that tend to pull everyone together onto the same side. Basic human intuitions of fairness, cooperation, curiosity, and autonomy are hard to argue with, and are cumulative in a way that our more destructive impulses often aren’t. It is easy to argue that children shouldn’t die of disease if we can prevent it, and easy from there to argue that everyone’s children deserve that right equally. From there it is not hard to argue that we should all band together and apply our intellects to achieve this outcome. Few disagree that people should be punished for attacking or hurting others unnecessarily, and from there it’s not much of a leap to the idea that punishments should be consistent and systematic across people. It is similarly intuitive that people should have autonomy and responsibility over their own lives and choices. These simple intuitions, if taken to their logical conclusion, lead eventually to rule of law, democracy, and Enlightenment values. If not inevitably, then at least as a statistical tendency, this is where humanity was already headed. AI simply offers an opportunity to get us there more quickly—to make the logic starker and the destination clearer.
我认为文化的价值观是一种胜利的战略，因为它们是一百万个小决定的总和，这些决定具有明显的道德力量，并倾向于将每个人团结在同一个阵营。公平、合作、好奇心和自主等基本的人类直觉很难被反驳，并且以一种我们更具破坏性的冲动通常不具备的方式累积。如果我们能够预防，很容易就认为孩子们不应该死于疾病，并且从这里很容易就认为每个孩子的权利都应平等地得到保障。从这里开始，很容易就认为我们应该团结起来，运用我们的智慧来实现这一目标。很少有人不同意，人们应该因为无故攻击或伤害他人而受到惩罚，并且从这里到惩罚应该对所有人一致和系统化的观点并不难跳跃。同样直观的是，人们应该对自己的生活和选择拥有自主权和责任感。这些简单的直觉，如果按照它们的逻辑结论来考虑，最终会导致法治、民主和启蒙价值观。 如果不是不可避免地，那么至少作为一个统计趋势，人类已经走向了这里。人工智能只是提供了一个更快到达那里的机会——使逻辑更清晰，目的地更明确。

Nevertheless, it is a thing of transcendent beauty. We have the opportunity to play some small role in making it real.
尽管如此，它是一件超越的美。我们有幸在使之成为现实的过程中扮演一些小角色。

Thanks to Kevin Esvelt, Parag Mallick, Stuart Ritchie, Matt Yglesias, Erik Brynjolfsson, Jim McClave, Allan Dafoe, and many people at Anthropic for reviewing drafts of this essay.
感谢 Kevin Esvelt、Parag Mallick、Stuart Ritchie、Matt Yglesias、Erik Brynjolfsson、Jim McClave、Allan Dafoe 以及 Anthropic 的许多人审阅这篇论文的草稿。

To the winners of the 2024 Nobel prize in Chemistry, for showing us all the way.
向 2024 年诺贝尔化学奖获得者致敬，感谢你们为我们指明道路。

Footnotes 脚注

  1. 1https://allpoetry.com/All-Watched-Over-By-Machines-Of-Loving-Grace ↩
    1 https://allpoetry.com/All-Watched-Over-By-Machines-Of-Loving-Grace ↩ 1 https://allpoetry.com/被慈爱的机器所监护的万物 ↩

  2. 2I do anticipate some minority of people’s reaction will be “this is pretty tame”. I think those people need to, in Twitter parlance, “touch grass”. But more importantly, tame is good from a societal perspective. I think there’s only so much change people can handle at once, and the pace I’m describing is probably close to the limits of what society can absorb without extreme turbulence. ↩
    我预计有一部分人的反应会是“这相当温和”。我认为这些人需要，用 Twitter 的说法，“摸摸草”。但更重要的是，从社会角度来看，温和是好的。我认为人们一次只能处理这么多变化，而我描述的步伐可能接近社会在没有极端动荡的情况下所能吸收的极限。

  3. 3I find AGI to be an imprecise term that has gathered a lot of sci-fi baggage and hype. I prefer "powerful AI" or "Expert-Level Science and Engineering" which get at what I mean without the hype. ↩
    我发现 AGI 是一个不精确的术语，它积累了很多科幻的包袱和炒作。我更喜欢“强大的 AI”或“专家级科学与工程”，这些术语能更准确地表达我的意思，而不带有炒作。

  4. 4In this essay, I use "intelligence" to refer to a general problem-solving capability that can be applied across diverse domains. This includes abilities like reasoning, learning, planning, and creativity. While I use "intelligence" as a shorthand throughout this essay, I acknowledge that the nature of intelligence is a complex and debated topic in cognitive science and AI research. Some researchers argue that intelligence isn't a single, unified concept but rather a collection of separate cognitive abilities. Others contend that there's a general factor of intelligence (g factor) underlying various cognitive skills. That’s a debate for another time. ↩
    在这篇文章中，我使用“智力”一词来指代一种可以应用于不同领域的通用问题解决能力。这包括推理、学习、规划和创造力等能力。虽然我在整篇文章中都用“智力”作为简称，但我承认智力在认知科学和人工智能研究中的本质是一个复杂且存在争议的话题。一些研究人员认为，智力不是一个单一、统一的概念，而是一系列分离的认知能力的集合。其他人则认为，存在一个智力的普遍因素（g 因素），它是各种认知技能的基础。那是一个留待另一次讨论的话题。

  5. 5This is roughly the current speed of AI systems – for example they can read a page of text in a couple seconds and write a page of text in maybe 20 seconds, which is 10-100x the speed at which humans can do these things. Over time larger models tend to make this slower but more powerful chips tend to make it faster; to date the two effects have roughly canceled out. ↩
    这是人工智能系统的大致当前速度——例如，它们可以在几秒钟内阅读一页文本，也许在 20 秒内写一页文本，这比人类完成这些事情的速度快 10-100 倍。随着时间的推移，更大的模型往往会使这个过程变慢，但更强大的芯片往往会使其变快；到目前为止，这两个效果大致相互抵消。

  6. 6This might seem like a strawman position, but careful thinkers like Tyler Cowen and Matt Yglesias have raised it as a serious concern (though I don’t think they fully hold the view), and I don’t think it is crazy. ↩
    这可能看起来像是一个稻草人立场，但像 Tyler Cowen 和 Matt Yglesias 这样的谨慎思考者已经将其作为一个严重的问题提出（尽管我认为他们并没有完全持有这种观点），而且我认为这并不疯狂。

  7. 7The closest economics work that I’m aware of to tackling this question is work on “general purpose technologies” and “intangible investments” that serve as complements to general purpose technologies. ↩
    我所知的解决这个问题的最接近的经济学研究是关于“通用技术”和作为通用技术补充的“无形投资”的研究。

  8. 8This learning can include temporary, in-context learning, or traditional training; both will be rate-limited by the physical world. ↩
    此学习可以包括临时情境学习或传统培训；两者都将受到物理世界的限制。

  9. 9In a chaotic system, small errors compound exponentially over time, so that even an enormous increase in computing power leads to only a small improvement in how far ahead it is possible to predict, and in practice measurement error may degrade this further. ↩
    在一个混沌系统中，小错误随时间指数级累积，因此即使计算能力大幅提升，也只能带来预测范围的小幅改善，而在实际测量中，误差可能会进一步降低这一改善。

  10. 10Another factor is of course that powerful AI itself can potentially be used to create even more powerful AI. My assumption is that this might (in fact, probably will) occur, but that its effect will be smaller than you might imagine, precisely because of the “decreasing marginal returns to intelligence” discussed here. In other words, AI will continue to get smarter quickly, but its effect will eventually be limited by non-intelligence factors, and analyzing those is what matters most to the speed of scientific progress outside AI. ↩
    另一个因素当然是，强大的 AI 本身可能被用来创建更强大的 AI。我的假设是，这可能（事实上，很可能）会发生，但其影响将比你想象的要小，这正是由于这里讨论的“智能的边际收益递减”所导致的。换句话说，AI 将继续快速变得更聪明，但最终其影响将受到非智能因素的制约，而分析这些因素是科学进步速度（除 AI 外）最重要的。

  11. 11These achievements have been an inspiration to me and perhaps the most powerful existing example of AI being used to transform biology. ↩
    这些成就对我产生了启发，可能是目前最强大的将人工智能应用于生物学的例子。

  12. 12“Progress in science depends on new techniques, new discoveries and new ideas, probably in that order.” - Sydney Brenner ↩
    科学进步依赖于新技术、新发现和新思想，可能按此顺序。 - 西德尼·布伦纳

  13. 13Thanks to Parag Mallick for suggesting this point. ↩
    感谢 Parag Mallick 提出这个观点。

  14. 14I didn't want to clog up the text with speculation about what specific future discoveries AI-enabled science could make, but here is a brainstorm of some possibilities:
    我没有想要在文本中充斥着关于 AI 赋能科学可能做出具体未来发现的推测，但这里是一些可能性的头脑风暴：
    — Design of better computational tools like AlphaFold and AlphaProteo — that is, a general AI system speeding up our ability to make specialized AI computational biology tools.
    ——设计更好的计算工具，如 AlphaFold 和 AlphaProteo——也就是说，一个通用人工智能系统，它能加速我们开发专业人工智能计算生物学工具的能力。
    — More efficient and selective CRISPR.
    更高效、更具选择性的 CRISPR。
    — More advanced cell therapies.
    更先进的细胞疗法。
    — Materials science and miniaturization breakthroughs leading to better implanted devices.
    材料科学和微型化突破，引领更先进的植入式设备。
    — Better control over stem cells, cell differentiation, and de-differentiation, and a resulting ability to regrow or reshape tissue.
    更好的对干细胞、细胞分化和去分化进行控制，以及因此能够再生或重塑组织的能力。
    — Better control over the immune system: turning it on selectively to address cancer and infectious disease, and turning it off selectively to address autoimmune diseases. ↩
    更好的免疫系统控制：选择性激活以应对癌症和传染病，选择性关闭以应对自身免疫病。

  15. 15AI may of course also help with being smarter about choosing what experiments to run: improving experimental design, learning more from a first round of experiments so that the second round can narrow in on key questions, and so on. ↩
    当然，AI 还可以帮助更智能地选择要运行的实验：改进实验设计，从第一轮实验中获得更多经验，以便第二轮实验可以缩小关键问题的范围，等等。

  16. 16Thanks to Matthew Yglesias for suggesting this point. ↩
    感谢马修·伊格莱西亚斯提出这个观点。

  17. 17Fast evolving diseases, like the multidrug resistant strains that essentially use hospitals as an evolutionary laboratory to continually improve their resistance to treatment, could be especially stubborn to deal with, and could be the kind of thing that prevents us from getting to 100%. ↩
    快速演变的疾病，如本质上将医院用作进化实验室以不断改善其治疗耐药性的多重耐药菌株，可能特别难以对付，可能是阻止我们达到 100%的那种事情。

  18. 18Note it may be hard to know that we have doubled the human lifespan within the 5-10 years. While we might have accomplished it, we may not know it yet within the study time-frame. ↩
    18 注意，我们可能在 5-10 年内将人类寿命翻倍，这一点可能很难察觉。虽然我们可能已经实现了这一目标，但在研究时间范围内我们可能还不知道。 ↩

  19. 19This is one place where I am willing, despite the obvious biological differences between curing diseases and slowing down the aging process itself, to instead look from a greater distance at the statistical trend and say “even though the details are different, I think human science would probably find a way to continue this trend; after all, smooth trends in anything complex are necessarily made by adding up very heterogeneous components. ↩
    这是我在疾病治愈和延缓衰老过程本身之间明显存在生物学差异的情况下，愿意从更远的角度看待统计数据趋势，并说“尽管细节不同，我认为人类科学可能会找到继续这种趋势的方法；毕竟，任何复杂事物中的平滑趋势必然是由非常异质成分的总和构成的。”。

  20. 20As an example, I’m told that an increase in productivity growth per year of 1% or even 0.5% would be transformative in projections related to these programs. If the ideas contemplated in this essay come to pass, productivity gains could be much larger than this. ↩
    作为例子，我被告知，每年生产力增长 1%甚至 0.5%将对这些计划的相关预测产生变革性影响。如果本文中考虑的想法得以实现，生产力增长可能会比这更大。

  21. 21The media loves to portray high status psychopaths, but the average psychopath is probably a person with poor economic prospects and poor impulse control who ends up spending significant time in prison. ↩
    媒体喜欢描绘高地位的精神病态者，但普通的精神病态者可能是一个经济前景不佳、冲动控制能力差的人，最终会在监狱中度过大量时间。

  22. 22I think this is somewhat analogous to the fact that many, though likely not all, of the results we’re learning from interpretability would continue to be relevant even if some of the architectural details of our current artificial neural nets, such as the attention mechanism, were changed or replaced in some way. ↩
    我认为这某种程度上类似于这样一个事实：尽管可能并非所有，但我们从可解释性中学到的许多结果，即使在我们当前的人工神经网络的一些架构细节（如注意力机制）发生变化或以某种方式被替换的情况下，仍将继续保持相关性。

  23. 23I suspect it is a bit like a classical chaotic system – beset by irreducible complexity that has to be managed in a mostly decentralized manner. Though as I say later in this section, more modest interventions may be possible. A counterargument, made to me by economist Erik Brynjolfsson, is that large companies (such as Walmart or Uber) are starting to have enough centralized knowledge to understand consumers better than any decentralized process could, perhaps forcing us to revise Hayek’s insights about who has the best local knowledge. ↩
    我怀疑它有点像经典混沌系统——被不可还原的复杂性困扰，必须以大部分去中心化的方式来管理。尽管如我在本节后面所说，可能存在更多适度的干预。经济学家埃里克·布林约尔森向我提出的反论是，大公司（如沃尔玛或优步）开始拥有足够的集中知识，能够比任何去中心化过程更好地理解消费者，这或许迫使我们重新审视哈耶克关于谁拥有最佳地方知识的见解。

  24. 24Thanks to Kevin Esvelt for suggesting this point. ↩
    感谢 Kevin Esvelt 提出这个观点。

  25. 25For example, cell phones were initially a technology for the rich, but quickly became very cheap with year-over-year improvements happening so fast as to obviate any advantage of buying a “luxury” cell phone, and today most people have phones of similar quality. ↩
    例如，手机最初是富人的技术，但很快随着每年技术的飞速进步而变得非常便宜，以至于购买“奢侈品”手机的任何优势都消失了，如今大多数人都有质量相似的手机。

  26. 26This is the title of a forthcoming paper from RAND, that lays out roughly the strategy I describe. ↩
    这是 RAND 即将发表的一篇论文的标题，概述了我所描述的大致策略。

  27. 27When the average person thinks of public institutions, they probably think of their experience with the DMV, IRS, medicare, or similar functions. Making these experiences more positive than they currently are seems like a powerful way to combat undue cynicism. ↩
    当普通人想到公共机构时，他们可能想到的是他们与 DMV、IRS、医疗保险或类似功能的经历。让这些经历比现在更加积极，似乎是一种强有力的方式来对抗不必要的怀疑。

  28. 28Indeed, in an AI-powered world, the range of such possible challenges and projects will be much vaster than it is today. ↩
    确实，在一个由人工智能驱动的世界中，这样的可能挑战和项目的范围将比今天大得多。

  29. 29I am breaking my own rule not to make this about science fiction, but I’ve found it hard not to refer to it at least a bit. The truth is that science fiction is one of our only sources of expansive thought experiments about the future; I think it says something bad that it’s entangled so heavily with a particular narrow subculture. ↩
    我违反了自己不将此与科幻小说联系起来的规则，但我发现很难完全不提及它。事实上，科幻小说是我们关于未来的扩展性思维实验的唯一来源之一；我认为它如此紧密地与一个特定的狭窄亚文化纠缠在一起，这表明了某种不好的东西。

Back to top 返回顶部