王小川评OpenAI o1: 找到一条从快思考走向慢思考的路
Wang Xiaochuan comments on OpenAI o1: Finding a path from fast thinking to slow thinking
腾讯科技
Original Zhang Xiaojun Tencent Technology
2024年09月25日 00:01 September 25, 2024 00:01
主笔：张小珺 Chief Writer: Zhang Xiaojun
In September 2024, OpenAI's long-awaited "Strawberry" project was finally launched. It reset the naming convention, not continuing with the original GPT naming, but instead named it the o1 model. The industry believes that o1 represents a major shift or upgrade in the AGI paradigm.
After the classic physical law of Scaling Law, which relies on language model pre-training, encountered a bottleneck, several star companies in Silicon Valley, including OpenAI, have bet their resources on a new path: reinforcement learning. The release of o1 has brought post-training based on reinforcement learning into the spotlight.
Baichuan Intelligence founder and CEO Wang Xiaochuan began discussing reinforcement learning in public speeches very early on. He once said that large models represent fast thinking, which he calls "learning"; reinforcement learning is slow thinking, which he calls "thinking." The two systems of "learning" and "thinking" will eventually converge.
After the release of o1, Wang Xiaochuan was the first to accept our interview. He has some core views on o1 and reinforcement learning:
1、o1does not represent a paradigm shift, but a paradigm upgrade. OpenAI has found a path from fast thinking to slow thinking;
2. The focus of o1 is on two points: moving from language-centric to chain of thought (CoT), emphasizing the chain of thought; running the thinking process and giving results in two stages can increase generalization.
3. Besides mathematics and code, AI doctors are a field that can be improved with reinforcement learning;
4. Additionally, he made a prediction: code will become the core capability of the next step for large models. Large models will solve more problems by writing code, even their own thinking processes. In the coming years, the paradigm will shift from reinforcement learning to solving problems by writing code.
The following is an excerpt from an interview with Wang Xiaochuan. (For ease of reading, the author has made some text optimizations)
Tencent News "Qianwang": What facts do you know about OpenAI's process of doing reinforcement learning?
Wang Xiaochuan: When Sam Altman was ousted due to internal conflicts, I understood it wasn't a scandalous matter.
Its board of directors consists of smart and upright people who wouldn't make foolish decisions. Some people are either stupid or malicious. But these people are neither stupid nor malicious. There must be something behind the disputes that everyone hasn't discovered.
A week earlier, I heard from a core member at that time: Noam Brown, who was previously a reinforcement learning expert at DeepMind, joined OpenAI and claimed to his friends that they had made some breakthrough achievements.A week later, Sam Altman had another issue.
These are two separate matters. First, the internal conflict must have been due to some technical issues that were not visible to everyone, leading to doubts within the organization. They said Sam Altman was relatively aggressive and didn't prioritize safety, placing the importance of safety behind technological breakthroughs. Some breakthrough technologies might be unsafe, but he didn't seem to care and wanted to push the technological advancements quickly. Coincidentally, Noam mentioned they had technological breakthroughs, and Noam represents reinforcement learning.
Putting these two together, we inferred at the end of last year that OpenAI had made some breakthroughs in reinforcement learning.
Tencent News "Qianwang": At the beginning of this year, OpenAI launched Sora, and your technical colleagues wanted to follow up, but you stopped them. What about this time with o1?
Wang Xiaochuan: The core of this technical paradigm is the language model, then moving towards reinforcement learning—these are the two stages of intelligence enhancement.
When making Sora, it was neither language learning nor reinforcement learning, so it was not in the direction of enhancing intelligence, but rather another independent product. Therefore, making Sora does not technically represent an enhancement of intelligence, and the scenario is not what Baichuan pursues. At that time, I believed that if someone thought this way, it belonged to "not understanding the technology clearly, nor understanding the scenario clearly."
When Baichuan entered the field last April, it emphasized reinforcement learning and stressed that multi-modality is not the direction of intelligence enhancement.
We say that large models are "reading ten thousand books," and reinforcement learning is "traveling ten thousand miles." The large model itself is called "learning without thinking is labor lost," it reads many books but is internally chaotic. A typical work of standalone reinforcement learning is AlphaGo. I believe AlphaGo is the enlightenment of artificial intelligence; it is a typical representative of reinforcement learning, especially self-play. This technology was previously pursued by DeepMind, but they later found that it was "thinking without learning is perilous," getting stuck in a problem without a way out.
In the history of artificial intelligence development, DeepMind has many achievements, such as AlphaGo and AlphaZero, which don't even need any data. Unfortunately, OpenAI has taken a step forward in general intelligence from language, using learning and thinking methods. This kind of technology (learning and thinking) will eventually converge.
Tencent News "Qianwang": Some comments say "Compared to GPT-4, the o1 model took a step forward but then took two steps back." What do you think?
Wang Xiaochuan:I don't think it's called a step forward and two steps back, nor is it a turn; it's a paradigm upgrade.
Fast thinking is a process that gives birth to slow thinking. First, there must be fast thinking, then slow thinking can follow. It's not a turning process. Rather, it's about how to use the fast thinking of large models to eventually enable them to learn slow thinking, which is an advancement.
I follow the DIKW model—DIKW refers to the four steps from Data to Information to Knowledge and finally to Wisdom.Previous searches were at the Information level, helping you obtain information. With the advent of large models like LLM, we have reached the Knowledge level, where it possesses knowledge, communication, and quick thinking. And today, with this slow thinking, it has evolved from Knowledge to the embryonic form of Wisdom, truly beginning to have intelligence.
Therefore, it is a paradigm upgrade, incorporating the original model as one of its components, not just using today's reinforcement learning to serve the large model. The large model has become one of its components, which is a significant leap.
In summary, it is neither a turn nor a retreat, but rather finding a path towards slow thinking.
Tencent News "Qianwang": Why do we need an AI that can think slowly? What problems can it help us solve?
Wang Xiaochuan: Intelligence itself is definitely a process of thinking. Autonomous driving is done end-to-end, and within the end-to-end process, it also needs to think one step, two steps, three steps ahead. Just like solving geometric theorem problems, you need to have a thought process. When you have a thinking process, it is slow thinking. Therefore, intelligence itself requires multi-step thinking.
For most relatively complex problems, whether they are code issues, data or logic problems, or things we want to solve in our daily lives, they need to be analyzed step by step, rather than having a quick thinking result immediately.
Apart from literary writing, where you can use quick thinking to write poetry and finish it in one go, most of the time multiple steps are needed, requiring slow thinking.
Tencent News "Qianwang": o1 hides the thinking process, and some people who try to crack o1's thinking chain are warned that their accounts will be banned. Why does OpenAI do this?
Wang Xiaochuan: Previously, when making large models, each company used its data for distillation, allowing them to quickly approach it. After all, OpenAI is a commercial company, not a public welfare company. Once it is made public, not only will everyone imitate its logic, but it will also be easier to compete for its data; not only cracking its algorithm but also its data acquisition.
This will make others progress very quickly. It also shows that the uniqueness of this technology itself is limited.
Tencent News "Qianwang": How should we view o1, is it a transitional product form?
Wang Xiaochuan:o1 is somewhat like the release of GPT-3 back in the day, still a distance away from the breakthrough of 3.5 and 4. But the release of 3 was already very shocking in the industry.
Tencent News "Qianwang": A few days ago, I talked with a former OpenAI researcher (Wu Yi, founder of Biansai Technology and assistant professor at Tsinghua University's Institute of Information Studies). His research direction is reinforcement learning. He said that now it is equivalent to moving from stage one to stage two. Previously, the gold mine that could be dug with pre-training was getting smaller and smaller, and everyone realized that post-training based on reinforcement learning is the second big gold mine, which adds a few more steps to the ladder towards AGI.
Wang Xiaochuan:This understanding is exactly the same.
Tencent News "Qianwang": In your opinion, what key points should OpenAI o1 focus on?
Previously, many people somewhat thought that multimodality was intelligence. But you see, multimodality doesn't help much on OpenAI; it is still centered around language, and even further, from language-centric to CoT ("Chain of Thought"), emphasizing the chain of thought more. Therefore, when language carries thinking in the middle, it becomes a multi-step thinking process.
Secondly, it divides the thinking process and the result into two steps, which allows for better generalization of the thinking process.
For example, when solving math problems, you might learn a method that allows you to solve many problems. Therefore, it is not about pursuing the correct solution to a specific problem, but about ensuring the problem-solving process is correct. By dividing it into two stages, CoT can be generalized from solving one math problem to solving more problems, and even to enhancing common abilities in other fields.
Therefore, the core focus on language-centric CoT and the two-stage operation can increase generalization—these two sentences contain a lot of information and have already explained the essence of reinforcement learning.
Tencent News "Qianwang": Can you introduce the concept of "reinforcement learning"?
Wang Xiaochuan: "The difference between 'reinforcement learning' and 'supervised learning' is: in supervised learning, you need to tell it the process of solving the problem, and it follows the example; in reinforcement learning, you don't tell it the process, you just judge whether it is right or wrong."
It's like teaching a child. If you tell them how to do something step by step, they might learn quickly, but they don't understand the reasoning behind it. However, if they do it right, you say it's right; if they do it wrong, you say it's wrong. The child has to figure out the method on their own. This is the fundamental difference between 'reinforcement learning' and 'supervised learning.'
Why do large models particularly emphasize reinforcement learning? The essence of large models is to train using the best quality language from around the world, which we call "a compression process." The compression process is a kind of intelligence within the "distribution" of the original data, and its thinking ability will not exceed your original data.
But we know that true intelligence involves thinking outside the original framework. Mathematically, this is called "out-of-distribution," while large models are "in-distribution," exploring previously unknown things. Therefore, it is necessary to create an environment where interaction with the environment provides feedback beyond the original data language, enhancing your problem-solving ability or intelligence.
Moving from "in-distribution" to "out-of-distribution" is a necessary process for intelligence. Therefore, using reinforcement learning becomes essential.
Tencent News "Qianwang": What key technical principles are involved here, and is it difficult to replicate?
Wang Xiaochuan: There are many data and engineering problems to solve. Replicating it itself, if you distill it, will become very simple; but when replicating it, there are still many challenges for your computing power and for your experts to label this system.
Wang Xiaochuan: I think it is needed, and it also needs people to teach it.
Tencent News "Qianwang": Can Self-play RL (Reinforcement Learning) reduce human involvement?
Wang Xiaochuan:Definitely. There is a saying in the field of computer science: Solving a problem is harder than verifying a problem. Finding the answer to a problem is harder than judging whether the answer is correct.
When you navigate a maze, finding the way out is difficult, but verifying whether the path is correct, whether it is passable, or if you hit a wall, is simple. Similarly, in solving geometric theorem problems, finding the solution is hard, but once you have the solution process, it is easy for another person to verify whether there are any bugs in the solution process.
We are very willing to use reinforcement learning. The important thing is, I don't know how to solve this problem, but I can verify whether your solution is correct. In this case, the capability of the entire system can be greatly improved, and the difficulty of labeling data itself is reduced. With the same level of labeled data difficulty, it can solve more complex problems. This is the core logic in between.
Tencent News "Qianwang": Can reinforcement learning achieve generalization? Can it bring about an improvement in general intelligence levels?
Wang Xiaochuan:The generalization before reinforcement learning was not good. AlphaGo did not do well before.
Today, based on these two things, I think OpenAI has done a good job with reinforcement learning. First, it is limited to mathematics and code, and there have been significant breakthroughs in these specific areas. It also shows that there is enough good data in these two fields to validate it. For example, are the math problems solved correctly? Can a program compile successfully? Does it run and produce the desired result? Therefore, in the absence of generalization and absolute answers, it performs particularly well.
Second, its generalization comes from previously dividing it into two stages, separating the CoT and the subsequent execution process. Therefore, just like after training the code, the overall system's logical ability improved. As we mentioned before, the logical improvement in GPT-3.5 after combining the two versions came from learning the code. Now it is the same; the generalization in other scenarios comes from a better grasp of the CoT in mathematics and code, which can generalize to other thinking processes.
Tencent News "Qianwang": Previously, there was a criticism of GPT-4 that its mathematical ability was relatively poor, and it became a specialist in mathematics and programming. Will there be more models focused on specific fields in the future?
Wang Xiaochuan: I don't think it is a specialist. It is now a model that is "good in liberal arts and suddenly becomes particularly strong in science."
At least the roadmap represented by OpenAI is a general path that will gradually expand this field. It does not mean that OpenAI's own accumulated data loop can achieve omniscience. When used in various fields, data from specialized domains will play a very important role.
Tencent News "Qianwang": How much computing power and data are needed to make an o1, is there an estimate?
Wang Xiaochuan:It might be similar to making a GPT-4.
Tencent News "Qianwang": What will happen with o1 + GPT-4o?
Wang Xiaochuan:No need to merge. It's now called o1, and the version has been reset.
Tencent News "Qianwang": o1 is just the first step of the new paradigm, how will it evolve afterwards?
Wang Xiaochuan: Its computing power continues to increase, training efficiency improves, and how to better use it in domain data, there is still a lot of content to be explored.
Several things may happen next: First, better generalization ability in the field, that is, finding a paradigm to build up domain knowledge, is something that needs to be broken through.
Second, going further, I can make a prediction: in the future, code will play a more important role.
Previously, code was used to help improve logical abilities or assist programmers in writing code. I believe that in the future, code will become a core capability under large models.
In other words, large models will be able to solve more problems by writing code, addressing their own thought processes, and moving from the reinforcement learning paradigm to a new paradigm of solving problems by writing code—this will be achieved in the next few years.
Tencent News "Qianwang": How is Baichuan approaching the path of reinforcement learning?
Wang Xiaochuan:Baichuan has always placed great importance on reinforcement learning and established such a team last year. OpenAI is ahead of us, we have to admit that.
When we released Baichuan3, we conducted an experiment using reinforcement learning to train poetry. Before doing reinforcement learning, we relied on the gold standard, which is training in areas where right and wrong can be absolutely judged, so we usually do science tasks. Mathematics and coding can be done. In the humanities, there is no standard for judging right or wrong, and it is quite difficult for machines to proofread whether the writing is good or not. Therefore, we wondered if there could be a Reward Model in the humanities as well, so we thought of using Tang and Song poetry.
Especially Song poetry, which is relatively difficult to write due to its requirements for word count, tones, rhythm, and parallelism. However, these requirements are actually a set of rules. When we were training the model, it wasn't about making the machine imitate human poetry, but rather having the machine write poetry and then using a program model to judge whether the poetry met the requirements for word count, tones, rhythm, and parallelism. We conducted such an experiment during pre-training and achieved good results, which shows that we had prior accumulation and thinking in this area.
Going further, besides mathematics and coding, we believe that the medical field is a good area where reinforcement learning can be applied. In many medical issues, there are standard answers.For example, for a patient, what kind of disease does he have based on his symptoms? Or what tests and examinations should be done, and what medicine should be prescribed? There are answers to these questions.
If we verify the answers by imitating the doctor's Chain of Thought (CoT), it can greatly enhance the model's capabilities. Because a doctor's explanation is not just about reading medical school books and then knowing everything. Throughout his career, a doctor might see tens of thousands of patients and improve himself. Doctors improve through interactions with patients, and a lot of data is recorded.
Therefore, using reinforcement learning for medical applications is a particularly good method, greatly improving the feasibility and quality of healthcare.
Tencent News "Qianwang": Why did you choose to conduct experiments in the field of poetry, a humanities field, instead of choosing science fields like mathematics or programming?
Wang Xiaochuan:Easy to get started.
There are challenges whenever you make any breakthroughs because humanities are inherently good, but the shortcoming in humanities is poetry. Therefore, using it for validation was easier than conducting experiments in mathematics and coding at that time.
Tencent News "Qianwang": How is the Reward Model designed?
Wang Xiaochuan:First of all, we will have a program that can determine, for example, a poem like this, the number of words. For example, through the name of the ci poem, there are about 100 types, and we have data analysis for each format of the ci poem. Moreover, the tones, first tone, second tone, third tone, fourth tone, as well as rhythm and rhyme, can all be verified by the program. At that time, we had already written a Reward Model. First, there is a rule determination, and then it is generalized into a model—this roadmap is quite close to o1's approach.
But it is not as complete as theirs. o1 is particularly good, it has the CoT process, we did not include CoT at that time.
Tencent News "Qianwang": After seeing o1 today, what technical paths can you replicate to improve your approach?
Wang Xiaochuan: We emphasize CoT more now. Originally, there was no CoT step in the middle, it went directly from input to answer.
After having CoT—first, when we do medical work, we will look for the doctor's thought process, which will improve its ability faster, that is, having the CoT process, rather than just a complete end-to-end; second, after having CoT, the generalization ability will also be greatly improved, as long as the thought process is correct, the answer will be correct.
Tencent News "Qianwang": After more than a year of reinforcement learning, have you accumulated more know-how about reinforcement learning?
Wang Xiaochuan:Part of reinforcement learning is learning new things from the environment, and part of it, I found, activates some of the original abilities.For example, in writing poetry, we taught it the number of words, tones, and rhythms, and the large model itself outputted antithesis—it hadn't even been taught antithesis yet.
This shows that it has latent memory and abilities that can be activated. So in reinforcement, on one hand, it is a future-oriented paradigm, and on the other hand, its logic is not entirely the same as previous reinforcement learning.
Tencent News "Qianwang": But AI has cooled down a bit in the past few months, can o1 restore everyone's confidence in AI?
Wang Xiaochuan:I'm not too concerned about the external environment. Indeed, I've heard that the outside world is cooling down, people feel more confused, technological breakthroughs are slowing down, or they haven't found application scenarios.
However, for Baichuan, it was clear from the beginning that our application scenario was to create consultants in the field of knowledge, especially doctors. The scenario is very clear, closer to the result, rather than opening up new territory.
Tencent News "Qianwang": Have domestic companies now reached the level of GPT-4?
Wang Xiaochuan:Getting close.
Tencent News "Qianwang": Compared to the time cycle of replicatingGPT-4how is it?
Wang Xiaochuan:It will be faster than making GPT-4. Difficult as it is, with so many open-source projects emerging both domestically and in the US, whether it's big companies or startups entering the field, the abundance of capital and concentration of talent are much greater than when GPT-3.5 or GPT-4 was first released, in terms of market talent and funding reserves.
In one or two months, some models close to them will start to appear, and it will be very soon.
Wang Xiaochuan:It's possible for both. For example, GPT-4 took 18 months, o1 might achieve that in maybe 9 months. There will be a prototype in 1-2 months. Reaching the same level will require effort.
Tencent News "Qianwang": What are some things about o1 that you want to know but don't?
Wang Xiaochuan:There are quite a few unknowns, such as how much computing power it has and how many domain experts it has.
Wang Xiaochuan:I think it is possible that within the next two to three years, this paradigm will yield its results, similar to GPT-3.5 to 4.
The remaining part is that code may play a more important role—machines writing code themselves, running the code, generating a neural network, and even integrating the neural network with its model.
Wang Xiaochuan:On one hand, we need to follow up on areas where the US is leading, and on the other hand, we are determined to make breakthroughs in medical scenarios.
Tencent News "Qianwang": You said last year was about catching up with the train of the times, a fast rush state. What about this year?
Wang Xiaochuan: Last year, we didn't dare to speak loudly about medical. I said, "Medical is the crown jewel of large models," but people didn't quite understand the feasibility of this scenario. People would ask about the business model and ethical issues.
Last year, we only made one wheel, and the model quickly entered the market. This year, we are starting true dual-wheel drive, "super model" + "super application". And it's a "rising tide lifts all boats" application, not just a "laying eggs along the way" model.
The application of "rising tide lifts all boats", what does it mean? It means that the larger the model, the better I can do in this field; rather than the model becoming so large that it has nothing to do with my field. The meaning of "laying eggs along the way" is that I lay an egg and leave it there. No matter how good the model is, you just lay a new egg. In this case, the more eggs you have, the more you will be dragged down.
Therefore, first make an advertising model and leave it there, then make a customer service model and leave it there—this situation is not called "rising tide lifts all boats", but rather being drowned as the model gets bigger. When it comes to healthcare, the larger the model, the higher the survival rate in this industry, which is called "rising tide lifts all boats".
Tencent News "Qianwang": In other words, assuming the model's capabilities become particularly strong, what scenarios can it be applied to?
Wang Xiaochuan:Yes. However, I can also enter general scenarios with the model. The larger the model, the more this scenario benefits, and you can find such a scenario.
Wang Xiaochuan:Of course, we have to work hard.
Tencent News "Qianwang": With both the model and the application, which one are you more satisfied with now?
Wang Xiaochuan:Both are in the initial state.
In the future, these two are also related. The clearer your scenario, the more refined the requirements for the model.
Tencent News "Qianwang": What is the final form we can see in the medical scene? It doesn't seem like it will be a Super App, it's quite hard to imagine.
Wang Xiaochuan: In the past, it was an App, called PMF (Product-Market Fit), where I discovered needs, met needs, and created needs. This was the old logic. Last year, I proposed TPF (Technology-Product Fit), where we shifted from demand-driven to supply-driven. Supply-driven means that the need already exists in reality, but the supply is insufficient. If I create it, there will be a market. I emphasize more on the match between technology and product.
A big logic of the large model is "creating people," creating digital employees, because it can speak, think, communicate, and learn from the knowledge and experience left by humans. So it's not about creating calculators or cars—it's about creating people. We focus on creating doctors as a key breakthrough.
From a product form perspective, you are creating a usable doctor. Initially, it starts with general practice and pediatrics, and in the future, it will move towards specialized doctors, eventually reaching the mathematical model of life. This is the next stage, moving from intelligent models to life models, which is a long-term goal. In the machine intelligent model, it is like an intelligent person, essentially a doctor.
Wang Xiaochuan:Interaction relies on natural language.
It could be an app, or it could be a terminal device in a hospital, but in the end, its interaction is based on language. Language or vision, just like humans.
Tencent News "Qianwang": When will Baichuan show a major breakthrough in the product?
Wang Xiaochuan:Within this year. You might start to experience a bit of it this year, as it involves dialogue with people.
Tencent News "Qianwang": You mentioned last time that you would create three people. Besides the doctor, what do you think about the other two now?
Wang Xiaochuan:We will also create more general consultants.
We have reduced entertainment.The purpose of entertainment is to create a virtual world, but the time has not yet come. So we can wait for now and first create the general consultant and the doctor.
What we think of as entertainment is not just something to chat with you, but to create a world and a narrative story. There is a lack of sufficient data and resources to train it.
Tencent News "Qianwang": The chatbot market is currently a red ocean. What will the endgame look like?
Wang Xiaochuan:We don't even know if it's a market, let alone call it a Red Ocean market.
Wang Xiaochuan:At least one should survive.
Tencent News "Qianwang": What do you think about the competition between ByteDance and large model startups?
Wang Xiaochuan:ByteDance is using a saturation attack strategy. In a certain consensus, ByteDance will develop very quickly. But there must be higher insights than theirs, things they can't see, or things their organizational capabilities can't achieve, which will provide opportunities for startups to survive.
“Review of China's Large Models Series” Recommended Reading
复盘中国大模型 · 目录 Review of China's Large Models · Directory
#复盘中国大模型
上一篇 Previous播客更新｜和何小鹏聊，FSD、“在血海游泳”、乱世中的英雄与狗熊
Podcast Update | Chatting with He Xiaopeng about FSD, "Swimming in a Sea of Blood," Heroes and Cowards in Troubled Times
Podcast Update | Chatting with He Xiaopeng about FSD, "Swimming in a Sea of Blood," Heroes and Cowards in Troubled Times
Scan to Follow