Forget data labeling: Tencent’s R-Zero shows how LLMs can train themselves

A new training framework developed by researchers at Tencent Ai Lab And Washington University in St. Louis enables great language models (LLMs) to improve themselves without necessary any data labeled by people. The technology, called R-ZeroMakes reinforcement learning to generate its own training data completely, whereby one of the most important bottlenecks is tackled in creating self-inventing AI systems. R-Zero works by having two independent models evolve together by communicating each other and challenging each other.

Experiments show that R-Zero significantly improves reasoning opportunities in various LLMS, which could lower the complexity and training costs of advanced AI. For companies, this approach could accelerate the development of specialized models for complex reasoning tasks without the enormous costs for compiling labeled data sets.

The challenge of self -evolving LLMS

The idea behind Self-Vulving LLMS is to create AI systems that can generate, refine and learn from their own experiences autonomously. This offers a scalable path to more intelligent and capable AI. A big challenge, however, is that training these models requires large quantities of high -quality tasks and labels that act as supervisory signals for the AI ​​to learn from.

Trust in human annotators to make this data is not only expensive and slow, but also creates a fundamental bottleneck. It effectively limits the potential possibilities of an AI to what people can teach it. To tackle this, researchers have developed label -free methods that directly deduce remuneration signals from the own outputs of a model, for example by measuring trust in an answer. Although these methods eliminate the need for explicit labels, they still rely on an existing series of tasks, which limits their applicability in real self -evolving scenarios.

Other approaches include having models that generate their own tasks to learn from. In domains such as an open reasoning, where there is no easy way to check for correctness (such as a code executor), it is an important obstacle of the quality of this self-generated data.

How R-Zero works

R-Zero is a framework that is designed to train reasoning LLMs that can evolve from zero external data. The process starts with a single basic model, which is split into two roles: a “challenger” and a “solution”. These two models are independently optimized but evolve together through a continuous cycle of interaction.

The aim of the challenger is to make new tasks that are only on the threshold of the current skills of the solver, neither too easy nor impossible. The solver in turn is rewarded for solving these increasingly complex tasks. In written comments to Venturebeat, Chengsong Huang, co-author of the newspaper and a doctoral student at Washington University in St. Louis, explained that this dynamic is crucial because generating high-quality questions is often more complicated than finding the answers.

“What we found in a practical setting is that the biggest challenge is not generating the answers … but rather generating high -quality, new and increasingly difficult questions,” said Huang. “We believe that good teachers are much rarer than good students. The co-revolutionary dynamic automates the establishment of this ‘teacher’, so that a stable and dynamic curriculum is guaranteed that the possibilities of the solver push far beyond what a static, already existing dataset could achieve.”

As soon as the challenger generates enough questions, they are filtered for diversity and compiled in a training dataset. In the training phase of the solver, it is refined about these challenging questions. The “correct” answer for each question is determined by a majority of votes from their own earlier attempts by the solver.

This entire process repeats itself and creates a self -strengthening loop that works without human intervention, so that the two models can push each other to become more and more capable of each iteration.

R-zero in action

The researchers tested R-Zero on various open-source LLMs, including models of the QWEN3 and Octothinker families. They first trained the models about math problems and then tested whether the learned reasoning skills could generalize to other complex, general benchmarks such as such as such as MMLU-PRO (Multi-language understanding and reasoning tasks) and SuperGPQa (Science and reasoning tasks).

The results showed that R-Zero is a very effective, model-agency framework. For example, it increased the score of the QWEN3-4B-Base model by +6.49 on average in the mathematics reasoning benchmarks. The training process improved the performance consistently and substantially, with a profit that accumulated over different iterations. In the larger QWEN3-8B-Base model, the average mathematical score was climbing with +5.51 points after three iterations.

An important finding was the immediate jump for performance after the first iteration, which validated the effectiveness of the role of the challenger in creating a high -quality learning curriculum. “This confirms that the intelligent curriculum generated by the RL trained challenger is considerably more effective than that of a non-trained generator,” the researchers write in their paper.

In particular, the skills that were learned from math problems were effectively transferred to general reasoning tasks, which improved the underlying possibilities of the models. For example, the same QWEN3-4B-Base model showed an improvement of +7.54 at reasoning benchmarks for the general domain. Another interesting finding is that R-Zero can serve as a decisive step for the training. Models were improved for the first time by R-Zero, achieved even higher performance when later a refined traditional labeled data, which suggests that the framework acts as a performance amplifier.

For companies, the “from Nul data” approach can be a game changer, especially in niched domains where there are high-quality data or not. Huang emphasizes that the most important advantage of R-Zero is the ability to circumvent the most expensive and time-consuming part of AI development: data management.

“Our approach completely bypasses the fundamental bottleneck of high -quality data sets, labeling and compiling,” he said. “This is not just about a cost -saving measure; it is a path to creating AI that can surpass human possibilities, because it is no longer limited by the scope of human knowledge or data.”

However, the co-revolutionary process also revealed a crucial challenge. As the challenger successfully generates more and more difficult problems, the ability of the solver to produce reliable “correct” answers through the majority voice. The researchers discovered that the actual accuracy of these self -generated labels fell from 79% in the first iteration to 63% by the thirdCompared to a strong Oracle LLM such as GPT -4. This decrease in data quality is an important assessment and a potential bottleneck for the long -term performance of the system.

Huang acknowledged that this is a fundamental problem for self -evolving paradigm. “Our work is a proof of concept that demonstrates the potential of this approach, but we acknowledge that maintaining stable, long -term improvement without plates is an important obstacle,” he said. “Solving this problem will be a crucial next step for the entire research community.”

The researchers also emphasize an important limitation of the framework: the current mechanism is most suitable for domains such as mathematics where correctness can be objectively determined. So, how can this powerful paradigm be extended to more subjective industries such as generating marketing copy or summarizing reports?

Huang suggests that a potential path is ahead to add a third, co-inventing AI agent to the mix: a “verifier” or “critic”.

“Instead of evaluating to a simple ‘correct’ answer, this verifier would be trained to evaluate the quality of the output of the solver on the basis of more nuanced criteria,” he explained. “The co-evolutionary dynamics would then entail the challenger who creates the prompt, the solver that generates the reaction and the verifier that offers a quality signal, where all three models improve together.”

Although this remains a direction for future research, it points to a future where completely autonomous AI systems can not only control objective logic, but also subjective reasoning.