Meta’s SPICE framework lets AI systems teach themselves to reason
Researchers at Meta HONEST and the National University of Singapore have developed a novel reinforcement learning framework for self-improving AI systems.
Called Self-play in Corpus environments (SPICE)the framework pits two AI agents against each other, creating its own challenges and gradually improving without human supervision.
While currently a proof-of-concept, this self-play mechanism could provide a foundation for future AI systems that can dynamically adapt to their environments, making them more robust against the unpredictability of real-world applications.
The challenge of AI self-improvement
The goal of self-improving AI is to create systems that can increase their capabilities by interacting with their environment.
A common approach is reinforcement learning with verifiable rewards (RLVR), where models are rewarded for providing correct answers to problems. This is often limited by the reliance on human-curated problem sets and domain-specific reward engineering, making it difficult to scale.
Self-play, where a model improves by competing with itself, is another promising paradigm. But existing self-play methods for language models are often limited by two critical factors.
-
Ffactual errors in generated questions and answers worsen, leading to a feedback loop of hallucinations.
-
When the problem generator and the solver have information symmetry (i.e. share the same knowledge base), they fail to generate truly new challenges and fall into repetitive patterns.
As the researchers note in their article, “These systematic empirical failures indicate that self-improvement requires interaction with an external source that provides diverse, verifiable feedback, rather than pure closed-loop introspection.”
How SPICE works
SPICE is a self-play framework where a single model fulfills two different roles.
-
A ‘challenger’ compiles a curriculum with challenging problems from a large corpus of documents.
-
A “Reasoner” then tries to solve these problems without access to the source documents.
This setup breaks the information symmetry that limits other self-play methods, as the Reasoner does not have access to the documents and knowledge that the Challenger uses to generate the problems.
Organizing the tasks in a vast and diverse corpus of documents prevents hallucinations by anchoring questions and answers in real-world content. This is important because AI systems can reliably improve themselves when they need external grounding sources. Therefore, LLM agents must learn from interactions with people and the real world, and not just from their own output, to avoid compound errors.
The adversarial dynamic between the two roles creates an automatic curriculum.
The Challenger is rewarded for generating problems that are both diverse and at the limits of the Orator’s capabilities (neither too easy nor impossible).
The Orator is rewarded for the correct answer. This symbiotic interaction forces both agents to continually discover and overcome new challenges.
Because the system uses raw documents instead of predefined question-answer pairs, it can generate different task formats, such as multiple-choice and free-form questions.
This flexibility allows SPICE to be applied to any domain, breaking the bottleneck that limited previous methods to narrow fields like math and code. It also reduces reliance on expensive, human-curated datasets for specialized domains such as legal or medical analysis.
SPIJS in action
The researchers evaluated SPICE on several basic models, including Qwen3-4B-Base and OctoThinker-3B-Hybrid-Base.
They compared performance against baselines such as the basic model without training, a Reasoner model trained with a fixed “Strong Challenger” (Qwen3-32B-Instruct), and pure self-play methods such as R-Zero and Absolute Zero. The evaluation covered a wide range of mathematical and general reasoning benchmarks.
Across all models, SPICE consistently outperformed baselines, yielding significant improvements in both mathematical and general reasoning tasks.
The results show that the reasoning abilities developed through corpus-based self-play transfer broadly across models, thanks to the diverse external knowledge corpus they used.
A key finding is that the adversarial dynamic creates an effective automatic curriculum. As training progresses, the Challenger learns to generate increasingly difficult problems.
In one experiment, the Reasoner’s success rate on a fixed set of problems rose from 55% to 85% over time, showing that its capabilities have improved.
Meanwhile, later versions of the Challenger were able to generate questions that dropped a Reasoner’s success rate from 55% to 35% early on, confirming that both roles are successfully evolving together.
The researchers conclude that this approach entails a paradigm shift in self-improving reasoning methods: from “closed self-play that often stagnates due to hallucinations, to open-ended improvement through interaction with the vast, verifiable knowledge embedded in the corpora of web documents.”
Currently, the corpus used for SPICE represents the human experience captured in text. The ultimate goal is for self-improving systems to generate queries based on interactions with reality, including the physical world, the Internet, and human interactions through multiple modalities such as video, audio, and sensor data.
