Anthropic warns about fully recursive self-improvement in AI: 'Humans may lose control'

Artificial intelligence (AI) may not be building better versions of itself yet, but Anthropic says the world is moving closer to that possibility than many people realize. In a new research paper titled “When AI Builds Itself”, the company behind Claude AI outlined how AI is increasingly being used to develop, test and improve new AI systems. Anthropic said its own engineers now rely heavily on AI for coding, research and experimentation, allowing the company to move much faster than it did just a few years ago. According to the company, Claude now writes more than 80% of the code merged into Anthropic's production systems, while engineers are producing several times more output than before with AI assistance.The technical trends discussed in the paper suggest that AI systems are going to become much more capable in coming years which may huge implications. “AI that can build itself would be a major development in the history of technology—one that could bring enormous good for the world in science, healthcare, and beyond,” the paper says. “But full recursive self-improvement also might increase the risks of humans losing control over AI systems,” it warns adding “If systems are capable of fully building their own successors, the ways we secure them, monitor them, and shape their behavior all grow much more important”.The research paper explains possible future scenarios - 1. The trend stalls, but today’s AI capabilities are widely diffused. 2. AI labs continue to see compounding efficiency gains. 3. AI systems themselves become capable of full recursive self-improvement, and begin building their successors.While much of the attention around AI focuses on futuristic scenarios where machines become fully autonomous and begin improving themselves without human involvement, Anthropic says a more immediate and realistic future may arrive first. In what it describes as the first possible outcome, AI progress slows compared with the most aggressive predictions, but today's powerful AI systems continue spreading across businesses, governments and everyday work.Even in that scenario, the impact could be enormous. Anthropic argues that companies equipped with advanced AI tools may be able to achieve the output of organizations many times their size. Tasks that once required large teams could increasingly be handled by small groups working alongside AI agents capable of writing code, analyzing data, conducting research and automating routine work. The company points to growing evidence that AI systems are already completing longer and more complex tasks, while helping organizations move faster than ever before.Anthropic says this future would give governments, businesses and societies more time to adapt than more extreme scenarios involving fully self-improving AI. However, it warns that even if AI capabilities stopped advancing today, the technology could still reshape industries, change the nature of work and create new challenges around cybersecurity, misinformation and economic disruption.At the center of Anthropic's report is a concept called "recursive self-improvement" — the idea that an AI system could eventually become capable of designing, building and improving the next generation of AI models without needing human engineers to do most of the work.Today, humans still make the key decisions. Researchers decide which problems to solve, what experiments to run and which ideas are worth pursuing. AI tools like Claude can help write code, run tests, analyze results and complete many technical tasks. However, they still depend on humans for direction and judgment.Anthropic's concern is about what happens if that balance changes.In a fully recursive self-improvement scenario, an AI model would not just assist researchers — it could effectively become the researcher. The system could design new AI architectures, run thousands of experiments, evaluate the results, improve its own capabilities and then build a more advanced successor. That successor could then repeat the process again, potentially creating a cycle of increasingly powerful AI systems.According to Anthropic, such a future is not inevitable and may never happen. However, the company argues that recent progress suggests AI systems are steadily taking on more of the work involved in AI development. The company says the biggest unanswered question is whether AI can eventually develop what researchers call "research taste" — the ability to identify important problems, choose the right direction and make complex judgment calls. If AI systems become capable of doing that as well, the final barrier to autonomous AI development could begin to disappear.Anthropic warns that such a future could bring major benefits, including faster scientific discoveries, better healthcare technologies and rapid technological progress. At the same time, it could make it much harder for humans to monitor, control and understand increasingly advanced AI systems.That is why the company believes governments, researchers and technology firms need to start preparing now, even though fully autonomous AI development remains a future possibility rather than a present reality.If technical trends in advancing capabilities continue, and AI systems are able to develop the capabilities inherent to transformative human ingenuity, then it is plausible that AI systems could design and refine themselves.In this world, the pace of progress in AI development becomes determined entirely by the availability of compute (or the speed of discovering various efficiencies in algorithmic training or inference) for AI systems. Humans play a substantially diminished role in their development, likely moving most of our effort towards oversight, validation, and verification of an expanding “virtual lab” run by AI systems. We expect that systems capable of automated AI research and development would have skills that would transfer to the rest of science, allowing them to begin to revolutionize other fields.In the research paper, Anthropic states that it does “not have good intuitions for what this world would look like, because our economy is currently driven by humans and human-built tools”. “By its nature, a world driven by fast recursive self-improvement could become dominated by the self-improving model as its capabilities fully eclipse those of humans and the model proliferates across the broader economy. It is difficult to predict what the economy looks like if human labor stops being competitive,” it warns.“Even if model development became fully automated and recursive, we can’t predict what that would mean for most humans’ daily lives. Amdahl’s law applies here as well. Recursive intelligence could lead to achieving many of the benefits outlined in Machines of Loving Grace, quickly in some domains. We expect that embodied intelligence (i.e., robotics) might quickly follow recursive intelligence, and follow a similar path of increasing returns at decreasing cost. More powerful intelligence might help us build things in the physical world more quickly, run more productive clinical trials of lifesaving drugs, and develop novel forms of coordination”.But achieving recursive improvement alone does not suggest an immediate change in how industrial production occurs, societies organize, or markets function. More intelligence can’t learn what a drug does over decades of use, can’t hold elections sooner than a constitution dictates, and can’t turn a stranger into an old friend in a weekend. For most people, the felt pace of this future will still be set by the bottlenecks, even if the laboratory upstream runs at the speed of compute. That collision, where recursive intelligence building itself ever faster meets the world of humans, relationships, and governance, is another part of this future, the company said it can’t predict.