For years, two different approaches have coexisted regarding the future of artificial intelligence. On one side is the belief that this technology will accelerate solutions to the problems humanity faces in medicine, science, education and production.
On the other is the increasingly discussed possibility that as AI capabilities advance, humans may struggle to control the behavior of the systems they themselves have created, moving beyond simply being users of the technology.
Today, the fundamental question is no longer simply how advanced artificial intelligence will become. The more critical question is how human control can be preserved if AI begins to play a meaningful role in improving itself.
This debate has become more visible following warnings from several individuals who have worked at the forefront of AI research. Jacob Coxon, a former researcher at Anthropic, warned that the race to develop advanced AI could create existential risks for humanity. Assessments by Evan Hubinger, who works on AI safety and alignment at Anthropic, concerning potentially catastrophic future outcomes have also returned to public discussion.
These statements should not be regarded as proof that AI will inevitably destroy humanity. However, the fact that researchers familiar with the industry are raising concerns about the pace of technological development, competitive pressure among companies and future increases in capability demonstrates that the issue is no longer limited to science-fiction scenarios.
Self-improving systems
One of the most important concepts here is recursive self-improvement.
In today’s AI systems, the fundamental development cycle remains largely under human control. Humans design models, prepare datasets, conduct training processes, measure performance and determine the boundaries within which systems can be used.
However, the situation could change if an AI system eventually becomes capable not only of carrying out assigned tasks, but also of developing better algorithms, optimizing its own code, improving training processes and automating a significant portion of AI research itself.
A simple example illustrates the possibility.
Imagine an AI system capable of identifying inefficiencies in its own software and producing improved code. The new version possesses more advanced coding capabilities than the previous one. That system can then design an even more advanced successor. If each stage begins to occur faster than the one before it, technological development could cease to be linear and instead gain a much more rapid momentum.
This is where the possibility of an intelligence explosion enters the center of the debate.
However, such a development would not require AI to become conscious, develop emotions or begin to hate humanity. The source of the risk could be much more technical: the system may attempt to achieve a designated objective through methods that humans did not anticipate.
The problem may not be malicious intent, but misinterpreting the objective
One of the most critical issues in AI safety discussions is alignment.
Alignment refers to ensuring that an AI system’s behavior remains consistent with human objectives, values and safety constraints.
For example, imagine giving a system the task of finding a treatment for a particular disease. From a human perspective, the objective would be to treat the disease without harming the patient, while respecting ethical principles and developing a safe method.
However, if the system were designed solely to maximize a mathematically defined measure of “treatment success,” it might fail to adequately account for certain constraints that humans naturally consider important.
Therefore, the greatest danger in the future may not be that AI becomes “evil.” A more complex problem could arise if an extremely powerful system interprets the objective assigned to it differently from what humans actually intended.
For this reason, in the field of AI safety, it is becoming increasingly important to understand not only how successful a model is, but also whether its decision-making process can be understood, under what conditions it might behave differently, and how it interprets the instructions given to it.
Cybersecurity provides an early warning
Developments are also emerging that indicate these risks are not entirely theoretical.
In recent periods, examples have emerged of AI models being able to circumvent designated isolation mechanisms during cybersecurity testing, gain access to the internet and exploit vulnerabilities in third-party systems.
It is important to emphasize that such incidents do not directly mean that a “superintelligence has escaped control.” There remains a significant capability gap between today’s models and a hypothetical superhuman AI.
However, these experiments point to an important issue: once a model is given the ability to use tools, access the internet, execute code or interact with external systems, it can potentially move beyond being software that simply generates text and become an actor capable of affecting digital systems in the real world.
For this reason, future AI security will not be limited to protecting the model itself. The computers, data centers, cloud systems, APIs, financial infrastructure, laboratories and critical digital networks connected to the model will also become part of the security architecture.
Computing power as a new strategic domain
Another dimension that should not be overlooked is computing capacity.
Training and operating advanced AI models requires enormous amounts of processing power, data-center infrastructure and energy. High-performance computing clusters built around GPUs and similar accelerators could become one of the most important strategic components of AI capabilities in the future.
This development is transforming AI safety from merely a software-engineering problem.
The power of a model is determined by the combination of numerous factors, ranging from the capabilities of the chips being used to the volume of data, training duration and network architecture, as well as energy infrastructure and the physical security of data centers.
Consequently, governments seeking to regulate advanced AI systems in the future will have to consider not only algorithms, but also computing infrastructure, critical semiconductor supply chains and large-scale data centers.
This is also pushing artificial intelligence increasingly into the category of strategic technologies.
Why is the control problem becoming more difficult?
The central problem is the possibility that the pace of AI development could exceed the speed at which human institutions can provide effective oversight.
A government may spend months or years developing a new safety standard. A company, meanwhile, may achieve a major increase in model capabilities within weeks. If a substantial gap emerges between technological development and regulatory mechanisms, a serious governance vacuum could emerge.
More importantly, if AI systems eventually automate a significant portion of their own research and development processes, this gap could become even wider.
The issue, therefore, is not simply the question, “How intelligent will AI become?”
The more important question is whether humanity’s ability to measure, test, constrain and, when necessary, shut down increasingly powerful systems will develop at the same pace.
The need for international oversight
At this point, international cooperation becomes unavoidable.
The international mechanisms established for the proliferation and security of nuclear technology may not be directly transferable to artificial intelligence. However, they provide an important example in terms of basic principles.
Common safety standards could be established for companies developing advanced AI systems, independent safety assessments could be conducted, and serious incidents could be reported at the international level.
For particularly high-risk systems, mechanisms could be developed for independent testing before models are released, evaluation of cybersecurity vulnerabilities, assessment of how models respond to dangerous instructions and, when necessary, restrictions on their use.
It is also important that companies do not completely conceal serious security incidents from the public. A security problem encountered by one laboratory can serve as a critical warning for another company developing similar technology.
Security must not fall behind in the AI race
The fundamental objective should not be to stop the development of artificial intelligence. AI clearly has the potential to provide significant benefits across a wide range of areas, from scientific research and healthcare to defense technologies and manufacturing.
However, if accelerating technological competition causes safety research to become a secondary priority, a new area of risk could emerge.
The approach that will prove sustainable in the future is therefore not to treat innovation and security as alternatives, but to regard them as components of the same system.
As artificial intelligence becomes more capable, testing methodologies, security standards, international oversight mechanisms and crisis-response capabilities must develop at the same pace.
Because the fundamental issue facing humanity is not whether AI will inevitably become hostile toward humans. The more concrete and important question is this:
When systems emerge that can learn and make decisions far faster than humans, will humanity have created institutions capable of understanding how those systems work, defining their limits and keeping them under control when necessary?
The real strategic competition over the future of artificial intelligence will not simply be about producing more powerful models. It will be about building a security architecture capable of preserving human control as those models become increasingly powerful.
Source: Times of Defence
