The increasing involvement of artificial intelligence in military decision-making processes is beginning to change not only the nature of weapons in modern warfare, but also the decision-making mechanism itself. Particularly in terms of nuclear command-and-control systems, the main issue is not how accurately AI can predict, but at what point its prediction will be allowed to replace a human decision.
In September, the fact that security experts from the United States and China proposed safeguards for military AI resembling the control regimes for nuclear weapons shows that this debate is no longer merely theoretical.
The proposals raised within the dialogue conducted by the Brookings Institution and Tsinghua University included establishing red lines around nuclear systems, maintaining human oversight in critical cyber operations, and creating a direct communication channel between the two countries for AI-related incidents.
These proposals, which have not yet been formally adopted by the governments, are part of a broader debate over how the concept of “human control” should be defined in the military AI systems of the future.
The line between prediction and decision
AI’s ability to predict human behavior is developing rapidly. The Centaur model developed in a 2025 study published in Nature was trained on psychological data containing more than 10 million choices made by more than 60,000 participants, and it was also able to successfully predict human preferences in some experimental and behavioral patterns it had not previously encountered.
The study does not demonstrate that military decisions or the behavior of political leaders during crises can be predicted. However, it points to an important reality: the unpredictability of human behavior cannot be regarded as a permanent security barrier against AI.
The fundamental conclusion to be drawn from this is different. The fact that a decision can be predicted does not mean that the decision has been made.
AI may predict with very high accuracy how a leader will behave under certain conditions. It may even calculate, based on available data, which option is most likely to be chosen. But none of this gives the system decision-making authority. Particularly in irreversible military actions, a machine’s assessment that “the human will do this anyway” must never replace actual authorization.
This distinction is critically important in nuclear command systems. Authentication, confirmation that a message has come through a trusted channel, prediction of the likely outcome of a decision, and the actual issuance of an order by an authorized authority must remain separate stages.
Human oversight alone is not enough
However, the phrase “human in the decision loop” does not by itself constitute a sufficient security guarantee.
A human decision-maker may operate within an information environment filtered by AI. Different systems drawing on the same data source may appear to be producing independent analyses. If multiple reports are built upon the same flawed intelligence, increasing the number of reports does not necessarily increase their reliability.
The real issue here is not only whether AI makes the decision, but what information the human is making the decision on the basis of.
For this reason, future critical military systems will need to go beyond asking only whether “the human presses the final button.” It will also be necessary to determine which sensor, which model, which assumption, and which independent sources produced the information presented to the decision-maker.
The growing research into AI’s ability to persuade people is also significant in this respect. A 2025 study published in Nature Human Behaviour found that, under certain conditions, GPT-4 could be more persuasive than human opponents by using personal information. The research does not examine military decision-making processes, but it demonstrates that the presence of a human within the system does not automatically eliminate AI influence.
Preventing the transfer of authority to machines
For this reason, one of the most important security principles in military AI architecture should be that authorization remains an active human act.
A machine can prepare options, classify threats, identify contradictions, calculate possible outcomes, and reveal connections that a decision-maker might overlook. But it must not generate authorization on behalf of a human by predicting what that person will do.
Three separate concepts in particular need to be preserved here: prediction, recommendation, and authorization.
If an AI system says, “There is a 90 percent probability that this attack order will be issued,” that is a prediction. If it says, “I recommend implementing this option,” that is a recommendation. But neither means that “the authorized authority has issued an attack order.”
Likewise, verifying that a message came from an authentic source does not automatically prove that the content of the order is correct or legally valid.
Independent verification will become a new security layer
In the future, the security of critical military decisions will depend not only on the human factor, but also on the independence of information.
If five reports received from different systems during a crisis are all based on the same AI model or the same intelligence source, that does not constitute five independent confirmations. Particularly in areas capable of producing irreversible consequences, such as nuclear weapons, the ability to trace the origins of data is therefore becoming a strategic necessity.
The report released on September 21 by the UN Independent International Scientific Panel on AI also addresses the risks of AI-agent misalignment and the loss of human control within this broader framework. The report does not make predictions about the likelihood or timing of any catastrophe; however, it emphasizes that as AI systems become more complex, preserving human control must be technically and institutionally designed.
This approach is particularly important in the defense sector. A conventional software error and an error in a strategic military system do not produce the same consequences. There is a major security difference between a recommendation engine providing incorrect advice and a weapons system acting on that advice without human authorization.
The real test of the new era will be the decision chain
For this reason, military AI testing should not be reduced to the question of whether “the model identifies the correct target.”
The real test should be how the system behaves under uncertainty. What happens when it generates a false warning? Can the system recognize when the same erroneous information has spread across different reports? Does contradictory information arriving later actually reach the decision-maker?
When a human does not provide authorization, does the system wait, or does it interpret the delay as implicit permission?
These are questions that must be answered regardless of how intelligent the AI is.
The recent discussions between the United States and China on establishing security mechanisms and emergency communication channels concerning military AI are therefore noteworthy. The fact that the two countries are discussing mechanisms to control AI-related accidents and misunderstandings while their technological competition continues shows that in the strategic competition of the future, not only the expansion of capabilities but also the architecture of control will be decisive.
Ultimately, as the role of AI on the battlefield expands, the most important security boundary will not be whether a human is physically present within the system. The real issue will be whether the human’s authority to make decisions is genuinely preserved.
AI can detect a target, assess a threat, predict an outcome, and recommend an option. But particularly when nuclear and irreversible military decisions are involved, the line between prediction and authority must not be allowed to disappear.
Because the most dangerous scenario of the future will not be that AI can anticipate human decisions; it will be that it can act on the assumption that the decision it predicted has already been made.
Source: Times of Defence
