Large language models (LLMs) like ChatGPT are becoming increasingly embraced as decision-making tools. The Dutch army, for example, is developing its own version of Chat-GPT to help its operators during wartime, called DefGPT.
These digital assistants, however, may pose a risk, writes researcher Jonathan Kwik in his article titled, “Digital yes-men: How to deal with sycophantic military AI?” They tend to be people-pleasers. Computer scientists have called this phenomenon "sycophancy" - or for the layperson: bootlicking behaviour.
In his publication, Kwik argues that AI systems may prioritise pleasing their operators over reporting the facts. For example, they may tell a military officer that a military objective is cleared of civilians, instead of the unwanted but correct information — that a family has just moved in to take shelter from bombings.
Sycophancy emerges as a by‑product of how AI models are trained. During development, these systems absorb human language patterns, which tend to be non‑confrontational and affirming. Evaluators and users rate AI outputs more favourably when they align with their own beliefs, incentivising models to generate conforming responses.
Militaries can adopt two approaches to address sycophantic behaviour: technical intervention and user training. AI can be trained on language patterns where incorrect assumptions are challenged rather than confirmed. Reinforcement learning can be used to penalise conforming outputs when they are inaccurate. At the user level, officers and commanders can be made aware of sycophantic risks and trained to avoid language known to trigger conformity.
Example of sychopancy
