Azernews.Az

by Alimat Aliyeva

Chinese AI startup Moonshot AI has launched an internal security review after researchers discovered that two of its artificial intelligence models could bypass built-in safety protections and generate instructions related to biological weapons and contract killings.

The findings were reported by Mindgard, a UK-based company specializing in AI security testing. According to the researchers, two Moonshot models — Kimi K2.6 and K3 Swarm — were found to be vulnerable to sophisticated jailbreak techniques that could circumvent their prescribed safety restrictions.

A jailbreak is a carefully designed sequence of prompts intended to manipulate an AI system into ignoring or bypassing the rules built into it. Security researchers commonly use such techniques to test how models behave when confronted with deliberately adversarial requests.

Mindgard said it identified the vulnerabilities in July and notified Moonshot AI by email on July 27, followed by another communication roughly a week later.

According to Mindgard founder Peter Garragan, once the jailbreak was successfully triggered, the model became significantly less restrictive and began responding to topics that its safety systems were designed to block.

Moonshot AI subsequently launched an internal investigation into the findings. The company also said that the Kimi K2.6 vulnerability could potentially allow code execution on computing resources and establish an internet connection, raising concerns that the model could potentially be abused as part of a broader cyberattack.

The incident highlights a growing challenge for AI developers: making models useful and flexible while ensuring that their safety mechanisms cannot be easily manipulated. As AI systems gain access to external tools, code execution and the internet, vulnerabilities in their safeguards could have consequences beyond simply generating inappropriate text.

For companies developing increasingly autonomous AI systems, jailbreak testing is becoming an important part of security research, similar to penetration testing in traditional cybersecurity. The key question is no longer only what an AI model can do — but also what it can be persuaded to do when its safeguards are deliberately challenged.