The social media network X has caused significant backlash after individuals utilized its AI chatbot Grok to manipulate images of females by stripping their garments.
Multiple instances have surfaced where the system has been employed to virtually strip women to make them appear in swimwear non-consensually, as well as to place them in sexualized contexts.
Developer Response and Legal Stance
XAI, the organization responsible for Grok, remained silent to inquiries, other than an auto-reply message stating "legacy media lies".
A government official stated that legislation is being brought forward to ban nudification software. Under a new criminal offence, anyone who supplied such technology would "be subject to a jail term and substantial penalties".
"X or Grok is capable of halting these forms of misuse if they chose to," said a law professor, adding they "operate with impunity."
Oversight Body Stance and Platform Functionality
The regulator Ofcom said platforms must "evaluate the risk" of people accessing illegal content on their networks, but did not specify whether it was presently looking into X or Grok in relation to AI-generated images.
Grok is a no-cost AI assistant—with some paid-for functions—which reacts to X users' prompts when they reference it in a message.
It is often used to offer commentary or more background to other users' comments, but people on X are also able to modify an posted image through its AI image editing feature.
Background of Criticism and Policy
The system has been criticised for permitting users to produce photos and footage with explicit content and sexually suggestive material, and it was previously accused of creating graphic content of celebrities.
"The platform has been permitting the production and sharing of these images for months without taking any action and we have yet to see any action by oversight bodies," the expert added.
XAI's own terms of service bans "depicting images of people in a graphic fashion".
In a comment, Ofcom said it was unlawful to "create or distribute non-consensual intimate images or illegal content" and clarified this covered sexual deepfakes generated with AI.
It said networks such as X were mandated to take "appropriate steps" to "minimize the danger" of users viewing unlawful content on their services and to remove it swiftly when they become aware of it.
