🔗 Share this article

Tech firms and child protection agencies will receive authority to assess whether artificial intelligence systems can generate child exploitation material under recently introduced British legislation.

Significant Rise in AI-Generated Harmful Content

The announcement came as revelations from a protection watchdog showing that reports of AI-generated CSAM have more than doubled in the past year, rising from 199 in 2024 to 426 in 2025.

New Regulatory Structure

Under the changes, the government will allow designated AI developers and child safety groups to inspect AI systems – the underlying systems for conversational AI and image generators – and ensure they have sufficient safeguards to prevent them from producing depictions of child sexual abuse.

"Fundamentally about preventing abuse before it occurs," stated the minister for AI and online safety, noting: "Experts, under strict protocols, can now identify the danger in AI systems promptly."

Tackling Legal Obstacles

The changes have been introduced because it is illegal to produce and own CSAM, meaning that AI developers and others cannot generate such content as part of a evaluation regime. Until now, officials had to wait until AI-generated CSAM was published online before addressing it.

This law is aimed at preventing that issue by enabling to stop the creation of those materials at their origin.

Legislative Structure

The amendments are being introduced by the government as revisions to the criminal justice legislation, which is also implementing a prohibition on owning, creating or distributing AI systems developed to create exploitative content.

Practical Impact

This recently, the minister toured the London headquarters of a children's helpline and listened to a simulated call to counsellors involving a account of AI-based exploitation. The interaction depicted a adolescent requesting help after being blackmailed using a sexualised AI-generated image of himself, created using AI.

"When I learn about young people facing extortion online, it is a cause of extreme anger in me and rightful anger amongst families," he stated.

Alarming Data

A leading internet monitoring organization stated that instances of AI-generated exploitation material – such as webpages that may contain multiple images – had more than doubled so far this year.

Instances of category A content – the most serious form of abuse – increased from 2,621 images or videos to 3,086.

• Female children were predominantly targeted, making up 94% of illegal AI images in 2025
• Portrayals of infants to toddlers rose from five in 2024 to 92 in 2025

Industry Response

The legislative amendment could "constitute a vital step to ensure AI products are secure before they are released," commented the head of the online safety organization.

"AI tools have enabled so victims can be targeted repeatedly with just a few clicks, providing offenders the ability to create possibly limitless amounts of advanced, lifelike child sexual abuse material," she continued. "Content which additionally commodifies survivors' trauma, and makes children, particularly female children, more vulnerable both online and offline."

Counseling Interaction Data

Childline also published information of counselling sessions where AI has been referenced. AI-related risks discussed in the conversations comprise:

• Employing AI to rate weight, body and looks
• Chatbots dissuading young people from consulting safe guardians about abuse
• Facing harassment online with AI-generated material
• Online blackmail using AI-manipulated pictures

Between April and September this year, Childline conducted 367 counselling sessions where AI, conversational AI and associated terms were discussed, four times as many as in the equivalent timeframe last year.

Fifty percent of the references of AI in the 2025 sessions were related to psychological wellbeing and wellness, encompassing using AI assistants for support and AI therapy apps.