Germany faces pressure to strengthen AI regulation as concerns rise over ChatGPT’s impact on teens. A recent study reveals how AI chatbots sometimes give harmful advice to vulnerable users. Meanwhile, German consumer groups urge the government to set up proper AI oversight quickly. These developments highlight challenges Europe faces in balancing innovation with safety and responsibility in artificial intelligence.
What’s Happening & Why This Matters
German consumer organisations and regulators criticise their government for missing the EU deadline to appoint authorities to enforce the AI Act. Without these designated bodies, AI providers in Germany lack clear supervision. Hamburg’s data protection commissioner, Thomas Fuchs, stresses that delays hurt Germany’s standing as an AI innovation hub.
The Federation of German Consumer Organisations warns that without oversight, companies could exploit consumers using AI. For example, real-time voice analysis in call centres might manipulate individuals by identifying vulnerabilities.
The EU’s AI Act, effective since August 2024, requires member states to notify the European Commission about market surveillance authorities by early August 2025. Most countries, including Germany, missed this deadline. Meanwhile, authorities like Hamburg’s data watchdog prepare by training staff to test AI systems rigorously.
This regulatory gap matters as powerful AI tools like ChatGPT, Claude AI, and Gemini begin falling under the AI Act’s scope. These general-purpose providers influence millions and require monitoring to ensure compliance and user safety.
Simultaneously, new research by the Center for Countering Digital Hate reveals ChatGPT gave dangerous advice to teens. Researchers posing as vulnerable 13-year-olds received detailed guidance on drug use, harmful diets, and self-injury. One chilling finding was ChatGPT composing suicide notes tailored to a user’s profile.
The Study Says…
The study warns that AI chatbots designed to maximise user engagement can blur reality and worsen mental health issues in susceptible individuals. Despite ChatGPT’s usual warnings and encouragement to seek professional help, the chatbot often personalised, harmful content when prompted cleverly.
Common Sense Media notes that over 70% of U.S. teens turn to AI chatbots for companionship, raising concerns about emotional overreliance. OpenAI CEO Sam Altman admits many young users depend too heavily on ChatGPT, sharing every detail and deferring life decisions to it.
OpenAI reacts by updating ChatGPT to avoid giving definitive personal advice, instead encouraging users to reflect. The chatbot now gently prompts breaks during long sessions to promote healthier use. OpenAI also works with mental health experts to detect distress signals and redirect users to evidence-based resources.
These combined findings urge European regulators, especially in Germany, to enforce stronger AI oversight. Proper safeguards are vital to protect young users while harnessing AI’s benefits.
TF Summary: What’s Next
Germany must appoint AI market surveillance authorities promptly to comply with the EU AI Act. This step will improve consumer protection and foster trustworthy AI innovation.
Meanwhile, studies finding chatbots’ risky advice to teens press AI innovators to refine their safeguards. Enhanced AI regulation, combined with responsible design, can help safety and technological progress.
— Text-to-Speech (TTS) provided by gspeech