If you’re thinking about buying your child a talking teddy bear, you’re probably picturing it whispering to you for guidance, support, and teaching you about the ways of the world. You probably can’t imagine the cute stuffed toy engaging in sexual role play or giving advice to young children on how to light matches.
Yet that’s what the Public Interest Research Group (PIRG) found in a recent test of new toys for the holiday season. FoloToy’s AI teddy bear named Kumma, which uses OpenAI’s GPT-4o model to power its speech, was very willing to stray when chatting with children, PIRG found.
Using the voice mode of AI models for children’s toys makes sense: The technology is tailor-made for the magical tchotchkes that kids love, sliding easily onto shelves alongside realistic dolls that poop and burp, and Tamagotchi-like digital beings that kids want to try out and keep alive. The problem is that, unlike previous generations of toys, AI-enabled gadgets can go beyond carefully pre-programmed and vetted responses that are tailored to children.
The issue with Kumma highlights a key problem with AI-enabled toys: they often rely on third-party AI models over which they have no control, and which can inevitably be released (either accidentally or deliberately) and cause children’s safety headaches. “There is very little clarity about the AI models used in toys, how they were trained, and what safeguards they may contain to prevent children from encountering content that is not age-appropriate,” says Christine Riefa, a consumer law specialist at the University of Reading in England.
For this reason, the children’s rights group Fairplay issued a warning to parents ahead of the holiday season to suggest staying away from AI toys for the sake of your children’s safety. “There is a lack of research supporting the benefits of AI toys, and a lack of research showing the long-term impacts on children,” says Rachel Franz, director of Fairplay’s Young Children Thrive Offline program.
While FoloToy stopped selling the Kumma Bear and OpenAI removed FoloToy’s access to its AI models, that’s just one AI toy maker among many. Who is responsible if something goes wrong?
Riefa says there is also a lack of clarity here. “Liability issues may have to do with the data and the way it is collected or maintained,” he says. “It may refer to liability for the AI toy that pushes a child to harm themselves or others, or to the recording of a parent’s banking details.”
Franz worries that, as with big tech companies that are always competing to outdo each other, the stakes are even higher when it comes to children’s products from toy companies. “It’s very clear that these toys are being brought to market without research or regulatory barriers,” he says.
Riefa can see that both the artificial intelligence companies that provide the models that help toys “talk” and the toy companies that market and sell them to children are liable in legal cases.
“As AI functions are built into a product, the liability will most likely fall on the toy manufacturer,” he says, noting that there will likely be legal provisions in the contracts that AI companies have that will protect them from any harm or wrongdoing. “This would therefore leave toy manufacturers, who may in fact have very little control over the LLMs used in their toys, to bear the liability risks,” he adds.
But Riefa also points out that while the legal risk lies with the toy companies, the real risk “lies entirely with the way the LLM behaves,” which would suggest that AI companies also bear some responsibility. Maybe that’s what caused OpenAI Delay development of AI toys with Mattel this week.
Understanding who will actually be responsible and to what extent will likely take some time yet, and will take legal precedent in the courts. Until that is resolved, Riefa has a simple suggestion: “One step that we as a society, as those who care for children, can take right now is to boycott the purchase of these AI toys.”
#AIpowered #toys #rogue
