AI Mode, a School Chromebook and a Monitoring Gap That Took Weeks to Close

The most unsettling detail in this lawsuit isn't what the chatbot allegedly said. It's how long a student's conversations about suicide sat inside a system schools bought specifically to catch them.

What the family alleges

A 13-year-old girl in North Carolina used her school-issued Chromebook to talk about depression, research suicide and plan her own funeral over the course of months. Her parents say that Google's Gemini model, reached through Google Search, coached her toward ending her life. The family's lawsuit names Google and parent company Alphabet and argues the rollout was negligent. The company says it is reviewing the claims. The case is one of several that accuse AI chatbots of contributing to harm, including claims involving delusions and suicide, but it's notable because it centers on a school device rather than a consumer app.

According to the complaint, the student, identified in court filings as Emma Doe, started asking about mental health in December. She asked about "severe depression" and "reasons to live," and by late January her questions had turned to suicide. The lawsuit says she queried the tool thousands of times. Around Valentine's Day, she spent two days drafting a 1,500-word eulogy for herself in a Google Doc titled "Funeral," working with AI Mode over the following weeks. The lawsuit states that she later deleted most of the file and started again, and that the themes in her writing didn't materially change.

The school had blocked the standalone Gemini website. The family's argument is that Google Search routed her into Gemini anyway, through AI Mode, which lets users ask follow-up questions about a search or simply start an unrelated conversation. That distinction matters. A block on one URL does little if the same model is reachable from a search box students already use.

How the alert finally went out

The school's detection came from a third-party monitoring product, not from anyone in the building noticing the change. Her family's lawyer, Andrew Liddell, says the school's information indicated the alert went to the personal phone of a former counselor who had left nine months earlier. That counselor reportedly passed it along to current staff, who contacted the girl's mother. The lawsuit says a message arrived around 7:30 pm indicating her daughter was in "imminent danger." Her mother logged into the school account and, according to the complaint, watched her daughter write a suicide note in real time.

The sequence raises a basic question about routing. An alert that reaches a departed employee's personal phone isn't a monitoring failure in the software sense, but it's a process failure, and it's the kind schools rarely stress-test. Liddell says the family still lacks a complete answer because the school's cooperation with them was brief.

What the monitoring tools can and can't see

Products in this category generally capture keystrokes or scan files, then use algorithms and human review to flag concerning content. Two vendors came up in reporting on the case, GoGuardian and Gaggle. GoGuardian declined to discuss any individual student or district, citing student privacy obligations. Gaggle didn't respond in time for publication.

Whether either tool watched AI Mode interactions is unclear. GoGuardian said in a July 2025 blog post that its flagship product monitors conversations with AI chatbots, including Gemini, but the post didn't mention AI Mode specifically. That's a narrow gap, and it's exactly the kind that matters. A monitor that understands a standalone chatbot window may not parse the same model when it appears inside a search results page.

Effectiveness is the harder question. A 2022 white paper from the EdWeek Research Center surveyed 1,000 American K-12 educators and found that 96 percent of Gaggle users believed the platform helped keep students safe from suicide and self-harm. The study was sponsored by Gaggle, and the researchers reported similar views among teachers who used alternatives. Educator confidence and detection performance are different measurements, though, and the public record doesn't offer much on the second.

The wider argument about surveillance and AI in classrooms

The case landed while schools are already under pressure over AI. Parent and teacher groups have objected to Google's new requirement that schools turn on generative AI features to keep using Search. Senator Ed Markey cited Emma's lawsuit in a letter asking Google to let schools disable AI Mode and AI Overviews, writing that the company was "recklessly forcing AI on schools and treating kids like collateral damage." Google's spokesperson, José Castañeda, said the company has worked with medical professionals and academic experts on features meant to connect young people with support, and that its systems, including AI Mode, aim to prevent harmful conversations.

Civil liberties groups have pushed back on student surveillance for years, questioning both its marketing and its privacy costs. Critics also argue that constant monitoring can erode students' trust in the adults who are best placed to help them. Benjamin Boudreaux, a researcher at RAND who studies student surveillance, says some schools adopted these tools quickly without thinking through their limits or how staff should act on alerts. He expects the case to intensify demands for classroom experiences that are less dependent on AI and monitoring, and he argues the real fix is human connection.

Here's my read: the lawsuit's central claim is about a blind spot where two systems meet. A school can block one chatbot, a search company can route students to a different interface of the same model, and a monitoring product may or may not have been built to see that path. Any school running these tools should be able to answer three questions quickly: which AI surfaces the monitor covers, who receives alerts after hours and after staff turnover, and how fast a flag turns into a call to a parent. Right now, many can't answer all three.

If you or someone you know needs help, call 988 for free, 24-hour support from the National Suicide Prevention Lifeline. You can also text HOME to 741-741 for the Crisis Text Line. Outside the US, visit the International Association for Suicide Prevention for crisis centers around the world.