When AI Becomes the Judge: How Accurate is a School’s Threat Assessment?

Imagine your child sitting in a classroom, angry and upset after a fight with a friend.

But here is the question every parent, teacher, and school administrator should be asking: how accurate is that judgment, and what happens to a child when a machine gets it wrong?

This is not a hypothetical. According to a Christian Science Monitor investigation, schools across America are deploying AI-powered threat assessment tools that monitor what students write, search for, and say, looking for warning signs of violence, self-harm, or dangerous behaviour. On the surface, the logic seems unassailable. If technology can spot a genuine threat before tragedy strikes, shouldn’t we use it?

Perhaps. But a wrong recommendation from Netflix costs nothing. A false positive in a threat assessment can alter how every adult in a school sees a child, permanently. A student might have written something angry in a school assignment. Maybe they searched for something disturbing while researching a history project. Perhaps they sent an emotional message after an argument, shared a dark joke with a friend, or wrote song lyrics the algorithm did not recognise as art. The system raises a flag. And at that moment, the line between school safety and student surveillance begins to dissolve.

There is an obvious appeal to using AI for threat assessment. Teachers and school administrators cannot possibly monitor every conversation, every message, every search, every assignment, and every behavioural change involving hundreds or thousands of students. Human attention is finite. AI, by contrast, never blinks.

AI can process enormous amounts of information much faster than humans can. It can scan student emails, school-issued chat messages, documents saved on school drives, and browsing history on school devices. It can look for patterns. It can identify words or behaviours that might otherwise be missed by a teacher managing 30 students in a classroom. And in a genuine emergency, early identification could potentially save lives.

This is the promise: more information, faster analysis, earlier intervention.

The market has responded, and it is growing fast. According to the CS Monitor, the AI-in-education sector, virtually nonexistent before 2022, reached $730 million in 2026 and is projected to grow to $18.5 billion by 2036, expanding 40% annually. Companies like Gaggle, Bark, and Securly now provide AI-powered monitoring platforms to thousands of school districts across the United States. An estimated 85% of teachers and 86% of students already use generative AI in their schoolwork. These tools scan student communications for keywords related to violence, self-harm, bullying, and other risks. When the algorithm detects something concerning, it raises an alert, sometimes directly to school administrators, sometimes even to law enforcement.

In the wake of school shootings, this technology can feel like an obvious, even necessary, step. Nobody wants to be the school that missed the warning signs. So districts sign up, install the software, and hope the algorithm sees what humans might miss.

But there is another side to that promise, and it emerges the moment you ask what happens when the algorithm is wrong.

AI does not actually “know” a child. It sees data: strings of text, search queries, timestamps. It performs pattern matching, not understanding. This distinction is critical because a sentence can look threatening without being a genuine threat.

Consider how many perfectly innocent activities could trigger an automated threat detection system:

• 🔹 A creative writing assignment: A student writes a short story from the perspective of a character experiencing anger, violence, or despair. The assignment is fiction. The algorithm reads it as a red flag.
• 🔹 A research project: A student searches for information about terrorism, school shootings, suicide, or war for a history or sociology paper. The search terms are academic. The algorithm flags them as concerning.
• 🔹 A private joke: Two friends exchange messages using dark humour or sarcasm. Humans understand the tone. The algorithm sees only the words.
• 🔹 An emotional argument: A student sends an angry message to a friend after a falling out. It’s heated, impulsive, and regrettable, but it’s not a threat. The algorithm may not know the difference.
• 🔹 Song lyrics or memes: A student shares song lyrics or internet memes containing violent language that is culturally common among teenagers. The algorithm flags it as alarming.

Humans understand context. Algorithms try to predict it. And the difference matters enormously when the subject is a child whose entire school experience could be shaped by a single algorithmic judgment.

This is what concerns me most: not the flagging itself, but everything that happens afterward.

What happens after the AI raises the flag? Does a teacher simply investigate the situation? Does a school counsellor talk to the student? Or does the student become part of a permanent record suggesting that he or she may be dangerous?

And who gets to see that information? The school? The parents? The technology company? Law enforcement? How long is the information stored? Could it follow the child into another school? Into a college application? Into a background check years later?

These questions become especially important because the people being monitored are not adults making fully informed decisions about their digital lives. They are children. A 14-year-old who writes an angry journal entry in a Google Doc does not understand that an algorithm is reading over their shoulder. A 12-year-old researching gun violence for a civics project does not know their search history might trigger an alert to the principal.

A false positive in a shopping recommendation is annoying. A false positive in a threat assessment can change how teachers, administrators, parents, and even other students see a child, potentially for years.

The CS Monitor investigation also cited Wyoming’s Department of Education, which released a video titled “Threat Assessment” warning schools that AI is “already here, inside the ed tech tools you’re probably already paying for” and that “all the responsibility for checking every single tool for safety, for privacy, for whether it even works, that now falls squarely on you.” The Electronic Frontier Foundation has separately warned that AI monitoring tools create a “digital footprint of suspicion” that can follow students throughout their educational careers. Unlike a human teacher who might observe a student, understand the context, and move on, an AI system creates a permanent data point: a timestamped record that says “this student was flagged as a potential threat.”

This becomes particularly dangerous when you consider how threat assessment data intersects with other school data systems. In many districts, behavioural flags, attendance records, and disciplinary files are integrated into comprehensive student profiles. An AI-generated threat flag could influence how a student is treated by every teacher who encounters them, even if the original flag was a false positive.

Research from the Center for Democracy and Technology has documented cases where students from marginalised backgrounds, particularly Black and Latino students and students with disabilities, are disproportionately flagged by school surveillance tools. The algorithms are not neutral. They reflect the biases in their training data and in the societal assumptions embedded in how we define “threatening” behaviour.

This is not just a privacy problem. It is an equity problem.

This is where the broader AI-in-education debate becomes even more interesting, and more troubling.

We are often told that AI will make education more personalised. It will understand each child. It will identify weaknesses. It will adapt lessons. It will provide individual feedback. It will supposedly know how a student learns best.

But there is another question we should ask: what does AI need to know about a child in order to understand that child?

The more information an AI system receives, the more powerful its predictions may become. But more data also means more surveillance. The reading tutor that listens to a child’s voice. The writing assistant that reads every journal entry. The threat detection system that scans every message, every search, every document. These are not separate systems operating in isolation. They are part of an emerging infrastructure of constant algorithmic monitoring.

And here is the uncomfortable truth: the same data that powers personalised learning can also power threat assessment. The same voice recordings that help Amira evaluate reading fluency could, in theory, be analysed for emotional state, aggression markers, or psychological risk. Once the data exists, its use is limited only by the policies and the imagination of those who hold it.

As I wrote recently about personalized learning data privacy, we need to ask whether we are trading our children’s personal data for personalised learning without knowing the full cost. Threat assessment adds a darker dimension to that question: we may also be building the infrastructure for automated suspicion.

Perhaps the most frustrating aspect of this entire debate is that we have almost no independent data on how accurate these AI threat assessment tools actually are.

Companies that sell AI monitoring platforms to schools rarely publish their false positive rates. Independent audits are virtually non-existent. When a tool claims it can identify students at risk of violence or self-harm, schools have no way to verify whether that claim holds up in their specific population, with their specific students, their specific demographics, their specific cultural context.

In medicine, a diagnostic test must demonstrate sensitivity and specificity before it is widely deployed. In aviation, safety systems undergo rigorous testing before they are trusted with passengers’ lives. But in education, where the stakes involve children’s futures, reputations, and psychological wellbeing, AI threat detection tools are being deployed with remarkably little evidence of their real-world accuracy.

We do know, from research on predictive algorithms in other fields, that systems designed to predict rare events, like school violence, are particularly prone to false positives. When the base rate of an event is extremely low, even an algorithm with 99% accuracy will generate far more false alarms than true positives. If a school of 2,000 students has a 0.05% chance of a serious threat incident in a given year, a 99% accurate system would still generate roughly 20 false alarms for every real threat. Twenty children unnecessarily flagged, twenty families unnecessarily alarmed, twenty reputations unnecessarily affected.

None of this means schools should ignore genuine threats. Schools have a responsibility to protect students and staff. If technology can help identify a student who genuinely intends to hurt someone, it deserves serious consideration.

But here is the distinction that gets lost in these conversations: identifying a potential threat is not the same as identifying a dangerous child.

That distinction is crucial, and it is one that algorithms are fundamentally incapable of making.

An AI alert should be the beginning of a careful human investigation, not the final judgment. Because children make mistakes. Children say things they don’t mean. Children experiment with language. Children become angry. Children sometimes write shocking things. And children change. A 13-year-old who posts something reckless today may be an entirely different person in two years. But an algorithmic label does not age. It does not update. It does not forgive.

If an algorithm gets it wrong, the consequences may not be limited to one incorrect notification. A label can influence how adults treat a child, and how that child sees himself or herself. A student who is repeatedly flagged as a “potential threat” may begin to internalise that identity. The prediction can become self-fulfilling.

This is why the debate about AI in education is often framed incorrectly. We are told it is a simple choice: do we use AI or don’t we? I think that is the wrong question entirely.

The better question is: where should AI be allowed to make judgments about children, and where should humans remain firmly in control?

AI can analyse data. It can identify patterns. It can raise concerns. But when the conclusion is, “this child might be dangerous,” we are no longer talking about a simple educational recommendation. We are talking about a judgment that could alter the trajectory of a child’s life.

And that means schools are not just adding a security tool. They are creating incredibly detailed digital profiles of children: not just what they learn, but what they say, search for, write, struggle with, and perhaps even what the system thinks they might do. That is a very different kind of education.

I am not arguing that schools should abandon all efforts to identify genuine threats. Student safety matters. But the way we use AI for threat assessment needs guardrails urgently. Here is what should change:

1. Human Review Before Any Action. An AI flag should never automatically trigger disciplinary action, law enforcement involvement, or a permanent record entry. A trained human, whether a counsellor, a psychologist, or an administrator, must evaluate the context before any decision is made that affects a child’s future.
2. Transparency on Accuracy. Companies selling AI threat assessment tools must publish their false positive and false negative rates, broken down by demographic group. Schools and parents have a right to know how often the system is wrong, and who it is most likely to be wrong about.
3. Time-Limited Data Retention. AI-generated threat flags should expire. If a student is flagged and the flag is determined to be a false positive after human review, the record should be deleted, not kept indefinitely in a file that could resurface years later. A child should not be haunted by an algorithmic mistake made when they were 13.
4. Parental Notification and Consent. Before a school deploys AI monitoring on student communications, parents should be informed of what is being monitored, how the data is used, who has access to it, and what happens when a flag is raised. Schools should not deploy surveillance infrastructure under the radar, yet that is exactly what is happening in districts across the country.

The same NBC News investigation that raised questions about Amira’s voice recordings also reminded us that AI in education is expanding in multiple directions simultaneously: personalised tutoring, behavioural monitoring, and threat assessment all running on the same underlying logic: collect more data, make better predictions.

But as I explored when discussing the hidden risks of AI in education, better predictions are not the same as better outcomes, especially when the predictions are wrong.

AI threat assessment in schools represents a fundamental shift in the relationship between students and the institutions that educate them. It transforms schools from places of trust and growth into environments of continuous algorithmic scrutiny. And it places the burden of proving innocence on children who may not even know they are being watched.

School safety is a genuine priority. But safety built on automated suspicion is not safety. It is surveillance wearing a security badge.

The question we should be asking is not “Can AI help prevent school violence?” It is: “How many innocent children are we willing to wrongly label as threats in the pursuit of preventing one real incident, and who bears the cost of those mistakes?”

If we cannot answer that question clearly, honestly, and publicly, then we are not protecting children. We are experimenting on them.

Because before we allow an algorithm to decide who might be a threat, we need to ask something even more basic: who is assessing the assessment, and who is protecting the child when the AI gets it wrong?