AI Tools May Be Highly Accurate, But Not Foolproof

It is estimated that about two-thirds of teachers report regularly using tools for detecting AI-generated content. At that scale, even tiny error rates can add up quickly and this was seen in one case.

Moira Olmsted was taking some time off from college early in the pandemic to start a family. When the threat subsided, she was eager to return to online school.

For months, she juggled a full-time job and a toddler to save up for a self-paced program that allowed her to learn remotely. Seven months pregnant with her second child, Olmsted enrolled in online courses at Central Methodist University in 2023, studying to become a teacher.

Just weeks into the fall semester, Olmsted submitted a written assignment in a required class—one of three reading summaries she had to do each week. Soon after, she received her grade: zero. When she approached her professor, Olmsted said she was told that an AI detection tool had determined her work was likely generated by artificial intelligence. In fact, the teacher said, her writing had been flagged at least once before.

For the 24-year old Olmsted, the accusation was a "punch in the gut." It was also a threat to her standing at the university. "It’s just kind of like, oh my gosh, this is what works for us right now—and it could be taken away for something I didn’t do," she says.

Olmsted disputed the accusation to her teacher and a student coordinator, stressing that she has autism spectrum disorder and writes in a formulaic manner that might be mistakenly seen as AI-generated, according to emails viewed by Bloomberg Businessweek. The grade was ultimately changed, but not before she received a strict warning: If her work was flagged again, the teacher would treat it the same way they would with plagiarism.

The best AI writing detectors are highly accurate, but they’re not really foolproof.

Businessweek made a test on two of the leading services — GPTZero and Copyleaks —o n a random sample of 500 college application essays submitted to Texas A&M University in the summer of 2022. These documents were obtained before the release of ChatGPT, effectively guaranteeing they weren’t AI-generated.

The essays were also collected through a public records request, meaning they weren’t part of the datasets on which AI tools are trained. Businessweek found the services falsely flagged 1 percent to 2 percent of the essays as likely written by AI, in some cases claiming to have near 100 percent certainty.