ChatGPT Is Convincing But Researchers Say They Can Spot AI Writing With 99% Accuracy

A group of scientists from the University of Kansas say they have developed an algorithm that can differentiate between academic writings written by ChatGPT and human writers with a 99% accuracy rate. The machine learning algorithm uses new features for discriminating academic writers from AI and says it can be done for other types of work.

The advancements in AI recently have started all sorts of conversations concerning the use of the technology. Whether it be about the dangers of AI, or simply how cool and exciting the possibilities it presents are, AI has been a central focus point for many lately. One of the areas artificial intelligence has been heavily talked about is its usage in creating written content, such as an article like this. Several companies, including ChatGPT creator OpenAI, have attempted to build AI-spotting technology that can effectively tell the difference between something written by a human and an AI author. Until now, all of those attempts have yielded poor results.

In a new research paper published in Cell Reports Physical Science, a group of scientists says they created an algorithm that takes advantage of certain habits human writers have that AI has yet to acquire. Some of those habits include things such as observing that scientists tend to write long paragraphs and have a liking for equivocal language, using words like "but," "however," and "although." It is this set of 20 features that the group says allows its algorithm to have such a high rate of success in detecting AI-written content.

Because the algorithm is written in a way to detect scientific writings, it is not effective at detecting an AI author in things like an article such as this one, or an essay by a college student on the migratory habits of moose in Alaska. However, this could change as AI advances and adapts its own writing habits.

"We tried hard to create an accessible method so that with little guidance, even high school students could build an AI detector for different types of writing," explained Heather Desaire, an author of the paper and a chemistry professor at the University of Kansas, in an interview with EurekAlert. "There is a need to address AI writing, and people don't need a computer science degree to contribute to this field."

The authors stated they sought two goals, "The first is to answer the question about the extent to which a field-leading approach for distinguishing AI- from human-derived text works effectively at discriminating academic science writing as being human-derived or from ChatGPT, and the second goal is to attempt to develop a competitive alternative classification strategy."

While the study is promising, it should be noted that it is not fool-proof, as a human could simply tweak an AI-written paper in order to subvert detection. The authors understand this, and explained, "Since the key goal of this work was a proof-of-concept study, the scope of the work was limited, and follow-up studies are needed to determine the extent of this approach's applicability."