Detecting AI Generated Text: GPTZero vs Watermarking

Detecting AI Generated Text: GPTZero vs Watermarking

Table of Contents:

1. Introduction
2. Two Ways of Detecting AI Generated Text 2.1 GPTZero Tool 2.2 Watermarking
3. Explaining GPTZero 3.1 Perplexity 3.2 Burstiness
4. Limitations of GPTZero 4.1 Spelling and Grammar Errors 4.2 Low Burstiness
5. Introducing Watermarking 5.1 How Watermarking Works 5.2 Detecting Watermarked Text 5.3 Advantages and Disadvantages of Watermarking
6. Potential Attacks on Watermarking 6.1 Word Substitutions 6.2 Emoji Attack 6.3 Limitations of Watermarking
7. Conclusion
8. FAQ

Detecting AI Generated Text: GPTZero and Watermarking

Introduction:

As artificial intelligence (AI) language models become more advanced, it is becoming increasingly difficult to distinguish between text written by humans and text generated by AI systems. This raises concerns about the authenticity and trustworthiness of written content. In this article, we will explore two methods of detecting AI generated text: GPTZero, a tool specifically designed for this purpose, and watermarking, a technique that shows promise in identifying text generated by language models such as ChatGPT.

Two Ways of Detecting AI Generated Text:

1. GPTZero Tool: The GPTZero tool measures perplexity and burstiness, two properties that vary between human-written content and machine-written content. Perplexity measures the familiarity of a produced text to a language model. Higher perplexity indicates unfamiliarity and suggests human authorship. Burstiness, on the other hand, measures sentence complexity. Humans tend to use a wide range of sentence lengths and rare words, resulting in a bumpy burstiness graph. Language models, however, exhibit more constant sentence structures. GPTZero analyzes these properties to determine the likelihood of text being AI generated.

2. Watermarking: Watermarking is an alternative method for detecting AI generated text. In this approach, the language model's output is marked with a unique fingerprint, undetectable to humans but identifiable through statistical analysis. The watermarking process involves randomly blacklisting a portion of the words in the language model's vocabulary. To detect generated text, one simply needs to count the number of blacklisted words. Text composed solely of whitelist words is highly likely to be AI generated. Watermarking shows promise in reliably detecting AI generated text, although it is not foolproof and can be vulnerable to certain attacks.

Explaining GPTZero:

1. Perplexity: Perplexity measures how probable it is for a language model to have generated a specific word given the previous words. GPTZero uses perplexity as an indicator of human-written text, assuming that higher perplexity reflects the presence of improbable words, suggesting human authorship.

2. Burstiness: Burstiness refers to the variation in sentence complexity. Humans tend to use a diverse range of sentence lengths and incorporate rare words. Language models, however, produce more consistent sentence structures. By examining the burstiness graph, GPTZero can distinguish between human-written and AI-generated text.

Limitations of GPTZero:

1. Spelling and Grammar Errors: GPTZero can be deceived by introducing small spelling mistakes or grammar errors, which can make AI-generated text appear human-written. By intentionally incorporating these errors, text can be misclassified by GPTZero.

2. Low Burstiness: GPTZero assumes that high burstiness indicates human authorship. However, if a piece of text has a low burstiness value, GPTZero might mistakenly label it as AI generated. This can lead to false positives and accusations of AI-generated content when it is actually human-written.

Introducing Watermarking:

1. How Watermarking Works: Watermarking involves adding a unique fingerprint to the language model's output. During decoding, a percentage of words are randomly blacklisted, using the last word of the input as a seed. The blacklist can be reconstructed by knowing the seed and the random number generator used. By counting the blacklisted words, one can identify text generated by the language model.

2. Detecting Watermarked Text: Watermarked text primarily consists of whitelist words since blacklisted words are avoided by the language model. Consequently, text composed mostly of whitelist words indicates AI-generated content with a high level of certainty. Watermarking also accounts for low entropy words, ensuring that important and commonly used words are not blacklisted.

3. Advantages and Disadvantages of Watermarking: Watermarking provides a more reliable method for detecting AI-generated text, with a lower likelihood of false positives. However, it relies on language model developers willingly incorporating watermarks into their models. Without mandatory regulations, not all language models will be watermarked in the future.

Potential Attacks on Watermarking:

1. Word Substitutions: Attackers can rewrite AI-generated text using different but synonymous words to bypass watermark detection. As long as humans are involved in rewriting the content, watermarking may not be able to detect the modified text.

2. Emoji Attack: Instructing the language model to add emojis or exchange letters in its output can disrupt the watermarking process. Although the resulting text may initially appear nonsensical, the attacker can automatically remove the irrelevant portions, making the watermark less effective.

Conclusion:

Detecting AI generated text is critical for maintaining transparency and trust in written content. Both GPTZero and watermarking offer potential approaches to achieve this. GPTZero relies on measuring perplexity and burstiness, while watermarking adds a unique fingerprint to the language model's output. While watermarking shows promise in reliably identifying AI-generated text, it is not immune to attacks. The implementation of watermarking and other detection methods depends on the willingness of language model developers to adopt such measures.

FAQ:

Q: Is GPTZero a reliable tool for detecting AI-generated text? A: While GPTZero can provide insights into AI-generated text, it has limitations and can be deceived by spelling and grammar errors or low burstiness values.

Q: Can watermarking be fooled or bypassed? A: Watermarking can be susceptible to attacks such as word substitutions or the "emoji attack," where the attacker instructs the language model to add emojis or modify output in strategic ways to disrupt the watermarking process.

Q: How effective is watermarking in detecting AI-generated text? A: Watermarking offers a relatively reliable way to identify AI-generated text, especially when the content includes a high proportion of whitelist words. However, it is not foolproof and can be vulnerable to certain attacks.

Q: Are there disadvantages to using watermarking? A: While watermarking provides a more reliable method for detecting AI-generated text, it depends on language model developers willingly incorporating watermarks into their models. Without regulations or widespread adoption, not all language models will be watermarked.

Q: What are some alternative methods for detecting AI-generated text? A: Besides GPTZero and watermarking, other approaches include analyzing sentence structures, comparing writing styles, or utilizing external verification sources.

Q: Will ChatGPT implement watermarking in the future? A: OpenAI has mentioned potential plans for watermarking ChatGPT or similar language models, but it remains uncertain when or if this implementation will be realized.

Q: Why is detecting AI-generated text important? A: Detecting AI-generated text helps ensure transparency, credibility, and trustworthiness in written content.