We analyzed 29,000 reviews about 784 companies at Capterra, a site for reviews about B2B software. Our AI detector examined the text of reviews to find how much content was likely generated by humans or AI.

Summary of Key Findings:

1. More than 30% of reviews on Capterra since the launch of ChatGPT are likely to be AI-generated. This represents an increase of 224%.
2. High ratings of 4 and 5 were 2.1 times more likely to have AI-generated reviews.

Study Method and Data:

1. The text from 29,000 Capterra reviews that were at least 100 words long were rated by Originality.AI’s AI detector as human or AI generated.
2. We tracked the percentage of reviews rated as AI-generated over time.
3. The product rating given along with each review was analyzed along with whether the review was written by a human or AI. The relationship between ratings and AI-generated text was analyzed. 

AI in Capterra Reviews Study Findings

More than 30% of Reviews on Capterra, since the launch of ChatGPT, are suspected of being AI generated

Our AI detector examined the text of reviews to find how many of them were likely generated by humans or AI. We calculated what percent of reviews posted each month were AI-generated and tracked how that percentage changed over time. The amount of AI-generated reviews grew year after year as GPT-2 and GPT-3 were introduced. Then there was an immediate spike when ChatGPT became available on November 30, 2022. The  14.8% of reviews were AI-generated in November 2022 and then shot up to 29.7% in December 2022. The first 11 months of 2002 averaged 15% of reviews being from AI, while December 2022 through September 2023 averaged 33.6%, an increase of 224%.

Note that the ~5% of reviews that were identified as AI generated before 2020 represent false positives that occur with AI detectors. Reference our accuracy study for more details - https://originality.ai/ai-content-detection-accuracy/

High Ratings of 4 or 5 Were 2.1 Times More Likely to Be AI-Generated

Capterra ratings are on a scale of 1 to 5. We found that 17.9% of reviews with a rating of 4 or 5 were written by AI, and that 8.4% of reviews with ratings of 1 to 3 were after the launch of ChatGPT. This difference made higher ratings 2.1 times more likely to have AI-generated reviews than lower ratings.

Summary

Using AI to generate text for reviews presents challenges for everyone trying to use those reviews to make decisions, whether for personal or business uses. How do you feel about a review that was AI-generated? There could be times when it could be appropriate to use ChatGPT to improve readability, but is that what is happening in more than 30% of Capterra reviews? Do you want to feel like you’re in a Turing test every time you read a review? The findings in this study show how detecting AI can change how we use sites like Capterra.