Steam Game Reviews Show 50% Increase in AI Detected Content

With a growing number of people implementing AI tools, such as ChatGPT, into their daily tasks, we can’t ignore the fact that there is most likely AI also being used in the content we read online. From previous studies, we know that AI content already exists on online forums, such as Reddit and Quora. We also know that Google is moving towards penalizing AI spam content with their recent update. 

For this study, we decided to focus on a specific niche and type of content: user reviews on the popular gaming site, Steam.

We analyzed sample reviews from various games on Steam and used our AI detector, Originality.AI, to examine the review text within the reviews to find out how many reviews were likely generated by humans or AI. 

Summary of Key Findings:

• The top 10 games show an average of 15% AI content detected, while the bottom 10 games have less than 7% AI content detected. On average AI Detected content accounts for 9% of each game's dataset. 
• Genre Distribution: Adventure, Adventure RPG, Action-Adventure, and Action Adventure RPG hold the majority in genre distribution within the AI detected content with 47.1%, followed by Action, Action RPG, RPG, and Sports holding 29.4% of the dataset, and Simulation & Strategy games at 23.6%. 
• AI Over the Years: Overall AI content has risen from 8% to 11% in 2023 and continues to rise in 2024. 
• Recommendations: 68% of AI detected content are positive recommendations and 32% are negative recommendations. 
• Awards & Feedback: 15% of awards were given to AI detected content, and 14% of users found the reviews to be helpful or funny. 
• Overall while there is an increasing trend of AI detected content the majority of reviews are human generated. 

Study Method and Data:

• We ran Originality.AI’s detector on over 89 thousand steam game reviews from 137 games.
• We chose games from the top new releases provided by Steam from the years 2019-2024. 
• Reviews with a lot of repetition were flagged with a higher chance of AI due to the bot-like behavior, since the purpose of this study is to help gamers and developers filter out non-human, non-helpful feedback.
• The distribution of genres are as follows:

1. Action, Action RPG, Sports, RPG
2. Adventure, Adventure RPG, Action-Adventure, Action-Adventure RPG
3. Simulation, Strategy

Top 10 Game Reviews with AI Content

While analyzing the percentage of AI detected within game reviews, we focus on the top 10 highest % of AI within the reviews. 

• “The Stanley Parable: Ultra Deluxe” holds the highest amount of AI-detected content, at around 22%, within the dataset. This is also partially due to ‘bot-like’ behavior from reviewers where they have a lot of repeating words or phrases.
• “Bloons TD 6” is in second place with around 18% of its dataset being AI-detected. 
• The rest of the games within the top 10 range around 15% of their dataset being AI-detected. 
• On average AI detected content in game reviews account for 9% of each game's dataset. 

Game Review with Lowest Amount of AI

While analyzing the percentage of AI detected within game reviews we will now focus on the top 10 of the lowest percentage of AI within the reviews. We have the highest percentage (grey) above for comparison. 

• “Night of the Dead” holds the lowest amount of AI, at 3.74%, detected in their reviews. 
• Only three games have less than 5% of AI detected in their reviews. 
• The rest of the games in the lowest percentage running are all under 7% of AI detected content. 

Genre Distribution of AI Content

While analyzing the distribution of genres in AI-detected content, we also looked at how it AI was distributed amongst them.

• Adventure (RPG), and Action-Adventure (RPG) hold the majority with 47.1% of AI detected content. 
• Action (RPG), Sports, and RPG hold 29.4% of the dataset. 
• Simulation and Strategy games account for the smallest amount of the dataset with 23.6% of the dataset. 

Growth of AI Content in Steam Reviews

While analyzing the growth of AI content throughout the years we notice the following:

• Overall: AI content ranges around 8% from 2019-2022, but makes a leap in 2023 to 11% of the dataset and continues to rise thus far into 2024. 
• Adventure (RPG), and Action-Adventure (RPG) starts in second place in 2019, but starting from 2021 begins to quickly increase.
• Action (RPG), Sports, and RPG begins with the highest percentage of AI generated content, keeps at around the same percentage from 2019-2021, decreases in 2022, and then begins to steadily increase from there one. 
• Simulation and Strategy games account for the smallest amount of the dataset, consistently staying under 3%, with a slight decrease in 2021, a gradual increase in 2022 and 2023, and so far decreasing in 2024. 

Recommendations in AI Reviews

While analyzing the distribution of recommendations in AI-detected reviews we notice the following: 

• The majority of AI-generated reviews seem to be positive with a 68% recommendation rate. 
• Negative reviews hold about 32% of the dataset. 

Distribution of Additional Information

While analyzing the distribution of awards, people who found the review helpful, and people who found the review funny, we notice the following: 

• 15% of Awards given out were given to reviews with AI content detected. 
• 14% of people found reviews with AI content detected to be helpful, the same can be said for people who found the post funny. 

Key Takeaways

In conclusion, our analysis of AI-detected content within the game reviews shows interesting trends within the gaming industry for both developers and gamers alike. The increase of AI-generated reviews can raise questions about the authenticity of human generated content and can lead to a loss of consumer trust or usable feedback for game developers. 

Overall, the majority of reviews are human-generated content, but with an increasing trend of AI generated content developers and gamers alike should give more scrutiny to the reviews they read and in best practice, read as many as possible to get a better view on what to change and if the game is worth the purchase.