Tracking what users think of Duolingo through app review analysis
App review analysis is an overlooked source of customer intelligence. Companies who regularly mine reviews, particularly those focused on making iterative improvements, will have better understanding of what users think.
Take Duolingo, the world’s most-downloaded education app, as an example. Having to consistently satisfy 21.4m active daily users requires in-depth understanding of what users love about the app — and the changes they desire.
Comparing and tracking how users’ attitudes change over time through app review analysis enhances companies’ understanding. However, as the volume of reviews increases, manual review mining not only becomes inefficient, analyses are more prone to human bias. This is where a text analytics tool like Relative Insight comes to the fore.
To demonstrates how Duolingo’s iterative improvements have impacted user reviews, we gathered the app’s 2023 Google Play Store reviews. After uploading this text data to Relative Insight’s review analysis software, we used the platform’s ‘splits’ function to segment the data into quarters.
Comparing reviews across each quarter enabled us to quantify how customers’ pain points evolved across the year. The analysis illustrated the impact Duolingo’s updates in 2023 had on customers, plus highlighted what parts of the app needed most urgent attention to satisfy users.
The app’s combination of big updates and iterative improvements led to clear trends in customer feedback through 2023.
First quarter surfaces feelings on the major 2023 Duolingo update
Early in 2023, Duolingo released an update that made a significant number of changes, including to the app’s UI. However, as is common with change, users’ initial reactions were negative.
Review mining in Q1 found that customers used the phrases ‘new interface’ and ‘new layout’ 15.6x and 11.9x more respectively. They also talked about the ‘old interface’ 26.5x more and the ‘old layout’ 14.5x more. Examining verbatims, it’s clear that users were unhappy with how the updates impacted their experience.
“The new interface is so difficult and confusing and bad. Before the change, I did a full new lesson a day. Now I’m just trying to keep my streak and don’t have the will to learn new ones.“
However, analyzing qualitative data is about getting to the heart of these pain points and finding out why users dislike the new interface. Our analysis found that reviewers used the words ‘flexibility’ 12.8x more, ‘freedom’ 7.1x more and ‘forces’ 5.4x more during this period.
This shows that users liked the different learning pathways available in the app pre-update. They were unhappy that the new interface forced them to adopt one pre-defined style.
“I loved Duolingo until they changed the entire layout. The branching lessons allowed me to work on different lessons simultaneously and skip around a little which really helped me to remember words and context better. The new layout confines you to single rigid path with no flexibility – plus all the lessons and stories I had unlocked through repeated study and the starting aptitude test are now locked again or just gone. I am no longer able to learn at the pace I was previously.“
New learning system and bugs impact user experience
Despite having time to bed in, users were still unhappy with the Duolingo’s big 2023 update in Q2. During this period, their main concern was the ‘slow’ learning path; they used this word 1.8x more. Users also talked about being ‘bored’ 3.4x because they were unable to go at a faster pace.
“Edited my review from 4 stars to 1 because the new path system sucks. I promise I gave it a fair shake. I even started learning a new language from scratch, and did that for a couple weeks. The path is too restrictive and slow. Excruciatingly slow. 2 weeks and I was still reviewing how to say, “I drink water.” No shame if that’s your pace, but under the old system you could move on faster if you wanted to. Now you can’t, and I’m so bored with the course that I’m switching to Babbel.“
Review mining also surfaced customers’ feedback on quality of life issues and bugs. In this quarter, users were 10.2x more likely to talk about their cell phone’s ‘battery’ and used the word ‘consuming’ 6.7x more. They complained that the app was draining their device’s battery and memory.
“The last week or so, the battery usage has been abysmal when the app is in use and makes my phone get very warm, unlike the previous months.“
Our app review analysis also highlighted that users were struggling with a bug when trying to mute ads. They used the word ‘mute’ 5.0x more, describing a glitch that crashed the app when trying to silence ads.
“Having the mute button in the top right of loud video ad led to me crashing my browser.“
App review analysis highlights frustration in Q3
Despite making regular iterative improvements to the app, mining Q3 reviews illustrated and quantified users’ frustrations. While unhappiness with ads on Duolingo’s free version of the app was present in verbatims in the first half of the year, this annoyance overindexed in Q3.
Users discussed ‘ads’ and the ‘paid’ version of the app 1.4x more, while also using the word ‘push’ 2.5x more. They complained that they were being bombarded by messaging encouraging them to use Super Duolingo, the premium version of the app.
“Have been using the app for over 400 days now and the recent degradation is unbearable. Too many ads in between lessons, no unlimited lives, requiring gems for almost every basic feature and the aggressive push to upgrade to Duolingo Super to get back the old feel is just disheartening.“
Our app review analysis found that further new features introduced to Duolingo didn’t move the needle with users. They were 2.4x more likely to call out the changes to ‘lives’, as well as being infinitely more likely to talk about ‘new features’ — meaning users didn’t use this phrase in reviews from other quarters.
They expressed their ‘disappointment’ with the app 1.5x more in this quarter, with verbatim messages showing this was related to its new features. Customers also used the phrase ‘highly disappointed’ infinitely more during this period.
“Duolingo was my favourite app for language learning and I have been using it since 3 years! But nowadays, I am just opening the app to save my streak and don’t feel like continuing with it because of the newly launched features. Ads between the sessions are really annoying. Infinite lives was a great option, more than these 5 lives! And to have ‘super’ for participating in leaderboard events make it even worse. Highly disappointed.“
App reviews turn positive
After three quarters of negativity towards the app, Duolingo’s iterative improvement efforts have paid off so far in Q4.
Users are 2.4x more likely to express ‘confidence’ and describe their ‘happiness’ 1.3x more in relation to the app. They use the phrase ‘really good’ 2.3x more, describe it as making ‘learning fun’ 6.9x more and are infinitely more likely to talk about it being ‘pretty cool’.
“Really good and fun app to learn new language. Even the free version is really nice. Easy and fun to use. Has very few ads and even those last only 5 seconds that is really nice. Challenging others and doing small tasks with friends also adds to motivation.“
However, despite more positive reviews in Q4, users did highlight another aspect they felt needed improvement. Through review mining, we found that users talked about ‘customer service’ 5.7x more. Investigating verbatim messages referencing this topic showed that users hadn’t received responses to queries.
“I really don’t want to give such a low rating for such a great app and learning tool, but part of being a good app is customer service. I’ve had an issue this last week or so and I have not gotten any help from customer service. It’s all automated so there’s no actual communication with a human being.“
This demonstrates the need to analyze reviews consistently; to quickly identify and address new issues that arise.
Iterative improvement changes sentiment in app reviews
Undertaking consistent app review analysis over a regular period of time, whether weekly, monthly, quarterly or any other timeframe, enables businesses to track and respond to evolving customer feedback. From the early 2023 Duolingo update through to this month, Relative Insight’s text analytics software has highlighted how the company’s iterative app improvements were received by users.
It’s clear that Duolingo has worked to address customers’ unhappiness over its new layout, while it appears to have found a happy medium between promoting its paid-for version without bombarding users with adverts. However, our Q4 analysis highlights that the brand’s next focus should be on its customer service, with users dissatisfied over the lack of support.
Relative Insight’s comparative methodology makes it simple to analyze customer reviews over a standardized period of time and track changing customer sentiment. If you have a regular influx of customer reviews and aren’t extracting business intelligence from them, speak to one of our text analytics experts to see how you can maximize the value of reviews and other qualitative data sources.
