Choros
Inspiration
Small teams, school clubs and local offices launch websites without user testing because testing with real people takes time and money. I wanted to know if AI personas could stand in for some of that early feedback, and how far I could trust what they say.
What it does
You describe a site and who uses it, for example "Test the IRS website with 5 taxpayers." Choros creates fictional personas (a retiree, a gig worker, a recent immigrant), opens a real browser for each one, and lets them use the site on their own. After each visit, the persona writes what confused or frustrated them, and every point is tied to a recorded step and screenshot. On public websites it only reads and navigates. It never submits a form, signs up or buys anything.
How I built it
Question: do AI personas find the same problems that real users complain about?
Data: I pulled 500 public App Store reviews of IRS2Go, the official IRS app, from January 2021 to November 2026. I kept the 411 rated 1 to 3 stars and sorted each into one of 8 themes, such as refund status, identity verification and errors. The top three were errors and outages (24%), refund status (22%) and identity verification (21%). One limit: people who leave reviews are usually angrier than the average user.
Method: I ran 20 Choros personas on IRS.gov across two runs, then sorted their complaints into the same 8 themes. As a baseline, I asked the same AI to guess what 10 taxpayers would complain about without visiting the site.
Results
- Choros personas found 3 of the 5 main real complaint themes. The no-browsing guess found 4 of 5.
- About 57% of persona complaints matched a real theme, compared with 60% for the guess.
- The two runs agreed on half of their themes, and 87% of persona complaints pointed to a step that actually happened.
So in this test, browsing did not beat guessing. The biggest reason is that personas can only browse, so they never reached the refund tracker or the identity login, which are the two biggest real complaints.
Challenges
My first check showed the personas' top complaint was errors and outages. When I read those complaints, 15 of them came from my own tool: clicks that timed out, and IRS.gov's bot protection showing "Access Denied" to automated browsers. A real person would not see those, so I removed them before calculating the results. In the first run, 5 of 10 personas also stopped early when connections to the AI dropped. I added automatic retries, and the second run finished with all 10.
What I learned
Checking the output mattered more than generating it. Without reading the complaints by hand, I would have reported a result that was mostly my own tool's errors. I also learned that the data has to match the product: my reviews were about the app, and the personas used the website.
What's next
- Compare against website feedback, not app reviews, so the data matches what the personas use.
- Let personas reach steps like the refund tracker on a safe test copy of a site.
- Test a few more sites with a small group of real users to measure how close the personas get.