Accuracy
How often does MivoChat make things up?
Run of 4 November 2026. Answers by MivoChat's answer model. The larger model available on Pro was not tested.
Headline results
The results
A single run across real software websites, including questions the source pages could not answer.
- 46 of 46
questions the docs could not answer were handed to a human, not guessed
- 96%
of code snippets reproduced exactly (45 of 47). On the original 16 code questions: 44% before our fix, 94% after
- 95%
of questions got the right decision: answer, draft for review, or hand off
- 4.5 / 5
average groundedness: how well each claim is backed by the site's own pages
How we tested
We also publish an anonymised study of SaaS website content health.
For each site we crawled its public pages, then wrote questions by hand the way visitors actually type them: short, sometimes misspelled, sometimes a follow-up to an earlier answer. No model wrote or edited the questions. Each one has a reference answer and the page it comes from.
The questions come in four kinds. Answerable (178): the site has the answer. Not in the docs: a plausible question the site does not answer. Out of scope: about something else entirely. Human only: refunds, contracts, account changes, things only a person should decide. The last three (46 in total) are the traps: the right result is a handoff to a person, not an answer.
Every question goes through the same reply and decision code that answers your visitors: search the site's pages, pick the passages that answer the question, write a reply, and decide whether to send it, draft it for your team, or hand the conversation over. A second model then grades each reply for correctness, groundedness and tone.
The test sites
| Site | What it is | Questions |
|---|---|---|
| Docusaurus | Open-source docs framework | 34 |
| Next.js | Web framework docs | 33 |
| Resend | Email API docs | 32 |
| Vercel | Hosting platform docs and pricing | 34 |
| MivoChart | Org chart SaaS (our own product) | 64 |
| SeatPlanning | Event seating SaaS (our own product) | 27 |
The third-party sites are public documentation we tested against. We are not affiliated with, sponsored by or endorsed by any of them, and their names belong to their owners.
Measured outcomes
Results
Every value below comes from this published run. The miss count is 12: 11 answerable questions were handed off instead of answered; 1 unanswerable question was declined as off-topic instead of being handed to a person.
- Unanswerable questions handed off
- 100% (46 of 46)
- Questions the site does not answer, questions about something else, and questions only a person should answer, such as refunds or a contract.
- Unanswerable questions answered automatically anyway
- 0
- The failure that matters most: a confident reply to something the docs do not cover.
- Right decision (answer, draft or hand off)
- 95%
- Across all questions. No miss invented an answer: 11 answerable questions went to a person instead of being answered, and 1 unanswerable question was declined as off-topic instead of being handed to a person.
- Answerable questions answered automatically
- 90%
- Of 178 answerable questions. The rest went to a person as a draft or a handoff.
- Code reproduced exactly
- 96% (45 of 47)
- Commands and code that must be copied character for character. In both misses the reply copied a different command that the retrieved pages also show, not the one the reference answer names.
- Correctness (1 to 5)
- 4.67
- Against a hand-written reference answer.
- Groundedness (1 to 5)
- 4.54
- 5 means every claim is supported by the pages MivoChat retrieved; 1 means it invents features, prices, limits or links.
- Replies with unsupported claims (groundedness 2 or lower)
- 0 of 224
- None in this run, and none in a second run of the same code. Earlier runs had one or two, so it can still happen: this is why your team sees every conversation.
- Errors (no reply at all)
- 0
- A failed reply would be handed to your team.
What the test made us fix
The first run reproduced code exactly only 7 times out of 16. The main cause was not the model. When we split pages into pieces for search, each code block became a piece of its own. A code block shares almost no words with the visitor's question, so search found the explanation but not the command, and the model could not copy what it never saw. A command that is almost right is worse than none: the visitor pastes it and it fails.
Now a code block always stays with the sentence that introduces it and is never cut in half. On those same 16 questions, exact code went from 7 to 15. We also grew the code questions to 47 so a small sample could not flatter us: 45 of 47 (96%) are now exact, with no drop in handoffs.
The next most common miss was also search. The reply only saw the top 8 search results, and for 16 of 178 answerable questions the page with the answer was not among them. Search now returns 20, a small, fast model call picks the passages that answer the question, and follow-ups and questions in another language get a second search in the documentation's own words. The page with the answer now reaches the model for 173 of 178 answerable questions, up from 162.
Where it still fails
No reply scored 2 or lower for groundedness in this run, but earlier runs had one or two replies with unsupported claims, so it can still happen.
11 answerable questions were passed to a person instead of answered. That is cautious behavior, but it means the visitor did not get an automatic answer. 1 unanswerable question was declined as off-topic instead of being handed to a person: no answer was invented, but nobody was asked to follow up. No unanswerable question was answered automatically.
Code accuracy is also incomplete: 2 of 47 code answers were not exact.
Limits of this test
- All 6 test sites are software products: four public documentation sites and two of our own SaaS websites. We have not yet measured other kinds of sites, such as online shops or service businesses. We plan to test those next.
- We wrote the 224 questions ourselves, by hand, from each site's pages. They are realistic, including typos and follow-ups, but they are not real visitor traffic.
- Replies are graded by a separate, stronger grading model, not by people. It sees the reference answer and the pages MivoChat used, and scores each reply from 1 to 5. A model grader can be wrong. Only answers grounded in your content count as replies; handoffs and non-answers are free.
- Search results, and the search queries rewritten for follow-ups and questions in other languages, were replayed from a frozen copy of each site, with a local embedding model standing in for the one we use in production. Live search will rank pages somewhat differently.
- Samples are small: 46 unanswerable questions and 47 code questions. 100% on 46 questions is a good sign, not a guarantee that it never happens.
- Most questions are in English. 26 are in German, French or Romanian.
- This is one run. We ran the same code a second time and the quality scores moved by up to 0.03 on a 1 to 5 scale, so read small differences as noise.
- This is one run of one model configuration on one day. We will publish new runs here when we change how answers work.
Test it on your own site
The numbers above are ours. The 14-day trial lets you ask your own questions before any visitor sees an answer. Questions about the method: hello@mivochat.app.