Crawler information

MivoChatBot

MivoChatBot visits only public pages a MivoChat customer adds as a knowledge source, so their website assistant can answer from that content.

Identify it

Requests identify themselves with this user-agent string. MivoChat does not publish fixed IP addresses or hostnames for the crawler.

MivoChatBot/1.0 (+https://mivochat.app/bot)

What it fetches and how often

It runs when a customer adds a website, sitemap, individual page, or llms-full.txt knowledge source, and when that source is refreshed. It does not crawl the web at large. For a website source it starts from the supplied page, follows links up to three clicks deep, and prefers important pages such as home, pricing, docs and help. Sitemaps are read to find pages. It follows robots.txt for every page and redirect hop: a MivoChatBot group takes precedence over the wildcard group, a missing file allows access, and a server error blocks fetching until it recovers. A blocked source is reported as “Disallowed by robots.txt”.

It reads HTML and XHTML pages. Sitemap discovery accepts XML; PDFs and other assets are skipped. JavaScript-rendered pages may appear as an empty shell because the crawler does not run the page in a browser.

Pages are limited to 2 MB each. Fetches have a 15-second timeout. Crawl-delay is not supported; use Disallow to limit access. The crawler honors the configured refresh schedule: daily or weekly; by default Pro sources refresh daily and Trial/Starter sources weekly. Refresh can also be turned off. New indexed pages are capped across the workspace at 200 on Trial, 200 on Starter and 1,000 on Pro; each source may also have its own lower page cap.

Allow it

The user-agent can be spoofed, so do not allow all requests based only on its name. Allow MivoChatBot only on public paths and apply a rate limit; ensure robots.txt permits those paths. For example, a Cloudflare WAF rule can Skip or Allow requests whose User-Agent contains MivoChatBot; check that Bot Fight Mode is not challenging those requests. In Vercel Firewall, add a custom rule to allow or bypass matching requests. For other WAFs/CDNs, use the equivalent user-agent allow rule or bot-scoring exception.

User-Agent contains: MivoChatBot
Action: Allow requests only on public paths, with a rate limit

User-agent: MivoChatBot
Allow: /

# or keep it out of one section
User-agent: MivoChatBot
Disallow: /private/

Pages behind a login cannot be read because the crawler cannot sign in. Scope any firewall exception to the paths you intend to make available.

Block it

Add this rule to robots.txt to stop MivoChatBot from fetching any path on your site. To block only a section, replace / with that path, such as /private/.

# Block MivoChatBot from the whole site
User-agent: MivoChatBot
Disallow: /

If the crawler behaves unexpectedly or you want a page removed from a customer's assistant, contact hello@mivochat.app with the host and request time. Learn more in the developer docs.