About strawbibot
If you've seen strawbibot/1.0 (+https://strawbi.com/bot) in your logs, that was us. strawbi is a service companies use to watch their own website: which pages exist, when their text changes, and whether the crawlers behind AI assistants can read them. Either someone asked us to look at your site, most likely someone who works there, or an AI answer to a question one of our customers follows linked to a page on your site, and we read that one page.
What it fetches
/robots.txt, your sitemaps,/llms.txtand/llms-full.txt.- HTML pages on the site, found from the sitemap and by following links. Never images, PDFs, scripts or other files.
- For a small sample of pages, the same address asked for as markdown and a
.mdsibling. - The homepage a few more times with other crawlers' user-agent strings, to see whether the site treats them differently.
- Pages that AI answers link to. When an answer cites a page on your site, we fetch that page (and your robots.txt) once, then about once a week while answers keep citing it. Nothing else on the site.
How fast
One request at a time per site, at least 1 second apart, up to 5,000 pages per visit, usually once a day. Pages are read once per visit; nothing is fetched more than once a day unless a person presses the recrawl button, which has its own cooldown.
How to keep it out
We read robots.txt before every visit and follow it. To block the whole site:
User-agent: strawbibot Disallow: /
To block part of it, use a narrower Disallow path. Changes apply on the next visit. If something looks wrong, email [email protected] and a person will answer.
What we do with it
We keep the text and the HTML of each page so the person who added the site can see what changed and when. The copy is shown only to them, inside their account. A page fetched because an answer cited it is kept the same way, so the customers whose answers cite it can see when it changed. None of it is published, resold or used to train anything.