How to recognize RivalEye requests
RivalEye reads only publicly accessible pages and states its name with every request; the Competitor Check also respects robots.txt. This page explains which requests exist, what they read and how you block the check.
1. Owner verification
When a customer confirms that a website belongs to them, RivalEye reads only the homepage and up to 4 subpages such as the imprint, contact or “About us” page of that customer’s own website in order to find the email address published there. Because this concerns the customer’s own website, this request does not evaluate a robots.txt. It identifies itself with:
RivalEye-Verify/1.0 (+https://rivaleye.pro/methodik.html)
2. Free Competitor Check
In the free Competitor Check, someone enters their own website and up to three competitor websites. RivalEye then reads each website entered at most once per check, automatically and according to fixed rules. At most, the following are read:
- the robots.txt – of the address entered, of the same address with or without “www.” and, if the homepage redirects to a different address, the robots.txt there as well,
- the homepage – if it does not respond over https, alternatively over http:// or with or without “www.”,
- a test of whether the address redirects from http:// to https://,
- the llms.txt,
- the sitemap (the one named in the robots.txt, otherwise sitemap.xml or sitemap_index.xml) and, for a sitemap index, up to 2 sub-sitemaps of the same domain.
If the homepage redirects to a different address (for example with “www.”), we also read llms.txt, the sitemap and the redirect test there where necessary. No subpages, no forms, no logins. Several requests to the same address within one check are combined.
Cache: If the same website is entered in another check shortly afterwards, we reuse the stored result and do not request it again – as a competitor for up to 6 hours, as your own website for up to 10 minutes. The number of checks per website is limited in addition.
Time limits: For the robots.txt and the homepage we wait at most 7 seconds, for llms.txt, the redirect test and sitemaps at most 4 seconds, and for one website at most 18 seconds in total. If llms.txt, the redirect test or the sitemap do not respond in time, we leave them out; the report then shows “not determined” instead of a deduction of points.
AI Council: After the check has been confirmed, the RivalEye AI Council writes and checks the texts of the report. It works only with the data already measured and does not call up any websites – there are no further requests with a different identifier.
The request identifies itself with:
Mozilla/5.0 (compatible; RivalEye-Check/1.0; +https://rivaleye.pro/bot.html)
How we respect the robots.txt
The robots.txt is read first and respected before every further request – including every redirect to a different address. We evaluate it according to the standard RFC 9309: the group User-agent: RivalEye-Check applies, otherwise a group User-agent: RivalEye, otherwise User-agent: *; within the group the longest matching rule decides, and in a tie Allow wins.
- No robots.txt (4xx error), not reachable or no response in time: then everything counts as allowed – as the standard provides.
- Server error (5xx): then, out of caution, we read nothing further. As a competitor the website appears as “not readable”; as your own website it cannot be checked in that case – because its robots.txt returned a server error, not because it blocks us.
- Very long robots.txt: we read the first 200 KB. If the group that applies to us has more than 1,000 rules, we treat it as a complete block.
Blocking via robots.txt:
User-agent: RivalEye-Check Disallow: /
A group with User-agent: RivalEye applies as well. If your website is blocked, the check reads only its robots.txt: as a competitor it appears in the comparison as “not readable”, as your own website it cannot be checked. Results that are already cached (see above) expire after 6 hours at the latest.
3. Regular monitoring in the subscription
The regular monitoring in the subscription will get its own identifier. We will publish it here with its user agent and robots.txt token before it runs for the first time.
4. Contact
Questions or complaints about requests: contact@semia-agent.de, +49 (0) 155 1038 9104 or via the contact page. We reply within one business day. More on how we work: methodology.