For site owners

Crawler policy

SearchITISYOUBot is the crawler behind our search index. This page explains exactly how it identifies itself, how it reads your robots.txt, how fast it crawls, and how to block it or ask us to remove content - in plain terms, with nothing left vague.

Note: Search.ITISYOU is a hybrid engine. General-web results labelled "Live web" come from the Brave Search API - SearchITISYOUBot does not crawl them. Only results labelled "ITISYOU indexed" come from this crawler, which visits an owner-approved seed set into our own independent, growing index.

Our user agent

Every request our crawler makes carries this exact User-Agent header:

SearchITISYOUBot/1.0 (+https://search.itisyou.app/crawler)

The address in parentheses points back to this page. When you write a rule for us in robots.txt, target the product token SearchITISYOUBot - matching is case-insensitive, and it is the clearest thing to use.

How we read robots.txt

Before requesting any page on your site, we fetch and cache your /robots.txt and follow it. Matching follows the conventions most crawlers share: named groups, longest-match precedence, and a conservative default when the file itself can't be trusted.

Choosing the right group

If your robots.txt has more than one User-agent group, we use the most specific one that names us - a group for SearchITISYOUBot beats a group for *. Only when nothing names us do we fall back to the * group. Inside whichever group applies, Allow and Disallow rules are compared by the length of the matching path pattern: the longest, most specific match wins, and Allow wins any exact tie.

Conservative failures

What happens next depends on how fetching your robots.txt goes:

Found and parsed (any successful response): we follow the rules in it.
No file published (a not-found or client-error response, other than "too many requests"): we treat your site as unrestricted, the same way most crawlers do.
Anything ambiguous - a server error, a "too many requests" response, or a network failure: we do not guess. The whole site is treated as off-limits until a later fetch succeeds. Ambiguity always resolves to not crawling, never to crawling anyway.

Re-checked after a successful fetch: 24 hours
Re-checked after a failed fetch: 1 hour

Crawl cadence and politeness

We are not trying to crawl the whole web, and we don't sprint. Our index is small and deliberately selective, and requests to any one host are spaced out rather than issued in a burst.

If your robots.txt declares a Crawl-delay, we honor it, bounded to between 1 and 60 seconds - a value outside that range is clamped rather than ignored. Without a declared delay, we still keep a polite minimum gap between requests to your host. A site that is slow to respond or returning errors gets crawled more cautiously, not more aggressively.

There is no AI deciding any of this. Crawl policy is a fixed, deterministic set of rules plus a resource budget, so what we do on your site is repeatable and inspectable, not a live model's judgment call.

Blocking the crawler

You are always in control through standard robots.txt. To block just our crawler from part of your site:

User-agent: SearchITISYOUBot
Disallow: /private/
Disallow: /account/

User-agent: *
Disallow: /admin/

To block us from your entire site:

User-agent: SearchITISYOUBot
Disallow: /

A change takes effect the next time we fetch your robots.txt - within 24 hours if our last fetch of it succeeded, sooner if our last attempt failed.

Ask us to remove content

Already-indexed pages can be removed even if you don't control the site's robots.txt, or if you'd rather not wait for us to recrawl. A verified removal request overrides everything else, including robots.txt, for that page or site.

Start a removal request

Contact

Questions about how we crawl, or about a specific request you've seen from us? Write to support@itisyou.app.