SEARCH.ITISYOULive V1

Live V1 architecture

How a question becomes a result here

Search.ITISYOU runs its own selective crawler, its own processing pipeline and its own versioned index. This page is the honest version of that: what actually runs, in the order it runs, with the limits stated rather than hidden.

  • General-web results come from Brave and are labelled "Live web". This page describes the other half: our own crawler and index, labelled "ITISYOU indexed".
  • We index a reviewed set of hosts. Our own index is not internet-wide coverage and never claims to be.
  • Crawling is asynchronous. Searching queries the index version that is active now; it never launches a crawl.
  • Ranking is deterministic and versioned, so the same query on the same index gives the same order.
  • The AI pipeline exists as a code boundary and is switched off in V1. Nothing here is written by a model.

Search the webRead the crawler policy

An optional 3D version of this diagram arrives in a later phase. It only ever restates the stages written below, so nothing is missing without it.

The pipeline, stage by stage

Nine stages, in order. Each one is written in plain language first, then in the terms the code itself uses.

  1. Stage 1 of 9

    Crawl - where addresses come from

    Nothing is crawled just because it exists. A host enters the queue only as an owner-approved seed, a reviewed public submission, a link found on a page we were already allowed to read, or a sitemap we have validated. Broad, uncontrolled crawling of the open web is switched off.

    Discovery normalises every URL before deduplication, and bounds depth, query-parameter combinations, path repetition, calendar ranges and session-like addresses so that one site cannot generate unlimited work.

    URL state
    discovered
    Module
    frontier

  2. Stage 2 of 9

    Policy gate - permission before any request

    Before a page is ever requested we check whether we are allowed to have it. The robots file for that scheme and host is fetched and cached, our declared user agent is matched against the correct group, and explicit disallow rules are respected. Owner blocks and verified removals override every other path.

    An ambiguous or failing robots fetch is treated conservatively and retried within limits - it is never read as permission. The declared user agent, contact address, blocking instructions and removal process are published on the crawler page.

    URL states
    policy_pending -> eligible | blocked | unsupported | unsafe | removed
    Module
    crawler-policy

  3. Stage 3 of 9

    Frontier - deciding what is next, and when

    Allowed addresses wait in a queue that remembers each host: the earliest moment we may knock again, recent outcomes, error rate and priority. Changed and frequently requested pages come first, and no single host is allowed to monopolise the crawl.

    Queue messages carry identifiers, never document bodies, and every consumer re-checks current state. Leases and idempotency keys mean a retry cannot create a duplicate document or duplicate postings. When budget runs short, work moves to budget_deferred rather than being dropped.

    URL states
    eligible -> queued -> leased | budget_deferred
    Module
    frontier (Queues + D1 state)

  4. Stage 4 of 9

    Fetch - retrieving the page safely

    An ordinary HTTP request is tried first. A full browser render happens only when a deterministic check says the essential content is missing, because rendering is expensive and the budget is finite.

    Loopback, private, link-local, multicast, reserved and cloud-metadata destinations are rejected before every connection and re-checked at every redirect hop. Headers, body size and total time are capped and bodies are streamed, so an oversized or slow page is abandoned instead of absorbed.

    URL states
    leased -> fetched | fetch_failed
    Module
    fetch

  5. Stage 5 of 9

    Extract - turning a page into text worth indexing

    We keep the canonical address, the title, the description, the headings, the main readable text, the language, and dates only where they are credible. Scripts, styles, repeated navigation and hidden content are discarded before anything is indexed.

    Each extraction records a content hash and the parser version that produced it. A document is re-indexed only when its material content changes or the indexing logic itself changes, so identical work is never repeated.

    URL states
    fetched -> parsed | parse_failed | unsupported
    Module
    extract

  6. Stage 6 of 9

    Dedup - one page, not five copies of it

    Identical documents are found by hashing the extracted content. Near-identical documents are grouped by a bounded fingerprint. Each cluster keeps one primary document and records the rest as aliases.

    Ranking never counts duplicates as independent authority. Quality exclusions - thin content, boilerplate ratio, unstable status, spam heuristics - are reason-coded, versioned and reversible, so an exclusion can be explained and undone rather than quietly disappearing a site.

    URL states
    parsed -> duplicate | quarantined
    Modules
    extract + index

  7. Stage 7 of 9

    Versioned index - built, checked, then switched on

    An index is never edited underneath you while you search. A new one is built under its own version identifier, validated, and only then activated as the version queries run against. If it turns out to be wrong, the previous version is switched back on.

    Versions are immutable and carry document counts and a checksum. Postings store document frequency, bounded per-document term frequency and field flags, with positions only where phrase search needs them. Compaction and orphan cleanup run as budget-aware idempotent jobs.

    URL states
    parsed -> indexed -> scheduled_for_recrawl
    Modules
    index + retrieval

  8. Stage 8 of 9

    Deterministic ranking - no hidden hand on the scale

    Order comes from a fixed, published formula. It is not a model forming an opinion, and it is not for sale. The same query against the same index version returns the same order every time.

    score = lexical relevance + title, heading and exact-phrase boosts + freshness and source-quality adjustments - duplicate and spam penalties. Every weight belongs to a named ranking version; changing one requires offline regression results, a decision record and a reversible activation.

    Guarantee
    same query + same index version = same order
    Module
    ranking

  9. Stage 9 of 9

    Results - what you actually see

    Your query is validated and bounded, matched against the index version that is active right now, ordered by the ranking above, and shown with snippets built from stored, sanitised text around your terms. When nothing in the index is a reliable match you are told that, instead of being handed weak filler.

    Only privacy-safe normalised result sets are cached. There is no account, no advertising profile and no permanent personal query history in V1, and a result always traces back to one exact indexed document version and its crawl timestamp.

    Response
    versioned contract, also served by /api/search
    Modules
    query + retrieval

What the engine is doing right now

These are the live values behind the stages above. Crawling is asynchronous: a search queries the index version named here and never starts a crawl of its own.

Operating mode
Normal - crawling, indexing and search all running
Active index version
idx_mtkitocb
Activated
Ranking version
rank-v1.0.0
Documents indexed
364
Hosts indexed
11

Live V1, and what is only future architecture

Everything marked Live is running behind this site today. Everything marked Future is designed for but not shipped: it does nothing yet, and nothing you can use here depends on it.

Live in V1

  • Live

    Selective crawl of reviewed hosts

    Seeds are owner-approved or reviewed before they are queued. This is a small index of chosen sources, not a copy of the web.

  • Live

    Robots, blocking and removals enforced

    Disallow rules, owner blocks and verified removal requests are applied ahead of every fetch and every result.

  • Live

    Versioned index with rollback

    Builds are validated before activation, and the previous version can be reactivated without a rebuild.

  • Live

    Deterministic, versioned ranking

    A published formula with named weights and regression fixtures. No paid placement and no manual boosts.

  • Live

    Budget-aware safe pause

    Crawling pauses before free-tier limits are reached. Search keeps serving the index that is already built.

  • Live

    Public search API

    The same bounded, versioned response contract the site itself uses, at /api/search.

Future - not running

  • Future

    Wider host coverage

    More reviewed sources, added deliberately. It will still not be internet-wide, and we will not describe it that way.

  • Future

    Reader-controlled ranking weights

    Letting you move freshness against source quality yourself, with the resulting order still fully explainable.

  • Future

    Live crawl metrics

    Real-time frontier and index telemetry on the public status page, built on measured values only.

  • Future

    Advanced query operators

    Field and phrase operators beyond the V1 set, exposed only once the index genuinely supports them.

FutureDisabled in V1

AI answer layer

A provider-neutral interface exists in the codebase so the option stays open. In V1 the flag is off, no AI request is ever made, and no answer, summary or ranking decision on this site comes from a model.

Advanced search operators are governed by a second flag and are off in V1 - the search box shows only what the index can genuinely answer.