robots.txt Generator / Tester

Based on RFC 9309 sections 2.2.1–2.2.4 (opens a new tab). Enter an exact crawler product token, not an HTTP User-Agent header. Case-insensitive exact-token groups merge; only when none match, all * groups merge. No applicable rule means Allowed; /robots.txt is implicitly allowed. Empty Allow and Disallow match nothing.

Paths match from their start, case sensitively. Longest specificity wins: count literal octets after percent normalization, excluding * and the final $. Allow wins equal specificity; otherwise first source line wins. * matches zero or more octets including / and query delimiters. A final $ anchors the entire path plus query; elsewhere use %24. Use %2A and %23 for literal * and #. # starts a document comment; blank lines do not end groups; Sitemap never ends a group.

Compare the original path plus query, including ? and =; omit origin and fragment. No URL resolving, dot-segment removal, plus-to-space conversion or Unicode normalization. UTF-8 non-ASCII octets are percent-normalized; hexadecimal case is ignored, unreserved percent escapes decode once, reserved escapes remain distinct from literal delimiters (so %2F is not /). Malformed percent escapes, controls, invalid Unicode and internal whitespace are refused. Percent-encoded bytes are compared as octets, not decoded Unicode strings.

This is a strict bounded editing/test dialect, not a compliant fetching crawler: malformed or unsupported directives refuse the entire test instead of a crawler's best-effort parsing. Only User-agent, Allow, Disallow and absolute HTTP(S) Sitemap are supported. Generator fields have one value per line and no comments; optional blank paths become empty Disallow. LF output is deterministic. Limit: 16 groups, 256 document lines, 4096 document/output UTF-16 characters, 1024 per other field/pattern. These editing limits are below RFC crawler retrieval requirements.

Ordinary native edits clip before Blazor transport at limit + 2; managed and Domain checks refuse every oversize value, never accepting clipped prefixes. Refusal clears all inputs and stale results; re-enter valid data or Clear. Native bounds do not certify hostile assignments or all IME behavior. No crawling, fetching, storage of payloads or changes to this website's robots.txt. Interactive Auto server events traverse the Blazor connection; WebAssembly runs on-device. Website active only; extension/Android inactive discovery metadata is not hard route denial. Crawler behavior can differ. Robots rules are not authentication and expose listed paths publicly.