What Is Robots Txt Rule Tester?
Robots.txt Rule Tester is a free online tool by TechnSols that checks pasted robots.txt rules against the URLs you type. You paste the rules, pick the user agent to test as, list the URLs one per line, and press Test URLs to see the verdict for every URL with the rule that decided it.
A robots.txt file is a small plain text file at the root of a website that tells search engine crawlers which paths they may fetch and which they must leave alone. One wrong line can hide a whole section from Google, or leave an admin area open to every crawler. A pattern that looks narrow can match far more than intended, and a longer rule always beats a shorter one, whatever order the lines appear. The tester applies the same matching rules a real crawler uses, so you see the verdict before the file goes live. If you still need to write the file itself, the Robots.txt Generator tool builds a fresh robots.txt from your choices.
Robots.txt Rule Tester differs from the generator: the generator creates a robots.txt file, while this tester checks rules you already have. It works on pasted rules only: it never fetches a live robots.txt file, and it never crawls anything. The result is a clear list of allowed and disallowed URLs, the matching rule for each one, and warnings about lines that look wrong, ready to download as a text report.
How To Use Robots Txt Rule Tester?
Testing robots.txt rules takes less than a minute, with no software to install and no account to make. The page opens with a working example already filled in, so the first click shows a real result.
Step 1: Paste The Rules And The URLs
Paste your robots.txt rules into the first box, keeping the User-agent lines and the Allow and Disallow lines exactly as they will appear in the file. Type the user agent to test as, such as Googlebot or the star that stands for every crawler, and list the URLs to check in the second box, one per line. A path such as /admin/dashboard works, and a full address works too, because the tester reads the path from it. To check the links on the pages those rules leave open, the Link Anchor Checker tool checks every anchor on a page and reports the broken ones.
Step 2: Press Test URLs And Read The Verdicts
Press Test URLs and read the list. Each line shows one URL, the word Allowed or Disallowed, and the rule that decided it, with its line number. A URL with no matching rule is allowed by default. Warnings appear at the top, for example a rule before any User-agent group, an unknown directive, or a Sitemap line that is not an absolute address.
Step 3: Download The Test Report
Press Download on the result row to save a text report with the user agent tested, every verdict with its matching rule, and the warnings. Keep the report with the robots.txt draft so the team can see what was checked before the file was published.
Why Use Robots Txt Rule Tester?
Robots.txt rules are short, and that is exactly why they are misread so often. A single star or a missing slash changes what a whole section of a site means to a crawler.
Longest Match Wins, Not First Match
When several rules match one URL, the crawler follows the longest matching pattern, not the first line in the file. A Disallow for /private blocks /private/page, but an Allow for /private/public/ opens that folder again, because the longer pattern wins. The tester applies the longest match rule to every URL, so the verdict you see is the verdict a crawler reaches. For the official matching rules, the robots.txt guide from Google explains how crawlers read each line.
One Group Applies To Each Crawler
A robots.txt file can hold several groups, one per crawler, and a crawler reads only the most specific group that names it, then falls back to the star group when no group names it. A rule written under Googlebot never applies to Bingbot, and a rule under the star never applies to a crawler that has its own group. The tester picks the right group for the user agent you type. When the site also needs its pages checked for crawl problems, the HTTP Header Checker tool reads the headers a live page actually sends.
Wildcards And End Anchors Surprise People
A star matches any run of characters, so Disallow for /*.pdf$ blocks every address ending in .pdf, while the dollar sign anchors the match to the end, so /file.pdf?x=1 stays allowed because the address does not end at .pdf. A pattern without a dollar sign is a prefix match, so Disallow for /news blocks /news/, /newsletter/ and /news-archive/ alike.
What Can You Do With Robots Txt Rule Tester?
The tester fits anywhere a robots.txt file is written, changed or checked, from a new site launch to a routine audit.
Check A Draft Before It Goes Live
A new robots.txt file is often written in a hurry at launch, and a single over broad rule can hide the whole shop or blog from search engines for weeks. Paste the draft, list the ten most important URLs on the site, and test them against the crawler you care about most. The verdict list shows in seconds whether the home page, the product pages and the blog stay open while the admin area stays closed. For the pages that must stay findable, the HTML Element Inspector tool shows the live markup of a page so its headings and links can be checked.
Find Why A Page Vanished From Search
When a page drops out of search results, a robots.txt rule is one of the first suspects, and one of the easiest to rule out. Paste the live file, test the missing page, and read the verdict. If the page is disallowed, the matching rule names the line to fix. If the page is allowed, the cause lies elsewhere, and the search moves on to indexing, meta tags or a server error. When a page needs its selectors checked next, the CSS Selector Tester tool shows which elements a selector matches on the page.
Prove A Fix To A Client Or A Team
A robots.txt change is invisible, so a client or a team lead cannot see it worked without a test. Run the tester before and after the change, download both reports, and the two files show the verdict flipping from disallowed to allowed on the URLs that mattered. The report carries the user agent, the rules and the line numbers, so anyone can repeat it.
Quality And Settings
The tester has no quality setting to tune, because matching robots.txt rules is exact logic rather than a matter of taste. The settings that matter are the user agent, the rules and the list of URLs. A rule the tester cannot read is reported as a warning with its line number, never guessed at.
- User agent field accepts a star for every crawler, a token such as Googlebot or Bingbot, or a full user agent string.
- Rules support Allow and Disallow lines, star wildcards, dollar end anchors, Sitemap lines, and Crawl-delay lines, which Google ignores.
- Empty Disallow values mean allow everything, and Disallow for a single slash blocks the whole site, exactly as the protocol defines.
- Matching is case sensitive on the path, so a rule for /Blog does nothing for a site that serves /blog/.
- Every verdict names the deciding rule and its line number, and a URL with no matching rule is allowed by default.
When a rule cannot be applied, the warning names the problem line and says what was done with it. The tester checks the rules you pasted, not the file your server serves today, so compare the verdicts with the live file before relying on them.
How Robots Txt Rule Tester Works?
The tester parses the pasted text into groups. Each User-agent line starts a new group, and the Allow and Disallow lines that follow belong to that group until the next User-agent line. Sitemap lines are collected and listed. Lines before any User-agent group and lines with an unknown directive are reported as warnings, because a crawler ignores them too.
For each test URL, the tester picks the group that applies to the user agent you typed: a case-insensitive token match first, then the star group. It treats the star as any run of characters and the dollar sign as an anchor to the end of the path, then checks every rule in the chosen group against the path and query of the URL. The longest matching pattern wins, and when an Allow and a Disallow match with exactly the same length, the Allow wins. That is the rule in RFC 9309, the robots exclusion protocol, which Google and most major crawlers follow. When the next step is to check the page itself, the Image Loading Sequence Tester tool shows the order in which the page loads its images, and the Lazy Loading Tester tool checks which images on the page load lazily.
Privacy
Robots.txt Rule Tester runs entirely in the browser. The rules that are pasted, the user agent that is typed and the URLs that are tested never leave the device, and nothing is uploaded to a server or stored anywhere. There is no account and no tracking. When the same audit also needs a page checked for sharing tags, the Open Graph Generator tool works the same way, with everything running locally and nothing sent away.
Because nothing is stored, there is nothing to delete and nothing to export. The downloaded text report is the only record.
Tips
A few habits make robots.txt files much safer to manage.
- Test the ten most important URLs on the site after every change, not just the URL the change was meant to affect.
- Keep a separate group for each crawler that needs different rules, and leave the star group as the default for all the rest.
- Remember that robots.txt controls crawling, not indexing, so a disallowed page can still appear in search if other pages link to it.
- Use a dollar sign when a rule should only match an address that ends a certain way, and test an address with a query string too.
- Check the live file after publishing, because a cached copy can make the site serve different rules from the ones tested.
When a page must not appear in search at all, use a noindex tag instead of a Disallow rule, because a crawler has to fetch the page to see it. For a full check of the site after a robots.txt change, the Website Resource Size Analyzer tool measures every resource the pages load and reports what still needs fixing.
More Tools For The Same Job
The tools below solve the jobs that sit next to robots.txt rules in the same site audit.
- Robots.txt Generator: builds a fresh robots.txt file from the rules you choose.
- Sitemap Generator: builds an XML sitemap listing the pages crawlers should find.
- Sitemap Validator: checks whether a sitemap file is well formed.
- HTTP Header Checker: reads the headers a live page sends, including status codes.
- Meta Tag Generator: builds the meta tags that tell crawlers how to treat a page.
- Canonical URL Checker: reads the canonical tag on a live page.
- URL Encoder: encodes special characters in an address so links work as intended.
- Open Graph Generator: builds the sharing tags that control how a page looks.
- SEO Audit Tool: scans the pages that matter and reports the SEO problems.
- JSON Formatter: formats and checks pasted JSON.
FAQ
The questions below cover the points that come up most often when robots.txt rules are tested for the first time.
What Does Disallow Slash Mean In Robots Txt?
Disallow with a single slash blocks the whole site for the crawlers in that group, because every path on the site starts with a slash. It is the strongest rule in the file, and it is usually only meant for a staging copy that must stay out of search.
Why Is My Page Disallowed When No Rule Mentions It?
A prefix rule matches more than its own folder, so Disallow for /news blocks /newsletter/ and /news-archive/ as well as /news/. A star rule reaches further still, so Disallow for /*.pdf$ blocks every address ending in .pdf. The tester shows the matching rule, so the surprise points at the line to change.
Does Robots Txt Stop A Page From Appearing In Search?
No, robots.txt controls crawling, not indexing. A disallowed page can still appear in search results, usually without a snippet, if other pages link to it, because Google may index the address without fetching the page. To remove a page from search, use a noindex tag and let crawlers fetch the page.
Which User Agent Group Applies To Googlebot?
Googlebot reads the group that names Googlebot, and only falls back to the star group when no group names it. A rule under the star never applies to Googlebot once its own group exists. Test with Googlebot in the user agent field to see the group that really applies.
Is Robots Txt Rule Tester By TechnSols Free?
Yes, Robots.txt Rule Tester by TechnSols is completely free to use online, with no sign up, no install and no limit on the number of tests. It runs in the browser, nothing is uploaded, and the text report downloads free too.
Robots.txt rules stop being a puzzle once the matching rule is known. Paste the rules, test the URLs that matter, read the verdict with its matching line, and ship with confidence.