What Is Website Link Crawler?
Website Link Crawler lists every link in a sitemap or in pasted HTML, then checks the ones it can fetch, so you can find dead links before visitors and search engines do. You paste a sitemap XML file or HTML pages, and the tool reports each link as OK, broken or not checkable, with the reason beside it. This free online crawler by TechnSols runs entirely in your browser, with no install and no sign-up.
Links break for ordinary reasons: a page is deleted, a post moves to a new address, a domain changes, or a URL is typed with a mistake. Each broken link sends a visitor to an error page and tells a search engine the site is not maintained. On a large site the dead links hide in old posts, footers and menus, where nobody looks until a visitor complains.
The crawler works on pasted material, which is its honest strength. A browser cannot fetch an arbitrary live site, because cross origin reads are blocked unless the server allows them, so this tool never claims to crawl a whole live site from a single address. The free site crawler from DIY SEO Hub runs the same job on its own servers, reading a sitemap and following internal links for up to 5,000 pages, which is the right next step when a whole site needs a deep crawl.
People use it to check the links in a new sitemap before submitting it, to audit the links in a set of pages after a redesign, to confirm a list of URLs from a spreadsheet, and to prove a fix after repairing dead links. A site owner can paste a sitemap and see the broken count in one run. An editor can paste the HTML of a long article and check every link it contains.
How To Use Website Link Crawler?
Here is how to crawl website links online in three quick steps. Paste your sitemap or HTML, give a base URL for relative links, and press the button. There is nothing to install and no account to make, and the list fills in as each link is checked. At most 50 links are checked per run.
Step 1: Paste A Sitemap Or HTML
Paste a sitemap XML file, with its loc entries, or paste the HTML of one or more pages, or paste bare URLs one per line. The tool reads loc entries from a sitemap, href values from HTML, and plain URLs from their own lines, then removes duplicates so each address is checked once. The URL Extractor pulls every link out of a page first, when the starting point is a single page rather than a sitemap.
Step 2: Set The Base URL
Type the site address into the base URL box when your HTML holds relative links, such as /blog/post-one, so they resolve to full addresses before they are checked. Without a base, relative links resolve against this page, which gives the wrong answers. The URL redirect checker follows a single address to see where it really lands, which helps when a base address itself redirects.
Step 3: Read The Link Report
Press Crawl links and read the report: OK means the link answered with a good status, BROKEN means it answered with an error status, and NOT CHECKABLE means the browser could not fetch it, usually because the server does not allow cross origin reads. Fix the broken ones, then paste again to prove the fix. The HTTP status checker confirms the status code of a single link on its own, so a result you doubt can be double checked.
Why Website Link Crawler?
Dead links are the most common fault on an older site, and the least visible, because each one sits quietly in a page that otherwise loads fine. A regular crawl keeps the link graph healthy, and this tool makes a small crawl take seconds.
Find Dead Links Before Visitors Do
A visitor who follows a link to an error page loses trust in the site, and a search engine that meets dead links often reads the site as poorly kept. The crawler lists the dead links with the address beside each one, so the fix has an address from the start. The XML sitemap generator builds a fresh sitemap after the links are fixed, so search engines find the corrected pages quickly.
Check A Sitemap Before You Submit It
A sitemap that lists dead pages wastes crawl time and weakens the signal it sends to search engines. Paste the sitemap here first, fix the broken entries, and only then submit the clean file, so every address in it is a live page. The robots.txt generator builds the file that tells crawlers which parts of the site to read, so the sitemap and the robots file agree.
Prove A Repair With Evidence
After repairing dead links, paste the same sitemap again and the report proves the repair, with the broken count at zero and every link marked OK. That evidence matters when a client asks whether the job is done, because a list of green OK marks beats a promise. The canonical tag generator builds the tag that tells search engines which address is the real one, so a repaired link points at the right canonical page.
What Can You Do With Website Link Crawler?
Link checks fit into real jobs: a redesign, a domain move, a content audit and a sitemap submission all need the same answer, which is which links still work. These are the jobs this crawler handles best.
Audit Links After A Redesign
After a redesign, paste the sitemap and check every listed page, because new templates often change link addresses in menus, footers and related post blocks. The broken list is the exact repair list for the launch week. The Link to PDF tool turns a repaired page into a PDF file, which suits a launch report that needs a fixed record.
Check The Links In A Long Article
A long article gathers links over years of edits, and some of its sources move or vanish. Paste the article HTML, run the crawl, and the broken lines name the sources to replace or remove. The PDF link extractor pulls the links out of a PDF version of the same article, so a download and a web page can be checked against each other.
Verify A List Of URLs From A Spreadsheet
When a list of URLs comes from a spreadsheet, a redirect map or an old audit, paste the addresses one per line and check them all at once. The report separates the live addresses from the dead ones, so the list can be cleaned before it is used. The keyword density checker reads the text of a repaired page, so the content check follows the link check in the same pass.
Quality And Settings
Each link is fetched with a real browser request, so the result reflects what a visitor’s browser would meet at that address. The report shows the status for every link it could fetch, and says plainly when a link could not be checked, instead of guessing.
- Three ways in: paste sitemap XML, paste HTML pages, or paste bare URLs one per line.
- Duplicates removed: each unique address is checked once, however many pages link to it.
- Real statuses: a link that answers with an error status is marked BROKEN with the code beside it.
- Honest limits: links the browser cannot fetch are marked NOT CHECKABLE, never guessed as OK or broken.
- Fair limit: at most 50 links per run, with a ten second timeout per link, so a long list stays fast.
There are no settings to tune beyond the base URL, because the crawl follows the links you paste: the sitemap or the HTML is the scope, and the browser is the checker. If a list holds more than 50 links, split it into parts and crawl each part on its own.
How Website Link Crawler Works
The crawler reads your input and extracts addresses: loc entries from sitemap XML, href values from anchor tags in HTML, and bare URLs on their own lines. Relative links resolve against the base URL you gave, or against this page when you gave none. Each unique address is then fetched with a GET request, and the response status decides the mark: a good status is OK, an error status is BROKEN.
Two limits are worth knowing. First, the browser can only fetch same origin pages and pages whose servers allow cross origin reads, so any other link is listed as NOT CHECKABLE; that is a browser rule, not a fault in the link. Second, the crawl never follows links it finds on fetched pages, because it works on the pasted list only. The Dead Link Crawler service walks a whole site from its own servers when a full recursive crawl is needed, which a browser tool cannot do.
Privacy
Your pasted sitemap and HTML never leave your device, except for the link checks themselves, which fetch the addresses you listed, exactly as a visitor’s browser would. Nothing is uploaded to this site, nothing is stored, and nothing is shared with anyone.
Because no account is needed, there is no name, email or history tied to your crawls. Closing the tab ends the session, and the browser clears the pasted text from memory. A sitemap you are auditing may list unreleased pages, and the same protection applies to every list you crawl here.
Tips
A few habits make link crawls faster and more honest, especially on a large or old site.
- Crawl after every move: run the sitemap again whenever a domain, folder or permalink changes.
- Set the base first: relative links give wrong answers when the base URL is missing or wrong.
- Fix the page, not the link: a dead link often means the target page moved, so restore or redirect the page.
- Read NOT CHECKABLE carefully: it means the browser could not fetch the link, not that the link is dead.
- Crawl monthly: a short monthly crawl catches new dead links before visitors report them.
When the report is clean, save the sitemap with the check date, so the next crawl has a baseline to compare against. A clean report today makes a new broken link tomorrow easy to spot.
More Tools For The Same Job
These tools handle the same kinds of jobs: checking links, pages and sitemaps before visitors and search engines meet them.
- HTTP Status Code Checker: confirms a page answers with the status code you expect.
- URL Redirect Checker: follows an address to see where it really lands before you test it.
- XML Sitemap Generator: builds a fresh sitemap after the links are fixed.
- Robots.txt Generator: builds the file that tells crawlers which parts of the site to read.
- Canonical Tag Generator: builds the tag that tells search engines which address is the real one.
- Keyword Density Checker: reads the text of a repaired page to check its keyword use.
- Meta Tags Generator: builds the meta tags a repaired page needs for search and sharing.
- URL Parser: breaks a link address into its parts, so a faulty address is easy to read.
- Open Graph Image Preview: shows how a repaired page will look when it is shared.
- Meta Description Length Checker: keeps the search snippet inside its length limit while you edit a page.
FAQ
Common questions about crawling links, and what a browser crawl can and cannot check.
Can Website Link Crawler Crawl A Whole Site?
No. It crawls the links you paste, from a sitemap, HTML or a URL list, and checks up to 50 per run. A browser cannot fetch an arbitrary live site, because cross origin reads are blocked. A server side crawler is needed for a full recursive crawl.
What Does NOT CHECKABLE Mean?
It means the browser could not fetch that link, usually because the server does not allow cross origin reads, or because the address could not be reached. It does not mean the link is broken. Open the link in a tab to check it by hand.
Does It Follow Redirects?
Yes. Each fetch follows redirects, so a link that moves to a new address is checked at its final destination. A link that redirects to an error page is marked BROKEN, and a link that redirects forever times out and is marked NOT CHECKABLE.
Can I Paste More Than 50 Links?
You can paste more, but only the first 50 unique links are checked per run. Split a longer list into parts and crawl each part on its own. That keeps each run fast and the browser responsive. Start with the newest pages first.
Is Website Link Crawler By TechnSols Free?
Yes, Website Link Crawler by TechnSols is free to use online, with no install and no sign-up. You can crawl as many sitemaps and pages as you like, fifty links at a time, and nothing you paste is uploaded or stored.
Dead links hide in plain sight, in old posts, footers and menus that nobody opens. Crawl the sitemap after every move, check long articles before they go live, and retest after each repair, so every link on your site leads somewhere real.