The crawler runs on our server — not in your browser. It follows links the same way Googlebot does, page by page, extracting real data from real HTML.
The crawler sends an HTTP request to your URL using a standard browser User-Agent. It follows redirects automatically and records the final URL as the canonical starting point.
Every <a href> on the page is parsed. Relative paths are resolved to absolute URLs. External links, file downloads (.pdf, .jpg, etc.), and API routes are excluded. Internal links go into the crawl queue.
Pages are fetched 6 at a time with a short delay between batches to avoid overwhelming your server. Each page is checked for noindex — those pages are skipped from the sitemap but their links are still followed.
For each indexable page the crawler extracts: title tag, meta description, H1, H2 headings (up to 6), word count, image count, and last-modified date from the HTTP Last-Modified header or article:modified_time meta tag.
URLs are sorted by depth then alphabetically. Priority is assigned by depth (homepage 1.0, top-level 0.8, deeper 0.7–0.6). Duplicate canonical URLs are deduplicated. The output is valid sitemap.org XML ready to submit to Google Search Console.
This tool automatically excludes pages with noindex meta tags, non-HTML resources, and pages that return 4xx or 5xx status codes. Review your output for any URLs that should not appear in Google.
Google uses lastmod to prioritize recrawls. The crawler reads the Last-Modified HTTP header or article:modified_time meta tag when available. If neither is present, today's date is used as a safe fallback.
Add Sitemap: https://yourdomain.com/sitemap.xml as the last line of your robots.txt. All compliant crawlers (Google, Bing, Yandex) will discover it automatically.
Upload the file to your domain root, then go to Google Search Console → Sitemaps → Submit the full URL. Re-submit whenever you add significant new content.
Google's limit is 50,000 URLs and 50 MB per sitemap file. Sites with more content should split into multiple sitemaps and reference them from a sitemap index file.
Crawl your marketing site, get every features/pricing/comparison page, generate a structured sitemap in one click.
Discover all published posts, check titles and word counts, spot missing meta descriptions in the results table.
Generate a baseline sitemap for every new client site. The pages table doubles as a quick content audit.
Validate routes are reachable, check canonical URLs, and verify redirects are being followed correctly.
The page detail view (title, H1, description, H2s, word count) surfaces thin-content and missing meta data at a glance.
Crawl category pages and key product URLs. Check that lastmod dates are being served correctly from your platform.
Yes. Enter your URL and our server-side crawler visits every internal page, follows links, extracts titles and headings, and generates a complete sitemap.xml — no AI, no guessing. The Paste URLs tab still works for manual control.
Up to 300 pages per crawl. For large sites you can run multiple crawls on sub-sections (e.g. /blog, /products), then merge the outputs — or use the manual paste mode for complete control.
Yes, completely free with no sign-up required. You can crawl and download as many sitemaps as you need.
For each page: the canonical URL, title tag, meta description, H1, up to 6 H2 headings, word count, image count, crawl depth, and last-modified date from HTTP headers or meta tags.
Yes. Pages with a noindex robots meta tag are discovered and their links followed, but they are excluded from the sitemap output.
No. The crawler sends unauthenticated requests. Pages behind a login, HTTP basic auth, or Cloudflare Access will return a 401/403 and be skipped — same as Googlebot would see them.