Robots.txt Generator
Generate a valid robots.txt file and control exactly what crawlers can access
User-agent: * Disallow: /admin Allow: / Sitemap: https://bilalvirk.com/sitemap.xml
Create a robots.txt file that tells search engine crawlers exactly where they can and can't go. Block sensitive admin areas, allow your important pages, and point crawlers to your sitemap — all with a visual builder.
What Is robots.txt?
The robots.txt file sits at the root of your domain (https://yoursite.com/robots.txt) and gives instructions to web crawlers about which pages they can and cannot access.
Important: robots.txt is a convention, not a security measure. Malicious bots ignore it. Only use it to manage crawl budget, not to hide sensitive data.
Common robots.txt Rules
# Allow all crawlers to access everything (default)
User-agent: *
Allow: /
# Block all crawlers from admin areas
User-agent: *
Disallow: /admin/
Disallow: /api/
# Block only GPTBot (OpenAI's web crawler)
User-agent: GPTBot
Disallow: /
# Point to sitemap
Sitemap: https://yoursite.com/sitemap.xmlWhat You Should Block
- Admin panels (
/admin/,/wp-admin/) - API endpoints (
/api/) - User-specific pages (
/dashboard/,/account/) - Checkout and payment pages
- Internal search result pages (duplicate content)
- Staging environments (better handled with
noindex)
Blocking AI Training Crawlers
Many website owners now block AI company crawlers to prevent their content from being used to train LLMs:
User-agent: GPTBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /Don't Forget the Sitemap Line
Every robots.txt should point to your sitemap so crawlers can find it without guessing. If you haven't generated one yet, the XML Sitemap Generator builds the file this tool's Sitemap: line should point to.
Crawl Budget and Large Sites
For sites with thousands of pages, robots.txt helps search engines prioritise crawl budget on your important pages. Book a consultation to review your crawl strategy.
Want this built against your real numbers?
A 30-minute call to scope the workflow, agent, or automation you actually need.
More seo tools
All tools
301 Redirect Rule Generator
Turn a list of old and new URLs into redirect rules for Apache htaccess, Nginx, Cloudflare or Netlify. Detects redirect chains and loops before you deploy them, and handles bulk migrations from a pasted list.
Canonical Tag Generator
Generate a correct rel=canonical tag from any URL. Normalises the scheme, strips tracking parameters, resolves trailing slash and case issues, and warns about the canonical mistakes that quietly deindex pages.
Content Brief Generator
Generate a complete SEO content brief from any keyword — ready to hand to a writer
Google SERP Preview Tool
See how your page will look in Google results on desktop and mobile before you publish. Renders the title link, breadcrumb URL and snippet with realistic truncation, and highlights query terms the way Google bolds them.
Hreflang Tag Generator
Build a complete, reciprocal hreflang set for a multilingual page and export it as link tags, an HTTP header block or XML sitemap entries. Validates language and region codes and adds x-default automatically.
Keyword Clustering Tool
Group any keyword list by search intent and topic — ready for content planning
Have a workflow that's burning hours every week?
Bring me one real bottleneck. I'll tell you whether it's worth automating, and what it would take.