Skip to content
Start a project
AI Opportunity Sprint. Find the right first job to automate

/ Technical SEO generator

Robots.txt Generator.

Create a robots.txt file with crawl rules, private-path blocks and a sitemap reference.

Generator settings

Generated robots.txt

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /login/
Disallow: /cart/
Disallow: /checkout/
Disallow: /wp-admin/
Sitemap: https://example.com/sitemap.xml
robots.txt should not be used to hide sensitive information. It only gives crawl instructions to search engines and does not protect private pages.

Deployment guidance

Upload this file to the root of your website so it appears at https://example.com/robots.txt. After publishing, check the URL in your browser and submit the sitemap in Google Search Console.

Free Website Growth Review

Not sure whether your site is being crawled properly?

Uppercut Labs can review your technical SEO setup and check whether search engines can properly access, understand, and index your website.

Book a Free Website Growth Review
/ More free tools