To create a robots.txt file and an XML sitemap optimised for Google, put the robots.txt at the root of your site (domain.com/robots.txt) with Disallow directives for the directories you want to exclude and a Sitemap directive pointing to your site plan. The XML sitemap lists every page you want indexed with its modification date and priority, and must then be submitted to Google Search Console.
25% of websites have a badly configured robots.txt file or no sitemap at all, according to Screaming Frog. Both files are invisible to visitors, yet a single syntax error in robots.txt can block the indexing of your entire site within minutes.
This guide explains what these two files do, the mistakes to avoid at all costs, and how to generate and submit them properly in under 10 minutes.
Generate your robots.txt and XML sitemap for free →
What are robots.txt and the XML sitemap for in Google search?
The robots.txt file and the XML sitemap are the two founding files of technical SEO: the first tells Googlebot which pages not to crawl, the second tells it which pages to crawl first. They work as a pair to make the most of your crawl budget, meaning the time Google spends exploring your site on each visit.
The robots.txt file is read by Google before it crawls any page. It holds simple directives:
The XML sitemap lists every page you want indexed, with its last modified date, its update frequency and its priority level (0.0 to 1.0). Without a sitemap, Google discovers your pages by following links one at a time, which can take weeks on a medium-sized site.
How does a generator build a valid robots.txt and XML sitemap?
A generator for valid robots.txt and XML sitemap files works by collecting your domain name, the directories to exclude and the URLs to index, then producing files that follow the official sitemaps.org specification and Google Search Console's recommendations. It removes the risk of syntax errors that could block your indexing.
The 3 steps of the generator:
Step 1: Enter your site URL
Fill in your domain name. The tool uses it to generate the correct absolute paths (which are compulsory in an XML sitemap).
Step 2: Set your robots.txt rules
List the directories to block. The tool pre-fills common rules (admin, basket, sorting pages) that you can then adjust.
Step 3: List your pages for the sitemap
Add the URLs of your main pages, their update frequency and their priority. The tool generates a compliant sitemap, ready to submit to Google Search Console.
Which robots.txt errors can destroy a site's search rankings?
The robots.txt errors that destroy a site's rankings are usually simple, but the consequences are severe: "Disallow: /" blocks the entire site in one line, a relative path instead of an absolute one in the sitemap makes the file invalid, and leaving out the Sitemap: directive forces Google to find your site plan some other way.
Common mistakes to avoid:
| Mistake | Consequence | Fix |
|---|---|---|
| "Disallow: /" (the whole site) | Entire site deindexed | Leave Disallow empty |
| Blocking CSS/JS resources | Google cannot render your pages | Remove the rules on /assets/ |
| No sitemap in the robots.txt | Google takes longer to find it | Add Sitemap: https://domain.com/sitemap.xml |
| Relative URLs in the sitemap | Invalid sitemap (rejected by Search Console) | Use absolute URLs (https://...) |
| Sitemap containing noindex pages | Inconsistency that wastes crawl budget | Remove noindex pages from the sitemap |
What to do once your files are generated
Generating the files is only the first step. Here is the full process to get Google to take them into account:
1. Upload robots.txt to the root of your server
Connect by FTP or through cPanel and drop the file at the root of your domain. Check that it is reachable by typing domain.com/robots.txt into your browser.
2. Place the sitemap.xml and check it is reachable
Same approach: put sitemap.xml at the root and confirm access at domain.com/sitemap.xml. The file must return structured XML, not a 404 error page.
3. Submit the sitemap in Google Search Console
In Search Console, go to "Sitemaps" (left-hand menu), enter your sitemap URL and click "Submit". Google starts crawling within hours.
4. Watch the indexing report
In Search Console, open "Pages" to see how many of your URLs are indexed versus submitted. If pages are "excluded", the report says why (noindex, 404 error, different canonical URL and so on).
5. Update the sitemap every time you add a page
If you add a page to your site and do not update the sitemap, Google can take weeks to find it on its own. Regenerate and resubmit the sitemap with every major publication.
Frequently asked questions about robots.txt and XML sitemaps
Do you have to submit your sitemap to Google Search Console?
No, it is not technically compulsory, because Google can find your sitemap through the directive in your robots.txt. Submitting it through Search Console does speed up indexing, though, and it gives you detailed reports: URLs submitted versus indexed, errors detected, date of the last crawl. For a site that is launching, or one that publishes content regularly, submitting it is strongly recommended.
My CMS already generates a sitemap automatically. Is it good enough?
Not always. Some CMS platforms generate sitemaps that include noindex pages (tag pages, archives, search results), which creates an inconsistency that wastes crawl budget. Others include duplicate URLs or error pages. Check what is in your automatic sitemap by opening domain.com/sitemap.xml and confirming that every URL listed is a page you actually want indexed.
Where exactly should the robots.txt and sitemap.xml files sit on your server?
The robots.txt file must sit at the root of the domain (domain.com/robots.txt), as it does not work from a subfolder. The XML sitemap can sit anywhere, but by convention it also goes at the root (domain.com/sitemap.xml). What matters is that the sitemap URL is entered correctly in the robots.txt and in Google Search Console.
How often should a sitemap be updated?
How often you update depends on how often you publish. For a blog publishing 2 to 4 articles a week, a weekly sitemap update is advisable. For a brochure site that barely changes, quarterly is enough. The changefreq value in the sitemap (daily, weekly, monthly) is a hint to Google, but Google does not follow it to the letter: it sets its own crawl frequency based on how fresh your content really is. Good practice: regenerate and resubmit your sitemap whenever you add more than 3 new pages or substantially rewrite existing content.
How do you know whether Google has indexed your pages after submitting the sitemap?
To check indexing, type "site:domain.com" into Google and count the results. A more precise method is Google Search Console: in the "Pages" report, compare the number of URLs "submitted via sitemap" with the number "indexed". A wide gap points to problems worth investigating: noindex pages, duplicate content, crawl errors, or content judged too thin to deserve indexing. At ConvertiLab, during SEO rebuilds, we have consistently raised indexing rates from 70% to 95%+ by cleaning up sitemaps and fixing robots.txt files. The average time for a new page to appear in Google after a sitemap submission is 3 days to 2 weeks for sites with a good crawl history. Generate my robots.txt and XML sitemap for free →
