A sitemap is a file, most often in XML format, that lists the URLs of a website to communicate them to search engine bots. Its primary role is to signal the existence of pages, not to force their crawling or indexing. This distinction changes the way it is designed and utilized.
XML Sitemap: a discovery signal, not an indexing guarantee
Google specifies in its documentation: a sitemap serves to indicate URLs, but it neither forces crawling nor indexing. The bots then decide, based on their own criteria, which pages deserve to be crawled and added to the index.
You may also like : Essential Equipment for a Comfortable and Well-Equipped Home
This nuance has practical consequences. Adding hundreds of pages with a 404 error, redirects, or marked as noindex in a sitemap produces nothing positive. The current best practice is to list only canonical, indexable URLs that return a 200 code.
A poorly filtered sitemap sends noise to the bots. Instead of speeding up the discovery of your strategic content, it dilutes the crawler’s attention across unnecessary pages. In practice, on the sitemap page of Excargot, each listed URL corresponds to a page that is actually accessible, which illustrates this principle of prior filtering well.
Related reading : Essential Tips for Easily Installing a Wooden Pergola at Home
Crawl budget and deep pages: when the sitemap becomes truly useful
For a showcase site with five pages linked together by a menu, the sitemap adds almost nothing. The bots find all the internal links themselves and crawl the entire site in a few seconds.
The sitemap becomes valuable in specific cases:
- Large e-commerce catalogs where some product sheets are buried several clicks deep from the homepage, without a direct link from the main navigation.
- Editorial archives or technical documentation, where old content gradually loses its internal links over time due to redesigns.
- Dynamic content (filtered search pages, personalized results pages) that does not appear in any fixed menu.
In these situations, the sitemap compensates for the shortcomings of internal linking. Pages that are poorly linked internally sometimes have no other way to be discovered by a bot. The sitemap provides them with a direct entry point.
It is not a comfort tool; it is a safety net for content that the traditional navigation structure does not cover.
Submitting a sitemap in 2025: the end of pinging and alternatives
For years, webmasters could notify Google of an update to their sitemap via an HTTP ping. Google deprecated this ping method in 2023. The two recommended methods are now submission via Google Search Console and declaring the sitemap URL in the robots.txt file.
The declaration in robots.txt is the simplest. You just need to add a line like:
Sitemap: https://www.example.com/sitemap.xml
Every bot that checks the robots.txt file (and the vast majority do so first) automatically discovers the location of the sitemap. No need for manual notification, no need for an account in a search engine console.
Google Search Console remains useful for monitoring the processing status of the sitemap: how many URLs have been submitted, how many are indexed, and especially what errors the bot encountered during crawling.
XML Sitemap and HTML Sitemap: two files, two distinct functions
The XML sitemap is aimed at bots. The HTML sitemap is aimed at human visitors. Confusing the two leads to ineffective design choices.
An HTML sitemap is a page on the site, accessible from the footer or menu, that presents the structure in the form of a list of clickable links. For visitors looking for a specific page without going through the internal search engine, this page provides an overview of the site’s structure.
The XML sitemap does not replace good internal linking or a coherent site architecture. A site that relies solely on its XML sitemap to make its pages accessible has a structural problem, not a sitemap problem.
The two formats complement each other. The XML helps bots map the site. The HTML helps visitors navigate when the main navigation is insufficient.
Filtering your sitemap: URLs to exclude
Listing all the URLs of a site in the sitemap is a common mistake. Some pages do not belong in this file:
- Pages with a noindex tag or a canonical directive pointing to another URL.
- Pages protected by a password or reserved for a member area.
- Pages of internal search results or combinatorial filters that generate thousands of URLs without unique content.
- URLs with a 301 redirect or returning a 404 code.
A clean sitemap contains only the pages that the site wants to see indexed. This discipline of filtering makes the bots’ work easier and improves the readability of reports in tracking tools.
On large sites, breaking down into multiple sitemap files (one per section, one per content type) simplifies maintenance and allows for quick identification of which part of the site is problematic during crawling.
The sitemap remains a discreet technical tool, but its impact entirely depends on the quality of what is included. A well-filtered file, declared in robots.txt and monitored in Search Console covers all the needs for the majority of websites.