A sitemap is a hint, not a directive. You hand it to a crawler and the crawler decides what to do with it. Google treats sitemap.xml as a discovery suggestion — a list of URLs you consider important — and reserves the right to crawl, index, or ignore each entry based on its own signals. Your sitemap strategy is therefore not “list everything.” It is “list what you want indexed, omit what you do not, and do not waste the crawler’s attention on URLs that will be filtered out anyway.”

The cost of getting this wrong is not theoretical. A sitemap stuffed with redirected URLs, noindex pages, soft-404s, and parameter duplicates trains Google to treat the file as low-signal. Crawl budget leaks into URLs that never had a chance of ranking, and the genuinely important pages you wanted indexed get discovered more slowly. I have seen Search Console reports where the only thing a sitemap was accomplishing was the steady discovery of 410-gone endpoints nobody had pruned in years. The fix is to be selective. Below is what belongs in the file, what does not, and how to keep the metadata honest.

What a sitemap is for

Per the sitemaps.org protocol, a sitemap is “an XML file that lists URLs for a site along with optional metadata about each URL.” The protocol is explicit that the optional metadata is advisory. The three optional tags are lastmod, changefreq, and priority. Crawlers may ignore them. Google has said publicly that it largely ignores changefreq and priority, and only uses lastmod if it is consistently accurate.

What the sitemap actually buys you is discovery. For a small, well-linked site, crawlers will find every page without one. A sitemap starts to matter when the site is large, freshly launched, poorly internally linked, or has isolated page clusters (for example, a tool launched without a card on the homepage). It is also the channel through which you tell Google about new pages faster than waiting for the next crawl of your site’s existing pages.

It is not a ranking signal. Listing a URL in a sitemap does not make it rank. Not listing a URL does not prevent it from ranking if it is internally linked and crawlable. Treat the file as a map of your canonical, indexable surface area, not a wish list.

Include: canonical, indexable, important pages

The URLs that belong in your sitemap are the ones that satisfy all three of these conditions:

  1. They return HTTP 200 with a canonical tag pointing to themselves (self-referential canonical).
  2. They are indexable. No noindex directive, no Disallow in robots.txt, no X-Robots-Tag: noindex.
  3. They are pages you actively want ranking in search results.

The third condition is the one people skip. A URL can be canonical and indexable and still not belong in the sitemap. Tag pages on a small blog, filter-result pages with thin content, paginated archives whose canonical resolves to page one. These are technically crawlable, but they are not pages you are optimizing for ranking. Each entry in the sitemap is a vote of attention. Spend those votes on the pages that have a real chance to rank.

For a developer-tools site like this one, that means the homepage, the tool pages themselves, the category hubs, the blog articles, and the legal pages. Each tool page is the target of an internal linking strategy from the homepage cards and the category pages, and each one is listed in the sitemap. A typical entry looks like this:

<url>
  <loc>https://devtidy.com/json-formatter/</loc>
  <lastmod>2026-07-14</lastmod>
</url>

No changefreq, no priority. They are noise. Keep each <url> to loc plus an honest lastmod.

Exclude: noindex, redirects, 4xx/5xx, parameter duplicates, variants

If a URL fails any of the three conditions above, leave it out. Specifically, drop:

  • noindex pages. Listing a URL in the sitemap while sending a noindex directive on the page itself is a contradiction. Google has to resolve it, and you have wasted a sitemap slot. Search Console will flag these as errors.
  • Redirected URLs. If /old-tool/ 301-redirects to /new-tool/, list /new-tool/ only. Listing the redirect source teaches Google the sitemap is stale.
  • 4xx and 5xx URLs. Every broken link in the sitemap is a credibility hit. Run a status check before publishing and remove anything that does not return 200.
  • Faceted sort and filter parameters. A product listing with ?sort=price and ?color=red variants produces dozens of URLs that resolve to substantially duplicate content. List the canonical version, not every parameter combination.
  • Duplicate variants. Print-friendly versions, AMP variants, www vs non-www, http vs https. If one is the canonical, list the canonical. The rest should 301 or carry rel=canonical to the canonical version and stay out of the sitemap.
  • Paginated archives beyond page one. If page two of your blog archive canonicals to page one, do not list page two. If each page is self-canonical, list the ones that have a real ranking purpose.

A useful heuristic: if you cannot explain in one sentence why this specific URL should rank for a specific query, it probably does not belong in the sitemap.

sitemap.xml vs a sitemap index

The sitemaps protocol caps a single sitemap file at 50,000 URLs and 50MB uncompressed. Past either limit you split the file and list the pieces in a sitemap index:

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>https://devtidy.com/sitemap-tools.xml</loc>
    <lastmod>2026-07-14</lastmod>
  </sitemap>
  <sitemap>
    <loc>https://devtidy.com/sitemap-blog.xml</loc>
    <lastmod>2026-07-14</lastmod>
  </sitemap>
</sitemapindex>

You do not need to wait for the 50,000-URL limit to split. Splitting by content type — tools, blog, docs, category pages — keeps each file human-auditable. When something breaks on the blog, you regenerate sitemap-blog.xml without touching the tools file. Most static-site generators, Astro included via @astrojs/sitemap, will produce an index automatically when you cross the threshold and let you partition by path if you want finer control.

Submit the index URL (typically /sitemap-index.xml or /sitemap.xml) in Google Search Console. The crawler follows the index to the child files.

lastmod — keep it honest

lastmod is the one optional tag Google actually uses, and only when it is accurate. The moment you ship a sitemap with bogus lastmod values, Google learns the field is unreliable on your site and starts ignoring it. The usual offenders are scripts that stamp every URL with today’s date regardless of when the page changed, or a stale date from two years ago that nobody ever updates. Once that trust is gone, getting it back is slow.

The rule is simple: lastmod should reflect when the page content materially changed, not when the sitemap was generated. A blog article that you edited on July 14 gets <lastmod>2026-07-14</lastmod>. A tool page whose copy has not changed since March keeps its March date, even if you regenerate the sitemap today.

For the deeper treatment of lastmod, changefreq, and priority, read the companion piece on reading a sitemap. It covers how Google weighs each tag, what the protocol actually specifies, and the failure modes of each. This article is about which URLs belong in the file at all; that one is about what to put inside each <url> element.

Validate, then publish

Before you publish a new sitemap, validate it. XML errors, unescaped ampersands in URLs, missing urlset wrappers, and oversized files are the usual failures. Run the file through the Sitemap Validator to catch structural problems, encoding issues, and protocol violations before Google does. A broken sitemap does not just fail silently. It can sit in Search Console flagged as an error for weeks while the real URLs you wanted indexed go uncrawled, and the error message Google gives you will rarely point at the actual line that broke the parse.

Discovery is the other half. The sitemaps protocol lets you declare your sitemap in robots.txt so any crawler can find it without you submitting to a specific console:

User-agent: *
Allow: /

Sitemap: https://devtidy.com/sitemap-index.xml

If you do not already have a robots.txt, generate one with the Robots TXT Generator and add the Sitemap: directive on its own line. Google will fetch robots.txt on every crawl and follow the sitemap reference from there. Belt and braces: submit the same URL through Search Console as well.

FAQ

Does every page need to be in the sitemap?

No. Only pages you want indexed, that return 200, that are self-canonical, and that are not blocked by noindex or robots.txt belong in the sitemap. Pages that fail any of those conditions should be omitted, not listed.

Will listing a URL make it rank faster?

No. A sitemap is a discovery hint, not a ranking signal. It helps Google find the URL; ranking depends on content, links, and relevance. A page can rank without ever appearing in a sitemap if it is internally linked.

Should I list redirected URLs?

No. List only the redirect destination. Listing the source of a 301 redirect wastes a sitemap slot and signals that the file is not being maintained.

How often should I regenerate the sitemap?

Regenerate whenever you publish or materially update pages — for a static site, at build time. The right cadence is tied to content change, not to a fixed schedule. What matters is that lastmod reflects the actual change.