How to Extract URLs from Sitemap for Better SEO Audits

When I first started doing SEO audits, I used to open a website's XML sitemap in the browser and manually copy each URL. It worked for small sites, but as soon as I hit a few hundred pages I knew there had to be a better way. Learning to extract URLs from sitemap files changed how I approach technical SEO and content optimization. It saves hours and reduces error.

Every site you audit has a sitemap. Whether it is generated by Yoast SEO, Rank Math, or a custom script, that file is the single source of truth for what pages the site wants Googlebot to crawl and index. But reading it as a human is impractical. You need a way to pull those URLs into a spreadsheet or an analysis tool quickly.

Why Bother Extracting URLs from a Sitemap?

A sitemap is not just a list of links. It reveals the structure of a website. When you extract URLs from sitemap files, you see every page the site considers important. You can then check for broken links, verify that canonical URLs are set correctly, and confirm that 301 redirects point where they should. This is foundational for any SEO audit.

I have seen too many audits where the reviewer only looks at the homepage or a handful of landing pages. That misses entire sections of a site. For example, a blog with hundreds of articles might have orphaned posts that never appear in the navigation. Those posts still need to be crawled and indexed. If they are not in the sitemap, you would never know they exist.

Methods to Extract URLs from Sitemap

There are many ways to do this. The right method depends on your technical comfort and the size of the sitemap. Below are the approaches I have used in real projects.

Using a Free Online Tool

Several free SEO toolkits can extract URLs from sitemap files for you. You paste the sitemap URL, and the tool returns a list of all URLs. This is the quickest option for a one-off check. The site invisible.is offers a free tool that does exactly this. It also helps with content optimization and technical SEO analysis after you have the list.

The trade-off is that you are uploading your sitemap to a third-party service. For public sites that is usually fine, but for internal staging sites you might want a local method.

Using cURL and grep

If you are comfortable with the command line, cURL is fast and reliable. A simple command like:

curl -s https://example.com/sitemap.xml | grep -oP '\K[^<]+'

This pulls every URL from a standard sitemap. If the sitemap uses a sitemap index file, you first download the index, then loop through each child sitemap. It is not elegant, but it works on any system with cURL installed. I have used this on servers where no other tools were available.

Using Python and Beautiful Soup

For more control, a short Python script with Beautiful Soup can parse the XML, handle namespaces, and output clean data. This is my go-to method when I need to extract URLs from sitemap files regularly, especially when I want to combine the output with other data like page titles or response codes.

The script fetches the sitemap, finds all tags, and prints the URLs. It can also handle the sitemap index file by recursively fetching each sub-sitemap. The flexibility is valuable, but it does require Python and Beautiful Soup installed.

Using Screaming Frog SEO Spider

Screaming Frog is a desktop tool that many SEO professionals use. It can crawl a site from a seed list or directly import a sitemap. Once loaded, you can export all discovered URLs. This tool also checks for broken links, missing meta descriptions, and duplicate content. The downside is that the free version limits you to 500 URLs, which may not be enough for large sites.

What to Do After You Have the URLs

Getting the list is step one. The real work begins after extraction. Here are the checks I run on every sitemap I extract.

Check for Broken Links

Run the list against a crawler or use Google Search Console to see which pages return 4xx or 5xx errors. Broken links in a sitemap tell Googlebot that the site is poorly maintained. Fix them with 301 redirects or remove them from the sitemap.

Verify Canonical URLs

Every URL in the sitemap should match the canonical URL for that page. If the sitemap lists a page with a trailing slash but the canonical points to the non-slash version, you have a signal conflict. I have seen this happen after a site migration where 301 redirects were not updated properly.

Look for Indexability Issues

Check each URL against the index status in Google Search Console. Sometimes pages that you want indexed are not, or pages you want noindexed are still in the sitemap. This is a common mistake with Yoast SEO and Rank Math when the plugin settings are not reviewed after a site update.

Common Pitfalls

One mistake I see often is extracting URLs from sitemap files but ignoring the sitemap index. A sitemap index file points to multiple child sitemaps. If you only fetch the index, you get no actual page URLs. You must parse the index and then extract from each child.

Another pitfall is assuming the sitemap is complete. Many sites do not include all pages, especially if the CMS has a limit. For example, Yoast SEO by default includes only the first 200 posts in a post type sitemap if you have not set a custom limit. Always cross-check the extracted list with a full crawl from a tool like Ahrefs or SEMrush to confirm coverage.

Finally, beware of caching. Some CDNs or caching plugins serve a stale version of the sitemap. If you are using a tool like Screaming Frog or a Python script, force a fresh fetch by adding a cache-busting parameter or clearing the server cache first.

When to Automate

If you manage multiple sites, manual extraction becomes tedious. I set up a weekly cron job that uses cURL and a simple Bash script to extract URLs from sitemap files for all my client sites. The script emails me the list, and I compare it against the previous week to spot new pages or missing ones. This catches issues before they affect indexing.

Automation is also useful for monitoring the health of a sitemap during a site migration. When you move domains or restructure URLs, the sitemap should change. Running an automated extraction daily lets you see when the old URLs drop out and new ones appear. You can then verify that 301 redirects are in place and that Googlebot is seeing the right version.

Tools That Complement Sitemap Analysis

After you extract URLs from sitemap files, you will likely want to analyse them further. Tools like Ahrefs and SEMrush let you paste a list of URLs and run bulk checks for backlinks, traffic, and ranking keywords. Google Search Console provides the index coverage report for the same URLs. If you find pages that are not indexed, you can request indexing directly from the console.

For a deeper technical SEO audit, I combine the sitemap list with a full crawl from Screaming Frog. This shows me every page the site has, not just the ones in the sitemap. The overlap and the gaps are where the problems live. Pages that are in the sitemap but not discovered by the crawler often point to a linking or navigation issue.

Final Thoughts

Learning to extract URLs from sitemap files is a small skill that pays off repeatedly. It is one of those tasks that seems simple but reveals so much about a site. Whether you use a free online tool, a command-line utility, or a script, the key is to make it part of your regular workflow. The data you get is the foundation for every content optimization and technical SEO decision you make.

The next time you start an SEO audit, do not guess what pages a site has. Extract the sitemap, verify it, and work from a complete list. Your future self will thank you when you catch that orphaned landing page or that broken link that was hiding in plain sight.