How to check all pages on a website
How to find every page on a website, see which ones Google indexes, and spot the broken, thin and orphaned pages a sitemap never lists.
·4 min read
To check all pages on a website, build the list first from four sources, then test the list. A sitemap gives the pages the owner wants found, Search Console gives the pages Google knows, a crawl gives the pages that actually exist, and a site: query gives a rough cross-check. Any page that appears in one list and not the others is the finding, not the noise.
The order matters. Checking titles on a page that returns a 404 wastes the check.
Step 1: Build the page inventory
| Source | What it lists | Where to get it |
|---|---|---|
| XML sitemap | Pages the site wants indexed | /sitemap.xml or the Sitemap: line in robots.txt |
| Search Console Pages report | URLs Google has seen, indexed or not | Indexing, Pages, export |
| A crawl | Pages reachable by following links | A crawler run from the homepage |
site: query |
A rough index count | Google search, or Bing for a second opinion |
| Server logs | URLs actually requested | Access log for the last 30 days |
The sitemap and the crawl disagree by design. A sitemap can list a page that is no longer linked, and a crawl finds a page that was never added to the sitemap. Both are real problems, and each one needs a different fix.
A sitemap also has hard limits: 50,000 URLs and 50 MB uncompressed per file, with a sitemap index above that. A file that breaks either limit is rejected, and the pages inside it may never be read.
Our sitemap validator fetches a sitemap, follows a sitemap index and reports invalid XML, blocked URLs, wrong hosts and pages that return an error.
Step 2: Check which pages Google will index
For each URL on the list, four things decide indexability.
- Status code. A 200 qualifies. A 301 sends Google elsewhere, and a 404 or 410 removes the page.
- Meta robots.
noindexin the head keeps the page out of the index. - X-Robots-Tag. The same instruction can arrive as a response header, which is easy to miss in the page source.
- Canonical. A canonical pointing at another URL moves the page's signals there, so the page itself is not the one indexed.
Checking these one URL at a time does not scale. Our indexing checker reads all four signals for a URL in one request.
Then read the Search Console Pages report. It groups URLs by status: indexed, crawled but not indexed, discovered but not indexed, excluded by a noindex tag, alternate page with a proper canonical, and so on. The exclusions explain more than the indexed count does.
Step 3: Test every URL's status code
After indexability, check that each URL on the list still returns what it should. A site with 2,000 pages usually has a few dozen that moved or died and were never redirected.
Our HTTP status bulk checker tests a list of URLs and returns the status and final URL for each one. Run it against the sitemap export, not against a hand-typed list, so the test covers what the site claims to publish.
Look for three patterns:
- A 404 or 410 on a page that earns links. Restore it or redirect it to the closest replacement.
- A 301 chain. Two hops or more should collapse to one.
- A 200 that redirects through a trailing slash or host.
wwwand the apex are different hosts; pick one and redirect the other.
Step 4: Find the pages that are not linked
A page with no internal links is an orphan. Google can still find it through the sitemap, but it gets almost no internal link equity, so it usually ranks below the pages in the navigation.
Compare the crawl to the sitemap. Every URL in the sitemap with no path from the homepage is a candidate. Our orphan page finder does that comparison.
The reverse also matters: pages reachable by crawl that are missing from the sitemap. Those are the pages a broken template or a paginated archive is leaking, and they are usually duplicates.
Step 5: Check the pages that do get indexed
Once the list is clean, check what is on the pages. Four checks catch most of it:
| Check | Pass | Tool |
|---|---|---|
| Title | Unique, 50 to 60 characters, specific | On-page checker |
| Meta description | Unique, 150 to 160 characters, specific | On-page checker |
| Headings | One H1, logical H2s | On-page checker |
| Internal links | Linked from a relevant page, not only the sitemap | Broken link finder |
Duplicate titles are the most common finding on a site with more than a few hundred pages, usually from a template that reuses one title across a category. Sort the exported titles, and any group larger than one is a page worth rewriting.
Step 6: Turn the inventory into a schedule
A one-off crawl goes stale in a month. Keep it current with three habits:
- Regenerate the sitemap on publish, so it never lists a deleted page.
- Check the sitemap weekly with the validator. A malformed sitemap fails silently.
- Read the Pages report monthly, and compare the excluded count with last month's.
One person can check 20 pages by hand. A site with 2,000 pages needs the list built once and the test repeated, which is what a crawler is for. Our website audit checklist covers the rest of the audit around the page inventory, and orphan pages explains the internal link side in more detail.
Check the sitemap before you check the pages
The sitemap is the cheapest way to see what a site claims to publish. Our sitemap validator reads a sitemap or a sitemap index and reports the problems that keep Google from using it. Fix that first, then check indexability, status codes and the pages themselves.



