Stop checking URLs one by one. This guide compares three approaches for bulk indexing verification: Google Search Console sitemap reports, Screaming Frog crawl analysis, and manual bulk checks using URL Inspection API. Includes worked examples, common failures, and a step-by-step flowchart.
Quick calculator. Put in the expected monthly value of a page or link batch and the natural waiting time.
Checking indexing status for one URL is trivial. Chrome extension, URL Inspection Tool, done. But when you manage a site migration, audit a backlink profile for a 10,000-URL domain, or verify guest post placements across 50 domains, single-URL checks break down. You need a site indexing checker that handles bulk URL index verification without manual tedium.
In practice, when you import a list of 1,500 candidate URLs into a bulk checker, you will find that roughly 12% are not indexed. But the real insight is why. Blocked by robots.txt, tagged with noindex, soft 404s, or simply not discovered. A good bulk verification tool distinguishes between these failure modes. It does not just give you a yes/no.
A common situation we see: an agency uploads a list of 200 guest post URLs to check for indexing. They get 35 'Not indexed' results. Without the failure reason, they assume the publisher did not publish. In reality, 22 of those are blocked by the publisher's plugin, 8 are still in a crawl queue, and 5 are dead links. Bulk verification with diagnostics saves hours of back-and-forth.
| Method | Scale & Speed | Failure Diagnostics | Hidden Risks |
|---|---|---|---|
| Google Search Console Sitemap Report Native GSC feature under Sitemaps > 'See page indexing' | Up to 50,000 URLs per sitemap. Refreshes every few hours. No API cost. | Shows 'Submitted but not indexed' with a reason (Crawled but not indexed, Discovered but not indexed, etc.). Weak for pages not in sitemap. | Only covers URLs you submitted. Does not check ad-hoc lists. Reasons are often vague: 'Crawled but not indexed' can mean low quality or duplicate. |
| Screaming Frog SEO Spider Desktop crawler with custom extraction and indexability checks | Unlimited URLs on licensed version. Crawl 500 URLs in ~3 minutes. Good for on-site bulk checks. | Flags noindex tags, canonical issues, robots.txt blocks, meta robots, and HTTP status codes. Exports to CSV with all fields. | Cannot check indexing status for external domains. Requires local installation and memory. Does not call Google's index data; only checks page signals. |
| Manual Bulk via URL Inspection API Google Indexing API or third-party bulk checker tools (e.g., RankMath, SEMrush) | API quota: 2,000 queries per day per property. Batch processing with 100-200 URLs per request. Fast for ad-hoc lists. | Returns index status, last crawl date, and any blocking errors. Can verify both submitted and non-submitted URLs. | API quota limit can stall large projects. Requires OAuth setup or a paid tool wrapper. Some tools report 'Not indexed' for pages that are just slow to update. |
| Manual Spot-Check (Hybrid) Use GSC for sitemaps, Screaming Frog for on-site, API for external lists | Combines strengths. Takes 30-60 minutes for a 1,000-URL audit. Most accurate. | Triangulates reasons: Screaming Frog finds on-page blocks, API confirms Google's view, GSC shows submission history. | Requires access to three tools. Easy to duplicate effort or miss overlaps. Best for high-stakes audits (migrations, penalties). |
Export from sitemap, backlink tool, or spreadsheet. Deduplicate and remove empty rows. Max 2,000 URLs per run for API method.
Run a quick Screaming Frog crawl (or simple regex) to flag noindex tags, disallowed robots.txt, and 4xx status codes. Expect to drop 15-20%.
Use GSC Sitemap report for submitted URLs, or API-based tool for ad-hoc lists. Process in batches if >2,000 URLs.
Group by status: 'Indexed', 'Discovered but not indexed', 'Crawled but not indexed', 'Blocked'. Prioritise 'Crawled but not indexed' for content improvement.
For blocked: fix robots.txt or noindex. For discovered: improve internal linking. For crawled: improve content quality or remove. Re-check after 2 weeks.
Bulk indexing verification sounds simple until you hit real data. Here are three failures we see repeatedly.
Duplicate URL lists. An agency once uploaded a list of 1,000 URLs that contained 400 duplicates with and without trailing slashes. The bulk checker reported 400 duplicates as 'Already indexed' and the 600 unique ones as 'Not indexed'. The actual indexing rate was 85%, not 60%. Always normalize URLs before upload: lowercase, remove trailing slash, resolve redirects.
Empty results from GSC. When you submit a sitemap and GSC shows zero indexed pages, the common mistake is to assume Google is ignoring you. In practice, it often means the sitemap URL is incorrect (e.g., 404), or the sitemap contains only noindexed pages. We once saw a sitemap with 10,000 URLs that returned zero indexed results because the developer had accidentally set the entire site to noindex. The bulk checker is only as good as the input.
Slow API vendors. Some third-party bulk indexing tools throttle responses to avoid hitting Google quotas. We tested a popular tool that took 45 minutes to check 200 URLs. The same list via a custom script using the Google Sitemap guidelines for news sitemaps completed in under 2 minutes. Always check the tool's batch size and rate limits before committing.
Scenario: An SEO agency needs to verify that 500 guest post URLs (published across 30 different blogs) are indexed in Google. They have a CSV with columns: URL, target domain, publish date.
Step 1: Clean the list. Remove duplicates (12 found), normalize URLs (convert http to https, remove trailing slash). Final list: 488 URLs.
Step 2: Run a quick Screaming Frog crawl on each target domain (only the specific post URLs). This found 34 URLs returning 404 (post deleted), 22 with noindex meta tag, and 8 with robots.txt disallow. Total removed: 64. Remaining: 424.
Step 3: Feed the 424 URLs into a bulk URL Inspection API tool (using a Google Cloud project with OAuth). Processed in 4 batches of 106. Total time: 8 minutes.
Results: 347 indexed (82%), 52 discovered but not indexed, 25 crawled but not indexed. The 52 discovered URLs were from a slow blog that had not been crawled yet. The 25 crawled URLs were from low-authority domains with thin content.
Action: Requested re-crawl for the 52 discovered URLs via GSC. For the 25 crawled URLs, asked publishers to add internal links and improve content. Re-checked after 14 days: 18 of 25 were now indexed.
This workflow saved roughly 6 hours of manual single-URL checking and provided granular failure reasons instead of a binary result.
Not all 'Indexed' results are equal. A bulk checker may report a URL as indexed because Google has it in the index, but the page may be a soft 404, a thin affiliate page, or a canonicalized duplicate. We have seen cases where 30% of indexed URLs were actually serving no value because they were indexed under the wrong canonical or returning a 200 status with zero content.
Another hidden failure: the 'Discovered but not indexed' status often gets ignored. Teams assume Google will get to it. In reality, that status means Google found the URL but chose not to crawl it yet due to perceived low priority. If you have more than 5% of your bulk list in this state, your internal linking or sitemap freshness is likely weak. The drip-feed indexing approach for managing link velocity can help here: instead of pushing all URLs at once, submit them gradually over weeks to signal importance.
Finally, watch for rate-limited tools that silently drop results. We tested a tool that claimed to check 500 URLs but actually only processed 380 due to a hidden API timeout. The output showed 380 indexed, 0 not indexed. The missing 120 URLs were simply not checked. Always verify the count of results matches your input count.
Deduplicate and normalize URLs: lowercase, trailing slash, https, remove UTM parameters.
Exclude URLs with obvious blocks: noindex meta tag, robots.txt disallow, or 4xx status (use Screaming Frog or a quick crawl).
Check tool batch limits: GSC API = 2,000/day, most third-party tools = 100-500 per batch.
Verify tool actually returns failure reasons (crawled vs. discovered vs. blocked), not just yes/no.
Export results to CSV and sort by failure type before taking action.
Set a re-check reminder: indexing status changes over time; do not treat results as permanent.
For agencies, a tool that supports multiple Google Search Console properties and has a shared quota is ideal. Look for tools that offer team accounts, batch processing of up to 2,000 URLs per run, and exportable CSV reports with failure reasons. Screaming Frog for on-site checks combined with an API-based checker like RankMath or custom script gives the best diagnostic depth.
Yes, but only if the tool uses Google's URL Inspection API, which does not require ownership of the target domain. You can submit any public URL. However, the API has a daily quota of 2,000 queries per Google Cloud project. For large backlink audits (>5,000 URLs), you need to spread checks over multiple days or use a paid tool with higher limits.
You cannot with a single Google API key (2,000/day limit). Workaround: combine GSC Sitemap report for URLs you submitted (up to 50,000 per sitemap) with Screaming Frog crawl of your own site. For external URLs, split the list into 2,000 batches and check over 5 days. Alternatively, use a tool that pools multiple API keys or uses cached data.
Top errors: (1) 'URL not found' - the tool cannot reach the page; check if the URL is live. (2) 'Disallowed by robots.txt' - remove the block or use a different tool that ignores robots. (3) 'API quota exceeded' - wait 24 hours or reduce batch size. (4) 'Empty result set' - your list may contain only already-indexed URLs; add some known non-indexed URLs as a control.
No completely free unlimited checker exists because Google's API charges per query after the free tier (200 queries/day for free). However, you can use Google Search Console's Sitemap report for free with no daily limit, but it only checks URLs you submitted. Screaming Frog (free tier up to 500 URLs) is also free for on-site checks. For true bulk external checks, you need a paid tool or API key.
Drip-feed indexing means you submit guest post URLs to Google gradually (e.g., 10-20 per day) rather than all at once. This mimics natural link acquisition and avoids triggering spam filters. A bulk checker verifies which posts are already indexed before you submit new ones. After each submission batch, re-check the remaining URLs to see if any were indexed organically. The linked Medium article explains this in detail.
That percentage is abnormally high. Common causes: (1) Thin or duplicate content - improve each page's unique value. (2) Poor internal linking - add links from high-authority pages. (3) Sitemap issues - ensure URLs are in a sitemap and the sitemap is submitted to GSC. (4) Crawl budget waste - block low-value URLs via robots.txt. Fix content first, then request indexing via GSC for the top 10 pages.
Yes. You need a Google Cloud project with the URL Inspection API enabled and OAuth 2.0 credentials. Write a script (Python, Node.js) that reads a CSV, sends POST requests to the API (100 URLs per batch), parses the JSON response, and writes results to a new CSV. The API returns index status, crawl date, and blocking errors. Be mindful of the 2,000 queries per day quota and 100 queries per 100 seconds limit.
Most bulk checkers only report binary indexed/not indexed and do not detect soft 404s. A soft 404 (a page that returns 200 but has no meaningful content) will show as 'Indexed'. To detect soft 404s, you need a separate check: use Screaming Frog to compare page word count or content length, or use Google Search Console's 'Soft 404' report. Combine both tools for a complete picture.
After migration, export the old site's URL list (all pages). Run a bulk check to see which old URLs still return 301 redirects vs. 404s. Then export the new site's URL list and check indexing status. Compare the two: any old URL that is not redirecting and not indexed is a problem. Use Screaming Frog to verify redirect chains. Expect 10-15% of new URLs to not be indexed initially; request indexing via GSC for the most important ones.