How to Automate IndexNow for Faster Search Indexing

When you publish or update a page, there's a gap between "this page exists" and "a search engine has actually crawled and indexed it." For a site that publishes or updates content daily, that gap adds up — pages can sit unindexed for days if you're relying entirely on a search engine to rediscover your sitemap on its own schedule. IndexNow is a protocol (backed by Bing, and consumed by a few other search engines) that lets you directly notify search engines the moment a URL changes, instead of waiting for them to recrawl your sitemap. I have it wired directly into my content pipeline, so every publish or edit fires off a submission automatically. Here's exactly how that's built, and what I'd do differently if I were setting it up again.

What IndexNow Actually Does (and Doesn't Do)

IndexNow directly notifies search engines when a website URL changes instead of waiting for sitemap recrawling

IndexNow is a simple HTTP-based protocol: you host a small verification key file on your domain, and then you send a request — either a single URL or a batch — to an IndexNow-participating endpoint whenever a page is created, updated, or deleted. That's the entire mechanism. What it's genuinely useful for: telling search engines "this URL just changed" the moment it happens, rather than relying purely on periodic recrawling. What it doesn't do: guarantee immediate indexing, guarantee ranking, or replace a sitemap. It's a notification mechanism, not an indexing guarantee — submitting a URL tells a search engine it's worth checking soon, not that it will definitely be indexed or ranked well. It's also worth being clear that not every search engine participates; treat it as one additional signal alongside your sitemap, not a replacement for one.

The Setup: Verification Key and Endpoint

IndexNow verification key file and API request for submitting updated website URLs

Before submitting anything, IndexNow requires proving you control the domain. This is done with a plain text key file hosted at your domain root:

text

https://example.com/a1b2c3d4e5f6.txt

The file's content is just the key itself, and the filename must match it. Once that file is live and reachable, you can start submitting URLs to an IndexNow endpoint:

text

POST https://api.indexnow.org/indexnow
Content-Type: application/json
 
{
  "host": "example.com",
  "key": "a1b2c3d4e5f6",
  "keyLocation": "https://example.com/a1b2c3d4e5f6.txt",
  "urlList": [
    "https://example.com/blog/some-new-article",
    "https://example.com/blog/another-updated-page"
  ]
}

That's the entire API surface for submitting. The batching support (urlList accepting multiple URLs in one request) is what makes this practical to automate — you're not limited to one HTTP request per URL.

Wiring This Into a Publish Pipeline

IndexNow automated URL queue with batch submission every five minutes and automatic retry on failure

The naive version of this is: after saving a new or updated article to the database, immediately fire a submission request with that one URL. This works, but it has a real downside once you're publishing or updating multiple pages close together — you end up making many separate small HTTP requests to the IndexNow endpoint in quick succession, which is both wasteful and, depending on the endpoint's own rate limiting, can start failing. What I actually run is a batching queue: instead of submitting immediately on every single change, changed URLs get pushed into a small in-memory queue, and a scheduled job flushes that queue every few minutes as a single batched request.

const pendingUrls = new Set();
 
function queueForIndexNow(url) {
  pendingUrls.add(url);
}
 
async function flushIndexNowQueue() {
  if (pendingUrls.size === 0) return;
 
  const urlList = Array.from(pendingUrls);
  pendingUrls.clear();
 
  try {
    const res = await fetch('https://api.indexnow.org/indexnow', {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({
        host: 'example.com',
        key: process.env.INDEXNOW_KEY,
        keyLocation: `https://example.com/${process.env.INDEXNOW_KEY}.txt`,
        urlList,
      }),
    });
 
    if (!res.ok) {
      console.error(`[IndexNow] Submission failed: ${res.status}`);
      // re-queue on failure rather than silently dropping the URLs
      urlList.forEach(u => pendingUrls.add(u));
    }
  } catch (err) {
    console.error('[IndexNow] Request error:', err);
    urlList.forEach(u => pendingUrls.add(u));
  }
}
 
// flush every 5 minutes
setInterval(flushIndexNowQueue, 5 * 60 * 1000);

Every place in the codebase that publishes, updates, or unpublishes a page calls queueForIndexNow(url) — the article save handler, the course-import cron job, anything that changes a page's live content. None of those call sites need to know or care about batching, rate limits, or retry logic; they just add a URL to the queue and move on.

The Retry Logic Actually Matters

The first version of this didn't re-queue on failure — a failed request just logged an error and moved on, which meant a transient network blip or a rate-limit response could silently drop URLs that never got submitted at all. Re-adding the failed batch back into the pending set (as shown above) means a temporary failure just gets picked up on the next flush cycle instead of being lost. This is a small detail, but it's the difference between "this system is quietly failing some percentage of the time and I'd never know" and "this system degrades gracefully and catches up automatically."

What I'd Change If Starting Over

A few things I'd do differently in hindsight:

Track submission counts, not just fire-and-forget. Right now the system logs how many URLs were submitted per flush, but I don't have a longer-term record of submission volume over time. A simple counter or log file would make it easier to spot if the queue silently stopped flushing for some reason.
Deduplicate more aggressively across close-together edits. If a page gets edited three times in five minutes, it only needs to be submitted once — the Set in the code above already handles this within a single flush window, but a page edited just after a flush and then again just before the next one will get submitted twice. Not harmful, just slightly wasteful.
Separate the "new page" case from the "minor edit" case. A brand-new page and a typo fix on an existing page arguably deserve different urgency — right now both go through the identical queue with no distinction.

Frequently Asked Questions

Does IndexNow replace submitting a sitemap?

No. A sitemap remains the primary way search engines discover the full structure of your site, especially for a first crawl. IndexNow is a supplementary "this specific URL just changed" signal for faster pickup of individual changes — keep both in place.

Do I need to submit every single page on my site, or only new/changed ones?

Only new, updated, or removed pages need submitting. There's no benefit to resubmitting unchanged URLs — it just adds noise to your submission volume without giving the search engine any new information.

What happens if I submit too many URLs too quickly?

Behavior here depends on the specific endpoint's rate limiting, but batching your submissions (as shown above) rather than firing one request per URL is the practical way to avoid running into limits in the first place, especially on a site publishing or updating many pages per day.

Is this worth building for a small site that publishes rarely?

For a site publishing a handful of pages a month, manually submitting a URL through a search engine's own webmaster tools after each publish is realistically just as fast as building automation for it. This kind of pipeline starts paying for itself once you're publishing or updating frequently enough that manual submission would mean doing it multiple times a day.

Justin is a self-taught developer who builds and runs DeelCart himself — from the articles to the server it runs on. He manages his own Linux infrastructure and writes guides based on tools and workflows he actually uses day to day.

Automating IndexNow Submissions So New Pages Get Indexed Faster