I hope you have heard of IndexNow at this point.
We covered his announcement in October and in December I had Fabrice Canel, Senior Program Manager for Microsoft Bing, on the SEJ Show to talk about this revolution in the way search engines discover new and updated content. day.
But what is it, exactly, and what do you need to know about it for SEO?
Here’s what we know so far.
Why we need IndexNow
Search engines have been discovering content the same way for a long time.
In 1993, JumpStation became the first Internet discovery tool to send bots to follow (or crawl) links from one website to another and create a content map or index.
Over the next few years, other search engines adopted this process as a means of finding new content.
This process was included in an algorithm that also considered content, relevance and other factors – and search engine optimization as we know it.
It didn’t take long before SEO-savvy web publishers recognized the importance of indexing to ensure high search engine rankings. In fact, without indexed content, a website will not generate organic traffic.
But therein lies the catch: unless you are a large, heavily trafficked website, it can take days or even weeks for a search engine to crawl your site and index new pages. . Heck, even if you’re a big e-commerce site or marketplace, it can take weeks for the engines to pick up new arrivals or out-of-stock products too, especially when you’re relying on discoverability with basic analytics .
During this time, you may have lost significant traffic to competitors.
To streamline this process, Microsoft Bing launched a new initiative in October 2021 called IndexNow.
What is IndexNow?
IndexNow is an open-source protocol that allows website publishers to instantly index participating search engines, updating results with the latest content changes.
Simply put, it’s a simple ping that notifies search engines that a URL and its content has been added, updated, or removed.
By eliminating the need for exploratory analytics, IndexNow seeks to improve the overall efficiency of the Internet.
How does IndexNow work?
There are two different ways search engines get index data: pull and push.
Pull indexing occurs when a search engine visits your site to request web pages and collects data from the server.
This is how search engines traditionally work.
Push indexing occurs when the web publisher or content management system notifies the search engine that pages have been added or removed or other changes have been made.
That’s what’s really revolutionary about IndexNow – it allows all URLs submitted to any IndexNow-enabled search engine to be simultaneously submitted to all other search engines using the protocol.
You can read more about the technical aspects of how IndexNow works here.
Which search engines use IndexNow?
At press time, Bing and Yandex use IndexNow.
A handful of others are said to be testing the protocol, according to an anonymous expert.
So far, DuckDuckGo has not announced any plans to adopt IndexNow.
However, it should be noted that the privacy-focused search engine draws from up to 400 sources in addition to using its own web crawler DuckDuckBot in compiling its search results.
Since Bing and Yandex are among these sources, you can also see updated content appearing faster in DuckDuckGo thanks to IndexNow.
Even Bing’s biggest competitor, Google, the first of all search engines, has confirmed that it will test the IndexNow protocol.
And while they haven’t gone so far as to announce whether or not they’ll adopt him, the fact that they’re experimenting with him shows an appreciation of his abilities.
Google’s main concerns with IndexNow seem to be related to sustainability and efficiency.
Google currently uses HTTP/2, a foundational data transfer protocol effective for more than half of all crawls, and may ultimately decide that this is a better fit for its needs.
Or they can decide to build their own alternative if they determine that an API-based approach is necessary – and it’s not a foregone conclusion that they will.
But not everyone is on board.
There are many people in the SEO community who don’t see the need for this protocol, instead claiming that XML sitemaps already accomplish what IndexNow is trying to do.
You can read more about the debate here.
How do you use IndexNow?
Notifying search engines of updates to your web content using IndexNow is a simple process:
- Generate an API key – This is submitted with the URLs to ensure ownership of the domain. You can do this using an online key generator tool like the one found here.
- Host API key – Your API key is hosted on the root directory in txt format.
- Submit URLs with parameters – You can submit URLs individually or in bulk. Submit your key location as a parameter.
- Check your submissions – Using the Bing Webmaster Tools portal, you can check which URLs have been submitted and discovered.
For more detailed examples and instructions, you can visit the IndexNow website.
More advanced users can use Python for bulk indexing and URL submission automation.
This requires a basic understanding of Python syntax and familiarity with Python libraries and packages.
We have a handy step-by-step guide to walk you through this process here.
What tools sync with IndexNow?
Any website developer can use the process described here to take advantage of IndexNow, but what’s really exciting is that you don’t need to know the HTML of HDMI to reap the benefits.
A number of large companies, including LinkedIn, MSN, and GitHub, are all planning migrations to IndexNow.
Additionally, other search solution providers like BotifyOnCrawl, Onely, and Yext are adopting this protocol.
Content management systems (CMS) are also jumping on board.
WordPress already offers a IndexNowComment plugin. Duda supports this proactive approach to web crawling, and Wix plans to integrate it soon.
Bing also recently announced the integration of Rank Math and All In One SEO for IndexNow.
Content Delivery Networks (CDNs) supporting IndexNow include Cloudy and Akamai.
Through their proxy servers, this will speed up data delivery and bring it closer to users.
What are the benefits of using IndexNow?
In addition to emphasizing the evolution of indexing from pull to push, the main benefit of IndexNow for publishers is the elimination of the time between updates and their discovery by search engines.
By allowing webmasters to notify all participating URLs with a single API call, search engine content discovery is streamlined, making the web more efficient.
This is advantageous for site owners, as increasing demands are reduced on the server.
Search engine spiders no longer need to perform exploratory crawls to determine if a page has been updated.
This allows servers to operate more efficiently without the need for redundant site load, which in turn reduces power consumption.
And unlike an XML sitemap, which only redirects SEO crawlers to pages on your website, IndexNow lets you notify search engines of non-200 status code pages.
IndexNow Frequently Asked Questions
What happens after I submit a URL?
IndexNow pushes your changes to search engines, which in turn apply them to page rankings.
Using IndexNow does not guarantee that web pages will be crawled or indexed and it may take time for changes to be reflected.
How many URLs can I submit using IndexNow?
You can provide search engines with a list of up to 10,000 URLs with a single API call.
Will I rank higher in search results if I index my content faster?
Yes, the sooner your pages get indexed, the sooner they can start competing for top search engine placements.
What does all this mean?
The march of technology is inexorable.
And as a lifeblood of the modern world, it’s only natural that changes to the internet, and in particular search engines, are part of it.
By shifting the process of indexing sites from a crawling process to something faster and more efficient, IndexNow seems to be on the cusp of a revolution.
And the fact that companies like MSN, GitHub, and WordPress are all joining us seems to indicate that this protocol will soon become the norm.
More resources:
Featured image: ra2 studio/Shutterstock
Comments are closed.