Crawling, Indexing and Ranking Explained.
What you will Learn
Search engines examine all the pages on the World Wide Web, categorize them, and put them in to a logical order when you search for something. Understanding how this works can help your business. This post will cover:
• How search engines find web pages
• What they do with the web pages they find
• How they decide what to show on search results pages.
When you’re using a search engine to find the closest hair salon or Hotel, you’re probably not thinking about search engine technology. But later you might wonder, how did it do that? How did it sort through the entire Internet so quickly, and choose the results you saw on the page?
Each search engine uses its own software programs, but the way they work is pretty similar. They all perform three tasks:
- Crawl over your webpage: They examine content they learn about and have permission to see. (It’s called Crawling)
- Index the content on your webpage: They categorize each piece of content (that’s called indexing).
- They decide which content is most useful to searchers (that’s called ranking).
We’ve talked about how search engines work, but what about crawling? Let’s take a look at how these work. Click here to read about how search engines work
Crawling
Search engines “crawl” the Internet to discover content, like web pages, images, and videos. Each search engine uses computer programs called “bots” (short for robot), “crawlers” or “spiders” to make their way through the pages.
The bots crawl from page to page by following links to other pages. These bots never stop; their sole purpose is to visit and revisit pages looking for new links and new content to include in the index.
Indexing
Indexing is the second part of the process. The index is a huge list of all the web pages and content found by the bots. The search engine uses this index as the source of information displayed on the search results pages.
But, not everything the bots find makes it into a search engine’s index. For example, search engines may find multiple copies of the exact same piece of content, located on different websites—so it may get listed more than once in an index entry!
How is that possible? Well, imagine you’re not searching for a Food store, but a Food manufacturer. You might notice that the top-of-the-page Mandu Food store has the same word-for-word description on the websites of many major retailers. The description might have been provided by the manufacturer… but now the search engine has decisions to make: which version to keep in the index? There’s no need for hundreds of duplicates, so it’s unlikely that every page will be added.
So if you own a website that’s selling Ready made Food, you’re likely better off writing your own description of the Mandu Food store.
Make sense? That covers crawling and indexing, which just leaves us with ranking.
Ranking
When you type in a search, the engine compares the words and phrases you use to its index, looking for matching results. Let’s say, for example, the search engine finds 230 million matching results. how does the search engine know what to display on top? That’s where ranking comes in.
The way search engines rank pages is top secret—it’s their ‘special sauce.’ There are hundreds of ways search engines determine rank, including things like the words on the page, the number of other websites linking to it, and the freshness of the content.
Ranking is a combination of the popularity and authority of your website. Think of it as a popularity contest between web pages: if you have more friends, it means you’re popular; if you have more friends that are influential, it means you are even more popular; and in case you don’t have either, sorry for breaking the news to you – no one will probably notice.
But no matter what formula they use to determine rank, the goal remains the same: to try to connect the searcher with what they are looking for.
We have a dedicated post that can help you gain more insight into How Google Rank pages
Say you’ve read about Green Tea and you want to try it. If you search for “Green tea near me” the search engine will show you nearby shops selling the drink, because your search indicated your location. You might even see a map to help you find them.
There are many factors Search engines use to determine which page to rank on SERP.
Visit here to read more about how to rank high on Google SERP
Conclusion
The crawlers and indexers at Google, Bing, and other big sites are like little roving bands of search-engine enthusiasts. They constantly scour the Web for content that can be turned into searchable information.
When they find something interesting—such as an image or video—they’ll keep it. Then, when someone searches for that exact phrase or phrase combination on one of those other websites, it will show up as a result in the user’s search results and then display the most relevant results to searchers.
So, what have we learned? Let us in at the comment section.
Sources
Learn with Google
