Crawling
Crawling is the process by which search engines discover updated content on the web, such as new sites or pages, changes to existing sites, and dead links.
To do this, a search engine uses a program that can be referred to as a ‘crawler’, ‘bot’ or ‘spider’ (each search engine has its own type) which follows an algorithmic process to determine which sites to crawl and how often.
As a search engine’s crawler moves through your site it will also detect and record any links it finds on these pages and add them to a list that will be crawled later. This is how new content is discovered.
Indexing
Once a search engine processes each of the pages it crawls, it compiles a massive index of all the words it sees and their location on each page. It is essentially a database of billions of web pages.
This extracted content is then stored, with the information then organised and interpreted by the search engine’s algorithm to measure its importance compared to similar pages.
Servers based all around the world allow users to access these pages almost instantaneously. Storing and sorting this information requires significant space and both Microsoft and Google have over a million servers each.
Ranking
As SEO’s this is the area we are most concerned with and the part that allows us to show clients tangible progress.
Once a keyword is entered into a search box, search engines will check for pages within their index that are a closest match; a score will be assigned to these pages based on an algorithm consisting of hundreds of different ranking signals.
These pages (or images & videos) will then be displayed to the user in order of score.
So in order for your site to rank well in search results pages, it’s important to make sure search engines can crawl and index your site correctly – otherwise they will be unable to appropriately rank your website’s content in search results.
To help give you even more of a basic introduction to this process, here is a useful video from Google which explains it quite well. Each search engine follows a similar methodology to this.
If you are having issues with crawling, indexing or rankings issues that you need help with. Get in touch and we can help you.
