Googlebot, What It Is, And How A Site Looks In Its Eyes

Two simple tools to check which version of a web page is seen by Googlebot or the Mountain View company crawlers. A crawler is a software that scans the content of web pages by following the hypertext links present (the well-known anchor tag) and collecting as much information as possible.

Search engines generally use multiple crawlers to delve deeply into the web. Google, for example, uses two crawlers, one to simulate accessing websites from a desktop and the other from a mobile device.

In the future, the desktop crawler will disappear with the attention that Google, especially since March 2021, intends to reserve for consultation from a mobile device. The crawler only has the (arduous) task of following the links in the pages: the possible indexing of content in the search engine is carried out in a second step. It implies using a complex ranking algorithm that evaluates the previously scanned content, assesses its authority, and decides its positioning in the SERPs or the results pages proposed by the search engine, subsequently when entering a particular query .

Anyone who administers one or more websites knows very well how valuable Google Search Console is, which offers information on the status of pages within the search engine. There are not only two Google crawlers: the Mountain View company uses others, in many cases one for each service managed by the company. The list of Google crawlers is published here. Every time a Google crawler reaches a web page, it presents itself with a specific user agent or string that certifies its identity.

The browser also transmits the user agent string you are using at the moment, and each browser uses an “ad hoc” one, different from the others. Indeed, the user agent changes according to the browser release installed, the components used locally, and the version of the operating system in use. To find your user agent string, check the user agent in the Google search box. Google noted that Chrome would discontinue using the user agent string, but it’s Googlebots will still broadcast their user agent.

Check How A Web Page Looks To Googlebot

Google provides some tools to check how a web page looks in the “eyes” of its crawlers.

One of them is called URL Inspection and is integrated into the Search Console. Pasting the URL of the page you want to check into the box provided and then clicking View Crawled Page will take you to the code that displays Googlebot’s crawl. However, access to this information is only possible for pages added to the Google index or published on the search engine ( Test published URL button ).

ALSO READ: How To Improve Your Company’s Wi-Fi Connection

However, by modifying the browser’s user agent string, it is possible to simulate access by Googlebot on any web page. Given that the procedure applies to any web browser (see How to change user agent with the main web browsers ) to simulate access to a page from Googlebot with Chrome, it is sufficient to proceed as follows. You will notice which and how many websites have different contents depending on the client, and you can check how your pages behave.

1. Visit the page with the list of user agent strings used by Google bots, then copy and paste that of Googlebot (Desktop) or Googlebot (smartphone) elsewhere.
2. Launch Google Chrome and press the F12 key to access Developer Tools. Before using the F12 key, it could be helpful to open an incognito browser window to be released, for example, from any cookie set on the system: Incognito browsing, when to use it.
3.  Press the CTRL+SHIFT+P key combination to bring up the search box shown in the figure.
4. Type network conditions, then click the Show network conditions item.
5. In the Network conditions tab, deactivate the Select automatically box, select Custom in the drop-down menu below, then paste the Googlebot user agent string.
6. By typing check user agent in the Google search box, you will see the Googlebot user agent appear.
7. Visiting any web page will simulate a visit by Googlebot, and you can check how it will appear “in the eyes” of the Google crawler.
8. To undo the change, close the Chrome Developer Tools.