Preface
This epic quest was started years ago when I was trying to understand why a JavaScript widget was not being picked up by Google. After many experiments and rabbit holes, I came up with some theories, which I will sum up here.
This analysis was done at the start of 2025. Things got in the way, so it sat in draft for a while. It seems the subject is hot now (See Dave Smarts’ article on the 5-second render limit myth), so I’m publishing what I found. One day I might be able to continue the research.
The rest of the article covers the experiments and my conclusions.
Tony’s Theory
Googlebot creates a predictable enviroment where it has control over its own warped time. While working, or waiting for network resources, it stops time, otherwise time flies.
When Googlebot time exceeds 5 seconds, it prepares for the end. The viewport is adjusted and the page has its last chance to load more content. A short time later, space and time is stopped, And that is when the HTML is captured.
If this has not happend in 52 seconds of real time, it stops all network requests letting it quickly reach it’s 5 second target and finish things up.
For the real crawl, rendering can be delayed, caching is ripe and unimportant resources are canceled on the fly.
Observational Frameworks: The External Gaze Upon Googlebot
Google provides a few tools to get insights into what Googlebot is up to. e.g. the URL Inspection tool in the Search Console and the stand-alone Rich Result Test tool. From these, we can observe:
- Structured Data from the rendered page
- The rendered HTML
- Screenshot (Testing tools only)
- Page resources (Testing tools and sometimes with the real Googlebot)
- JavaScript console messages (Testing tools but not with the real Googlebot)
The real Googlebot that adds pages to the index behaves differently from the testing tools, as they have different motivations, so we need to test both scenarios. e.g., the real Googlebot wants to be efficient, so it caches aggressively.
We could also glean information via server logs, but that’s hard, and I discovered I don’t need it.
We’ll focus on the rendered HTML. Using a small logging script, we can insert data into the rendered HTML, which we can then observe via the tools. The function htmlLog(message) inserts the message into a text script (nice to view in the testing tools), as well as a copy visible on the page. Each line starts with the current performance.now() value to track the milliseconds since the browser started. Times are all in UTC. This is what is logged by default (from a normal browser):
performance.now()
154.1999999880791: User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36
154.1999999880791: Browser Start (performance.timeOrigin): 2025-01-05T20:52:36.293Z
154.1999999880791: Server Start (HTML requested): 2025-01-05T20:52:36.373Z
154.3999999761581: Body End Script
228.5999999940395: Event: DOMContentLoaded
254.0000000000000: Event: loadThen, I spent a lot of time experimenting, discovering new things, and developing theories. Here are some of my findings.
Unveiling Googlebot: The Entity Behind the Search
A simple demonstration of this observation method is to gather details about Googlebot from the navigator API. This is a snapshot of the real Googlebot:
13.0000000000000: navigator: {
"cookieEnabled": true,
"appName": "Netscape",
"appCodeName": "Mozilla",
"platform": "Linux armv8l",
"product": "Gecko",
"appVersion": "5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.6778.139 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)",
"userAgent": "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.6778.139 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)",
"language": "en-US",
"onLine": true,
"deviceMemory": 8,
"hardwareConcurrency": 128,
"vendor": "Google Inc.",
"javaEnabled": false
}I’ve seen a few page speed hackers disable resources for Linux machines. This is because the page speed testing tools tend to be on Linux, and they can make loading look much faster if half the page is disabled. This is bad for Linux users and could also stop Googlebot from properly loading the pages.
The Intricacies of Randomness and Predictive Patterns within Googlebot
Google’s randomness is quite predictable. In fact, it’s not random at all.
While trying to set up a cache buster to ensure it would always crawl a page’s resources, I discovered my random numbers were always the same. Here’s a log of a sequence of random numbers from an indexed page via the real Googlebot:
1.0000000000000: User Agent: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.6778.139 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
2.0000000000000: Browser Start (performance.timeOrigin): 2025-01-05T00:00:00.000Z
3.0000000000000: Server Start (HTML requested): 2025-01-05T21:16:43.275Z
4.0000000000000: Math.random() 1: 0.6393021999392658
5.0000000000000: Math.random() 2: 0.5943970666266978
6.0000000000000: Math.random() 3: 0.08336732489988208
7.0000000000000: Math.random() 4: 0.3935315718408674
8.0000000000000: Math.random() 5: 0.9578430985566229
9.0000000000000: crypto random: {
"0": 866266943,
"1": 2265860932,
"2": 1996670117,
"3": 3830160457,
"4": 1005482941
}Let’s see what a live test looks like, using the test Googlebot:
1.0000000000000: User Agent: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.6778.139 Mobile Safari/537.36 (compatible; Google-InspectionTool/1.0)
2.0000000000000: Browser Start (performance.timeOrigin): 2025-01-07T01:16:01.000Z
3.0000000000000: Server Start (HTML requested): 2025-01-07T01:16:02.035Z
4.0000000000000: Math.random() 1: 0.6393021999392658
5.0000000000000: Math.random() 2: 0.5943970666266978
6.0000000000000: Math.random() 3: 0.08336732489988208
7.0000000000000: Math.random() 4: 0.3935315718408674
8.0000000000000: Math.random() 5: 0.9578430985566229
9.0000000000000: crypto random: {
"0": 866266943,
"1": 2265860932,
"2": 1996670117,
"3": 3830160457,
"4": 1005482941
}Finally, the Rich Result Test tool:
1.0000000000000: User Agent: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.6778.139 Mobile Safari/537.36 (compatible; Google-InspectionTool/1.0)
2.0000000000000: Browser Start (performance.timeOrigin): 2025-01-07T01:13:14.000Z
3.0000000000000: Server Start (HTML requested): 2025-01-07T01:13:14.649Z
4.0000000000000: Math.random() 1: 0.6393021999392658
5.0000000000000: Math.random() 2: 0.5943970666266978
6.0000000000000: Math.random() 3: 0.08336732489988208
7.0000000000000: Math.random() 4: 0.3935315718408674
8.0000000000000: Math.random() 5: 0.9578430985566229
9.0000000000000: crypto random: {
"0": 866266943,
"1": 2265860932,
"2": 1996670117,
"3": 3830160457,
"4": 1005482941
}Do things seem familiar? Different times, different tools, all the same “random” numbers.
In a standard browser, those numbers change on every load.
848.3000000119209: User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36
848.3000000119209: Browser Start (performance.timeOrigin): 2025-01-07T01:13:07.101Z
848.3000000119209: Server Start (HTML requested): 2025-01-07T01:13:06.241Z
848.3999999761581: Math.random() 1: 0.22958714071254382
848.3999999761581: Math.random() 2: 0.07547011299586193
848.3999999761581: Math.random() 3: 0.5682621863335187
848.3999999761581: Math.random() 4: 0.6977130401477343
848.3999999761581: Math.random() 5: 0.9050868471807068
848.6999999880791: crypto random: {
"0": 3328632500,
"1": 2480456081,
"2": 2071372819,
"3": 639006855,
"4": 719674988
}1379.9000000357628: User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36
1380.0000000000000: Browser Start (performance.timeOrigin): 2025-01-07T01:20:52.090Z
1380.0000000000000: Server Start (HTML requested): 2025-01-07T01:20:51.418Z
1380.0000000000000: Math.random() 1: 0.5222172700746359
1380.0000000000000: Math.random() 2: 0.4440564579172728
1380.0000000000000: Math.random() 3: 0.7365465502143878
1380.0000000000000: Math.random() 4: 0.16355262929037173
1380.0000000000000: Math.random() 5: 0.14030345842617864
1380.1000000238419: crypto random: {
"0": 2077931315,
"1": 3346506675,
"2": 487056322,
"3": 549096286,
"4": 732635138
}In reality, computers typically generate Pseudorandom numbers in a sequence based on a seed like the current time. I suspect the random seed is a constant for Googlebot.
Tony’s Conclusion
Googlebot has unrandomized random numbers so that page content is more predictable. They don’t want content to randomly change every time they look at it.
Temporal Divergences: Googlebot Time Versus Real World Time
A long time ago, I was trying to solve an issue with a third-party review widget and its structured data not being seen by Googlebot. This started my analysis of how the bot worked. I discovered that when the Googlebot clock struck around 5.5 seconds, it stopped space and time and processed the rendered HTML.
It was a Schrödinger’s bot theory, which Dave Smart (Tame The Bots) pointed out to me. Googlebot ticks to a different rhythm. Further tests confirmed this, but it has taken until now to take a deep dive into Googlebot’s idea of time. I was right that Googlebot thought it was around 5.5 seconds, but time on our planet said otherwise.
A test logging performance.now() in a 10ms timeout loop. This is what Googlebot shows at the end of its crawl:
js-performance.now: 5250.0999999046326Bridging Realities: Enabling Googlebot’s Glimpse into the Real World
How do we observe how Googlebot ticks? As the observer, we somehow have to show its time progress relative to our real time.
Our method of observation is from within, by logging data to the HTML that Googlebot provides us.
A simple first step is to have our server insert the real time for when the server generated the HTML. So we know exactly when Googlebot requested it.
3.0000000000000: Server Start (HTML requested): 2025-01-05T21:16:43.275ZWe also want to observe as time progresses. My solution was to return the real-time in the resources the test pages requested. Then, log those real times on the page for later observation. The server becomes the source of how real-time passes as Googlebot processes the page.
27.0000000000000: Resource "async1": Server Request: 2025-01-05T23:23:49.3750522Z (+4237ms)
28.0000000000000: Resource "async1": Server Delay: 100
29.0000000000000: Resource "async1": Server Response: 2025-01-05T23:23:49.4750194Z (+4337ms)Note: After a lot of confusion, I realised that Googlebot was very good at caching these resources (see later), and therefore, it saw echoes of past server responses. To get around this, I added cache busters to the requests, and that’s when I discovered we had no access within Googlebot to a random number or even the time…
The Genesis of Googlebot Time
In a standard browser and the testing tools, the browser’s time starts in real time:
2.0000000000000: Browser Start (performance.timeOrigin): 2025-01-05T21:09:26.000Z
3.0000000000000: Server Start (HTML requested): 2025-01-05T21:09:27.223ZHere you can see that the browser start time is very close to the server start time (our observed real-world time). However, if we look at the crawled page HTML, things don’t match up:
2.0000000000000: Browser Start (performance.timeOrigin): 2025-01-05T00:00:00.000Z
3.0000000000000: Server Start (HTML requested): 2025-01-05T21:06:20.911ZThe real Googlebot redacts the time of day!
Tony’s Conclusion
Time of day is a constant to remove randomness from a pages content.
Time Dilation: The Slow Tick of Googlebot’s Clock
I ran a test that repeatedly called Date.now() 5 times without any other actions. I then repeated the test with delays between each Date.now() request (adding and removing 1,000 elements). This is what it looks like in a normal browser:
performance.now()
941.5000000000000: Browser Start (performance.timeOrigin): 2025-01-05T22:15:25.858Z
941.5000000000000: Server Start (HTML requested): 2025-01-05T22:15:25.723Z
941.5999999940395: JS Date elapsed ms: 942
941.5999999940395: JS Date elapsed ms: 942
941.5999999940395: JS Date elapsed ms: 942
941.5999999940395: JS Date elapsed ms: 942
941.5999999940395: JS Date elapsed ms: 942
972.0999999940395: delay complete
972.0999999940395: JS Date elapsed ms: 972
1000.0000000000000: delay complete
1000.0999999940395: JS Date elapsed ms: 1000
1021.4000000059605: delay complete
1021.4000000059605: JS Date elapsed ms: 1022
1042.0000000000000: delay complete
1042.0000000000000: JS Date elapsed ms: 1042
1062.8000000119209: delay complete
1062.8000000119209: JS Date elapsed ms: 1063
1066.5000000000000: Body End Script
1067.5000000000000: Event: DOMContentLoaded
1093.9000000059605: Event: load
2941.9000000059605: 2 seconds timeout before: JS Date elapsed ms: 2942
3065.0000000000000: 2 seconds timeout after: JS Date elapsed ms: 3065As expected, the first 5 requests were instant (all simultaneously), then the next picked up the delay times. Date.now() and performance.now() closely matched.
Then I ran the test with Googlebot:
performance.now()
2.0000000000000: Browser Start (performance.timeOrigin): 2025-01-05T22:11:04.000Z
3.0000000000000: Server Start (HTML requested): 2025-01-05T22:11:04.881Z
4.0000000000000: JS Date elapsed ms: 0
5.0000000000000: JS Date elapsed ms: 1
6.0000000000000: JS Date elapsed ms: 2
7.0000000000000: JS Date elapsed ms: 3
8.0000000000000: JS Date elapsed ms: 4
9.0000000000000: delay complete
10.0000000000000: JS Date elapsed ms: 5
11.0000000000000: delay complete
12.0000000000000: JS Date elapsed ms: 6
13.0000000000000: delay complete
14.0000000000000: JS Date elapsed ms: 7
15.0000000000000: delay complete
16.0000000000000: JS Date elapsed ms: 8
17.0000000000000: delay complete
18.0000000000000: JS Date elapsed ms: 9
19.0000000000000: Body End Script
20.0000000000000: Event: DOMContentLoaded
21.0000000000000: Event: load
2000.0000000000000: 2 seconds timeout before: JS Date elapsed ms: 2000
2001.0000000000000: 2 seconds timeout after: JS Date elapsed ms: 2001What is going on here?
The first thing to notice is that it reports no time spent between the browser starting and the test running, while in a real browser, there is a delay (941ms) as the page is initially set up.
And the delays in the test take no time to run.
Tony’s Conclusion
Googlebot time stands still when the CPU is busy
We can see that the time increments by 1ms every time I ask for the time. And the time returned differs between performance.now() and Date.now(). Each API increments separately when they are called.
Tony’s Conclusion
Googlebot adds 1ms increments to their time APIs, to ensure time does seem to keep moving on.
e.g. Some bad coding practices may implement a timer based on a loop that checks the current time. If time stands still, the loop goes on forever, and a small explosion is seen on planet Google.
The final timeout checks are run before and after the test. Both are showing up as 2 seconds, indicating that the internal time does not use these times given to us. Let’s make time fly:
performance.now()
4.0000000000000: Date.now() = 0
5.0000000000000: 10,000 * Date.now(). Date.now() = 10001
5006.0000000000000: 5,000 * performance.now(). Date.now() = 10002
5007.0000000000000: Body End Script
5008.0000000000000: Event: DOMContentLoaded
5009.0000000000000: Event: load
5010.0000000000000: 2 seconds timeout: JS Date elapsed ms: 10003
5011.0000000000000: 5 seconds timeout: JS Date elapsed ms: 1000410,000 calls to Date.now() incremented Date.now() but had no effect on performance.now().
5,000 calls to performance.now() incremented performance.now() but had no effect on Date.now().
The timeouts happened after the tests were completed, indicating that they use a different time than what those APIs return.
There was a 6-second timeout, which was never logged, indicating Googlebot stopped sometime between 5 and 6 seconds (It’s internal time).
The Googlebot Clock and Time Distortion
We’ve already discovered that time seems to stop when the browser is busy running code. But we’ve not yet tested to see how it compares to real time. Let’s test a simple 5-second timer that triggers a resource request to get the real time.
10.0000000000000: Resource "start": Server Request: 2025-01-05T23:05:35.2352750Z (+307ms)
11.0000000000000: Resource "start": Server Delay: 0
12.0000000000000: Resource "start": Server Response: 2025-01-05T23:05:35.2352759Z (+307ms)
13.0000000000000: Body End Script
14.0000000000000: Event: DOMContentLoaded
15.0000000000000: Event: load
5019.8999996185303: Resource "5seconds": Server Request: 2025-01-05T23:05:35.3907412Z (+462ms)
5020.8999996185303: Resource "5seconds": Server Delay: 0
5021.8999996185303: Resource "5seconds": Server Response: 2025-01-05T23:05:35.3907418Z (+462ms)There’s an initial resource request to get the time when rendering started. On the 5-second timeout, the server indicated a time change of 455ms. That’s a quick 5 seconds in Googlebot time.
There we go; we can put Googlebot into bullet time.
Tony’s Conclusion
When Googlebot has nothing to do, its clock speeds up and it goes into bullet time.
Let’s do a variant on this test. This time, the server will cause the 5-second delay.
3.0000000000000: Server Start (HTML requested): 2025-01-05T23:24:19.974Z
10.0000000000000: Resource "5seconds": Server Request: 2025-01-05T23:24:20.1581075Z (+184ms)
11.0000000000000: Resource "5seconds": Server Delay: 5000
12.0000000000000: Resource "5seconds": Server Response: 2025-01-05T23:24:25.1553453Z (+5181ms)
13.0000000000000: Body End Script
14.0000000000000: Event: DOMContentLoaded
15.0000000000000: Event: loadHmm, so while Googlebot waits on a network resource, time stands still. The same happens with multiple resources with 100ms server-side delays. Note how the sync requests block while the async requests are after DCL:
10.0000000000000: Resource "sync1": Server Request: 2025-01-05T23:22:08.2030851Z (+372ms)
11.0000000000000: Resource "sync1": Server Delay: 100
12.0000000000000: Resource "sync1": Server Response: 2025-01-05T23:22:08.3125034Z (+481ms)
13.0000000000000: Resource "sync2": Server Request: 2025-01-05T23:22:08.1848526Z (+353ms)
14.0000000000000: Resource "sync2": Server Delay: 100
15.0000000000000: Resource "sync2": Server Response: 2025-01-05T23:22:08.2819139Z (+450ms)
16.0000000000000: Resource "sync3": Server Request: 2025-01-05T23:22:08.2125961Z (+381ms)
17.0000000000000: Resource "sync3": Server Delay: 100
18.0000000000000: Resource "sync3": Server Response: 2025-01-05T23:22:08.3123490Z (+481ms)
19.0000000000000: Resource "sync4": Server Request: 2025-01-05T23:22:08.1845047Z (+353ms)
20.0000000000000: Resource "sync4": Server Delay: 100
21.0000000000000: Resource "sync4": Server Response: 2025-01-05T23:22:08.2819573Z (+450ms)
22.0000000000000: Resource "sync5": Server Request: 2025-01-05T23:22:08.1845055Z (+353ms)
23.0000000000000: Resource "sync5": Server Delay: 100
24.0000000000000: Resource "sync5": Server Response: 2025-01-05T23:22:08.2820980Z (+451ms)
25.0000000000000: Body End Script
26.0000000000000: Event: DOMContentLoaded
27.0000000000000: Resource "async1": Server Request: 2025-01-05T23:22:08.4207167Z (+589ms)
28.0000000000000: Resource "async1": Server Delay: 100
29.0000000000000: Resource "async1": Server Response: 2025-01-05T23:22:08.5152473Z (+684ms)
30.0000000000000: Resource "async2": Server Request: 2025-01-05T23:22:08.4207005Z (+589ms)
31.0000000000000: Resource "async2": Server Delay: 100
32.0000000000000: Resource "async2": Server Response: 2025-01-05T23:22:08.5153157Z (+684ms)
33.0000000000000: Resource "async3": Server Request: 2025-01-05T23:22:08.6234730Z (+792ms)
34.0000000000000: Resource "async3": Server Delay: 100
35.0000000000000: Resource "async3": Server Response: 2025-01-05T23:22:08.7184885Z (+887ms)
36.0000000000000: Resource "async4": Server Request: 2025-01-05T23:22:08.6234720Z (+792ms)
37.0000000000000: Resource "async4": Server Delay: 100
38.0000000000000: Resource "async4": Server Response: 2025-01-05T23:22:08.7184249Z (+887ms)
39.0000000000000: Resource "async5": Server Request: 2025-01-05T23:22:08.8357820Z (+1004ms)
40.0000000000000: Resource "async5": Server Delay: 100
41.0000000000000: Resource "async5": Server Response: 2025-01-05T23:22:08.9370955Z (+1106ms)Tony’s Conclusion
While network resources are pending, Googlebot stops its clock, giving the page more real time to get things done.
The Halt at Threshold: When Googlebot Stops as Real Time Peaks
Let’s go extreme again and see where time really ends. Googlebot won’t give the page forever to load its resources. What happens if we keep Googlebot busy for a long time?
Let’s first cause a sequence of 15 resources, all loading one after the other with no idle time in between:
3.0000000000000: Server Start (HTML requested): 2025-01-05T23:28:32.854Z
10.0000000000000: Resource "start": Server Request: 2025-01-05T23:28:33.2884745Z (+434ms)
11.0000000000000: Resource "start": Server Delay: 0
12.0000000000000: Resource "start": Server Response: 2025-01-05T23:28:33.2884751Z (+434ms)
13.0000000000000: Body End Script
14.0000000000000: Event: DOMContentLoaded
20.0000000000000: Resource "resource1": Server Request: 2025-01-05T23:28:33.5357619Z (+681ms)
21.0000000000000: Resource "resource1": Server Delay: 500
22.0000000000000: Resource "resource1": Server Response: 2025-01-05T23:28:34.0365020Z (+1182ms)
30.0000000000000: Resource "resource2": Server Request: 2025-01-05T23:28:34.1441734Z (+1290ms)
31.0000000000000: Resource "resource2": Server Delay: 500
32.0000000000000: Resource "resource2": Server Response: 2025-01-05T23:28:34.6453833Z (+1791ms)
40.0000000000000: Resource "resource3": Server Request: 2025-01-05T23:28:35.0077308Z (+2153ms)
41.0000000000000: Resource "resource3": Server Delay: 500
42.0000000000000: Resource "resource3": Server Response: 2025-01-05T23:28:35.5081458Z (+2654ms)
50.0000000000000: Resource "resource4": Server Request: 2025-01-05T23:28:35.6180168Z (+2764ms)
51.0000000000000: Resource "resource4": Server Delay: 500
52.0000000000000: Resource "resource4": Server Response: 2025-01-05T23:28:36.1182873Z (+3264ms)
60.0000000000000: Resource "resource5": Server Request: 2025-01-05T23:28:36.2274883Z (+3373ms)
61.0000000000000: Resource "resource5": Server Delay: 500
62.0000000000000: Resource "resource5": Server Response: 2025-01-05T23:28:36.7272883Z (+3873ms)
69.9000000953674: Resource "resource6": Server Request: 2025-01-05T23:28:36.8345834Z (+3980ms)
70.9000000953674: Resource "resource6": Server Delay: 500
71.9000000953674: Resource "resource6": Server Response: 2025-01-05T23:28:37.3345489Z (+4480ms)
80.0000000000000: Resource "resource7": Server Request: 2025-01-05T23:28:37.4454102Z (+4591ms)
81.0000000000000: Resource "resource7": Server Delay: 500
82.0000000000000: Resource "resource7": Server Response: 2025-01-05T23:28:37.9459214Z (+5091ms)
90.0000000000000: Resource "resource8": Server Request: 2025-01-05T23:28:38.0527098Z (+5198ms)
91.0000000000000: Resource "resource8": Server Delay: 500
92.0000000000000: Resource "resource8": Server Response: 2025-01-05T23:28:38.5533910Z (+5699ms)
100.0000000000000: Resource "resource9": Server Request: 2025-01-05T23:28:38.6675001Z (+5813ms)
101.0000000000000: Resource "resource9": Server Delay: 500
102.0000000000000: Resource "resource9": Server Response: 2025-01-05T23:28:39.1677752Z (+6313ms)
110.0000000000000: Resource "resource10": Server Request: 2025-01-05T23:28:39.2779558Z (+6423ms)
111.0000000000000: Resource "resource10": Server Delay: 500
112.0000000000000: Resource "resource10": Server Response: 2025-01-05T23:28:39.7779867Z (+6923ms)
120.0000000000000: Resource "resource11": Server Request: 2025-01-05T23:28:39.9087097Z (+7054ms)
121.0000000000000: Resource "resource11": Server Delay: 500
122.0000000000000: Resource "resource11": Server Response: 2025-01-05T23:28:40.4091094Z (+7555ms)
130.0000000000000: Resource "resource12": Server Request: 2025-01-05T23:28:40.5195647Z (+7665ms)
131.0000000000000: Resource "resource12": Server Delay: 500
132.0000000000000: Resource "resource12": Server Response: 2025-01-05T23:28:41.0198993Z (+8165ms)
139.9000000953674: Resource "resource13": Server Request: 2025-01-05T23:28:41.1261368Z (+8272ms)
140.9000000953674: Resource "resource13": Server Delay: 500
141.9000000953674: Resource "resource13": Server Response: 2025-01-05T23:28:41.6253674Z (+8771ms)
149.9000000953674: Resource "resource14": Server Request: 2025-01-05T23:28:41.7356655Z (+8881ms)
150.9000000953674: Resource "resource14": Server Delay: 500
151.9000000953674: Resource "resource14": Server Response: 2025-01-05T23:28:42.2363540Z (+9382ms)
160.0000000000000: Resource "resource15": Server Request: 2025-01-05T23:28:42.3410490Z (+9487ms)
161.0000000000000: Resource "resource15": Server Delay: 500
162.0000000000000: Resource "resource15": Server Response: 2025-01-05T23:28:42.8404948Z (+9986ms)Googlebot time does tick a bit, but it only reaches 162ms in 9,986ms of real time. Reverse bullet time. Being a consistent test, we can see how smoothly Googlebot time slowly increases in real time:
Let’s see how long it can keep it up:
3.0000000000000: Server Start (HTML requested): 2025-01-05T23:50:35.140Z
10.0000000000000: Resource "start": Server Request: 2025-01-05T23:50:36.0203262Z (+880ms)
11.0000000000000: Resource "start": Server Delay: 0
12.0000000000000: Resource "start": Server Response: 2025-01-05T23:50:36.0203269Z (+880ms)
14.0000000000000: Body End Script
15.0000000000000: Event: DOMContentLoaded
20.0000000000000: Resource "resource1": Server Request: 2025-01-05T23:50:36.1277737Z (+987ms)
21.0000000000000: Resource "resource1": Server Delay: 5000
22.0000000000000: Resource "resource1": Server Response: 2025-01-05T23:50:41.1327657Z (+5992ms)
30.0000000000000: Resource "resource2": Server Request: 2025-01-05T23:50:41.2385013Z (+6098ms)
31.0000000000000: Resource "resource2": Server Delay: 5000
32.0000000000000: Resource "resource2": Server Response: 2025-01-05T23:50:46.2256705Z (+11085ms)
40.0000000000000: Resource "resource3": Server Request: 2025-01-05T23:50:46.3316579Z (+11191ms)
41.0000000000000: Resource "resource3": Server Delay: 5000
42.0000000000000: Resource "resource3": Server Response: 2025-01-05T23:50:51.3351709Z (+16195ms)
50.0000000000000: Resource "resource4": Server Request: 2025-01-05T23:50:51.4470433Z (+16307ms)
51.0000000000000: Resource "resource4": Server Delay: 5000
52.0000000000000: Resource "resource4": Server Response: 2025-01-05T23:50:56.4601614Z (+21320ms)
60.0000000000000: Resource "resource5": Server Request: 2025-01-05T23:50:56.5689034Z (+21428ms)
61.0000000000000: Resource "resource5": Server Delay: 5000
62.0000000000000: Resource "resource5": Server Response: 2025-01-05T23:51:01.5538808Z (+26413ms)
63.0000000000000: 50ms
69.9000000953674: Resource "resource6": Server Request: 2025-01-05T23:51:01.6662019Z (+26526ms)
70.9000000953674: Resource "resource6": Server Delay: 5000
71.9000000953674: Resource "resource6": Server Response: 2025-01-05T23:51:06.6634371Z (+31523ms)
80.0000000000000: Resource "resource7": Server Request: 2025-01-05T23:51:06.7742567Z (+31634ms)
81.0000000000000: Resource "resource7": Server Delay: 5000
82.0000000000000: Resource "resource7": Server Response: 2025-01-05T23:51:11.7883764Z (+36648ms)
90.0000000000000: Resource "resource8": Server Request: 2025-01-05T23:51:11.8966331Z (+36756ms)
91.0000000000000: Resource "resource8": Server Delay: 5000
92.0000000000000: Resource "resource8": Server Response: 2025-01-05T23:51:16.8827229Z (+41742ms)
100.0000000000000: Resource "resource9": Server Request: 2025-01-05T23:51:16.9915984Z (+41851ms)
101.0000000000000: Resource "resource9": Server Delay: 5000
102.0000000000000: Resource "resource9": Server Response: 2025-01-05T23:51:21.9922450Z (+46852ms)
110.0000000000000: Resource "resource10": Server Request: 2025-01-05T23:51:22.2700833Z (+47130ms)
111.0000000000000: Resource "resource10": Server Delay: 5000
112.0000000000000: Resource "resource10": Server Response: 2025-01-05T23:51:27.2722370Z (+52132ms)
113.0000000000000: 100ms
121.0000000000000: Event: load
2010.0000000000000: 2 seconds
5009.9000000953674: 5 secondsAt 52,132ms of real time, the server requests stopped, and Googlebot’s bullet time was still at 112ms.
However, Googlebot continued on to trigger timeouts and reach its target time:
js-performance.now: 5249.9000000953674Tony’s Conclusion
Running code and making network requests stops the Googlebot clock. At around 52 real time seconds of this, it pulls the plug and cancels all network requests. But it continues to run. At this point, it should have nothing much to do, giving its clock a chance to reach its target 5.25 seconds, where it stops space and time and takes a snapshot.
Exploring the Infinite: Boundless Possibilities with Googlebot
Google does not scroll (I check for scroll events), but it is rumoured to have a deep viewport to pick up content added as if it did scroll.
I tested a dynamically deepening page by adding divs with a set height. I added a lot, eventually added one every Googlebot millisecond and set their height to 1000 pixels.
Initially, this was the dimensions for my test page before any of the divs were added (count=0):
4.0000000000000: initial: {
"count": 0,
"window": {
"innerWidth": 981,
"innerHeight": 1742,
"outerWidth": 412,
"outerHeight": 732
},
"screen": {
"width": 412,
"height": 732,
"availHeight": 732,
"availWidth": 412
},
"visualViewport": {
"width": 980.0000610351562,
"height": 1741.1651611328125,
"scale": 0.4204081594944
}
}I’m not an expert on windows, screens and viewports. The key thing for Googlebot is that the page is prompted to trigger the loading of dynamic content that is lower down.
Googlebot fires two resize events. The first, at 5 seconds of Googlebot time, it fired this resize event:
5000.9000000953674: Event: resize
5001.9000000953674: resize: {
"count": 1250,
"window": {
"innerWidth": 981,
"innerHeight": 1742,
"outerWidth": 1,
"outerHeight": 1
},
"screen": {
"width": 412,
"height": 732,
"availHeight": 732,
"availWidth": 412
},
"visualViewport": {
"width": 981,
"height": 1742,
"scale": 1
}
}This changes the window’s outer dimensions to 1 pixel (strange) and the viewport scale from 0.42 to 1 🤷♂️.
Then, at 5.3 seconds, there is another resize:
5299.9000000953674: Event: resize
5300.9000000953674: resize: {
"count": 1325,
"window": {
"innerWidth": 981,
"innerHeight": 1251074,
"outerWidth": 1,
"outerHeight": 1
},
"screen": {
"width": 412,
"height": 732,
"availHeight": 732,
"availWidth": 412
},
"visualViewport": {
"width": 981,
"height": 1251074,
"scale": 1
}
}And bam! The height is now over 1 million pixels. I told you I added a lot. A million is enough, so I didn’t go further. I don’t want to cause another explosion on planet Google. Take that JR Oakes and your 12,140 🙂
Let’s move to the more realistic scenario of infinite scroll. This is often implemented using the IntersectionObserver to detect when the user is at the end of a section and then adding a new section after it, which repeats. This next test emulates including a resource request to pretend to get the content of the following section.
This was not so exciting. On the first resize, the first page was identified as visible. It loaded the resource and added the second page. On the second resize, it was seen as visible. It managed to load its resource and add the third page, but then time stopped. It seems that the IntersectionObserver was only firing after the resize events.
Related. If you use 100vh for your hero image to have the image fill the screen. You will see strange inspect URL previews, as that image will be very tall, because the viewport they use is very tall. I suspect this may cause some ranking issues as the visible content is always outside the viewport.
Here Martin confirms a bit about this and what I observed. He believes this is not a problem for SEO and Google seeing the content, but advises adding a max-height to limit it.
Caching Chronicles: Warping space and time
Crawling the resources needed to render a page will chip away from the crawl budget of the hostname that’s hosting the resource. To ameliorate this, WRS attempts to cache every resource (JavaScript and CSS) referenced in pages it renders. The time to live of the WRS cache is unaffected by HTTP caching directives; instead WRS caches everything for up to 30 days, which helps preserve the site’s crawl budget for other crawl tasks.
https://developers.google.com/search/blog/2024/12/crawling-december-resources
This caching is only done by the real Googlebot. The test tools want to work with the most recent resources so you get accurate results.
The Interlude: Decoding the Pause Between Crawling and Rendering in Googlebot
Google is open about how they first crawl the HTML, then later render the page, and finally add it to the index.
These days, they say the delay from crawl to render is typically short. Here’s an example showing a 23-second gap from the server providing the HTML to the page loading its first resource when rendering:
3.0000000000000: Server Start (HTML requested): 2025-01-05T21:06:20.911Z
10.0000000000000: Resource "start": Server Request: 2025-01-05T21:06:44.4748315Z (+23563ms)Digital Voids: The Black Hole Effect in Googlebot’s Universe
I added some tests on a real site to see what else I could discover.
Vanishing Act: Resource requests that never escape Googlebot
The URL Inspection sometimes reports resource errors. Quite often as “Other error”. To find out more, I used the PerformanceObserver to record resources, as well as intercepted fetch requests to get errors.
The times in the resources were mostly Googlebot times, with some strange moments of real time. Nothing I could trust, but it indicated there was some real-time knowledge inside Googlebot.
resource: {
"name": "https://.../Resource?name=resource8",
"entryType": "resource",
"startTime": 0,
"duration": 10,
"initiatorType": "script",
"deliveryType": "",
"nextHopProtocol": "",
"renderBlockingStatus": "non-blocking",
"workerStart": 0,
"redirectStart": 0,
"redirectEnd": 0,
"fetchStart": 0,
"domainLookupStart": 0,
"domainLookupEnd": 0,
"connectStart": 0,
"secureConnectionStart": 0,
"connectEnd": 0,
"requestStart": 0,
"responseStart": 1325.7000002861023,
"firstInterimResponseStart": 0,
"responseEnd": 10,
"transferSize": 390,
"encodedBodySize": 90,
"decodedBodySize": 90,
"responseStatus": 200,
"serverTiming": [
{
"name": "cfL4",
"duration": 0,
"description": "?proto=TCP&rtt=6359&min_rtt=6344&rtt_var=1355&sent=5&recv=8&lost=0&retrans=0&sent_bytes=3478&recv_bytes=1080&delivery_rate=657276&cwnd=252&unsent_bytes=0&cid=7bc84fa51e91646e&ts=269&x=0"
}
]
}Frequently, a resource request would return a status of 0. This typically means it is a cross-origin request without CORS, or the request was cancelled before completion. I found font related resources, 404s, and disallowed by robots.txt requests being cancelled. I’m not sure why I can’t see the status code for 404s.
Tony’s Conclusion
The Googlebot headless browser intercepts all requests and cancels those that are disallowed, fail, or it does not think are important, like ones related to fonts.
The URL inspection resources report must get the status information in its own way, maybe logging from within the code that intercepts resources. The report does have some quirks, like only the first of a series of POST requests that only vary by the payload is reported on.
Unexpected Exodus: Google-Specific Resources That Break Free from Googlebot
Interestingly, Google does not disallow Google Analytics via robots.txt. The base gtag script loads, but the subsequent event requests are cancelled. We even see the ga cookie being set.
Ads tracking is blocked by robots.txt, but GA4 doubleclick sometimes makes it. This can be seen in the Page resources reports.
Facebook loads the tag, but events are blocked by robots.txt
DIY Analysis: Conducting Your Own Googlebot Experiments
Here’s some simplified code to perform your own tests. Place it high up in the head section and call htmlLog() to add messages that you can see in the rendered HTML Googlebot creates.
<script type="text/plain" id="html-log">
</script>
<script>
const htmlLogTag = document.getElementById("html-log");
function htmlLog(message) {
htmlLogTag.appendChild(document.createTextNode(message+'\n'));
}
htmlLog('User Agent: ' + navigator.userAgent);
htmlLog('Browser Start (performance.timeOrigin): ' + (new Date(performance.timeOrigin)).toISOString());
addEventListener("DOMContentLoaded", (event) => {
htmlLog('Event: DOMContentLoaded');
});
addEventListener("scroll", (event) => {
htmlLog('Event: scroll '+document.documentElement.scrollTop);
});
addEventListener("load", (event) => {
htmlLog('Event: load');
});
addEventListener("visibilitychange", (event) => {
htmlLog('Event: visibilitychange hidden='+document.hidden);
});
addEventListener("error", (event) => {
htmlLog('Event: error '+JSON.stringify(event));
});
</script>