Keeping GoogleBot Happy
One of the interesting side effects I hadn’t considered when we rolled out some fairly significant caching updates on GitHub in the beginning of January was how much Google’s crawler…
One of the interesting side effects I hadn’t considered when we rolled out some fairly significant caching updates on GitHub in the beginning of January was how much Google’s crawler would take full advantage of the speed increase.
The graph lays it all out pretty clearly what happens when your site is more responsive. The number of pages the bot is able to index goes up dramatically when the time spent downloading the page is reduced dramatically. More pages indexed on Google inevitably means more revenue for us as our traffic grows, so this is a solid win beyond making sure existing GitHubbers are happy.
Written by
Related posts
What 986 million code pushes say about the developer workflow in 2025
Nearly a billion commits later, the way we ship code has changed for good. Here’s what the 2025 Octoverse data says about how devs really work now.
Introducing Agent HQ: Any agent, any way you work
At Universe 2025, GitHub’s next evolution introduces a single, unified workflow for developers to be able to orchestrate any agent, any time, anywhere.
Octoverse: A new developer joins GitHub every second as AI leads TypeScript to #1
In this year’s Octoverse, we uncover how AI, agents, and typed languages are driving the biggest shifts in software development in more than a decade.