Keeping GoogleBot Happy
One of the interesting side effects I hadn’t considered when we rolled out some fairly significant caching updates on GitHub in the beginning of January was how much Google’s crawler…
One of the interesting side effects I hadn’t considered when we rolled out some fairly significant caching updates on GitHub in the beginning of January was how much Google’s crawler would take full advantage of the speed increase.
The graph lays it all out pretty clearly what happens when your site is more responsive. The number of pages the bot is able to index goes up dramatically when the time spent downloading the page is reduced dramatically. More pages indexed on Google inevitably means more revenue for us as our traffic grows, so this is a solid win beyond making sure existing GitHubbers are happy.
Written by
Related posts
Why age assurance laws matter for developers
Youth safety requirements are moving down the tech stack to operating systems and app stores—raising new questions for open source developers.
How researchers are using GitHub Innovation Graph data to reveal the “digital complexity” of nations
Researchers share in an interview how they used GitHub data to predict GDP, inequality, and emissions in ways that traditional economic data misses, along with our Q4 2025 data release.
An update on GitHub availability
Here’s what we’ve done—and what we’re still doing—to improve our availability and reliability.