On Nov 24, GSC changed our lives! Yes, in this #OnCrawlOpenMic we'll be talking about the new #CrawlStats report.
🎙Join the discussion and tell us what you think!
https://twitter.com/googlesearchc/status/1331268722397503494
How Google crawls a website and what #googlebot finds used to be information you could only get in log files ...but as #technicalSEO players, we all know how hard it can be to get our hands on those.
#OnCrawlOpenMic
How do you monitor how your site is crawled by Google?
@googlesearchc
#OnCrawlOpenMic
So what's in the Crawl Stats report? Head to GSC > Settings > Crawl Stats and explore! Or let @TomekRudzki walk you through it.
https://twitter.com/TomekRudzki/status/1331315982422351872
#OnCrawlOpenMic
This new report is đź’° particularly if you don't have access to your (or your client's) log files. We like @micahfk's list of pros and cons:
https://twitter.com/micahfk/status/1331383631021961217
#OnCrawlOpenMic
Our TOP 3 favorite things about the Crawl Stats report:
1. Easy-to-read, high-level googlebot crawl stats, and hints like the green/yellow/red status indicators for the host status. This report is meant for everyone.
#OnCrawlOpenMic
2. The documentation, which not only answers 99% of your questions, but also gives best practices and tips for server health, red flags, crawl frequency management and basic googlebot wrangling.
https://support.google.com/webmasters/answer/9679690
#OnCrawlOpenMic
3. CRAWL PURPOSE BABY and, as @jbobbink adds, Googlebot page resource load, or when Google returns just to download a resource. This Google-specific stuff isn't in your log files!
https://twitter.com/jbobbink/status/1331499666740011008
#OnCrawlOpenMic
So, wait… if the GSC Crawl Stats report is magic (and it is!), should we still be using log files? AB-SO-LUTELY! Here's why ⤵️
#OnCrawlOpenMic
1. Logs tell you about more than just googlebot: bingbot, organic traffic, referring pages, orphan pages…
https://twitter.com/dawnieando/status/1220063605837783041
#OnCrawlOpenMic
2. Many people quickly noticed differences of 20-40% between the GSC report and their log files. This is a known issue in the Crawl Stats—but not in your logs!
https://twitter.com/Marie_Haynes/status/1331964978115514369
#OnCrawlOpenMic
3. Crawl Stats report = dashboard; logs = full data. You can see ALL requests in your logs, not just a sample, or filter your logs based on countries, dates, URLs... #OnCrawlOpenMic
4. Like @nikrangerseo, we love using the GSC report to find what to look for in logs. Then, you're free to look up ancient history, transform, analyze, merge, data viz-ify, do machine learning… #OnCrawlOpenMic https://twitter.com/nikrangerseo/status/1331367979330596866
5. Log files are always in style and won't go out of date. The reasons won't fit in a single tweet, but trust us (and @gfiorelli1) on this one! #OnCrawlOpenMic https://twitter.com/gfiorelli1/status/1319223762789650432
But back to Crawl Stats. We're stoked to see more people getting access to this type of info! Like @ohgm, we have tons of ideas. And like @pechnet, we want to know: when will there be an API? #OnCrawlOpenMic
https://twitter.com/Pechnet/status/1331305830671593473 https://twitter.com/ohgm/status/1331512151962488832
What's YOUR favorite thing to do with information in the new GSC Crawl Stats report (or log files)?
#OnCrawlOpenMic @bhartzer @senormunoz @OmiSido @serbayarda @muratyatagan @5le @chrisgreen87 @jennyhalasz @diije @naguibtoihiri @LukaszZelezny @RobinRozhon @KorayGubur @Suganthanmn
You can follow @OnCrawl.
Tip: mention @twtextapp on a Twitter thread with the keyword “unroll” to get a link to it.

Latest Threads Unrolled:
Those who think consciousness raising & movement building are disconnected, or even a distraction from electoral politics have not paid attention to the history of social change.In this thread, we
Read more
The 2020s will be known as the Remote Work decadeA few predictions of what is likely to emerge[ a thread ] Third Space: Office and Working from Home will
Read more
1/29: Have you ever had a concept explained to you that helps frame complex issues you’ve been wrestling with and opens your eyes to new possibilities? A concept that I
Read more
By continuing to use the site, you are consenting to the use of cookies as explained in our Cookie Policy to improve your experience.