Log In Sign Up

google.com
robots.txt

Robots Exclusion Standard data for google.com

Resource Scan

Scan Details

Site Domain google.com
Base Domain google.com
Scan Status Ok
Last Scan2024-10-25T13:31:20+00:00
Next Scan 2024-11-08T13:31:20+00:00

Last Scan

Scanned2024-10-25T13:31:20+00:00
URL https://google.com/robots.txt
Redirect https://www.google.com/robots.txt
Redirect Domain www.google.com
Redirect Base google.com
Domain IPs 2404:6800:4003:c02::65, 2404:6800:4003:c02::66, 2404:6800:4003:c02::71, 2404:6800:4003:c02::8b, 74.125.68.100, 74.125.68.101, 74.125.68.102, 74.125.68.113, 74.125.68.138, 74.125.68.139
Redirect IPs 2404:6800:4003:c03::63, 2404:6800:4003:c03::67, 2404:6800:4003:c03::68, 2404:6800:4003:c03::69, 74.125.130.103, 74.125.130.104, 74.125.130.105, 74.125.130.106, 74.125.130.147, 74.125.130.99
Response IP 172.253.118.106
Found Yes
Hash af8e6c36fcd2518a2710c95898b618d9aad448b5ea46cac4ee6bbc94903201e8
SimHash 3908436da47e

Groups

*

Rule Path
Disallow /search
Allow /search/about
Allow /search/static
Allow /search/howsearchworks
Disallow /sdch
Disallow /groups
Disallow /index.html?
Disallow /?
Allow /?hl=
Disallow /?hl=*&
Allow /?hl=*&gws_rd=ssl$
Disallow /?hl=*&*&gws_rd=ssl
Allow /?gws_rd=ssl$
Allow /?pt1=true$
Disallow /imgres
Disallow /u/
Disallow /setprefs
Disallow /default
Disallow /m?
Disallow /m/
Allow /m/finance
Disallow /wml?
Disallow /wml/?
Disallow /wml/search?
Disallow /xhtml?
Disallow /xhtml/?
Disallow /xhtml/search?
Disallow /xml?
Disallow /imode?
Disallow /imode/?
Disallow /imode/search?
Disallow /jsky?
Disallow /jsky/?
Disallow /jsky/search?
Disallow /pda?
Disallow /pda/?
Disallow /pda/search?
Disallow /sprint_xhtml
Disallow /sprint_wml
Disallow /pqa
Disallow /gwt/
Disallow /purchases
Disallow /local?
Disallow /local_url
Disallow /shihui?
Disallow /shihui/
Disallow /products?
Disallow /product_
Disallow /products_
Disallow /products%3B
Disallow /print
Disallow /books/
Disallow /bkshp?*q=*
Disallow /books?*q=*
Disallow /books?*output=*
Disallow /books?*pg=*
Disallow /books?*jtp=*
Disallow /books?*jscmd=*
Disallow /books?*buy=*
Disallow /books?*zoom=*
Allow /books?*q=related%3A*
Allow /books?*q=editions%3A*
Allow /books?*q=subject%3A*
Allow /books/about
Allow /booksrightsholders
Allow /books?*zoom=1*
Allow /books?*zoom=5*
Allow /books/content?*zoom=1*
Allow /books/content?*zoom=5*
Disallow /ebooks/
Disallow /ebooks?*q=*
Disallow /ebooks?*output=*
Disallow /ebooks?*pg=*
Disallow /ebooks?*jscmd=*
Disallow /ebooks?*buy=*
Disallow /ebooks?*zoom=*
Allow /ebooks?*q=related%3A*
Allow /ebooks?*q=editions%3A*
Allow /ebooks?*q=subject%3A*
Allow /ebooks?*zoom=1*
Allow /ebooks?*zoom=5*
Disallow /patents?
Disallow /patents/download/
Disallow /patents/pdf/
Disallow /patents/related/
Disallow /scholar
Disallow /citations?
Allow /citations?user=
Disallow /citations?*cstart=
Allow /citations?view_op=new_profile
Allow /citations?view_op=top_venues
Allow /scholar_share
Disallow /s?
Disallow /maps?
Allow /maps?*output=classic*
Allow /maps?*file=
Disallow /mapstt?
Disallow /mapslt?
Disallow /mapabcpoi?
Disallow /maphp?
Disallow /mapprint?
Disallow /maps/
Allow /maps/search/
Allow /maps/dir/
Allow /maps/d/
Allow /maps/reserve
Allow /maps/about
Allow /maps/match
Disallow /maps/api/js/
Allow /maps/api/js
Disallow /mld?
Disallow /staticmap?
Disallow /help/maps/streetview/partners/welcome/
Disallow /help/maps/indoormaps/partners/
Disallow /lochp?
Disallow /center
Disallow /ie?
Disallow /blogsearch/
Disallow /blogsearch_feeds
Disallow /advanced_blog_search
Disallow /uds/
Disallow /chart?
Disallow /transit?
Allow /calendar$
Allow /calendar/about/
Disallow /calendar/
Disallow /cl2/feeds/
Disallow /cl2/ical/
Disallow /coop/directory
Disallow /coop/manage
Disallow /trends?
Disallow /trends/music?
Disallow /trends/hottrends?
Disallow /trends/viz?
Disallow /trends/embed.js?
Disallow /trends/fetchComponent?
Disallow /trends/beta
Disallow /trends/topics
Disallow /trends/explore?
Disallow /trends/embed
Disallow /trends/api
Disallow /musica
Disallow /musicad
Disallow /musicas
Disallow /musicl
Disallow /musics
Disallow /musicsearch
Disallow /musicsp
Disallow /musiclp
Disallow /urchin_test/
Disallow /movies?
Disallow /wapsearch?
Allow /safebrowsing/diagnostic
Allow /safebrowsing/report_badware/
Allow /safebrowsing/report_error/
Allow /safebrowsing/report_phish/
Disallow /reviews/search?
Disallow /orkut/albums
Disallow /cbk
Disallow /recharge/dashboard/car
Disallow /recharge/dashboard/static/
Disallow /profiles/me
Allow /profiles
Disallow /s2/profiles/me
Allow /s2/profiles
Allow /s2/oz
Allow /s2/photos
Allow /s2/search/social
Allow /s2/static
Disallow /s2
Disallow /transconsole/portal/
Disallow /gcc/
Disallow /aclk
Disallow /cse?
Disallow /cse/home
Disallow /cse/panel
Disallow /cse/manage
Disallow /tbproxy/
Disallow /imesync/
Disallow /shenghuo/search?
Disallow /support/forum/search?
Disallow /reviews/polls/
Disallow /hosted/images/
Disallow /ppob/?
Disallow /ppob?
Disallow /accounts/ClientLogin
Disallow /accounts/ClientAuth
Disallow /accounts/o8
Allow /accounts/o8/id
Disallow /topicsearch?q=
Disallow /xfx7/
Disallow /squared/api
Disallow /squared/search
Disallow /squared/table
Disallow /qnasearch?
Disallow /app/updates
Disallow /sidewiki/entry/
Disallow /quality_form?
Disallow /labs/popgadget/search
Disallow /buzz/post
Disallow /compressiontest/
Disallow /analytics/feeds/
Disallow /analytics/partners/comments/
Disallow /analytics/portal/
Disallow /analytics/uploads/
Allow /alerts/manage
Allow /alerts/remove
Disallow /alerts/
Allow /alerts/$
Disallow /ads/search?
Disallow /ads/plan/action_plan?
Disallow /ads/plan/api/
Disallow /ads/hotels/partners
Disallow /phone/compare/?
Disallow /travel/clk
Disallow /travel/entity
Disallow /travel/search
Disallow /travel/flights/s/
Disallow /travel/hotels/entity
Disallow /travel/hotels/stories
Disallow /hotelfinder/rpc
Disallow /hotels/rpc
Disallow /commercesearch/services/
Disallow /evaluation/
Disallow /chrome/browser/mobile/tour
Disallow /compare/*/apply*
Disallow /forms/perks/
Disallow /shopping/suppliers/search
Disallow /ct/
Disallow /edu/cs4hs/
Disallow /trustedstores/s/
Disallow /trustedstores/tm2
Disallow /trustedstores/verify
Disallow /adwords/proposal
Disallow /shopping?*
Disallow /shopping/product/
Disallow /shopping/seller
Disallow /shopping/ratings/account/metrics
Disallow /shopping/ratings/merchant/immersivedetails
Disallow /shopping/reviewer
Disallow /shopping/search
Disallow /shopping/deals
Disallow /storefront
Disallow /storepicker
Allow /about/careers/applications/
Allow /about/careers/applications-a/
Allow /about/careers/applications/teams/
Allow /about/careers/applications-a/teams/
Allow /about/careers/applications/locations/
Allow /about/careers/applications-a/locations/
Allow /about/careers/applications/benefits/
Allow /about/careers/applications-a/benefits/
Allow /about/careers/applications/students/
Allow /about/careers/applications-a/students/
Allow /about/careers/applications/jobs/results/$
Allow /about/careers/applications-a/jobs/results/$
Allow /about/careers/applications/cloud/
Allow /about/careers/applications-a/cloud/
Allow /about/careers/applications/hardware/
Allow /about/careers/applications-a/hardware/
Allow /about/careers/applications/eeo/
Allow /about/careers/applications-a/eeo/
Allow /about/careers/applications/how-we-hire/
Allow /about/careers/applications-a/how-we-hire/
Allow /about/careers/applications/interview-tips/
Allow /about/careers/applications-a/interview-tips/
Disallow /about/careers/applications/
Disallow /about/careers/applications-a/
Disallow /about/careers/applications/jobs/results?page=
Disallow /about/careers/applications/jobs/results/?page=
Disallow /about/careers/applications/jobs/results?*&page=
Disallow /about/careers/applications/jobs/results/?*&page=
Disallow /landing/signout.html
Disallow /webmasters/sitemaps/ping?
Disallow /ping?
Disallow /gallery/
Disallow /landing/now/ontap/
Allow /searchhistory/
Allow /maps/reserve
Allow /maps/reserve/partners
Disallow /maps/reserve/api/
Disallow /maps/reserve/search
Disallow /maps/reserve/bookings
Disallow /maps/reserve/settings
Disallow /maps/reserve/manage
Disallow /maps/reserve/payment
Disallow /maps/reserve/receipt
Disallow /maps/reserve/sellersignup
Disallow /maps/reserve/payments
Disallow /maps/reserve/feedback
Disallow /maps/reserve/terms
Disallow /maps/reserve/m/
Disallow /maps/reserve/b/
Disallow /maps/reserve/partner-dashboard
Disallow /about/views/
Disallow /intl/*/about/views/
Disallow /local/cars
Disallow /local/cars/
Disallow /local/dealership/
Disallow /local/dining/
Disallow /local/place/products/
Disallow /local/place/reviews/
Disallow /local/place/rap/
Disallow /local/tab/
Disallow /localservices/*
Allow /finance
Allow /js/
Disallow /nonprofits/account/
Disallow /uviewer
Disallow /landing/cmsnext-root/

adsbot-google

Rule Path
Disallow /maps/api/js/
Allow /maps/api/js
Disallow /maps/api/place/js/
Disallow /maps/api/staticmap
Disallow /maps/api/streetview

twitterbot

Rule Path
Allow /imgres
Allow /search
Disallow /groups
Disallow /hosted/images/
Disallow /m/

facebookexternalhit

Rule Path
Allow /imgres
Allow /search
Disallow /groups
Disallow /hosted/images/
Disallow /m/

anthropic-ai
applebot-extended
bytespider
ccbot
chatgpt-user
claudebot
cohere-ai
diffbot
facebookbot
gptbot
imagesiftbot
meta-externalagent
meta-externalfetcher
omgilibot
perplexitybot
timpibot

Rule Path
Disallow /maps*
Back to top

Other Records

Field Value
sitemap https://www.google.com/sitemap.xml
Back to top

Comments

  • AdsBot
  • Crawlers of certain social media sites are allowed to access page markup when google.com/imgres* links are shared. To learn more, please contact images-robots-allowlist@google.com.
  • Explicit rules for common LLM bots that might not attribute source data.
Back to top