video.geo.hosted.espn.com
robots.txt

Robots Exclusion Standard data for video.geo.hosted.espn.com

Resource Scan

Scan Details

Site Domain video.geo.hosted.espn.com
Base Domain espn.com
Scan Status Ok
Last Scan2024-06-09T16:21:12+00:00
Next Scan 2024-07-09T16:21:12+00:00

Last Scan

Scanned2024-06-09T16:21:12+00:00
URL https://video.geo.hosted.espn.com/robots.txt
Domain IPs 2600:1f13:cd1:c300:49a0:5386:d72b:eb65, 2600:1f13:cd1:c300:a807:bbfd:be97:5e8d, 2600:1f13:cd1:c300:d99:c65f:8315:bdc, 2600:1f13:cd1:c301:736e:bcd3:43f:db34, 2600:1f13:cd1:c301:f1e7:431:7431:45b4, 2600:1f13:cd1:c302:d2e4:da9d:6828:4b2a, 2600:1f13:cd1:c302:dcd0:8ef6:3862:bde8, 2600:1f13:cd1:c302:eb0a:c04:8a18:cb7a, 34.209.45.126, 35.155.74.204, 35.161.242.155, 35.165.150.54, 52.34.242.50, 52.39.153.145, 54.184.243.78, 54.201.158.2
Response IP 35.161.242.155
Found Yes
Hash 647f81095f8642bd7151e32f8dad5ae98995296f3e2dac744c58990132989a46
SimHash 047b20816db0

Groups

claritybot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

*

Rule Path
Disallow /*admin/
Disallow /*conversation/
Disallow /*conversations/
Disallow /*deportes/
Disallow /*flash/
Disallow /*format/
Disallow /*order/
Disallow /*search?
Disallow /*search/
Disallow /*sort/
Disallow /*util/
Disallow /*webslices/
Disallow /*print?id
Disallow /ad/
Disallow /cgi
Disallow /community/
Disallow /composer/
Disallow /contests/
Disallow /espn/now
Disallow /espnradio/podcast/feeds/easports/
Disallow /index?sport=*&topId=*
Disallow /index?sport=*&type=replay
Disallow /sports/*/index?topId=*
Disallow /members/
Disallow /personalization/
Disallow /travel/passport/activity
Disallow /travel/passport/add
Disallow /travel/passport/event
Disallow /travel/passport/events
Disallow /travel/passport/invite
Disallow /travel/passport/map
Disallow /travel/passport/photos
Disallow /travel/passport/rankings
Disallow /travel/passport/stats
Disallow /travel/passport/venues
Disallow /video/search
Disallow /video/clipDeportes
Disallow /*.com
Disallow /*.net
Disallow /*.org
Disallow /*.co.uk
Disallow /*.com.au
Disallow /*.es
Disallow /*.me
Disallow /*.ly
Disallow /*.me
Disallow /*view/
Disallow /*start/
Disallow /*photoId/
Disallow /*type/
Disallow /*cat/
Disallow /*split/
Disallow /*calendar/
Disallow /*date/
Disallow /*seasontype/
Disallow /*season/200
Disallow /*year/200
Disallow /*_/year/20
Disallow /*_/group/
Disallow /*_/scoreboard/
Disallow /*_/week/
Disallow /*?ex_cid=espnapi_public
Disallow /*%26photoId%3D
Disallow /*%26full%3D
Disallow /*playbyplay?
Disallow /*boxscore?
Disallow /*conversation?
Disallow /*recap?
Disallow /*preview?
Disallow /*databaseresults/
Disallow /mlb/stats/*?*minage=*
Disallow /mlb/stats/*?*split=*

Comments

  • robots.txt for espn.go.com - last updated 20230719
  • Prevents crawl loop issues with incorrectly hyperlinked URLs in posts
  • Exclusions for redundant content that consume crawl budget needlessly
  • Exclusions for content with little to no value for search engines