broadband.espn.com
robots.txt

Robots Exclusion Standard data for broadband.espn.com

Resource Scan

Scan Details

Site Domain broadband.espn.com
Base Domain espn.com
Scan Status Ok
Last Scan2024-09-15T07:10:33+00:00
Next Scan 2024-10-15T07:10:33+00:00

Last Scan

Scanned2024-09-15T07:10:33+00:00
URL https://broadband.espn.com/robots.txt
Domain IPs 100.20.97.142, 2600:1f13:cd1:c300:7132:3a65:dead:be47, 2600:1f13:cd1:c300:8822:329:26b:73f, 2600:1f13:cd1:c300:d5ea:bc93:711a:69ae, 2600:1f13:cd1:c300:ec37:de33:4ab3:a688, 2600:1f13:cd1:c301:2bff:5d89:b6d4:f87, 2600:1f13:cd1:c301:bebc:6757:aa3a:173c, 2600:1f13:cd1:c301:f7c2:e26e:ae12:8992, 2600:1f13:cd1:c302:7867:c05a:ea8:7df3, 34.218.203.223, 35.166.10.115, 50.112.146.18, 52.26.138.238, 52.42.95.47, 54.189.49.60, 54.190.253.132
Response IP 54.149.245.199
Found Yes
Hash 647f81095f8642bd7151e32f8dad5ae98995296f3e2dac744c58990132989a46
SimHash 047b20816db0

Groups

claritybot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

*

Rule Path
Disallow /*admin/
Disallow /*conversation/
Disallow /*conversations/
Disallow /*deportes/
Disallow /*flash/
Disallow /*format/
Disallow /*order/
Disallow /*search?
Disallow /*search/
Disallow /*sort/
Disallow /*util/
Disallow /*webslices/
Disallow /*print?id
Disallow /ad/
Disallow /cgi
Disallow /community/
Disallow /composer/
Disallow /contests/
Disallow /espn/now
Disallow /espnradio/podcast/feeds/easports/
Disallow /index?sport=*&topId=*
Disallow /index?sport=*&type=replay
Disallow /sports/*/index?topId=*
Disallow /members/
Disallow /personalization/
Disallow /travel/passport/activity
Disallow /travel/passport/add
Disallow /travel/passport/event
Disallow /travel/passport/events
Disallow /travel/passport/invite
Disallow /travel/passport/map
Disallow /travel/passport/photos
Disallow /travel/passport/rankings
Disallow /travel/passport/stats
Disallow /travel/passport/venues
Disallow /video/search
Disallow /video/clipDeportes
Disallow /*.com
Disallow /*.net
Disallow /*.org
Disallow /*.co.uk
Disallow /*.com.au
Disallow /*.es
Disallow /*.me
Disallow /*.ly
Disallow /*.me
Disallow /*view/
Disallow /*start/
Disallow /*photoId/
Disallow /*type/
Disallow /*cat/
Disallow /*split/
Disallow /*calendar/
Disallow /*date/
Disallow /*seasontype/
Disallow /*season/200
Disallow /*year/200
Disallow /*_/year/20
Disallow /*_/group/
Disallow /*_/scoreboard/
Disallow /*_/week/
Disallow /*?ex_cid=espnapi_public
Disallow /*%26photoId%3D
Disallow /*%26full%3D
Disallow /*playbyplay?
Disallow /*boxscore?
Disallow /*conversation?
Disallow /*recap?
Disallow /*preview?
Disallow /*databaseresults/
Disallow /mlb/stats/*?*minage=*
Disallow /mlb/stats/*?*split=*

Comments

  • robots.txt for espn.go.com - last updated 20230719
  • Prevents crawl loop issues with incorrectly hyperlinked URLs in posts
  • Exclusions for redundant content that consume crawl budget needlessly
  • Exclusions for content with little to no value for search engines