api.espn.go.com
robots.txt

Robots Exclusion Standard data for api.espn.go.com

Resource Scan

Scan Details

Site Domain api.espn.go.com
Base Domain go.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-09-25T13:40:23+00:00
Next Scan 2025-12-24T13:40:23+00:00

Last Successful Scan

Scanned2023-08-14T17:00:46+00:00
URL https://api.espn.go.com/robots.txt
Domain IPs 35.83.192.132, 44.232.98.230, 44.236.85.8, 44.241.164.35, 52.13.133.36, 52.13.199.106, 54.148.105.143, 54.187.14.81
Response IP 52.13.133.36
Found Yes
Hash 34197f0005d1abdd58ec9e19b41d8f1bdfaec4f50e3772516534dafee6d428ae
SimHash 042b20816cb0

Groups

*

Rule Path
Disallow /*admin/
Disallow /*conversation/
Disallow /*conversations/
Disallow /*deportes/
Disallow /*flash/
Disallow /*format/
Disallow /*order/
Disallow /*search?
Disallow /*search/
Disallow /*sort/
Disallow /*util/
Disallow /*webslices/
Disallow /*print?id
Disallow /ad/
Disallow /cgi
Disallow /community/
Disallow /composer/
Disallow /contests/
Disallow /espn/now
Disallow /espnradio/podcast/feeds/easports/
Disallow /index?sport=*&topId=*
Disallow /index?sport=*&type=replay
Disallow /sports/*/index?topId=*
Disallow /members/
Disallow /personalization/
Disallow /travel/passport/activity
Disallow /travel/passport/add
Disallow /travel/passport/event
Disallow /travel/passport/events
Disallow /travel/passport/invite
Disallow /travel/passport/map
Disallow /travel/passport/photos
Disallow /travel/passport/rankings
Disallow /travel/passport/stats
Disallow /travel/passport/venues
Disallow /video/search
Disallow /video/clipDeportes
Disallow /*.com
Disallow /*.net
Disallow /*.org
Disallow /*.co.uk
Disallow /*.com.au
Disallow /*.es
Disallow /*.me
Disallow /*.ly
Disallow /*.me
Disallow /*view/
Disallow /*start/
Disallow /*photoId/
Disallow /*type/
Disallow /*cat/
Disallow /*split/
Disallow /*calendar/
Disallow /*date/
Disallow /*seasontype/
Disallow /*season/200
Disallow /*year/200
Disallow /*_/year/20
Disallow /*_/group/
Disallow /*_/scoreboard/
Disallow /*_/week/
Disallow /*?ex_cid=espnapi_public
Disallow /*%26photoId%3D
Disallow /*%26full%3D
Disallow /*playbyplay?
Disallow /*boxscore?
Disallow /*conversation?
Disallow /*recap?
Disallow /*preview?
Disallow /*databaseresults/
Disallow /mlb/stats/*?*minage=*
Disallow /mlb/stats/*?*split=*

Comments

  • robots.txt for espn.go.com - last updated 20160719
  • Prevents crawl loop issues with incorrectly hyperlinked URLs in posts
  • Exclusions for redundant content that consume crawl budget needlessly
  • Exclusions for content with little to no value for search engines