watch.api.espn.com
robots.txt

Robots Exclusion Standard data for watch.api.espn.com

Resource Scan

Scan Details

Site Domain watch.api.espn.com
Base Domain espn.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-04-09T03:33:39+00:00
Next Scan 2024-07-08T03:33:39+00:00

Last Successful Scan

Scanned2023-08-18T15:00:22+00:00
URL http://watch.api.espn.com/robots.txt
Domain IPs 23.2.16.9, 23.202.35.48, 2600:1413:b000:14::b857:c147, 2600:1413:b000:14::b857:c15a
Response IP 23.33.126.141
Found Yes
Hash dc2d439e070ee0f794b1f2e6ee0a0b9d68ec03c49ac7ca74548144db308ac773
SimHash 007b20816cb0

Groups

claritybot

Rule Path
Disallow /

*

Rule Path
Disallow /*admin/
Disallow /*conversation/
Disallow /*conversations/
Disallow /*deportes/
Disallow /*flash/
Disallow /*format/
Disallow /*order/
Disallow /*search?
Disallow /*search/
Disallow /*sort/
Disallow /*util/
Disallow /*webslices/
Disallow /*print?id
Disallow /ad/
Disallow /cgi
Disallow /community/
Disallow /composer/
Disallow /contests/
Disallow /espn/now
Disallow /espnradio/podcast/feeds/easports/
Disallow /index?sport=*&topId=*
Disallow /index?sport=*&type=replay
Disallow /sports/*/index?topId=*
Disallow /members/
Disallow /personalization/
Disallow /travel/passport/activity
Disallow /travel/passport/add
Disallow /travel/passport/event
Disallow /travel/passport/events
Disallow /travel/passport/invite
Disallow /travel/passport/map
Disallow /travel/passport/photos
Disallow /travel/passport/rankings
Disallow /travel/passport/stats
Disallow /travel/passport/venues
Disallow /video/search
Disallow /video/clipDeportes
Disallow /*.com
Disallow /*.net
Disallow /*.org
Disallow /*.co.uk
Disallow /*.com.au
Disallow /*.es
Disallow /*.me
Disallow /*.ly
Disallow /*.me
Disallow /*view/
Disallow /*start/
Disallow /*photoId/
Disallow /*type/
Disallow /*cat/
Disallow /*split/
Disallow /*calendar/
Disallow /*date/
Disallow /*seasontype/
Disallow /*season/200
Disallow /*year/200
Disallow /*_/year/20
Disallow /*_/group/
Disallow /*_/scoreboard/
Disallow /*_/week/
Disallow /*?ex_cid=espnapi_public
Disallow /*%26photoId%3D
Disallow /*%26full%3D
Disallow /*playbyplay?
Disallow /*boxscore?
Disallow /*conversation?
Disallow /*recap?
Disallow /*preview?
Disallow /*databaseresults/
Disallow /mlb/stats/*?*minage=*
Disallow /mlb/stats/*?*split=*

Comments

  • robots.txt for espn.go.com - last updated 20230719
  • Prevents crawl loop issues with incorrectly hyperlinked URLs in posts
  • Exclusions for redundant content that consume crawl budget needlessly
  • Exclusions for content with little to no value for search engines