api-app.espn.com
robots.txt

Robots Exclusion Standard data for api-app.espn.com

Resource Scan

Scan Details

Site Domain api-app.espn.com
Base Domain espn.com
Scan Status Ok
Last Scan2024-04-23T07:44:29+00:00
Next Scan 2024-05-23T07:44:29+00:00

Last Scan

Scanned2024-04-23T07:44:29+00:00
URL https://api-app.espn.com/robots.txt
Domain IPs 184.87.193.71, 184.87.193.90, 2600:1413:b000:1f::17d1:2e05, 2600:1413:b000:1f::17d1:2e0a
Response IP 42.99.140.195
Found Yes
Hash 647f81095f8642bd7151e32f8dad5ae98995296f3e2dac744c58990132989a46
SimHash 047b20816db0

Groups

claritybot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

*

Rule Path
Disallow /*admin/
Disallow /*conversation/
Disallow /*conversations/
Disallow /*deportes/
Disallow /*flash/
Disallow /*format/
Disallow /*order/
Disallow /*search?
Disallow /*search/
Disallow /*sort/
Disallow /*util/
Disallow /*webslices/
Disallow /*print?id
Disallow /ad/
Disallow /cgi
Disallow /community/
Disallow /composer/
Disallow /contests/
Disallow /espn/now
Disallow /espnradio/podcast/feeds/easports/
Disallow /index?sport=*&topId=*
Disallow /index?sport=*&type=replay
Disallow /sports/*/index?topId=*
Disallow /members/
Disallow /personalization/
Disallow /travel/passport/activity
Disallow /travel/passport/add
Disallow /travel/passport/event
Disallow /travel/passport/events
Disallow /travel/passport/invite
Disallow /travel/passport/map
Disallow /travel/passport/photos
Disallow /travel/passport/rankings
Disallow /travel/passport/stats
Disallow /travel/passport/venues
Disallow /video/search
Disallow /video/clipDeportes
Disallow /*.com
Disallow /*.net
Disallow /*.org
Disallow /*.co.uk
Disallow /*.com.au
Disallow /*.es
Disallow /*.me
Disallow /*.ly
Disallow /*.me
Disallow /*view/
Disallow /*start/
Disallow /*photoId/
Disallow /*type/
Disallow /*cat/
Disallow /*split/
Disallow /*calendar/
Disallow /*date/
Disallow /*seasontype/
Disallow /*season/200
Disallow /*year/200
Disallow /*_/year/20
Disallow /*_/group/
Disallow /*_/scoreboard/
Disallow /*_/week/
Disallow /*?ex_cid=espnapi_public
Disallow /*%26photoId%3D
Disallow /*%26full%3D
Disallow /*playbyplay?
Disallow /*boxscore?
Disallow /*conversation?
Disallow /*recap?
Disallow /*preview?
Disallow /*databaseresults/
Disallow /mlb/stats/*?*minage=*
Disallow /mlb/stats/*?*split=*

Comments

  • robots.txt for espn.go.com - last updated 20230719
  • Prevents crawl loop issues with incorrectly hyperlinked URLs in posts
  • Exclusions for redundant content that consume crawl budget needlessly
  • Exclusions for content with little to no value for search engines