Log In Sign Up

video.wttw.com
robots.txt

Robots Exclusion Standard data for video.wttw.com

Resource Scan

Scan Details

Site Domain video.wttw.com
Base Domain wttw.com
Scan Status Ok
Last Scan2024-05-03T12:40:09+00:00
Next Scan 2024-06-02T12:40:09+00:00

Last Scan

Scanned2024-05-03T12:40:09+00:00
URL https://video.wttw.com/robots.txt
Domain IPs 54.225.206.152
Response IP 50.16.19.106
Found Yes
Hash de85d797cae677e9b2c1eb5218cf2c2b2a917c5c6068b98f41b6d90a2af30528
SimHash 2e28b452c6f7

Groups

mediapartners-google

Rule Path
Disallow

*

Rule Path
Disallow /shows-page
Disallow /latest-episode
Disallow /watchlist/
Disallow /watchlist/page
Disallow /viewing-history/
Disallow /viewing-history/page
Disallow /favorite-shows/
Disallow /favorite-shows-page
Disallow /search
Disallow /search-videos
Disallow /login/
Disallow /logout/
Disallow /personal/
Disallow /profile
Disallow /feedback/submit/
Disallow /cc-info/
Disallow /troubleshooting/
Disallow /passport/
Allow /passport/learn-more/

Other Records

Field Value
crawl-delay 10

gptbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /
Back to top

Comments

  • FYI: This file only applies to Station Video Portals
  • PBS.org's robots.txt is maintained by Ops
  • shows data page
  • all personal pages
  • all search and search videos
  • all profile views
  • disallow most passport views
  • but allow the Passport Learn More page
  • RWEB-8073 blocking AI bots
Back to top