pbs.org
robots.txt

Robots Exclusion Standard data for pbs.org

Resource Scan

Scan Details

Site Domain pbs.org
Base Domain pbs.org
Scan Status Ok
Last Scan2024-10-22T04:01:13+00:00
Next Scan 2024-11-05T04:01:13+00:00

Last Scan

Scanned2024-10-22T04:01:13+00:00
URL https://pbs.org/robots.txt
Redirect https://www.pbs.org/robots.txt
Redirect Domain www.pbs.org
Redirect Base pbs.org
Domain IPs 54.225.198.196, 54.225.206.152
Redirect IPs 216.137.52.100, 216.137.52.114, 216.137.52.17, 216.137.52.27, 2600:9000:21f8:3200:0:3833:24c0:93a1, 2600:9000:21f8:6200:0:3833:24c0:93a1, 2600:9000:21f8:6c00:0:3833:24c0:93a1, 2600:9000:21f8:8c00:0:3833:24c0:93a1, 2600:9000:21f8:a200:0:3833:24c0:93a1, 2600:9000:21f8:ca00:0:3833:24c0:93a1, 2600:9000:21f8:e000:0:3833:24c0:93a1, 2600:9000:21f8:ee00:0:3833:24c0:93a1
Response IP 3.164.68.62
Found Yes
Hash 33db22e41d59f22107d9d0db0fe399c99e8e9b7db55c037a9078e16188e4f3d2
SimHash 03a7b575f730

Groups

*

Rule Path
Disallow /wgbh/pages/frontline/digitalnation/%3F
Disallow /wgbh/pages/frontline/digitalnation/%3F*
Disallow /wgbh/frontline/*elqat*
Disallow /wgbh/pages/frontline/*elqat*
Disallow /wgbh/pages/frontline/bushswar*
Disallow /independentlens/getinvolved/cinema/
Disallow /cgi-bin/*
Disallow /cgi-registry/*
Disallow /survey/
Disallow /whatson/stations/
Disallow /whatson/schedulex/
Disallow /whatson/pbsyou/
Disallow /whatson/dbs/
Disallow /remotecontrol/bestpractices/prototype/
Disallow /remotecontrol/bestpractices/jl-prototype/
Disallow /pressroom/
Disallow /als/shakespeare_pilot/
Disallow /newshour/bb/military/military.html/
Disallow /producers/webmanual/
Disallow /devcon2007/
Disallow /smcot/
Disallow /pbsholidaycard/
Disallow /gpra/
Disallow /tlqa/
Disallow /remote.control/
Disallow /teachers/connect/
Disallow /engage/sites/all/files/blog_images/monkeys.gif
Disallow /teachers/edcar/
Disallow /teachers/dll_research/
Disallow /peerconnection/feeds/
Disallow /teachers/innovators/toolkit/
Disallow /mt4/
Disallow /klru/mt4/
Disallow /newshour/mt4/
Disallow /mediashift/mt4/
Disallow /kcet/mt4/
Disallow /newshour/mt3/
Disallow /now/mt3/
Disallow /itvs/mt3/
Disallow /pbs/mt3/
Disallow /pbs/mt3o4/
Disallow /pbs/mt4/
Disallow /pbs/teachers/mt3/
Disallow /pbs/kids/mt3/
Disallow /pbs/kids/mt4/
Disallow /pbs/src/mt4/
Disallow /pbs/parents/mt4/
Disallow /pbsi/mt3/
Disallow /moyers/mt3/
Disallow /moyers/mt4/
Disallow /nbr/mt3/
Disallow /nbr/mt4/
Disallow /cringely/mt3/
Disallow /cringely/mt4/
Disallow /wgbh/mt4/
Disallow /wnet/expose/mt4/
Disallow /wnet/mt4/
Disallow /opb/mt4/
Disallow /search/search_results.html?q=*
Disallow /wgbh/nova/search/
Disallow /wgbh/nova/search/results/*
Disallow /wgbh/nova/beta/search/results/*
Disallow /nbr/info/search.html?*
Disallow /independentlens/video/index.php?*
Disallow /independentlens/video/?filter=*
Disallow /wnet/religionandethics/*/index.html?*
Disallow /wnet/religionandethics/search_results.html*
Disallow /teachers/search/resources/?q=*
Disallow /teacherline/catalog/browse/
Disallow /thewar/search_results.php?*
Disallow */emailthispage.html?*
Disallow /pov_mt/mt4/
Disallow /newshour/indepth_coverage/middle_east/iraq/honorroll/search.php?*
Disallow /includes/
Disallow */emailthispage.html
Disallow /about/faq/station-finder/?state-text=*
Disallow /about/faq/station-finder/?zip-text=*
Disallow /wgbh/nova/search-results.html
Disallow /wgbh/americanexperience/about/search/search_results.html
Disallow /moyers/journal/about/search_google.html
Disallow /foundation/contact/
Disallow *?replytocom
Disallow /social_media/
Disallow /independentlens/film-lists/watch-online/
Disallow /show/koth-show/
Disallow /my-list-videos/
Disallow /my-list-shows/
Disallow /my-list-viewing-history/
Disallow /parents/activity-finder/
Disallow /parents/search/

Other Records

Field Value
crawl-delay 1

baiduspider

Rule Path
Disallow /independentlens/getinvolved/cinema/
Disallow /independentlens/films/watch-online/*

gptbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.pbs.org/sitemap.xml
sitemap http://www.pbs.org/mercy-street/sitemap.xml
sitemap https://www.pbs.org/newshour/sitemaps/sitemap.xml
sitemap https://www.pbs.org/weta/washingtonweek/sitemap.xml

Comments

  • Disallow: /.