Log In Sign Up

csmonitor.com
robots.txt

Robots Exclusion Standard data for csmonitor.com

Resource Scan

Scan Details

Site Domain csmonitor.com
Base Domain csmonitor.com
Scan Status Ok
Last Scan2024-05-21T23:13:00+00:00
Next Scan 2024-05-28T23:13:00+00:00

Last Scan

Scanned2024-05-21T23:13:00+00:00
URL https://csmonitor.com/robots.txt
Redirect https://www.csmonitor.com/robots.txt
Redirect Domain www.csmonitor.com
Redirect Base csmonitor.com
Domain IPs 13.33.30.30, 13.33.30.34, 13.33.30.48, 13.33.30.83
Redirect IPs 13.33.30.30, 13.33.30.34, 13.33.30.48, 13.33.30.83, 2600:9000:229f:4800:1d:c25c:d00:93a1, 2600:9000:229f:5200:1d:c25c:d00:93a1, 2600:9000:229f:7800:1d:c25c:d00:93a1, 2600:9000:229f:9a00:1d:c25c:d00:93a1, 2600:9000:229f:d800:1d:c25c:d00:93a1, 2600:9000:229f:dc00:1d:c25c:d00:93a1, 2600:9000:229f:ee00:1d:c25c:d00:93a1, 2600:9000:229f:fa00:1d:c25c:d00:93a1
Response IP 13.33.30.34
Found Yes
Hash bf234f844efb8ad55a13a927243185296628518ceef64fe371e40c5dfe1602c1
SimHash d01c5a90a7a2

Groups

mediapartners-google*

Rule Path
Disallow /includes/
Disallow /keepalive.html

netshelter contentscan

Rule Path
Disallow /

netshelter

Rule Path
Disallow /

*

Rule Path
Disallow /sudoku
Disallow /404
Disallow /textedition/
Disallow /digitalpdf/
Disallow /content/search
Disallow /content/advancedsearch
Disallow /content/download/
Disallow /layout/set/text/
Disallow /layout/set/print/
Disallow /layout/set/newsletter/
Disallow /layout/set/ap/
Disallow /layout/set/doc/
Disallow /layout/set/htm/
Disallow /layout/set/lmt/
Disallow /layout/set/newsml/
Disallow /layout/set/nfb/
Disallow /layout/set/nitf/
Disallow /layout/set/txt/
Disallow /layout/set/txt_pq/
Disallow /layout/set/xml/
Disallow /layout/set/csm/
Disallow /layout/set/basic/
Disallow /layout/set/rssfull/
Disallow /content/view/full/
Disallow /1010203/
Disallow /tags/topic/
Disallow /csmlists/topic/
Disallow /csmlists/addbookmark/
Disallow /services/
Disallow /auth/

Other Records

Field Value
crawl-delay 1
Back to top

Other Records

Field Value
sitemap https://www.csmonitor.com/sitemap-index.xml
Back to top

Warnings

  • 2 invalid lines.
Back to top