knewz.com
robots.txt

Robots Exclusion Standard data for knewz.com

Resource Scan

Scan Details

Site Domain knewz.com
Base Domain knewz.com
Scan Status Ok
Last Scan2024-10-31T17:58:28+00:00
Next Scan 2024-11-07T17:58:28+00:00

Last Scan

Scanned2024-10-31T17:58:28+00:00
URL https://knewz.com/robots.txt
Domain IPs 104.26.10.205, 104.26.11.205, 172.67.68.221, 2606:4700:20::681a:acd, 2606:4700:20::681a:bcd, 2606:4700:20::ac43:44dd
Response IP 104.26.11.205
Found Yes
Hash 8db9fe3ecbaafee83de90bbd2efc779dbf883ea3918552a3f61ff486f382a65f
SimHash 0915c081879b

Groups

crowsnest

Rule Path
Allow /

msnbot

Rule Path
Allow /

slurp

Rule Path
Allow /

teoma

Rule Path
Allow /

twiceler

Rule Path
Allow /

gigabot

Rule Path
Allow /

scrubby

Rule Path
Allow /

robozilla

Rule Path
Allow /

nutch

Rule Path
Allow /

ia_archiver

Rule Path
Allow /

baiduspider

Rule Path
Allow /

*

Rule Path
Allow /
Allow /wp-admin/admin-ajax.php
Disallow /wp-admin/
Disallow /wp-includes
Disallow /wp-content/themes/sloth/
Disallow /readme.html
Disallow /cgi-bin/

archive.org_bot
grapeshotcrawler
semrushbot
yandex
lunabot
amazonbot
bingbot
yandexbot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.knewz.com/sitemap_index.xml
sitemap https://knewz.com/pn/sitemap.xml
sitemap https://knewz.com/pn/gnewssitemap.xml