complaintsbook.co.za
robots.txt

Robots Exclusion Standard data for complaintsbook.co.za

Resource Scan

Scan Details

Site Domain complaintsbook.co.za
Base Domain complaintsbook.co.za
Scan Status Ok
Last Scan2024-10-02T00:34:07+00:00
Next Scan 2024-10-09T00:34:07+00:00

Last Scan

Scanned2024-10-02T00:34:07+00:00
URL https://complaintsbook.co.za/robots.txt
Domain IPs 104.21.38.162, 172.67.136.20
Response IP 172.67.136.20
Found Yes
Hash c13c5882a72a6cd8fef61db50e6eede516cb956052f05f68ac14251cd2e51354
SimHash eb71bb086000

Groups

*

Rule Path
Disallow /admin/
Disallow /attachments/*
Disallow /bo/
Disallow /complaints/follow/*
Disallow /compare/*
Disallow /assets/*
Disallow /docs/*
Disallow /images/
Disallow /user/*
Disallow /manual/*
Disallow /search
Disallow /search/*
Disallow /*?q=*
Disallow /*?p=*
Disallow /*.pdf$
Disallow *.pdf

baiduspider

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

python-urllib

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

megaindex.com

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

qwantify

Rule Path
Disallow /

sentibot

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

slurp

Rule Path
Disallow /

yahoo! slurp china

Rule Path
Disallow /

Other Records

Field Value
sitemap https://complaintsbook.co.za/sitemap.xml

Comments

  • SITEMAPS