cancerhealth.com
robots.txt

Robots Exclusion Standard data for cancerhealth.com

Resource Scan

Scan Details

Site Domain cancerhealth.com
Base Domain cancerhealth.com
Scan Status Ok
Last Scan2024-10-30T11:04:54+00:00
Next Scan 2024-11-06T11:04:54+00:00

Last Scan

Scanned2024-10-30T11:04:54+00:00
URL https://cancerhealth.com/robots.txt
Redirect https://www.cancerhealth.com/robots.txt
Redirect Domain www.cancerhealth.com
Redirect Base cancerhealth.com
Domain IPs 100.29.92.245, 44.199.10.197
Redirect IPs 100.29.92.245, 44.199.10.197
Response IP 44.199.10.197
Found Yes
Hash 8750befb1b62600b3d4ca449fc5d55434b19f70953386f795957cb4b76e98d8f
SimHash 6b4f5851ccb0

Groups

*

Rule Path
Disallow */get-pdf

Other Records

Field Value
crawl-delay 12

mj12bot
seekportbot

Rule Path
Disallow */get-pdf

Other Records

Field Value
crawl-delay 20

awariorssbot
awariosmartbot
blexbot
dataforseobot
dotbot
panscient.com
petalbot
turnitinbot

Rule Path
Disallow */get-pdf

Other Records

Field Value
crawl-delay 16

ahrefsbot
ccbot
claudebot
daum
friendlycrawler
gptbot
hawaiibot
linkedinbot
megaindex.com
megaindex.ru
netestate
netestate%20ne%20crawler
panscient.com
seokicks
seokicks-robot
serpstatbot
treato-bot

Rule Path
Disallow */get-pdf

Other Records

Field Value
crawl-delay 14

bingbot
slurp

Rule Path
Disallow */get-pdf

Other Records

Field Value
crawl-delay 8

Other Records

Field Value
sitemap https://www.cancerhealth.com/sitemap.xml

Comments

  • automation managed
  • block get-pdf
  • default 10 sec delay
  • 20 sec delay
  • 16 sec delay
  • 14 sec delay
  • 8 sec delay