carethy.net
robots.txt

Robots Exclusion Standard data for carethy.net

Resource Scan

Scan Details

Site Domain carethy.net
Base Domain carethy.net
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a server error.
Last Scan2024-06-14T17:54:30+00:00
Next Scan 2024-06-21T17:54:30+00:00

Last Successful Scan

Scanned2024-06-06T17:53:43+00:00
URL https://carethy.net/robots.txt
Redirect https://www.carethy.net/robots.txt
Redirect Domain www.carethy.net
Redirect Base carethy.net
Domain IPs 104.26.14.90, 104.26.15.90, 172.67.68.164, 2606:4700:20::681a:e5a, 2606:4700:20::681a:f5a, 2606:4700:20::ac43:44a4
Redirect IPs 104.26.14.90, 104.26.15.90, 172.67.68.164, 2606:4700:20::681a:e5a, 2606:4700:20::681a:f5a, 2606:4700:20::ac43:44a4
Response IP 104.26.14.90
Found Yes
Hash e4466fba2bf604dabfbb7030f6867c2fec19244cfb52220c3814edc2ac43f278
SimHash 1877c1435a90

Groups

baiduspider
ahrefsbot
compraraquibot
voilabot
yandexbot
mj12bot
ezooms
heritrix
bingbot
blexbot
jobdiggerspider
kulturarw3
xovibot
siteexplorer
vagabondo
eccolabot
seokicks
dotbot
ltx71
scrapy
worldwebheritage
semrushbot
semrushbot-sa
prooxibot
applebot
spbot
nekstbot

Rule Path
Disallow /

bingbot

Rule Path
Disallow /admin/
Disallow /application/
Disallow /css/
Disallow /data/
Disallow /feeds/
Disallow /imgs/
Disallow /js/
Disallow /lib/
Disallow /plugins/
Disallow /scripts/
Disallow /templates/
Disallow /rtp.php
Allow /

Other Records

Field Value
crawl-delay 30

googlebot
googlebot-news
googlebot-image
googlebot-video
googlebot-mobile
mediapartners
mediapartners-google
adsbot-google

Rule Path
Disallow /admin/
Disallow /application/
Disallow /data/
Disallow /feeds/
Disallow /lib/
Disallow /plugins/
Disallow /scripts/
Disallow /templates/
Disallow /rtp.php
Allow /

Other Records

Field Value
crawl-delay 1

*

Rule Path
Disallow /admin/
Disallow /application/
Disallow /css/
Disallow /data/
Disallow /feeds/
Disallow /imgs/
Disallow /js/
Disallow /lib/
Disallow /plugins/
Disallow /scripts/
Disallow /templates/
Disallow /search/
Disallow /rtp.php
Allow /

Other Records

Field Value
crawl-delay 30

Other Records

Field Value
sitemap https://www.carethy.net/xmls/sitemap.xml

Comments

  • Google ignores this, but in case...

Warnings

  • 1 invalid line.