carb-counter.net
robots.txt

Robots Exclusion Standard data for carb-counter.net

Resource Scan

Scan Details

Site Domain carb-counter.net
Base Domain carb-counter.net
Scan Status Ok
Last Scan2024-09-24T23:49:13+00:00
Next Scan 2024-10-01T23:49:13+00:00

Last Scan

Scanned2024-09-24T23:49:13+00:00
URL https://carb-counter.net/robots.txt
Redirect https://www.carb-counter.net/robots.txt
Redirect Domain www.carb-counter.net
Redirect Base carb-counter.net
Domain IPs 44.231.242.138
Redirect IPs 44.231.242.138
Response IP 44.231.242.138
Found Yes
Hash 7b3f9a68ccd35305499817f639303c42233107e69f542527d43cf0be48d02b23
SimHash eb1c444242b1

Groups

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

psbot

Rule Path
Disallow /

*

Rule Path
Disallow /redirect.aspx

mj12bot
semrushbot-sa
semrushbot
ahrefsbot
dotbot
mauibot
mauibot (crawler.feedback+wc@gmail.com)
megaindex.ru/2.0
seekport crawler
sputnikbot/2.3
the knowledge ai
turnitinbot
ucbrowser
yacybot
yeti
seznambot
coccocbot-web

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.carb-counter.net/sitemap_index.xml