directorycircle.com
robots.txt

Robots Exclusion Standard data for directorycircle.com

Resource Scan

Scan Details

Site Domain directorycircle.com
Base Domain directorycircle.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2024-09-19T09:03:09+00:00
Next Scan 2024-11-18T09:03:09+00:00

Last Successful Scan

Scanned2024-07-22T09:01:18+00:00
URL https://directorycircle.com/robots.txt
Domain IPs 104.21.63.85, 172.67.144.86, 2606:4700:3032::ac43:9056, 2606:4700:3033::6815:3f55
Response IP 172.67.144.86
Found Yes
Hash 440c6437e09af3b26d055bc9aedd809c0100f6cae465149d9fd5ec2005f2f3f3
SimHash 5bcfd450c228

Groups

*

Rule Path
Allow /
Disallow *.csv

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

bingpreview

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

applebot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

petalbot

Rule Path
Disallow /

xxxxahrefsbot

Rule Path
Disallow /

bdcbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

seokicks

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

indeedbot

Rule Path
Disallow /

seekport

Rule Path
Disallow /

nekstbot

Rule Path
Disallow /

nimbostratus-bot

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

mojeekbot

Rule Path
Disallow /

ltx71

Rule Path
Disallow /

xxxxsemrushbot

Rule Path
Disallow /

megaindex.ru/2.0

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

xxxxsemrushbot

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

serpstatbot

Rule Path
Disallow /

clark-crawler2

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

neevabot

Rule Path
Disallow /

geedobot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://directorycircle.com/sitemap.xml