cremicro.com
robots.txt

Robots Exclusion Standard data for cremicro.com

Archived Snapshots

Resource Scan

Scan Details

Site Domain	cremicro.com
Base Domain	cremicro.com
Scan Status	Failed
Failure Stage	Fetching resource.
Failure Reason	Server returned a client error.
Last Scan	2024-11-10T06:40:25+00:00
Next Scan	2024-11-24T06:40:25+00:00

Last Successful Scan

Scanned	2024-09-14T01:38:50+00:00
URL	https://cremicro.com/robots.txt
Domain IPs	104.21.38.22, 172.67.218.8, 2606:4700:3031::ac43:da08, 2606:4700:3035::6815:2616
Response IP	172.67.218.8
Found	Yes
Hash	6b86f8959e6dca8f9c74e120f5b597059ac30c3451ef0d71a52d8a4c1944c45e
SimHash	4bf2480049b3

Groups

alexibot

Rule	Path
Disallow	/

Rule

Path

Disallow

aqua_products

Rule	Path
Disallow	/

Rule

Path

Disallow

backdoorbot/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

backlinkcrawler

Rule	Path
Disallow	/

Rule

Path

Disallow

bookmark search tool

Rule	Path
Disallow	/

Rule

Path

Disallow

botalot

Rule	Path
Disallow	/

Rule

Path

Disallow

bullseye/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

copyrightcheck

Rule	Path
Disallow	/

Rule

Path

Disallow

dotbot

Rule	Path
Disallow	/

Rule

Path

Disallow

download ninja

Rule	Path
Disallow	/

Rule

Path

Disallow

httrack

Rule	Path
Disallow	/

Rule

Path

Disallow

httrack 3.0

Rule	Path
Disallow	/

Rule

Path

Disallow

keyword density/0.9

Rule	Path
Disallow	/

Rule

Path

Disallow

linkscan/8.1a unix

Rule	Path
Disallow	/

Rule

Path

Disallow

linkwalker

Rule	Path
Disallow	/

Rule

Path

Disallow

mister pix

Rule	Path
Disallow	/

Rule

Path

Disallow

mj12bot

Rule	Path
Disallow	/

Rule

Path

Disallow

mlbot

Rule	Path
Disallow	/

Rule

Path

Disallow

openbot

Rule	Path
Disallow	/

Rule

Path

Disallow

propowerbot/2.14

Rule	Path
Disallow	/

Rule

Path

Disallow

prowebwalker

Rule	Path
Disallow	/

Rule

Path

Disallow

rma

Rule	Path
Disallow	/

Rule

Path

Disallow

surveybot

Rule	Path
Disallow	/

Rule

Path

Disallow

teleport

Rule	Path
Disallow	/

Rule

Path

Disallow

teleportpro

Rule	Path
Disallow	/

Rule

Path

Disallow

true_robot

Rule	Path
Disallow	/

Rule

Path

Disallow

true_robot/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

url control

Rule	Path
Disallow	/

Rule

Path

Disallow

url_spider_pro

Rule	Path
Disallow	/

Rule

Path

Disallow

webcapture 2.0

Rule	Path
Disallow	/

Rule

Path

Disallow

webcopier

Rule	Path
Disallow	/

Rule

Path

Disallow

webcopier v.2.2

Rule	Path
Disallow	/

Rule

Path

Disallow

webcopier v3.2a

Rule	Path
Disallow	/

Rule

Path

Disallow

webzip

Rule	Path
Disallow	/

Rule

Path

Disallow

webzip/4.0

Rule	Path
Disallow	/

Rule

Path

Disallow

webzip/4.21

Rule	Path
Disallow	/

Rule

Path

Disallow

webzip/5.0

Rule	Path
Disallow	/

Rule

Path

Disallow

xenu

Rule	Path
Disallow	/

Rule

Path

Disallow

xenuâ€™s

Rule	Path
Disallow	/

Rule

Path

Disallow

xenuâ€™s link sleuth 1.1c

Rule	Path
Disallow	/

Rule

Path

Disallow

yeti

Rule	Path
Disallow	/

Rule

Path

Disallow

ranksignals

Rule	Path
Disallow	/

Rule

Path

Disallow

*

Rule	Path
Disallow	/wp-admin/
Allow	/wp-admin/admin-ajax.php
Allow	/wp-content/uploads/

Rule

Path

Disallow

/wp-admin/

Allow

/wp-admin/admin-ajax.php

Allow

/wp-content/uploads/

Other Records

Field	Value
sitemap	https://cremicro.com/sitemap_index.xml

Field

Value

sitemap

https://cremicro.com/sitemap_index.xml

Comments

CCCCCCCC

cremicro.comrobots.txt

Resource Scan

Scan Details

Last Successful Scan

Groups

alexibot

aqua_products

backdoorbot/1.0

backlinkcrawler

bookmark search tool

botalot

bullseye/1.0

copyrightcheck

dotbot

download ninja

httrack

httrack 3.0

keyword density/0.9

linkscan/8.1a unix

linkwalker

mister pix

mj12bot

mlbot

openbot

propowerbot/2.14

prowebwalker

rma

surveybot

teleport

teleportpro

true_robot

true_robot/1.0

url control

url_spider_pro

webcapture 2.0

webcopier

webcopier v.2.2

webcopier v3.2a

webzip

webzip/4.0

webzip/4.21

webzip/5.0

xenu

xenuâ€™s

xenuâ€™s link sleuth 1.1c

yeti

ranksignals

*

Other Records

Comments

cremicro.com
robots.txt