map-it.azurewebsites.net
robots.txt

Robots Exclusion Standard data for map-it.azurewebsites.net

Archived Snapshots

Resource Scan

Scan Details

Site Domain	map-it.azurewebsites.net
Base Domain	map-it.azurewebsites.net
Scan Status	Ok
Last Scan	2025-08-19T11:58:26+00:00
Next Scan	2025-08-26T11:58:26+00:00

Last Scan

Scanned	2025-08-19T11:58:26+00:00
URL	https://map-it.azurewebsites.net/robots.txt
Domain IPs	138.91.16.18
Response IP	138.91.16.18
Found	Yes
Hash	1c5e3bff799a14ec1eba432098dd90c8ca3cd0816c1330a2457c93c6d5e7a11c
SimHash	825cd34023cd

Groups

dotbot

Rule	Path
Disallow	/

Rule

Path

Disallow

Other Records

Field	Value
crawl-delay	120

Field

Value

crawl-delay

120

semrushbot

Rule	Path
Disallow	/

Rule

Path

Disallow

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field	Value
crawl-delay	120

Field

Value

crawl-delay

120

yandex

No rules defined. All paths allowed.

Other Records

Field	Value
crawl-delay	120

Field

Value

crawl-delay

120

blexbot

No rules defined. All paths allowed.

Other Records

Field	Value
crawl-delay	120

Field

Value

crawl-delay

120

grapeshot

Rule	Path
Disallow	/

Rule

Path

Disallow

mozilla/5.0 (compatible; grapeshotcrawler/2.0; +http://www.grapeshot.co.uk/crawler.php)

Rule	Path
Disallow	/

Rule

Path

Disallow

serpstatbot

Rule	Path
Disallow	/

Rule

Path

Disallow

amazonbot

Rule	Path
Disallow	/

Rule

Path

Disallow

baiduspider

Rule	Path
Disallow	/

Rule

Path

Disallow

dataforseobot

Rule	Path
Disallow	/

Rule

Path

Disallow

ccbot

Rule	Path
Disallow	/

Rule

Path

Disallow

icc-crawler

Rule	Path
Disallow	/

Rule

Path

Disallow

grapeshot

Rule	Path
Disallow	/

Rule

Path

Disallow

ccbot

Rule	Path
Disallow	/

Rule

Path

Disallow

linespider

Rule	Path
Disallow	/

Rule

Path

Disallow

seokicks

Rule	Path
Disallow	/

Rule

Path

Disallow

seekportbot

Rule	Path
Disallow	/

Rule

Path

Disallow

zhanzhang

Rule	Path
Disallow	/

Rule

Path

Disallow

bytespider

Rule	Path
Disallow	/

Rule

Path

Disallow

y!j*

Rule	Path
Disallow	/

Rule

Path

Disallow

y!j-srd/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

y!j-mbs/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

y!j-brw/1.0 crawler

Rule	Path
Disallow	/

Rule

Path

Disallow

y!j-brj/yats crawler

Rule	Path
Disallow	/

Rule

Path

Disallow

y!j-brl/yatss crawler

Rule	Path
Disallow	/

Rule

Path

Disallow

y!j-brm/yatsd crawler

Rule	Path
Disallow	/

Rule

Path

Disallow

y!j-brn/yatsa crawler

Rule	Path
Disallow	/

Rule

Path

Disallow

y!j-bry/yatsh crawler

Rule	Path
Disallow	/

Rule

Path

Disallow

y!j-brz/yatsha crawler

Rule	Path
Disallow	/

Rule

Path

Disallow

y!j-bri/0.0.1 crawler

Rule	Path
Disallow	/

Rule

Path

Disallow

pingdom.com_bot

Rule	Path
Disallow	/

Rule

Path

Disallow

criteobot

Rule	Path
Disallow	/

Rule

Path

Disallow

ias-va

Rule	Path
Disallow	/

Rule

Path

Disallow

qwantify

Rule	Path
Disallow	/

Rule

Path

Disallow

gptbot

Rule	Path
Disallow	/

Rule

Path

Disallow

*

Rule	Path
Disallow

Rule

Path

Disallow

map-it.azurewebsites.netrobots.txt

Resource Scan

Scan Details

Last Scan

Groups

dotbot

Other Records

semrushbot

ahrefsbot

Other Records

yandex

Other Records

blexbot

Other Records

grapeshot

mozilla/5.0 (compatible; grapeshotcrawler/2.0; +http://www.grapeshot.co.uk/crawler.php)

serpstatbot

amazonbot

baiduspider

dataforseobot

ccbot

icc-crawler

grapeshot

ccbot

linespider

seokicks

seekportbot

zhanzhang

bytespider

y!j*

y!j-srd/1.0

y!j-mbs/1.0

y!j-brw/1.0 crawler

y!j-brj/yats crawler

y!j-brl/yatss crawler

y!j-brm/yatsd crawler

y!j-brn/yatsa crawler

y!j-bry/yatsh crawler

y!j-brz/yatsha crawler

y!j-bri/0.0.1 crawler

pingdom.com_bot

criteobot

ias-va

qwantify

gptbot

*

map-it.azurewebsites.net
robots.txt