mapcrow.info
robots.txt

Robots Exclusion Standard data for mapcrow.info

Resource Scan

Scan Details

Site Domain mapcrow.info
Base Domain mapcrow.info
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't establish SSL connection.
Last Scan2025-06-23T14:52:58+00:00
Next Scan 2025-09-21T14:52:58+00:00

Last Successful Scan

Scanned2024-08-28T13:48:49+00:00
URL https://mapcrow.info/robots.txt
Domain IPs 77.77.151.176
Response IP 77.77.151.176
Found Yes
Hash 8fe5853253ca4b32a86c3f00f6c4879e5f503fd3e3ac2ba3e6e92a17ec30f332
SimHash f31ed4e3ef90

Groups

*

Rule Path
Disallow /captcha.php

spbot

Rule Path
Disallow /

bdfetch

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

mlbot

Rule Path
Disallow /

msrbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

getintent crawler (http://getintent.com/bot.html)

Rule Path
Disallow /

clickagy intelligence bot v2

Rule Path
Disallow /

smtbot/1.0

Rule Path
Disallow /

*

Rule Path
Disallow /*sort_by*

*

Rule Path
Disallow /*landmarket*

*

Rule Path
Disallow /*driving*

*

Rule Path
Disallow /*Driving*

*

Rule Path
Disallow /*flight-time*

*

Rule Path
Disallow /*Travel-Time*

*

Rule Path
Disallow /*mobroutecnt*

blexbot

Rule Path
Disallow /

garlikcrawler/1.2

Rule Path
Disallow /

commoncrawler

Rule Path
Disallow /

istellabot

Rule Path
Disallow /

istellabot/t.1

Rule Path
Disallow /

istellabot/t.1.13

Rule Path
Disallow /

megaindex.ru/2.0

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

bubing

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

mj12bot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

ccbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

Comments

  • User-agent: Applebot/0.1
  • Disallow: /