myvoyage.pl
robots.txt

Robots Exclusion Standard data for myvoyage.pl

Resource Scan

Scan Details

Site Domain myvoyage.pl
Base Domain myvoyage.pl
Scan Status Failed
Failure ReasonScan timed out.
Last Scan2024-09-17T04:24:00+00:00
Next Scan 2024-12-16T04:24:00+00:00

Last Successful Scan

Scanned2024-05-19T23:59:46+00:00
URL https://myvoyage.pl/robots.txt
Domain IPs 5.252.228.188
Response IP 5.252.228.188
Found Yes
Hash efff3af3e4b77c7c5579b3004b86722c23287181d4eab131cac777350871f40c
SimHash 4aced85244a1

Groups

*

Rule Path
Disallow /wp-admin/
Disallow /wp-includes/
Disallow /wp-content/themes/
Disallow /wp-content/plugins/
Allow /wp-admin/admin-ajax.php

mj12bot

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

megaindex.com

Rule Path
Disallow /

obot

Rule Path
Disallow /

fr-crawler

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

python/3.5 aiohttp

Rule Path
Disallow /

adscanner

Rule Path
Disallow /

crawler4j

Rule Path
Disallow /

wbsearchbot

Rule Path
Disallow /

python/3.5 aiohttp

Rule Path
Disallow /

toweya.com

Rule Path
Disallow /

netestate

Rule Path
Disallow /

seekportbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

dotbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

omgili

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

awariorssbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

awariosmartbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

Other Records

Field Value
sitemap https://myvoyage.pl/sitemap_index.xml
sitemap https://myvoyage.pl/post-sitemap.xml

Comments

  • https://megaindex.com/crawler
  • http://filterdb.iss.net/crawler/
  • http://webmeup-crawler.com
  • http://seocompany.store
  • https://github.com/yasserg/crawler4j/
  • http://warebay.com/bot.html
  • http://www.website-datenbank.de/
  • https://bot.seekport.com
  • link contained in logs:
  • https://opensiteexplorer.org/dotbot
  • redirects to
  • https://moz.com/help/moz-procedures/crawlers/dotbot
  • https://www.abuseipdb.com/check/216.244.66.243
  • http://omgili.com/crawler.html
  • https://awario.com/bots.html