Log In Sign Up

aurainweb.pl
robots.txt

Robots Exclusion Standard data for aurainweb.pl

Resource Scan

Scan Details

Site Domain aurainweb.pl
Base Domain aurainweb.pl
Scan Status Ok
Last Scan2024-09-17T03:59:53+00:00
Next Scan 2024-10-17T03:59:53+00:00

Last Scan

Scanned2024-09-17T03:59:53+00:00
URL https://aurainweb.pl/robots.txt
Domain IPs 5.252.228.188
Response IP 5.252.228.188
Found Yes
Hash 49e435768fe2e8381b2cf8b4a1c36c908ef7a1b7b51d392f22521291270fa691
SimHash 4acad85244a1

Groups

scrapy

Rule Path
Allow /

*

Rule Path
Disallow /wp-admin/
Disallow /wp-includes/
Disallow /wp-content/themes/
Disallow /wp-content/plugins/
Allow /wp-admin/admin-ajax.php

mj12bot

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

megaindex.com

Rule Path
Disallow /

obot

Rule Path
Disallow /

fr-crawler

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

python/3.5 aiohttp

Rule Path
Disallow /

adscanner

Rule Path
Disallow /

crawler4j

Rule Path
Disallow /

wbsearchbot

Rule Path
Disallow /

python/3.5 aiohttp

Rule Path
Disallow /

toweya.com

Rule Path
Disallow /

netestate

Rule Path
Disallow /

seekportbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

dotbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

omgili

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

awariorssbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

awariosmartbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20
Back to top

Other Records

Field Value
sitemap https://www.aurainweb.pl/wp-sitemap.xml
Back to top

Comments

  • https://megaindex.com/crawler
  • http://filterdb.iss.net/crawler/
  • http://webmeup-crawler.com
  • http://seocompany.store
  • https://github.com/yasserg/crawler4j/
  • http://warebay.com/bot.html
  • http://www.website-datenbank.de/
  • https://bot.seekport.com
  • link contained in logs:
  • https://opensiteexplorer.org/dotbot
  • redirects to
  • https://moz.com/help/moz-procedures/crawlers/dotbot
  • https://www.abuseipdb.com/check/216.244.66.243
  • http://omgili.com/crawler.html
  • https://awario.com/bots.html
Back to top