Log In Sign Up

mitari.it
robots.txt

Robots Exclusion Standard data for mitari.it

Resource Scan

Scan Details

Site Domain mitari.it
Base Domain mitari.it
Scan Status Ok
Last Scan2024-11-09T13:11:16+00:00
Next Scan 2024-12-09T13:11:16+00:00

Last Scan

Scanned2024-11-09T13:11:16+00:00
URL https://mitari.it/robots.txt
Redirect https://www.mitari.it/robots.txt
Redirect Domain www.mitari.it
Redirect Base mitari.it
Domain IPs 2a01:4f9:3b:3bef::2, 65.21.127.141
Redirect IPs 2a01:4f9:3b:3bef::2, 65.21.127.141
Response IP 65.21.127.141
Found Yes
Hash bd2b96d445f0ae8404e4ce6e6bb6eaab7ff44b1f3204231a0b146b9949803a3e
SimHash 03d1516180b0

Groups

baiduspider

Rule Path
Disallow /

cazoodlebot

Rule Path
Disallow /

fasterfox

Rule Path
Disallow /

ingrid

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

jyxobot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

shopwiki

Rule Path
Disallow /

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

yahoo! slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

rogerbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

twengabot-2.0

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

megaindex.ru/2.0

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

megaindex.com

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

bubing

Rule Path
Disallow /

ucrawler/1.0

Rule Path
Disallow /

ucrawler

Rule Path
Disallow /

alphaseobot

Rule Path
Disallow /

alphaseobot-sa

Rule Path
Disallow /

yandeximages

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

vagabondo

Rule Path
Disallow /

velenpublicwebcrawler

Rule Path
Disallow /

pricebot

Rule Path
Disallow /

screaming frog seo spider

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

semrushbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

mail.ru

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 60
Back to top

Other Records

Field Value
sitemap https://www.mitari.it/sitemap_mitari_it.xml
sitemap https://www.mitari.it/sitemap_hijsjob_it.xml
Back to top

Comments

  • https://megaindex.com/crawler
Back to top