Log In Sign Up

anschlagmittel-shop.at
robots.txt

Robots Exclusion Standard data for anschlagmittel-shop.at

Resource Scan

Scan Details

Site Domain anschlagmittel-shop.at
Base Domain anschlagmittel-shop.at
Scan Status Ok
Last Scan2024-11-07T01:49:42+00:00
Next Scan 2024-12-07T01:49:42+00:00

Last Scan

Scanned2024-11-07T01:49:42+00:00
URL https://anschlagmittel-shop.at/robots.txt
Redirect https://www.anschlagmittel-shop.at/robots.txt
Redirect Domain www.anschlagmittel-shop.at
Redirect Base anschlagmittel-shop.at
Domain IPs 2a01:4f9:3b:3bef::2, 65.21.127.141
Redirect IPs 2a01:4f9:3b:3bef::2, 65.21.127.141
Response IP 65.21.127.141
Found Yes
Hash b175b7eaa575843114d677a20362e0dc99cfbcfd2d3dace6ecfd7e78559a2e6e
SimHash 03d1516146b0

Groups

*

Rule Path
Allow /
Disallow /?q=

baiduspider

Rule Path
Disallow /

cazoodlebot

Rule Path
Disallow /

fasterfox

Rule Path
Disallow /

ingrid

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

jyxobot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

shopwiki

Rule Path
Disallow /

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

yahoo! slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

rogerbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

twengabot-2.0

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

megaindex.ru/2.0

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

megaindex.com

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

bubing

Rule Path
Disallow /

ucrawler/1.0

Rule Path
Disallow /

ucrawler

Rule Path
Disallow /

alphaseobot

Rule Path
Disallow /

alphaseobot-sa

Rule Path
Disallow /

yandeximages

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

vagabondo

Rule Path
Disallow /

velenpublicwebcrawler

Rule Path
Disallow /

pricebot

Rule Path
Disallow /

screaming frog seo spider

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

semrushbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

mail.ru

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 60
Back to top

Other Records

Field Value
sitemap https://www.anschlagmittel-shop.at/sitemap_hijsjob_at.xml
Back to top

Comments

  • https://megaindex.com/crawler
Back to top