Log In Sign Up

arkivi-bildagentur.de
robots.txt

Robots Exclusion Standard data for arkivi-bildagentur.de

Resource Scan

Scan Details

Site Domain arkivi-bildagentur.de
Base Domain arkivi-bildagentur.de
Scan Status Ok
Last Scan2024-06-08T08:25:26+00:00
Next Scan 2024-07-08T08:25:26+00:00

Last Scan

Scanned2024-06-08T08:25:26+00:00
URL https://arkivi-bildagentur.de/robots.txt
Redirect https://www.arkivi-bildagentur.de/robots.txt
Redirect Domain www.arkivi-bildagentur.de
Redirect Base arkivi-bildagentur.de
Domain IPs 104.21.77.117, 172.67.207.91, 2606:4700:3035::ac43:cf5b, 2606:4700:3037::6815:4d75
Redirect IPs 104.21.77.117, 172.67.207.91, 2606:4700:3035::ac43:cf5b, 2606:4700:3037::6815:4d75
Response IP 104.21.77.117
Found Yes
Hash 7d7b2d8820ad63798eac793d79e196aade0a8531899e7d8c0a3cce2530e97706
SimHash a89508ad7154

Groups

sidewinder

Rule Path
Disallow /

microsoft.url.control

Rule Path
Disallow /

*

Rule Path
Disallow /tmp/
Disallow /test/
Disallow /*.gif$
Disallow /admin/
Disallow /orders/
Disallow /mark/
Disallow /account/
Disallow /*/add_to_cart
Disallow /*/to_luminary/*
Disallow /assets/
Disallow /search/*
Disallow /all_tagged_with/*
Disallow /*/tagged_with/*
Allow /assets/arkivi_bildarchiv*
Allow /assets/application*.css
Back to top

Other Records

Field Value
sitemap http://www.arkivi.de/sitemaps/sitemap.xml.gz
Back to top

Comments

  • See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file
  • To ban all spiders from the entire site uncomment the next two lines:
  • Disallow: /category/*/article/*
Back to top