Log In Sign Up

aerien.ch
robots.txt

Robots Exclusion Standard data for aerien.ch

Resource Scan

Scan Details

Site Domain aerien.ch
Base Domain aerien.ch
Scan Status Ok
Last Scan2024-10-10T14:09:48+00:00
Next Scan 2024-10-17T14:09:48+00:00

Last Scan

Scanned2024-10-10T14:09:48+00:00
URL https://aerien.ch/robots.txt
Redirect https://www.aerien.ch/robots.txt
Redirect Domain www.aerien.ch
Redirect Base aerien.ch
Domain IPs 108.163.147.49, 2607:f748:1200:11d:174:142:221:49
Redirect IPs 108.163.147.49, 2607:f748:1200:11d:174:142:221:49
Response IP 108.163.147.49
Found Yes
Hash 78554e2b04bda3c4617b714b34e6e627d9e304e6d481c00f81269b5790329ae8
SimHash 351b0c0fcdb5

Groups

*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

seekport crawler

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1200

petalbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1200

googlebot

Rule Path
Allow /

ia_archiver

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

lcc

Rule Path
Disallow /

adsbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.aerien.ch/sitemap_especes.xml
sitemap https://www.aerien.ch/sitemap_articles.xml
sitemap https://www.aerien.ch/sitemap_actu.xml
sitemap https://www.aerien.ch/sitemap_photos.xml
sitemap https://www.aerien.ch/sitemap_images.xml
Back to top

Comments

  • Higher crawl delay because dozens of thousands of bots crawl the website
  • but they do not display it in their search results!
  • Makes one wonder what they are looking for.
Back to top