energistafetten.no
robots.txt

Robots Exclusion Standard data for energistafetten.no

Resource Scan

Scan Details

Site Domain energistafetten.no
Base Domain energistafetten.no
Scan Status Ok
Last Scan2025-11-14T11:23:02+00:00
Next Scan 2025-12-14T11:23:02+00:00

Last Scan

Scanned2025-11-14T11:23:02+00:00
URL https://energistafetten.no/robots.txt
Redirect https://www.energistafetten.no/robots.txt
Redirect Domain www.energistafetten.no
Redirect Base energistafetten.no
Domain IPs 194.63.248.52, 2a01:5b40:0:248::52
Redirect IPs 176.111.200.83
Response IP 176.111.200.83
Found Yes
Hash bd44ce24b0ee5bc157ac3ee2bfe3ea38af3bfd500d58d8f9f583e8eefb9cb882
SimHash 63045070ee10

Groups

*

Rule Path
Disallow /error/blackhole
Disallow /storage/
Disallow /fileStorage/

dotbot

Rule Path
Disallow /

googlebot

Rule Path
Allow *.js
Allow *.css

Other Records

Field Value
crawl-delay 10

petalbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

seekportbot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.energistafetten.no/sitemap/events.xml
sitemap https://www.energistafetten.no/sitemap/blog.xml
sitemap https://www.energistafetten.no/sitemap/classifieds.xml
sitemap https://www.energistafetten.no/sitemap/forum.xml
sitemap https://www.energistafetten.no/sitemap/pages.xml