salsaposten.no
robots.txt

Robots Exclusion Standard data for salsaposten.no

Resource Scan

Scan Details

Site Domain salsaposten.no
Base Domain salsaposten.no
Scan Status Ok
Last Scan2024-06-07T11:30:07+00:00
Next Scan 2024-06-14T11:30:07+00:00

Last Scan

Scanned2024-06-07T11:30:07+00:00
URL https://salsaposten.no/robots.txt
Redirect https://www.salsaposten.no/robots.txt
Redirect Domain www.salsaposten.no
Redirect Base salsaposten.no
Domain IPs 2a02:c0:ac::e51:1, 87.238.38.1, 87.238.38.2
Redirect IPs 104.18.34.36, 172.64.153.220, 2606:4700:4400::6812:2224, 2606:4700:4400::ac40:99dc
Response IP 104.18.34.36
Found Yes
Hash 47d01716cfe6d5dae1de22b78ddec850593ba09e148a26a193b6d014e36f3240
SimHash 304688c2c557

Groups

*

Rule Path
Allow /ads.txt
Disallow /

ccbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

Comments

  • publication for testing, deny spiders
  • Start AI crawler block
  • End AI crawler block