Log In Sign Up

nupi.no
robots.txt

Robots Exclusion Standard data for nupi.no

Resource Scan

Scan Details

Site Domain nupi.no
Base Domain nupi.no
Scan Status Ok
Last Scan2025-09-16T02:09:56+00:00
Next Scan 2025-10-16T02:09:56+00:00

Last Scan

Scanned2025-09-16T02:09:56+00:00
URL https://nupi.no/robots.txt
Redirect https://www.nupi.no/robots.txt
Redirect Domain www.nupi.no
Redirect Base nupi.no
Domain IPs 151.101.130.216, 151.101.194.216, 151.101.2.216, 151.101.66.216, 2a04:4e42:200::728, 2a04:4e42:400::728, 2a04:4e42:600::728, 2a04:4e42:800::728, 2a04:4e42::728, 2a04:4e42:a00::728, 2a04:4e42:c00::728, 2a04:4e42:e00::728
Redirect IPs 151.101.130.216, 151.101.194.216, 151.101.2.216, 151.101.66.216, 2a04:4e42:200::728, 2a04:4e42:400::728, 2a04:4e42:600::728, 2a04:4e42:800::728, 2a04:4e42::728, 2a04:4e42:a00::728, 2a04:4e42:c00::728, 2a04:4e42:e00::728
Response IP 151.101.194.216
Found Yes
Hash 62aa14db26f331185621bfda36df81638cb340cfa26ce4a5761cfb40ac95f4ee
SimHash 175cd332e462

Groups

yandexbot

Rule Path
Disallow /

slurp

Rule Path
Disallow /

exabot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

sitebulb

Rule Path
Disallow /

sogou

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

*

Rule Path
Disallow /content/download
Disallow /mtmte
Disallow /sok
Disallow /*?pcform

Other Records

Field Value
crawl-delay 5
Back to top

Other Records

Field Value
sitemap https://www.nupi.no/sitemap.xml
Back to top

Comments

  • Yandex search, resource intensive
  • Yahoo search, can be intensive
  • Exalead search
  • Petal search, Huawei
  • Anthropic, AI, aggressive
  • Bytedance AI
  • Alexa AI
  • Common Crawl, creates large datasets
  • Mozilla SEO bot
  • Extensive SEO crawling bot
  • Aggressive SEO bot
  • SEO bot
  • Tencent
  • Distributed web crawler that generates lots of traffic
  • pcform is not unique pages, just a filter
Back to top