Log In Sign Up

app.scotsman.com
robots.txt

Robots Exclusion Standard data for app.scotsman.com

Resource Scan

Scan Details

Site Domain app.scotsman.com
Base Domain scotsman.com
Scan Status Ok
Last Scan2025-09-14T01:24:32+00:00
Next Scan 2025-09-28T01:24:32+00:00

Last Scan

Scanned2025-09-14T01:24:32+00:00
URL https://app.scotsman.com/robots.txt
Domain IPs 151.101.1.72, 151.101.129.72, 151.101.193.72, 151.101.65.72, 2a04:4e42:200::328, 2a04:4e42:400::328, 2a04:4e42:600::328, 2a04:4e42::328
Response IP 199.232.45.72
Found Yes
Hash b7a56ab8f7d12fc09bd06502809a786af8882e041ab8282138a41680ad6f81ff
SimHash 4824cab3efe3

Groups

applebot

Rule Path
Disallow /preview.html
Disallow /bolt.html
Disallow /bolt_preview.html
Disallow /search.html
Disallow /search/
Disallow /subs/
Disallow /api/

twitterbot

Rule Path
Disallow /preview.html
Disallow /bolt.html
Disallow /bolt_preview.html
Disallow /search.html
Disallow /search/
Disallow /subs/
Disallow /api/

*

Rule Path
Allow /.well-known/
Allow /ads.txt
Allow /app-ads.txt

*

Rule Path
Disallow /
Back to top

Comments

  • Allow Applebot, except for internal URLs
  • Allow Twitterbot, except for internal URLs
  • Disallow everyone else to everything else
Back to top