j2ski.com
robots.txt

Robots Exclusion Standard data for j2ski.com

Resource Scan

Scan Details

Site Domain j2ski.com
Base Domain j2ski.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-08-17T08:34:58+00:00
Next Scan 2025-10-16T08:34:58+00:00

Last Successful Scan

Scanned2025-06-18T20:47:45+00:00
URL https://j2ski.com/robots.txt
Redirect https://www.j2ski.com/robots.txt
Redirect Domain www.j2ski.com
Redirect Base j2ski.com
Domain IPs 35.176.209.29
Redirect IPs 18.155.68.55, 18.155.68.85, 18.155.68.95, 18.155.68.96
Response IP 35.176.209.29
Found Yes
Hash def61e871f6831669b6bc1b9918ba0bf917007931f84fedc7cffce10ed21ce0d
SimHash f41cd950e14a

Groups

*

Rule Path
Disallow /z/

baiduspider

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

baiduspider

Rule Path
Disallow /forum/
Disallow /ski-chat-forum/

petalbot

Rule Path
Disallow /

bingbot

Rule Path
Disallow /search/

Other Records

Field Value
crawl-delay 10

gptbot

Rule Path
Disallow /ski_hotels/
Disallow /ski_chalets/
Disallow /ski-chat-forum/

siteauditbot

Rule Path
Disallow /

semrushbot-ba

Rule Path
Disallow /

semrushbot-si

Rule Path
Disallow /

semrushbot-swa

Rule Path
Disallow /

semrushbot-ct

Rule Path
Disallow /

splitsignalbot

Rule Path
Disallow /

semrushbot-coub

Rule Path
Disallow /

dataforseobot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 60

dataforseobot

Rule Path
Disallow /apartments/
Disallow /luxury_ski_chalets/
Disallow /luxury_ski_hotels/
Disallow /ski_chalets/
Disallow /ski_hotels/

amazonbot

Product Comment
amazonbot Amazon's user agent
Rule Path
Disallow /apartments/
Disallow /luxury_ski_chalets/
Disallow /luxury_ski_hotels/
Disallow /ski_chalets/
Disallow /ski_hotels/
Disallow /ski-chat-forum/

awariorssbot
awariosmartbot

Rule Path
Disallow /

dotbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

meta-externalagent

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 60

Comments

  • SEMRush bots do a ridiculous amount of crawling
  • Slow these guys down, as they may be useful