Log In Sign Up

fatboy.com
robots.txt

Robots Exclusion Standard data for fatboy.com

Resource Scan

Scan Details

Site Domain fatboy.com
Base Domain fatboy.com
Scan Status Ok
Last Scan2024-06-18T13:59:35+00:00
Next Scan 2024-07-18T13:59:35+00:00

Last Scan

Scanned2024-06-18T13:59:35+00:00
URL https://fatboy.com/robots.txt
Redirect https://www.fatboy.com:443/robots.txt
Redirect Domain www.fatboy.com
Redirect Base fatboy.com
Domain IPs 52.209.22.3, 52.210.224.133, 99.81.23.71
Redirect IPs 2600:9000:21e1:2800:1d:d4ca:6ec0:93a1, 2600:9000:21e1:4000:1d:d4ca:6ec0:93a1, 2600:9000:21e1:5000:1d:d4ca:6ec0:93a1, 2600:9000:21e1:6200:1d:d4ca:6ec0:93a1, 2600:9000:21e1:8800:1d:d4ca:6ec0:93a1, 2600:9000:21e1:9e00:1d:d4ca:6ec0:93a1, 2600:9000:21e1:c000:1d:d4ca:6ec0:93a1, 2600:9000:21e1:c800:1d:d4ca:6ec0:93a1, 3.165.102.121, 3.165.102.126, 3.165.102.54, 3.165.102.87
Response IP 3.165.102.121
Found Yes
Hash 5625f6f01863c27d6647c871ac36b51799a08a5b79c3e7251324f77f21c34b7f
SimHash 2b6cd9ea8b30

Groups

blexbot

Rule Path
Disallow /

baidu spider

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

yandex bot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

duckduckgo bot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

*

Rule Path
Disallow /*cart
Disallow /*?variant_overview=1
Disallow /*?SID=
Disallow /sx-fr/*
Disallow /*channable
Disallow /*gtm_position
Disallow /*profileid
Disallow /*profilesource
Back to top

Other Records

Field Value
sitemap https://www.fatboy.com/sitemap.xml.gz
Back to top

Comments

  • Production
  • Crawlers Setup
  • Directories and query strings
  • Sitemap URL
Back to top