fabory.com
robots.txt

Robots Exclusion Standard data for fabory.com

Resource Scan

Scan Details

Site Domain fabory.com
Base Domain fabory.com
Scan Status Ok
Last Scan2024-09-11T13:35:29+00:00
Next Scan 2024-10-11T13:35:29+00:00

Last Scan

Scanned2024-09-11T13:35:29+00:00
URL https://fabory.com/robots.txt
Redirect https://www.fabory.com/robots.txt
Redirect Domain www.fabory.com
Redirect Base fabory.com
Domain IPs 13.73.158.53
Redirect IPs 13.107.246.59, 2620:1ec:bdf::59
Response IP 13.107.246.59
Found Yes
Hash 07fec67b36a938294cde41318e8a5532abf1daa75241c21fcb0b224a8477ba68
SimHash 3c563716edf9

Groups

*

Rule Path
Disallow /*/cart
Disallow /*/checkout
Disallow /*/my-account
Disallow /*/My-Favorites/
Disallow /*/product/stock
Disallow /*/product/price
Disallow /fr_XP*
Disallow /es_XP*
Disallow /nl_PM*
Disallow /en_PM*
Disallow /*/search*
Disallow /*?text=
Disallow /*/p/group/returns$

cazoodlebot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot/1.0

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.fabory.com/sitemap/en
sitemap https://www.fabory.com/sitemap/nl
sitemap https://www.fabory.com/sitemap/nl_BE
sitemap https://www.fabory.com/sitemap/fr
sitemap https://www.fabory.com/sitemap/fr_BE
sitemap https://www.fabory.com/sitemap/de_XP
sitemap https://www.fabory.com/sitemap/it_XP
sitemap https://www.fabory.com/sitemap/cs
sitemap https://www.fabory.com/sitemap/es
sitemap https://www.fabory.com/sitemap/hu
sitemap https://www.fabory.com/sitemap/pl
sitemap https://www.fabory.com/sitemap/pt
sitemap https://www.fabory.com/sitemap/ro
sitemap https://www.fabory.com/sitemap/sk
sitemap https://www.fabory.com/sitemap/en_GB

Comments

  • For all robots
  • Block access to specific groups of pages
  • Allow search crawlers to discover the sitemap
  • Block CazoodleBot as it does not present correct accept content headers
  • Block MJ12bot as it is just noise
  • Block dotbot as it cannot parse base urls properly
  • Block Gigabot