Log In Sign Up

laybyland.com.au
robots.txt

Robots Exclusion Standard data for laybyland.com.au

Resource Scan

Scan Details

Site Domain laybyland.com.au
Base Domain laybyland.com.au
Scan Status Ok
Last Scan2024-09-17T05:06:30+00:00
Next Scan 2024-10-17T05:06:30+00:00

Last Scan

Scanned2024-09-17T05:06:30+00:00
URL https://laybyland.com.au/robots.txt
Redirect https://www.laybyland.com.au/robots.txt
Redirect Domain www.laybyland.com.au
Redirect Base laybyland.com.au
Domain IPs 13.225.4.10, 13.225.4.115, 13.225.4.118, 13.225.4.98
Redirect IPs 13.225.4.10, 13.225.4.115, 13.225.4.118, 13.225.4.98
Response IP 13.225.4.10
Found Yes
Hash fd32d632c953e27b7344891bc35c6c1690bbf632f29d97114e8eee0cc88bacc6
SimHash 2814d9c2c68c

Groups

bingbot

Rule Path
Disallow /wishlist*

*

Rule Path
Allow /

siteauditbot

Rule Path
Disallow /

semrushbot-ba

Rule Path
Disallow /

semrushbot-si

Rule Path
Disallow /

semrushbot-swa

Rule Path
Disallow /

semrushbot-ocob

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

applebot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

youbot

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

geedoproductsearch

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

facebookcatalog/1.0

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

facebookexternalhit/1.1

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

facebookexternalhit/1.0 (+http://www.facebook.com/externalhit_uatext.php)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

*

Rule Path
Disallow /*?
Disallow /index.php/
Disallow /catalog/product_compare/
Disallow /catalog/category/view/
Disallow /catalog/product/view/
Disallow /wishlist/
Disallow /admin/
Disallow /catalogsearch/
Disallow /review/product/
Disallow /sendfriend/
Disallow /enable-cookies/
Disallow /LICENSE.txt
Disallow /LICENSE.html
Disallow /skin/
Disallow /js/
Disallow /checkout/
Disallow /onestepcheckout/
Disallow /customer/
Disallow /customer/account/
Disallow /customer/account/login/
Disallow /catalogsearch/
Disallow /catalog/product_compare/
Disallow /catalog/category/view/
Disallow /catalog/product/view/
Disallow /*?dir*
Disallow /*?dir=desc
Disallow /*?dir=asc
Disallow /*?limit=all
Disallow /*?mode*
Disallow /*.php$
Disallow /*?SID=
Disallow /app/
Disallow /bin/
Disallow /dev/
Disallow /lib/
Disallow /phpserver/
Disallow /pub/
Disallow /secomm-pma-tool/
Disallow /phpmyadmin/
Disallow /phpMyAdmin/
Disallow /pma/

googlebot

Rule Path
Disallow

googlebot-image

Rule Path
Disallow

mail.ru

Rule Path
Disallow /

yandex

Rule Path
Disallow /

twiceler

Rule Path
Disallow /
Allow /media/sitemaps/laybyland/

seekport crawler

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.laybyland.com.au/sitemap.xml
sitemap https://www.laybyland.com.au/media/sitemaps/laybyland/sitemap_1.xml
sitemap https://www.laybyland.com.au/media/sitemaps/laybyland/sitemap_2.xml
Back to top

Comments

  • block SemrushBot
  • AI bots
  • GeedoProductSearch
  • Facebook Crawler Setup
  • Crawlers Setup
  • Directories
  • Stop crawling user account and checkout pages by search engine robot:
  • Blocking native catalog and search pages:
  • Paths (no clean URLs)
  • More reasonable to use canonical tag on these pages.
  • Blocking CMS directories.
  • Block URL Tools
  • Google Crawler Setup
  • Google Image Crawler Setup
  • Allow Sitemap
  • Block Bad Bot
Back to top