casetify.com
robots.txt

Robots Exclusion Standard data for casetify.com

Resource Scan

Scan Details

Site Domain casetify.com
Base Domain casetify.com
Scan Status Ok
Last Scan2024-11-09T09:48:47+00:00
Next Scan 2024-11-23T09:48:47+00:00

Last Scan

Scanned2024-11-09T09:48:47+00:00
URL https://casetify.com/robots.txt
Redirect https://www.casetify.com/robots.txt
Redirect Domain www.casetify.com
Redirect Base casetify.com
Domain IPs 104.18.28.89, 104.18.29.89
Redirect IPs 104.18.28.89, 104.18.29.89
Response IP 104.18.29.89
Found Yes
Hash 5d87950093b6f14f92ea458590f2c832bdc8ef233bf7725383b0e6ebbc93daa2
SimHash 6d2e5ab6c5f7

Groups

*

Rule Path
Disallow /*index.php/
Disallow /*.php$
Disallow /*admin/
Disallow /bin/
Disallow /*build/
Disallow /*media/
Disallow /my/
Disallow /*checkout/
Disallow /*cron/
Disallow /*maintenance/
Disallow /*private/
Disallow /*scripts/
Disallow /*test/
Disallow /*testing/
Disallow /*tmp/
Disallow /*worker/
Disallow /*vendors/
Disallow /*includes/
Disallow /*setting/
Disallow /*action%3Dcheck_file_exist*
Disallow /*ajaxCall%3D1*

Other Records

Field Value
crawl-delay 1

baiduspider

Rule Path
Allow /
Disallow /es_MX/*
Disallow /de_DE/*
Disallow /zh_HK/*
Disallow /ja_JP/*
Disallow /ko_KR/*
Disallow /th_TH/*
Disallow /fr_FR/*
Disallow /id_ID/*

haosouspider

Rule Path
Allow /
Disallow /es_MX/*
Disallow /de_DE/*
Disallow /zh_HK/*
Disallow /ja_JP/*
Disallow /ko_KR/*
Disallow /th_TH/*
Disallow /fr_FR/*
Disallow /id_ID/*

sogou web spider

Rule Path
Allow /
Disallow /es_MX/*
Disallow /de_DE/*
Disallow /zh_HK/*
Disallow /ja_JP/*
Disallow /ko_KR/*
Disallow /th_TH/*
Disallow /fr_FR/*
Disallow /id_ID/*

sogou inst spider

Rule Path
Allow /
Disallow /es_MX/*
Disallow /de_DE/*
Disallow /zh_HK/*
Disallow /ja_JP/*
Disallow /ko_KR/*
Disallow /th_TH/*
Disallow /fr_FR/*
Disallow /id_ID/*

sogou spider2

Rule Path
Allow /
Disallow /es_MX/*
Disallow /de_DE/*
Disallow /zh_HK/*
Disallow /ja_JP/*
Disallow /ko_KR/*
Disallow /th_TH/*
Disallow /fr_FR/*
Disallow /id_ID/*

googlebot

Rule Path
Disallow

googlebot-image

Rule Path
Disallow

Other Records

Field Value
sitemap https://www.casetify.com/sitemap.xml

Comments

  • robotstxt.org/
  • Disallow: /*invite/
  • Disallow: /*showcase/
  • Disallow: /*tags[]=*
  • Disallow: /*sorting=*
  • Disallow: /*controllers*
  • Disallow: /*api-cache*
  • Disallow: /*layout_template*
  • Disallow: /*redirect*
  • Allow Google Ad Spiders