Log In Sign Up

foagroup.com
robots.txt

Robots Exclusion Standard data for foagroup.com

Resource Scan

Scan Details

Site Domain foagroup.com
Base Domain foagroup.com
Scan Status Ok
Last Scan2024-09-22T15:08:09+00:00
Next Scan 2024-10-22T15:08:09+00:00

Last Scan

Scanned2024-09-22T15:08:09+00:00
URL https://foagroup.com/robots.txt
Domain IPs 3.224.193.52, 34.205.237.7
Response IP 3.224.193.52
Found Yes
Hash f03bfe1288c9a00843fbd68d3a24bf85604ca1bfe7b3983f2e4e195f92704ac0
SimHash bd13b85143f3

Groups

amazonbot

Product Comment
amazonbot Amazon's user agent
Rule Path Comment
Disallow / disallow this directory

blexbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 250

adsbot-google
adsbot-google-mobile
adsbot-google-mobile-apps
applebot
applenewsbot
duckduckbot
duckduckgo-favicons-bot
facebot
googlebot
googlebot-image
googlebot-mobile
googlebot-news
googlebot-video
mediapartners-google
msnbot
msnbot-media
teoma
twitterbot

Rule Path
Disallow

*

Rule Path Comment
Disallow / -
Allow /*?p= -
Allow /index.php/blog/ -
Allow /catalog/seo_sitemap/category/ -
Allow /media/catalog/ -
Disallow /404/ -
Disallow /app/ -
Disallow /cgi-bin/ -
Disallow /downloader/ -
Disallow /errors/ -
Disallow /includes/ -
Disallow /lib/ -
Disallow /magento/ -
Disallow /software/ -
Disallow /ver13/ -
Disallow /media/captcha/ -
Disallow /media/css_secure/ -
Disallow /media/customer/ -
Disallow /media/dhl/ -
Disallow /media/downloadable/ -
Disallow /media/import/ -
Disallow /media/pdf/ -
Disallow /media/sales/ -
Disallow /media/tmp/ -
Disallow /media/wysiwyg/ -
Disallow /media/xmlconnect/ -
Disallow /pkginfo/ -
Disallow /report/ -
Disallow /scripts/ -
Disallow /shell/ -
Disallow /stats/ -
Disallow /var/ -
Disallow /index.php/ -
Disallow /catalog/product_compare/ -
Disallow /catalog/category/view/ -
Disallow /catalog/product/view/ -
Disallow /catalog/product/gallery/ -
Disallow /catalogsearch/ -
Disallow /checkout/ -
Disallow /control/ -
Disallow /contacts/ -
Disallow /customer/ -
Disallow /customize/ -
Disallow /newsletter/ -
Disallow /poll/ -
Disallow /review/ -
Disallow /sendfriend/ -
Disallow /tag/ -
Disallow /wishlist/ -
Disallow /ajax/ -
Disallow /cron.php -
Disallow /cron.sh -
Disallow /error_log -
Disallow /install.php -
Disallow /LICENSE.html -
Disallow /LICENSE.txt -
Disallow /LICENSE_AFL.txt -
Disallow /STATUS.txt -
Disallow /get.php Magento 1.5+
Disallow /*?p=* -
Disallow /*%26p%3D* -
Disallow /*?price=* -
Disallow /*%26price%3D* -
Disallow /*?color=* -
Disallow /*%26color%3D* -
Disallow /*?limit=* -
Disallow /*%26limit%3D* -
Disallow /*?mode=* -
Disallow /*%26mode%3D* -
Disallow /*?order=* -
Disallow /*%26order%3D* -
Disallow /*?dir=* -
Disallow /*%26dir%3D* -
Disallow /*?manufacturer=* -
Disallow /*%26manufacturer%3D* -
Disallow /*.php$ -
Disallow /*?SID= -
Disallow /?___from_store= -
Disallow /*___from_store%3D -
Disallow /rss* -
Disallow /*PHPSESSID -
Disallow /captcha/refresh -
Back to top

Comments

  • google.com landing page quality checks
  • google.com app resource fetcher
  • apple.com search engine
  • bing.com international search engine
  • User-agent: bingbot
  • User-agent: BingPreview
  • duckduckgo.com international privacy search engine
  • facebook.com social network
  • google.com international search engine
  • google.com adsense bot
  • bing.com international search engine
  • ask.com international search engine
  • twitter.com bot
  • crawling rule(s) for above bots
  • disallow all other bots
  • Allow is not an official standard
  • uncomment if you use Google CSE
  • Allow: /catalogsearch/result/
  • Directories
  • Disallow: /js/
  • Disallow: /media/
  • Disallow: /media/catalog/
  • Disallow: /media/css/
  • Disallow: /media/js/
  • Disallow: /skin/
  • Paths (clean URLs)
  • Files
  • Paths (no clean URLs)
  • Disallow: /*.js$
  • Disallow: /*.css$
Back to top