Log In Sign Up

handicappedpets.com
robots.txt

Robots Exclusion Standard data for handicappedpets.com

Resource Scan

Scan Details

Site Domain handicappedpets.com
Base Domain handicappedpets.com
Scan Status Ok
Last Scan2024-10-18T21:39:21+00:00
Next Scan 2024-11-17T21:39:21+00:00

Last Scan

Scanned2024-10-18T21:39:21+00:00
URL https://handicappedpets.com/robots.txt
Domain IPs 18.235.64.216
Response IP 18.235.64.216
Found Yes
Hash c65894075235bc267f1a6321e36ccb01940e5786d92bb7b1404728c720c192c0
SimHash 389c1f51cdfc

Groups

*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 60

*

Rule Path
Disallow /*/page/*
Disallow /wp-content/*
Disallow /*?src=isp_seo

googlebot

Rule Path
Allow /

rogerbot

Rule Path
Allow /

blog

Rule Path
Allow /

bingbot

Rule Path
Allow /

semrushbot

Rule Path
Allow /

facebookexternalhit

Rule Path
Allow /

googlebot-image

Rule Path
Allow /

yandexbot

Rule Path
Allow /

bingpreview

Rule Path
Allow /

baiduspider

Rule Path
Allow /

applebot

Rule Path
Allow /

firefox version 10 and lower - various robots

Rule Path
Allow /

screaming frog seo spider

Rule Path
Allow /

baidu ( catchall )

Rule Path
Allow /

linkcheck

Rule Path
Allow /

yandeximages

Rule Path
Allow /

yandexmobilebot

Rule Path
Allow /

adsbot-google

Rule Path
Allow /

dotbot

Rule Path
Allow /

feedfetcher-google

Rule Path
Allow /

tweetmemebot

Rule Path
Allow /

linkedinbot

Rule Path
Allow /

yahoo! slurp

Rule Path
Allow /

duckduckgo-favicons-bot

Rule Path
Allow /

googlebot-video

Rule Path
Allow /

google web preview

Rule Path
Allow /

google-adwords-instant

Rule Path
Allow /

msnbot

Rule Path
Allow /

hubspot crawler

Rule Path
Allow /
Disallow /*.tbd
Disallow /Articles/
Disallow /www/
Disallow /1/
Disallow /xmlrpc.php
Disallow /cron.php
Disallow /update.php
Disallow /install.*
Disallow /CHANGELOG.txt
Disallow /MAINTAINERS.txt
Disallow /LICENSE.txt
Disallow /UPGRADE.txt
Disallow /admin/
Disallow /aggregator/
Disallow /comment/reply/
Disallow /contact/
Disallow /logout/
Disallow /node/add/
Disallow /search/
Disallow /user/register/
Disallow /user/password/
Disallow /user/login/
Disallow /www/index.php/
Disallow /?q=admin%2F
Disallow /?q=aggregator%2F
Disallow /?q=comment%2Freply%2F
Disallow /?q=contact%2F
Disallow /?q=logout%2F
Disallow /?q=node%2Fadd%2F
Disallow /?q=search%2F
Disallow /?q=user%2Fpassword%2F
Disallow /?q=user%2Fregister%2F
Disallow /?q=user%2Flogin%2F
Disallow /2004/
Disallow /awstats/
Disallow /mediawiki/
Disallow /component/phocagallery/
Disallow /vets/
Disallow /gallery/
Disallow /xcart44/
Disallow /shoppingcart/
Disallow /k9brace/
Disallow /error400.html
Disallow /error403.html
Disallow /error404.html
Disallow /error500.html
Disallow /cgi_bin/
Disallow /orig/
Disallow /wp-content/plugins/*
Disallow /blog/wp-content/plugins/*
Disallow /*components*/
Disallow /*Itemid*/
Disallow /pet-care-articles/
Disallow *?replytocom
Disallow /allwizards/pages/
Disallow /cgi-bin/
Disallow /www
Back to top

Other Records

Field Value
sitemap https://www.handicappedpets.com/page-sitemap.xml
sitemap https://www.handicappedpets.com/product-sitemap.xml
sitemap https://www.handicappedpets.com/product_cat-sitemap.xml
sitemap https://woo.instantsearchplus.com/ext_sitemap?u=2a3988f9-90b8-4353-9961-cc62e8f390d6
Back to top

Comments

  • $Id: robots.txt,v 1.7.2.1 2007/03/23 18:57:07 drumm Exp $
  • robots.txt
  • This file is to prevent the crawling and indexing of certain parts
  • of your site by web crawlers and spiders run by sites like Yahoo!
  • and Google. By telling these robots where not to go on your site,
  • you save bandwidth and server resources.
  • This file will be ignored unless it is at the root of your host:
  • Used: http://example.com/robots.txt
  • Ignored: http://example.com/site/robots.txt
  • For more information about the robots.txt standard, see:
  • http://www.robotstxt.org/wc/robots.html
  • For syntax checking, see:
  • http://www.sxw.org.uk/computing/robots/check.html
  • sitemaps
  • Directories
  • Files
  • Paths (clean URLs)
  • Paths (no clean URLs)
  • Added by kate 8/2/16
  • Added by kate 8/9/16
  • For Blog funky urls 101916
  • 12/20/16 - kate
  • 08/14/17 - kate
  • START Instantsearch+
  • END Instantsearch+
Back to top