Log In Sign Up

peterjanes.ca
robots.txt

Robots Exclusion Standard data for peterjanes.ca

Resource Scan

Scan Details

Site Domain peterjanes.ca
Base Domain peterjanes.ca
Scan Status Ok
Last Scan2025-10-04T02:11:37+00:00
Next Scan 2025-11-03T02:11:37+00:00

Last Scan

Scanned2025-10-04T02:11:37+00:00
URL https://peterjanes.ca/robots.txt
Domain IPs 69.163.177.24
Response IP 69.163.177.24
Found Yes
Hash 7a438b42815613bfd4b64dc5ae44944e53fcb92390bf935554a244ecfd4c1fcb
SimHash 3edb59198771

Groups

googlebot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

googleother

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

charlotte

Rule Path
Disallow /

uw_cse_xwc

Rule Path
Disallow /

discobot

Rule Path
Disallow /

vadixbot

Rule Path
Disallow /

msrbot

Rule Path
Disallow /

twiceler

Rule Path
Disallow /

webalta

Rule Path
Disallow /

ilial

Rule Path
Disallow /

snapbot

Rule Path
Disallow /

ichiro

Rule Path
Disallow /

snapbot

Rule Path
Disallow /

dataspear

Rule Path
Disallow /

rufusbot

Rule Path
Disallow /

heritrix

Rule Path
Disallow /

converacrawler

Rule Path
Disallow /

gaisbot

Rule Path
Disallow /

deepindex

Rule Path
Disallow /

boitho.com-robot

Rule Path
Disallow /

lachesis

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

augurnfind

Rule Path
Disallow /

boitho

Rule Path
Disallow /

asterias

Rule Path
Disallow /

girafa

Rule Path
Disallow /

timbobot

Rule Path
Disallow /

sitecheck.internetseer.com

Rule Path
Disallow /

blogpulse

Rule Path
Disallow /

feedbucket

Rule Path
Disallow /

npbot

Rule Path
Disallow /

nutchcvs

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

naverrobot

Rule Path
Disallow /

msnbot

Rule Path
Disallow /

msiecrawler

Rule Path
Disallow /

microsoftprototypecrawler

Rule Path
Disallow /

http://www.almaden.ibm.com/cs/crawler

Rule Path
Disallow /

quepasacreep

Rule Path
Disallow /

rika

Rule Path
Disallow /

flickbot

Rule Path
Disallow /

voilabot

Rule Path
Disallow /

intelliseek

Rule Path
Disallow /

webzip

Rule Path
Disallow /

larbin

Rule Path
Disallow /

openfind

Rule Path
Disallow /

openbot

Rule Path
Disallow /

psbot

Rule Path
Disallow /

zyborg

Rule Path
Disallow /

mercator

Rule Path
Disallow /

vscooter

Rule Path
Disallow /

liberate

Rule Path
Disallow /

lmspider

Rule Path
Disallow /

vagabondo

Rule Path
Disallow /

zao

Rule Path
Disallow /

googlebot

Rule Path
Disallow /bot-redirect
Disallow /xhtml-redirect
Disallow /%28none%29
Disallow /~peterj/CBP/
Disallow /~peterj/CBParchive/
Disallow /CBP/
Disallow /CBParchive/
Disallow /mt/
Disallow /blog/comment
Disallow /blog/trackback
Disallow /blog/trackback/
Disallow /blog/search
Disallow /blog/page/
Disallow /blog/tag/
Disallow /blog/wp-admin/
Disallow /stats/
Disallow /blog/archives/categories/
Disallow /blog/archives/category/
Disallow /cgi-bin
Disallow /cgi-bin/
Disallow /cgi-bin/show
Disallow /dav/
Disallow /id/
Disallow /mp3/
Disallow /include/
Disallow /LenniFan/
Disallow /blog/2002/04/22/lenni-jabour-cest-what/
Disallow /styles/
Disallow /~peterj/comics/
Disallow /~peterj/personal
Disallow /~peterj/pyblagg
Disallow /~peterj/spycyroll/
Disallow /~peterj/newSong/
Disallow /~peterj/WhoIsThat/
Disallow /peterj/comics/
Disallow /peterj/personal
Disallow /peterj/pyblagg
Disallow /peterj/spycyroll/
Disallow /peterj/newSong/
Disallow /peterj/WhoIsThat/
Disallow /personal
Disallow /personal/
Disallow /newblog/
Disallow /family/images/
Disallow /family/media/
Disallow /family/themes/
Disallow /family/timeline.php
Disallow /family/calendar.php
Disallow /family/login.php
Disallow /family/reportengine.php
Disallow /family/search.php
Disallow /family/fanchart.php
Disallow /family/clippings.php
Disallow /family/sosabook.php
Disallow /family/aliveinyear.php
Disallow /family/ancestry.php
Disallow /family/descendancy.php
Disallow /family/famlist.php
Disallow /family/hourglass.php
Disallow /family/indilist.php
Disallow /family/patriarchlist.php
Disallow /family/pedigree.php
Disallow /family/placelist.php
Disallow /post
Disallow /blog/post
Disallow /resume
Disallow /resume.xhtml
Disallow /u/

slurp

Rule Path
Disallow /bot-redirect
Disallow /xhtml-redirect
Disallow /%28none%29
Disallow /~peterj/CBP/
Disallow /~peterj/CBParchive/
Disallow /CBP/
Disallow /CBParchive/
Disallow /mt/
Disallow /stats/
Disallow /blog/archives/categories/
Disallow /blog/archives/category/
Disallow /blog/category/
Disallow /blog/comment
Disallow /blog/trackback
Disallow /blog/trackback/
Disallow /blog/search
Disallow /blog/page/
Disallow /blog/tag/
Disallow /blog/wp-admin/
Disallow /blog/2007/12/28/upcoming-karla/feed/
Disallow /cgi-bin
Disallow /cgi-bin/
Disallow /cgi-bin/show
Disallow /dav/
Disallow /id/
Disallow /mp3/
Disallow /include/
Disallow /LenniFan/
Disallow /blog/2002/04/22/lenni-jabour-cest-what/
Disallow /styles/
Disallow /~peterj/comics/
Disallow /~peterj/personal
Disallow /~peterj/pyblagg
Disallow /~peterj/spycyroll/
Disallow /~peterj/newSong/
Disallow /~peterj/WhoIsThat/
Disallow /peterj/comics/
Disallow /peterj/personal
Disallow /peterj/pyblagg
Disallow /peterj/spycyroll/
Disallow /peterj/newSong/
Disallow /peterj/WhoIsThat/
Disallow /personal
Disallow /personal/
Disallow /newblog/
Disallow /family/images/
Disallow /family/media/
Disallow /family/themes/
Disallow /family/timeline.php
Disallow /family/calendar.php
Disallow /family/login.php
Disallow /family/reportengine.php
Disallow /family/search.php
Disallow /family/fanchart.php
Disallow /family/clippings.php
Disallow /family/sosabook.php
Disallow /family/aliveinyear.php
Disallow /family/ancestry.php
Disallow /family/descendancy.php
Disallow /family/family.php
Disallow /family/famlist.php
Disallow /family/hourglass.php
Disallow /family/indilist.php
Disallow /family/patriarchlist.php
Disallow /family/pedigree.php
Disallow /family/placelist.php
Disallow /post
Disallow /blog/post
Disallow /resume
Disallow /resume.xhtml
Disallow /u/

Other Records

Field Value
crawl-delay 30

*

Rule Path
Disallow /bot-redirect
Disallow /xhtml-redirect
Disallow /%28none%29
Disallow /mt/
Disallow /stats/
Disallow /blog/archives/categories/
Disallow /blog/archives/category/
Disallow /blog/category/
Disallow /blog/comment
Disallow /blog/trackback
Disallow /blog/trackback/
Disallow /blog/search
Disallow /blog/page/
Disallow /blog/tag/
Disallow /blog/wp-admin/
Disallow /cgi-bin
Disallow /cgi-bin/
Disallow /cgi-bin/show
Disallow /dav/
Disallow /id/
Disallow /mp3/
Disallow /include/
Disallow /LenniFan/
Disallow /blog/2002/04/22/lenni-jabour-cest-what/
Disallow /styles/
Disallow /~peterj/comics/
Disallow /~peterj/personal
Disallow /~peterj/pyblagg
Disallow /~peterj/spycyroll/
Disallow /~peterj/newSong/
Disallow /~peterj/WhoIsThat/
Disallow /peterj/comics/
Disallow /peterj/personal
Disallow /peterj/pyblagg
Disallow /peterj/spycyroll/
Disallow /peterj/newSong/
Disallow /peterj/WhoIsThat/
Disallow /personal
Disallow /personal/
Disallow /newblog/
Disallow /family/images/
Disallow /family/media/
Disallow /family/themes/
Disallow /family/timeline.php
Disallow /family/calendar.php
Disallow /family/login.php
Disallow /family/reportengine.php
Disallow /family/search.php
Disallow /family/fanchart.php
Disallow /family/clippings.php
Disallow /family/sosabook.php
Disallow /family/aliveinyear.php
Disallow /family/ancestry.php
Disallow /family/descendancy.php
Disallow /family/family.php
Disallow /family/famlist.php
Disallow /family/hourglass.php
Disallow /family/indilist.php
Disallow /family/patriarchlist.php
Disallow /family/pedigree.php
Disallow /family/placelist.php
Disallow /post
Disallow /blog/post
Disallow /resume
Disallow /resume.xhtml
Disallow /u/
Back to top

Comments

  • robots.txt for http://peterjanes.ca/
  • https://gizmodo.com/google-says-itll-scrape-everything-you-post-online-for-1850601486
  • I'll take my search results ranking based on content, not a "boost", thanks.
  • Lots of fast requests
  • Doesn't understand mixed case or special characters
  • divx.com media search engine
  • The next three are AltaVista spiders. After 800 requests they still only
  • have 2 pages listed in their index. What a waste of bandwidth... goodbye!
  • User-agent: Scooter
  • Disallow: /
  • Experimental AV spider
  • ???
  • If a user-agent matches, the * rule is no longer in effect
  • Limit requests by Slurp
  • http://help.yahoo.com/help/us/ysearch/slurp/slurp-03.html
  • Anything that doesn't match a user-agent above
Back to top