/.well-known/

Log In Sign Up

whatihavelearnedteaching.com
robots.txt

Robots Exclusion Standard data for whatihavelearnedteaching.com

Archived Snapshots

Resource Scan

Scan Details

Site Domain	whatihavelearnedteaching.com
Base Domain	whatihavelearnedteaching.com
Scan Status	Ok
Last Scan	2024-11-15T02:37:06+00:00
Next Scan	2024-11-22T02:37:06+00:00

Last Scan

Scanned	2024-11-15T02:37:06+00:00
URL	https://whatihavelearnedteaching.com/robots.txt
Domain IPs	104.18.37.69, 172.64.150.187, 2606:4700:4400::6812:2545, 2606:4700:4400::ac40:96bb
Response IP	104.18.37.69
Found	Yes
Hash	221c563ff4b2dfe7fa50666292a2ccabacec79f638b655a2e2c0e911d705d75a
SimHash	539e736ae3c3

Groups

chatgpt-user

Rule

Path

Disallow

a6-indexer

Rule

Path

Disallow

/

aboundex

Rule

Path

Disallow

/

asterias

Rule

Path

Disallow

/

backdoorbot/1.0

Rule

Path

Disallow

/

backlinkcrawler

Rule

Path

Disallow

/

black hole

Rule

Path

Disallow

/

blowfish/1.0

Rule

Path

Disallow

/

botalot

Rule

Path

Disallow

/

builtbottough

Rule

Path

Disallow

/

bullseye/1.0

Rule

Path

Disallow

/

bunnyslippers

Rule

Path

Disallow

/

ca-crawler

Rule

Path

Disallow

/

ccbot

Rule

Path

Disallow

/

ccbot/2.0

Rule

Path

Disallow

/

cegbfeieh

Rule

Path

Disallow

/

cheesebot

Rule

Path

Disallow

/

cherrypicker

Rule

Path

Disallow

/

cherrypickerelite/1.0

Rule

Path

Disallow

/

cherrypickerse/1.0

Rule

Path

Disallow

/

copyrightcheck

Rule

Path

Disallow

/

cosmos

Rule

Path

Disallow

/

crescent

Rule

Path

Disallow

/

crescent internet toolpak http ole control v.1.0

Rule

Path

Disallow

/

dbot

Rule

Path

Disallow

/

dittospyder

Rule

Path

Disallow

/

easouspider

Rule

Path

Disallow

/

eccp

Rule

Path

Disallow

/

emailcollector

Rule

Path

Disallow

/

emailsiphon

Rule

Path

Disallow

/

emailwolf

Rule

Path

Disallow

/

erocrawler

Rule

Path

Disallow

/

exabot/3.0

Rule

Path

Disallow

/

extractorpro

Rule

Path

Disallow

/

foobot

Rule

Path

Disallow

/

fr-crawler

Rule

Path

Disallow

/

friendly crawler

Rule

Path

Disallow

/

gigablastopensource

Rule

Path

Disallow

/

goodzer/2.0

Rule

Path

Disallow

/

grapeshotcrawler/2.0

Rule

Path

Disallow

/

harvest/1.5

Rule

Path

Disallow

/

heritrix/1.14.4

Rule

Path

Disallow

/

hloader

Rule

Path

Disallow

/

httplib

Rule

Path

Disallow

/

hubspot crawler 1.0

Rule

Path

Disallow

/

humanlinks

Rule

Path

Disallow

/

istellabot

Rule

Path

Disallow

/

infonavirobot

Rule

Path

Disallow

/

jennybot

Rule

Path

Disallow

/

kenjin spider

Rule

Path

Disallow

/

keyword density/0.9

Rule

Path

Disallow

/

konqueror/3.5

Rule

Path

Disallow

/

lexibot

Rule

Path

Disallow

/

libweb/clshttp

Rule

Path

Disallow

/

linkextractorpro

Rule

Path

Disallow

/

linkscan/8.1a unix

Rule

Path

Disallow

/

linkwalker

Rule

Path

Disallow

/

lipperhey seo service

Rule

Path

Disallow

/

lnspiderguy

Rule

Path

Disallow

/

lwp-trivial

Rule

Path

Disallow

/

lwp-trivial/1.34

Rule

Path

Disallow

/

mata hari

Rule

Path

Disallow

/

meanpathbot

Rule

Path

Disallow

/

microsoft url control - 5.01.4511

Rule

Path

Disallow

/

microsoft url control - 6.00.8169

Rule

Path

Disallow

/

miixpc

Rule

Path

Disallow

/

miixpc/4.2

Rule

Path

Disallow

/

mister pix

Rule

Path

Disallow

/

mister pix

Rule

Path

Disallow

/

mixrankbot

Rule

Path

Disallow

/

moget

Rule

Path

Disallow

/

moget/2.1

Rule

Path

Disallow

/

mozilla/4

Rule

Path

Disallow

/

mozilla/4.0 (compatible; bullseye; windows 95)

Rule

Path

Disallow

/

mozilla/4.0 (compatible; msie 4.0; windows 95)

Rule

Path

Disallow

/

mozilla/4.0 (compatible; msie 4.0; windows 98)

Rule

Path

Disallow

/

mozilla/4.0 (compatible; msie 4.0; windows nt)

Rule

Path

Disallow

/

mozilla/4.0 (compatible; msie 4.0; windows xp)

Rule

Path

Disallow

/

mozilla/4.0 (compatible; msie 4.0; windows 2000)

Rule

Path

Disallow

/

mozilla/4.0 (compatible; msie 4.0; windows me)

Rule

Path

Disallow

/

mozilla/5

Rule

Path

Disallow

/

nbot/2.0

Rule

Path

Disallow

/

netants

Rule

Path

Disallow

/

netzcheckbot/1.0

Rule

Path

Disallow

/

nicerspro

Rule

Path

Disallow

/

obot/2.3.1

Rule

Path

Disallow

/

offline explorer

Rule

Path

Disallow

/

openfind

Rule

Path

Disallow

/

openfind data gathere

Rule

Path

Disallow

/

pagesinventory

Rule

Path

Disallow

/

panscient.com

Rule

Path

Disallow

/

propowerbot/2.14

Rule

Path

Disallow

/

prowebwalker

Rule

Path

Disallow

/

queryn metasearch

Rule

Path

Disallow

/

repomonkey

Rule

Path

Disallow

/

repomonkey bait & tackle/v1.01

Rule

Path

Disallow

/

riddler

Rule

Path

Disallow

/

rma

Rule

Path

Disallow

/

ru_bot

Rule

Path

Disallow

/

screenerbot crawler beta 2.0

Rule

Path

Disallow

/

semrushbot/0.98~bl

Rule

Path

Disallow

/

seoengworldbot

Rule

Path

Disallow

/

seokicks-robot

Rule

Path

Disallow

/

seplinkbot

Rule

Path

Disallow

/

seplinkbot/1.0

Rule

Path

Disallow

/

seznambot

Rule

Path

Disallow

/

sistrix

Rule

Path

Disallow

/

sitesnagger

Rule

Path

Disallow

/

smtbot/1.0

Rule

Path

Disallow

/

spankbot

Rule

Path

Disallow

/

spbot

Rule

Path

Disallow

/

sogou web spider

Rule

Path

Disallow

/

spanner

Rule

Path

Disallow

/

suzuran

Rule

Path

Disallow

/

szukacz/1.4

Rule

Path

Disallow

/

teleport

Rule

Path

Disallow

/

teleportpro

Rule

Path

Disallow

/

telesoft

Rule

Path

Disallow

/

the intraformant

Rule

Path

Disallow

/

thenomad

Rule

Path

Disallow

/

tighttwatbot

Rule

Path

Disallow

/

titan

Rule

Path

Disallow

/

tocrawl/urldispatcher

Rule

Path

Disallow

/

true_robot

Rule

Path

Disallow

/

true_robot/1.0

Rule

Path

Disallow

/

turingos

Rule

Path

Disallow

/

urlappendbot

Rule

Path

Disallow

/

urly warning

Rule

Path

Disallow

/

vci

Rule

Path

Disallow

/

vci webviewer vci webviewer win32

Rule

Path

Disallow

/

voltron

Rule

Path

Disallow

/

wbsearchbot

Rule

Path

Disallow

/

web image collector

Rule

Path

Disallow

/

webauto

Rule

Path

Disallow

/

webbandit

Rule

Path

Disallow

/

webbandit/3.50

Rule

Path

Disallow

/

webcapture 2.0

Rule

Path

Disallow

/

webcheck 1.10.4

Rule

Path

Disallow

/

webcopierdanilo

Rule

Path

Disallow

/

c14542.sgvps.net webenhancer

Rule

Path

Disallow

/

webmasterworldforumbot

Rule

Path

Disallow

/

websauger

Rule

Path

Disallow

/

website quester

Rule

Path

Disallow

/

webster pro

Rule

Path

Disallow

/

webstripper

Rule

Path

Disallow

/

webzip

Rule

Path

Disallow

/

webzip/4.0

Rule

Path

Disallow

/

wesee

Rule

Path

Disallow

/

wget

Rule

Path

Disallow

/

wget/1.5.3

Rule

Path

Disallow

/

wget/1.6

Rule

Path

Disallow

/

www-collector-e

Rule

Path

Disallow

/

www.integromedb.org/crawler

Rule

Path

Disallow

/

xenu link sleuth

Rule

Path

Disallow

/

xenu's link sleuth 1.1c

Rule

Path

Disallow

/

zeus

Rule

Path

Disallow

/

zeus 32297 webster pro v2.9 win32

Rule

Path

Disallow

/

xovibot

Rule

Path

Disallow

/Danilo

zumbot

Rule

Path

Disallow

/

Disallow

/cgi-bin/

ahrefsbot

No rules defined. All paths allowed.

baiduspider
baiduspider
baiduspider+

Rule

Path

Disallow

/

sch-fast-se-crawl02.osl.basefarm.net

Rule

Path

Disallow

/

sch-fast-se-crawl04.osl.basefarm.net

Rule

Path

Disallow

/

moget
ichiro

Rule

Path

Disallow

/

naverbot
yeti

Rule

Path

Disallow

/

baiduspider
baiduspider-video
baiduspider-image

Rule

Path

Disallow

/

sogou spider

Rule

Path

Disallow

/

youdaobot

Rule

Path

Disallow

/

mj12bot

Rule

Path

Disallow

/

*

Rule

Path

Disallow

*/page/*

Disallow

*/?cat=*

Disallow

*/author%3D*

Disallow

*/?orderby=*

Back to top

Comments

Begin Exclusion From Directories from robots.txt
disallow OSL.basefarm.net

Back to top

Warnings

3 invalid lines.

Back to top