myfirstguide.com
robots.txt

Robots Exclusion Standard data for myfirstguide.com

Archived Snapshots

Resource Scan

Scan Details

Site Domain	myfirstguide.com
Base Domain	myfirstguide.com
Scan Status	Ok
Last Scan	2025-10-17T11:13:36+00:00
Next Scan	2025-10-31T11:13:36+00:00

Last Scan

Scanned	2025-10-17T11:13:36+00:00
URL	https://myfirstguide.com/robots.txt
Redirect	https://www.myfirstguide.com/robots.txt
Redirect Domain	www.myfirstguide.com
Redirect Base	myfirstguide.com
Domain IPs	104.21.39.237, 172.67.171.216, 2606:4700:3033::ac43:abd8, 2606:4700:3037::6815:27ed
Redirect IPs	104.21.39.237, 172.67.171.216, 2606:4700:3033::ac43:abd8, 2606:4700:3037::6815:27ed
Response IP	172.67.171.216
Found	Yes
Hash	1c69f65193acda462373cee42b6ba204b31308fc16fcaa874f011fcdfefa522d
SimHash	412552e12de7

Groups

*

Rule	Path
Allow	/.well-known/
Disallow	/v/login
Disallow	/v/cs/notice/delivery
Disallow	/v/license
Disallow	/mer/storage/json

Rule

Path

Allow

/.well-known/

Disallow

/v/login

Disallow

/v/cs/notice/delivery

Disallow

/v/license

Disallow

/mer/storage/json

baiduspider

Rule	Path
Disallow	/

Rule

Path

Disallow

yandex

Rule	Path
Disallow	/

Rule

Path

Disallow

vocusbot

Rule	Path
Disallow	/

Rule

Path

Disallow

alexibot

Rule	Path
Disallow	/

Rule

Path

Disallow

aqua_products

Rule	Path
Disallow	/

Rule

Path

Disallow

backdoorbot

Rule	Path
Disallow	/

Rule

Path

Disallow

backdoorbot/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

black.hole

Rule	Path
Disallow	/

Rule

Path

Disallow

blackwidow

Rule	Path
Disallow	/

Rule

Path

Disallow

blowfish

Rule	Path
Disallow	/

Rule

Path

Disallow

blowfish/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

botalot

Rule	Path
Disallow	/

Rule

Path

Disallow

botrighthere

Rule	Path
Disallow	/

Rule

Path

Disallow

builtbottough

Rule	Path
Disallow	/

Rule

Path

Disallow

bullseye

Rule	Path
Disallow	/

Rule

Path

Disallow

bullseye/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

bunnyslippers

Rule	Path
Disallow	/

Rule

Path

Disallow

cegbfeieh

Rule	Path
Disallow	/

Rule

Path

Disallow

cheesebot

Rule	Path
Disallow	/

Rule

Path

Disallow

cherrypicker

Rule	Path
Disallow	/

Rule

Path

Disallow

cherrypickerelite/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

cherrypickerse/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

chinaclaw

Rule	Path
Disallow	/

Rule

Path

Disallow

copernic

Rule	Path
Disallow	/

Rule

Path

Disallow

copyrightcheck

Rule	Path
Disallow	/

Rule

Path

Disallow

crescent

Rule	Path
Disallow	/

Rule

Path

Disallow

custo

Rule	Path
Disallow	/

Rule

Path

Disallow

disco

Rule	Path
Disallow	/

Rule

Path

Disallow

discofinder

Rule	Path
Disallow	/

Rule

Path

Disallow

dittospyder

Rule	Path
Disallow	/

Rule

Path

Disallow

eirgrabber

Rule	Path
Disallow	/

Rule

Path

Disallow

emailcollector

Rule	Path
Disallow	/

Rule

Path

Disallow

emailsiphon

Rule	Path
Disallow	/

Rule

Path

Disallow

emailwolf

Rule	Path
Disallow	/

Rule

Path

Disallow

erocrawler

Rule	Path
Disallow	/

Rule

Path

Disallow

extractorpro

Rule	Path
Disallow	/

Rule

Path

Disallow

eyenetie

Rule	Path
Disallow	/

Rule

Path

Disallow

flashget

Rule	Path
Disallow	/

Rule

Path

Disallow

foobot

Rule	Path
Disallow	/

Rule

Path

Disallow

frontpage

Rule	Path
Disallow	/

Rule

Path

Disallow

gaisbot

Rule	Path
Disallow	/

Rule

Path

Disallow

getright

Rule	Path
Disallow	/

Rule

Path

Disallow

getright/2.11

Rule	Path
Disallow	/

Rule

Path

Disallow

getright/3.1

Rule	Path
Disallow	/

Rule

Path

Disallow

getright/3.2

Rule	Path
Disallow	/

Rule

Path

Disallow

getright/3.3

Rule	Path
Disallow	/

Rule

Path

Disallow

getright/3.3.3

Rule	Path
Disallow	/

Rule

Path

Disallow

getright/3.3.4

Rule

Path

Disallow

getright/4.0.0

Rule

Path

Disallow

getright/4.1.0

Rule

Path

Disallow

getright/4.1.1

Rule

Path

Disallow

getright/4.1.2

Rule

Path

Disallow

getright/4.2

Rule

Path

Disallow

getright/4.2c

Rule

Path

Disallow

getright/4.3

Rule

Path

Disallow

getright/4.5

Rule

Path

Disallow

getright/4.5a

Rule

Path

Disallow

getright/4.5b

Rule

Path

Disallow

getright/4.5b1

Rule

Path

Disallow

getright/4.5b2

Rule

Path

Disallow

getright/4.5b3

Rule

Path

Disallow

getright/4.5b6

Rule

Path

Disallow

getright/4.5b7

Rule

Path

Disallow

getright/4.5c

Rule

Path

Disallow

getright/4.5d

Rule

Path

Disallow

getright/4.5e

Rule

Path

Disallow

getright/5.0beta1

Rule

Path

Disallow

getright/5.0beta2

Rule

Path

Disallow

go-ahead-got-it

Rule

Path

Disallow

grabnet

Rule

Path

Disallow

grafula

Rule

Path

Disallow

hmview

Rule

Path

Disallow

httrack

Rule

Path

Disallow

harvest

Rule

Path

Disallow

harvest/1.5

Rule

Path

Disallow

infonavirobot

Rule

Path

Disallow

interget

Rule

Path

Disallow

iron33/1.0.2

Rule

Path

Disallow

jennybot

Rule

Path

Disallow

jetcar

Rule

Path

Disallow

keyword.density

Rule

Path

Disallow

lnspiderguy

Rule

Path

Disallow

leechftp

Rule

Path

Disallow

lexibot

Rule

Path

Disallow

linkscan/8.1a.unix

Rule

Path

Disallow

linkwalker

Rule

Path

Disallow

linkextractorpro

Rule

Path

Disallow

miixpc

Rule

Path

Disallow

miixpc/4.2

Rule

Path

Disallow

msiecrawler

Rule

Path

Disallow

mj12bot

Rule

Path

Disallow

mj12bot/v1.4.0

Rule

Path

Disallow

mj12bot/v1.2.4

Rule

Path

Disallow

mj12bot/v1.2.3

Rule

Path

Disallow

mj12bot/v1.0.8

Rule

Path

Disallow

mj12bot/v1.0.7

Rule

Path

Disallow

mj12bot/v1.0.6

Rule

Path

Disallow

mj12bot/v1.0.5

Rule

Path

Disallow

nicerspro

Rule

Path

Disallow

npbot

Rule

Path

Disallow

npbot

Rule

Path

Disallow

navroad

Rule

Path

Disallow

nearsite

Rule

Path

Disallow

netants

Rule

Path

Disallow

netants/1.10

Rule

Path

Disallow

netants/1.23

Rule

Path

Disallow

netants/1.24

Rule

Path

Disallow

netants/1.25

Rule

Path

Disallow

netmechanic

Rule

Path

Disallow

netspider

Rule

Path

Disallow

octopus

Rule

Path

Disallow

offline.explorer

Rule

Path

Disallow

openbot

Rule

Path

Disallow

pagegrabber

Rule

Path

Disallow

perman

Rule

Path

Disallow

propowerbot/2.14

Rule

Path

Disallow

prowebwalker

Rule

Path

Disallow

python-urllib

Rule

Path

Disallow

rma

Rule

Path

Disallow

reget

Rule

Path

Disallow

realdownload

Rule

Path

Disallow

realdownload/4.0.0.40

Rule

Path

Disallow

realdownload/4.0.0.41

Rule

Path

Disallow

realdownload/4.0.0.42

Rule

Path

Disallow

sitesnagger

Rule

Path

Disallow

slysearch

Rule

Path

Disallow

smartdownload

Rule

Path

Disallow

spankbot

Rule

Path

Disallow

superbot

Rule

Path

Disallow

superhttp

Rule

Path

Disallow

superhttp/1.0

Rule

Path

Disallow

surfbot

Rule

Path

Disallow

szukacz/1.4

Rule

Path

Disallow

teleport

Rule

Path

Disallow

teleportpro

Rule

Path

Disallow

telesoft

Rule

Path

Disallow

the.intraformant

Rule

Path

Disallow

thenomad

Rule

Path

Disallow

tighttwatbot

Rule

Path

Disallow

titan

Rule

Path

Disallow

true_robot

Rule

Path

Disallow

true_robot/1.0

Rule

Path

Disallow

turnitinbot

Rule

Path

Disallow

turnitinbot/2.1

Rule

Path

Disallow

turnitinbot/2.0

Rule

Path

Disallow

turnitinbot/1.5

Rule

Path

Disallow

url_spider_pro

Rule

Path

Disallow

urly.warning

Rule

Path

Disallow

vci

Rule

Path

Disallow

voideye

Rule

Path

Disallow

www-collector-e

Rule

Path

Disallow

wwwoffle

Rule

Path

Disallow

web.image.collector

Rule

Path

Disallow

webauto

Rule

Path

Disallow

webbandit

Rule

Path

Disallow

webbandit/3.50

Rule

Path

Disallow

webcopier

Rule

Path

Disallow

webenhancer

Rule

Path

Disallow

webfetch

Rule

Path

Disallow

webleacher

Rule

Path

Disallow

webreaper

Rule

Path

Disallow

websauger

Rule

Path

Disallow

webstripper

Rule

Path

Disallow

webstripper/2.03

Rule

Path

Disallow

webstripper/2.10

Rule

Path

Disallow

webstripper/2.12

Rule

Path

Disallow

webstripper/2.13

Rule

Path

Disallow

webstripper/2.15

Rule

Path

Disallow

webstripper/2.16

Rule

Path

Disallow

webstripper/2.19

Rule

Path

Disallow

webwhacker

Rule

Path

Disallow

webzip

Rule

Path

Disallow

webzip/4.0

Rule

Path

Disallow

webmasterworldforumbot

Rule

Path

Disallow

website.quester

Rule

Path

Disallow

webster.pro

Rule

Path

Disallow

wget

Rule

Path

Disallow

wget/1.5.2

Rule

Path

Disallow

wget/1.5.3

Rule

Path

Disallow

wget/1.6

Rule

Path

Disallow

wget/1.7

Rule

Path

Disallow

wget/1.8

Rule

Path

Disallow

wget/1.8.1

Rule

Path

Disallow

wget/1.8.2

Rule

Path

Disallow

wget/1.9-beta

Rule

Path

Disallow

widow

Rule

Path

Disallow

asterias

Rule

Path

Disallow

b2w/0.1

Rule

Path

Disallow

cosmos

Rule

Path

Disallow

ecatch

Rule

Path

Disallow

ecatch/3.0

Rule

Path

Disallow

hloader

Rule

Path

Disallow

httplib

Rule

Path

Disallow

humanlinks

Rule

Path

Disallow

ia_archiver

Rule

Path

Disallow

larbin

Rule

Path

Disallow

libweb/clshttp

Rule

Path

Disallow

lwp-trivial

Rule

Path

Disallow

lwp-trivial/1.34

Rule

Path

Disallow

moget

Rule

Path

Disallow

moget/2.1

Rule

Path

Disallow

pavuk

Rule

Path

Disallow

pcbrowser

Rule

Path

Disallow

psbot

Rule

Path

Disallow

searchpreview

Rule

Path

Disallow

spanner

Rule

Path

Disallow

suzuran

Rule

Path

Disallow

takeout

Rule

Path

Disallow

tocrawl/urldispatcher

Rule

Path

Disallow

turingos

Rule

Path

Disallow

webfetch/2.1.0

Rule

Path

Disallow

semrushbot

Rule

Path

Disallow

ahrefs

Rule

Path

Disallow

Other Records

Field

Value

sitemap

https://www.myfirstguide.com/doyacimpl/sitemap/sitemap.xml

Comments

Disallow Bad Bots

myfirstguide.comrobots.txt

Resource Scan

Scan Details

Last Scan

Groups

*

baiduspider

yandex

vocusbot

alexibot

aqua_products

backdoorbot

backdoorbot/1.0

black.hole

blackwidow

blowfish

blowfish/1.0

botalot

botrighthere

builtbottough

bullseye

bullseye/1.0

bunnyslippers

cegbfeieh

cheesebot

cherrypicker

cherrypickerelite/1.0

cherrypickerse/1.0

chinaclaw

copernic

copyrightcheck

crescent

custo

disco

discofinder

dittospyder

eirgrabber

emailcollector

emailsiphon

emailwolf

erocrawler

extractorpro

eyenetie

flashget

foobot

frontpage

gaisbot

getright

getright/2.11

getright/3.1

getright/3.2

getright/3.3

getright/3.3.3

getright/3.3.4

getright/4.0.0

getright/4.1.0

getright/4.1.1

getright/4.1.2

getright/4.2

getright/4.2c

getright/4.3

getright/4.5

getright/4.5a

getright/4.5b

getright/4.5b1

getright/4.5b2

getright/4.5b3

getright/4.5b6

getright/4.5b7

getright/4.5c

getright/4.5d

getright/4.5e

getright/5.0beta1

getright/5.0beta2

go-ahead-got-it

grabnet

grafula

hmview

httrack

harvest

myfirstguide.com
robots.txt