westegg.com
robots.txt

Robots Exclusion Standard data for westegg.com

Archived Snapshots

Resource Scan

Scan Details

Site Domain	westegg.com
Base Domain	westegg.com
Scan Status	Ok
Last Scan	2024-10-10T18:29:54+00:00
Next Scan	2024-10-17T18:29:54+00:00

Last Scan

Scanned	2024-10-10T18:29:54+00:00
URL	https://westegg.com/robots.txt
Domain IPs	104.21.2.54, 172.67.128.204, 2606:4700:3031::6815:236, 2606:4700:3034::ac43:80cc
Response IP	172.67.128.204
Found	Yes
Hash	a4f23cd67ebbbf6a08c82053eb47dc53bd35f738aa7910af999b089450d1e10c
SimHash	cbbd65436080

Groups

xovibot

Rule	Path
Disallow	/

Rule

Path

Disallow

wotbox

Rule	Path
Disallow	/

Rule

Path

Disallow

wbsearchbot

Rule	Path
Disallow	/

Rule

Path

Disallow

unisterbot

Rule	Path
Disallow	/

Rule

Path

Disallow

turnitinbot

Rule	Path
Disallow	/

Rule

Path

Disallow

spbot

Rule	Path
Disallow	/

Rule

Path

Disallow

sogou spider

Rule	Path
Disallow	/

Rule

Path

Disallow

smtbot

Rule	Path
Disallow	/

Rule

Path

Disallow

siteexplorer

Rule	Path
Disallow	/

Rule

Path

Disallow

sistrix

Rule	Path
Disallow	/

Rule

Path

Disallow

seznambot

Rule	Path
Disallow	/

Rule

Path

Disallow

seokicks-robot

Rule	Path
Disallow	/

Rule

Path

Disallow

sentibot

Rule	Path
Disallow	/

Rule

Path

Disallow

semrushbot

Rule	Path
Disallow	/

Rule

Path

Disallow

searchmetricsbot

Rule	Path
Disallow	/

Rule

Path

Disallow

screenerbot

Rule	Path
Disallow	/

Rule

Path

Disallow

publiclibraryarchive.org

Rule	Path
Disallow	/

Rule

Path

Disallow

panscient.com

Rule	Path
Disallow	/

Rule

Path

Disallow

nutch

Rule	Path
Disallow	/

Rule

Path

Disallow

netestate ne crawler

Rule	Path
Disallow	/

Rule

Path

Disallow

mj12bot

Rule	Path
Disallow	/wp-includes/
Disallow	/wp-admin/

Rule

Path

Disallow

/wp-includes/

Disallow

/wp-admin/

Other Records

Field	Value
crawl-delay	10

Field

Value

crawl-delay

mixbot

Rule	Path
Disallow	/

Rule

Path

Disallow

memorybot

Rule	Path
Disallow	/

Rule

Path

Disallow

megaindex.ru/2.0

Rule	Path
Disallow	/

Rule

Path

Disallow

megaindex.ru

Rule	Path
Disallow	/

Rule

Path

Disallow

meanpathbot

Rule	Path
Disallow	/

Rule

Path

Disallow

mail.ru

Rule	Path
Disallow	/

Rule

Path

Disallow

lipperhey

Rule	Path
Disallow	/

Rule

Path

Disallow

linkpadbot

Rule	Path
Disallow	/

Rule

Path

Disallow

istellabot

Rule	Path
Disallow	/

Rule

Path

Disallow

heritrix

Rule	Path
Disallow	/

Rule

Path

Disallow

googlebot

Rule	Path
Disallow	/wp-admin/

Rule

Path

Disallow

/wp-admin/

feedbooster

Rule	Path
Disallow	/

Rule

Path

Disallow

ezooms

Rule	Path
Disallow	/

Rule

Path

Disallow

exb language crawler

Rule	Path
Disallow	/

Rule

Path

Disallow

easouspider

Rule	Path
Disallow	/

Rule

Path

Disallow

bubing

Rule	Path
Disallow	/

Rule

Path

Disallow

bpimagewalker/2.0

Rule	Path
Disallow	/

Rule

Path

Disallow

blexbot

Rule	Path
Disallow	/

Rule

Path

Disallow

advbot

Rule	Path
Disallow	/

Rule

Path

Disallow

aboundexbot

Rule	Path
Disallow	/

Rule

Path

Disallow

abonti

Rule	Path
Disallow	/

Rule

Path

Disallow

*

Rule	Path
Disallow	/wp-includes/
Disallow	/rss-feeds/
Disallow	/wp-admin/
Disallow	/cgi-bin/
Disallow	/tmp/
Allow	/wp-admin/admin-ajax.php

Rule

Path

Disallow

/wp-includes/

Disallow

/rss-feeds/

Disallow

/wp-admin/

Disallow

/cgi-bin/

Disallow

/tmp/

Allow

/wp-admin/admin-ajax.php

Other Records

Field	Value
crawl-delay	10

Field

Value

crawl-delay

westegg.comrobots.txt

Resource Scan

Scan Details

Last Scan

Groups

xovibot

wotbox

wbsearchbot

unisterbot

turnitinbot

spbot

sogou spider

smtbot

siteexplorer

sistrix

seznambot

seokicks-robot

sentibot

semrushbot

searchmetricsbot

screenerbot

publiclibraryarchive.org

panscient.com

nutch

netestate ne crawler

mj12bot

Other Records

mixbot

memorybot

megaindex.ru/2.0

megaindex.ru

meanpathbot

mail.ru

lipperhey

linkpadbot

istellabot

heritrix

googlebot

feedbooster

ezooms

exb language crawler

easouspider

bubing

bpimagewalker/2.0

blexbot

advbot

aboundexbot

abonti

*

Other Records

westegg.com
robots.txt