publishers.academy
robots.txt

Robots Exclusion Standard data for publishers.academy

Archived Snapshots

Resource Scan

Scan Details

Site Domain	publishers.academy
Base Domain	publishers.academy
Scan Status	Ok
Last Scan	2025-12-01T12:06:41+00:00
Next Scan	2025-12-31T12:06:41+00:00

Last Scan

Scanned	2025-12-01T12:06:41+00:00
URL	https://publishers.academy/robots.txt
Domain IPs	104.21.16.57, 172.67.210.34, 2606:4700:3032::6815:1039, 2606:4700:3035::ac43:d222
Response IP	172.67.210.34
Found	Yes
Hash	a18c4d237dae7a8324d8671bd7c1ad917f7f123241e561ee9f2b2c863fd8fea0
SimHash	fc85729136d4

Groups

mauibot

Rule	Path
Disallow	/

Rule

Path

Disallow

blexbot

Rule	Path
Disallow	/

Rule

Path

Disallow

seo spider

Rule	Path
Disallow	/

Rule

Path

Disallow

yandexbot

Rule	Path
Disallow	/

Rule

Path

Disallow

alexawebsearchplatform

Rule	Path
Disallow	/

Rule

Path

Disallow

baiduspider

Rule	Path
Disallow	/

Rule

Path

Disallow

betabot

Rule	Path
Disallow	/

Rule

Path

Disallow

petalbot

Rule	Path
Disallow	/

Rule

Path

Disallow

dotbot

Rule	Path
Disallow	/

Rule

Path

Disallow

screaming frog seo spider

Rule	Path
Disallow	/

Rule

Path

Disallow

crawl

Rule	Path
Disallow	/

Rule

Path

Disallow

exabot

Rule	Path
Disallow	/

Rule

Path

Disallow

gigabot

Rule	Path
Disallow	/

Rule

Path

Disallow

infoseek sidewinder

Rule	Path
Disallow	/

Rule

Path

Disallow

linkchecker

Rule	Path
Disallow	/

Rule

Path

Disallow

netsongbot

Rule	Path
Disallow	/

Rule

Path

Disallow

dataforseobot

Rule	Path
Disallow	/

Rule

Path

Disallow

dataforseo-bot

Rule	Path
Disallow	/

Rule

Path

Disallow

mj12bot/

Rule	Path
Disallow	/

Rule

Path

Disallow

linkdexbot

Rule	Path
Disallow	/

Rule

Path

Disallow

adsbot

Rule	Path
Disallow	/

Rule

Path

Disallow

awariosmartbot

Rule	Path
Disallow	/

Rule

Path

Disallow

feedspot

Rule	Path
Disallow	/

Rule

Path

Disallow

sogou web spider

Rule	Path
Disallow	/

Rule

Path

Disallow

feedspot/1.0

Rule	Path
Disallow	/

Rule

Path

Disallow

keybot

Rule	Path
Disallow	/

Rule

Path

Disallow

zoominfobot

Rule	Path
Disallow	/

Rule

Path

Disallow

feedspot

Rule	Path
Disallow	/

Rule

Path

Disallow

checkhost

Rule	Path
Disallow	/

Rule

Path

Disallow

orbbot

Rule	Path
Disallow	/

Rule

Path

Disallow

yandeximages

Rule	Path
Disallow	/

Rule

Path

Disallow

wellknownbot

Rule	Path
Disallow	/

Rule

Path

Disallow

surdotlybot

Rule	Path
Disallow	/

Rule

Path

Disallow

gdnplus.com

Rule	Path
Disallow	/

Rule

Path

Disallow

online-webceo-bot

Rule	Path
Disallow	/

Rule

Path

Disallow

scrapy

Rule	Path
Disallow	/

Rule

Path

Disallow

rogerbot

Rule	Path
Disallow	/

Rule

Path

Disallow

mj12bot

Rule	Path
Disallow	/

Rule

Path

Disallow

dotbot

Rule	Path
Disallow	/

Rule

Path

Disallow

alexibot

Rule	Path
Disallow	/

Rule

Path

Disallow

surveybot

Rule	Path
Disallow	/

Rule

Path

Disallow

xenu

Rule	Path
Disallow	/

Rule

Path

Disallow

exabot

Rule	Path
Disallow	/

Rule

Path

Disallow

gigabot

Rule	Path
Disallow	/

Rule

Path

Disallow

blekkobot

Rule	Path
Disallow	/

Rule

Path

Disallow

mecrawler

Rule	Path
Disallow	/

Rule

Path

Disallow

turnitinbot

Rule	Path
Disallow	/

Rule

Path

Disallow

python-requests

Rule	Path
Disallow	/

Rule

Path

Disallow

python-urllib/2.7

Rule

Path

Disallow

curl/7.35.0

Rule

Path

Disallow

wp_is_mobile

Rule

Path

Disallow

java/11.0.10

Rule

Path

Disallow

photon

Rule

Path

Disallow

photon/1.0

Rule

Path

Disallow

serendeputybot

Rule

Path

Disallow

webpage-inspector.com

Rule

Path

Disallow

twingly recon

Rule

Path

Disallow

bidtellect/0.0.958.0

Rule

Path

Disallow

slackbot-linkexpanding 1.0

Rule

Path

Disallow

df bot 1.0

Rule

Path

Disallow

who.is bot

Rule

Path

Disallow

sem rush bot

Rule

Path

Disallow

yandex bot

Rule

Path

Disallow

sogou bot

Rule

Path

Disallow

majestic bot

Rule

Path

Disallow

exalead bot

Rule

Path

Disallow

baidu bot

Rule

Path

Disallow

mediamathbot/1.0

Rule

Path

Disallow

trendictionbot0

Rule

Path

Disallow

omgili/0.5

Rule

Path

Disallow

iframely

Rule

Path

Disallow

cortex/1.0

Rule

Path

Disallow

trendictionbot0.5.0

Rule

Path

Disallow

domains project/1.3.7

Rule

Path

Disallow

trendictionbot

Rule

Path

Disallow

mediamathbot

Rule

Path

Disallow

omgili

Rule

Path

Disallow

universalfeedparser/5.2.1

Rule

Path

Disallow

woorankreview/2.0

Rule

Path

Disallow

feedbot/1.0

Rule

Path

Disallow

seekportbot

Rule

Path

Disallow

siteauditbot

Rule

Path

Disallow

daum/4.1

Rule

Path

Disallow

paperlibot/2.1

Rule

Path

Disallow

aboutusbot

Rule

Path

Disallow

neevabot/1.0

Rule

Path

Disallow

feedly/1.0

Rule

Path

Disallow

heritrix/3.3.0

Rule

Path

Disallow

wget

Rule

Path

Disallow

microsoft.url.control

Rule

Path

Disallow

ubicrawler

Rule

Path

Disallow

icc-crawler

Rule

Path

Disallow

sitecheck.internetseer.com

Rule

Path

Disallow

zealbot

Rule

Path

Disallow

webstripper

Rule

Path

Disallow

webcopier

Rule

Path

Disallow

httrack

Rule

Path

Disallow

libwww

Rule

Path

Disallow

baiduspider-image

Rule

Path

Disallow

k2spider

Rule

Path

Disallow

megaindex.ru

Rule

Path

Disallow

megaindex.com

Rule

Path

Disallow

linguee bot

Rule

Path

Disallow

mappy

Rule

Path

Disallow

garlikcrawler

Rule

Path

Disallow

feedbot

Rule

Path

Disallow

feedlyapp

Rule

Path

Disallow

feedly

Rule

Path

Disallow

wordpress

Rule

Path

Disallow

heritrix

Rule

Path

Disallow

fullstorybot

Rule

Path

Disallow

ias-va/3.1

Rule

Path

Disallow

xpanse-bot

Rule

Path

Disallow

phxbot/0.1

Rule

Path

Disallow

http://2ip.io

Rule

Path

Disallow

ahrefsbot

Rule

Path

Disallow

mojeebot

Rule

Path

Disallow

siteexplorer

Rule

Path

Disallow

uptimebot

Rule

Path

Disallow

screaming frog seo spider

Rule

Path

Disallow

phxbot

Rule

Path

Disallow

zgrab/0.x

Rule

Path

Disallow

wappalyzer

Rule

Path

Disallow

checkmarknetwork/1.0

Rule

Path

Disallow

duckduckgo-favicons-bot/1.0;

Rule

Path

Disallow

gptbot

Rule

Path

Disallow

chatgptbot

Rule

Path

Disallow

researchscan.comsys.rwth-aachen.de

Rule

Path

Disallow

awariobot/1.0

Rule

Path

Disallow

bitsightbot/1.0

Rule

Path

Disallow

internetmeasurement/1.0

Rule

Path

Disallow

duckduckgo-favicons-bot/1.0

Rule

Path

Disallow

yak/1.0

Rule

Path

Disallow

uptimerobot/2.0

Rule

Path

Disallow

awariorssbot/1.0

Rule

Path

Disallow

awariosmartbot/1.0

Rule

Path

Disallow

repolookoutbot/v1

Rule

Path

Disallow

netcraftsurveyagent/1.0

Rule

Path

Disallow

stractbot/0.1

Rule

Path

Disallow

yeti/1.1

Rule

Path

Disallow

mediatoolkitbot

Rule

Path

Disallow

serpstatbot/2.1

Rule

Path

Disallow

webwikibot/2.1

Rule

Path

Disallow

anderspinkbot/1.1

Rule

Path

Disallow

pandalytics/1.0

Rule

Path

Disallow

anthropic-ai

Rule

Path

Disallow

researchscan

Rule

Path

Disallow

hstspreload-bot

Rule

Path

Disallow

censysinspect/1.1

Rule

Path

Disallow

apache-httpclient/4.5.13

Rule

Path

Disallow

python/3.10 aiohttp/3.8.5

Rule

Path

Disallow

alittle client

Rule

Path

Disallow

dalvik/2.1.0

Rule

Path

Disallow

python/3.10

Rule

Path

Disallow

curl/7.29.0

Rule

Path

Disallow

crystal

Rule

Path

Disallow

ccbot

Rule

Path

Disallow

test-bot

Rule

Path

Disallow

startmebot/1.0

Rule

Path

Disallow

fidget-spinner-bot

Rule

Path

Disallow

guzzlehttp/7

Rule

Path

Disallow

claritybot

Rule

Path

Disallow

chatgpt-user

Rule

Path

Disallow

gptbot

Rule

Path

Disallow

spinn3r

Rule

Path

Disallow

yahoo pipes 1.0

Rule

Path

Disallow

kscrawler

Rule

Path

Disallow

bytespider

Rule

Path

Disallow

amazonbot

Rule

Path

Disallow

newsnow

Rule

Path

Disallow

moodlebot

Rule

Path

Disallow

schema-markup-validator

Rule

Path

Disallow

my-tiny-bot

Rule

Path

Disallow

thesis-research-bot

Rule

Path

Disallow

slack-imgproxy

Rule

Path

Disallow

cpanel-http-client/1.0

Rule

Path

Disallow

superfeedr bot/2.0

Rule

Path

Disallow

pine.blog/1.7

Rule

Path

Disallow

sitescorebot

Rule

Path

Disallow

apache-httpclient/unavailable

Rule

Path

Disallow

quora-bot/1.0

Rule

Path

Disallow

cpp-httplib/0.10.9

Rule

Path

Disallow

sitescorebot v20210315

Rule

Path

Disallow

anthill

Rule

Path

Disallow

amazonbot/0.1

Rule

Path

Disallow

feedbin

Rule

Path

Disallow

ev-crawler/1.0

Rule

Path

Disallow

python-requests/2.26.0

Rule

Path

Disallow

python-requests/2.31.0

Rule

Path

Disallow

python-requests/2.28.2

Rule

Path

Disallow

peach/1.01

Rule

Path

Disallow

feedburner/1.0

Rule

Path

Disallow

gametublmbot

Rule

Path

Disallow

imagesiftbot

Rule

Path

Disallow

velenpublicwebcrawler/1.0

Rule

Path

Disallow

archive.org_bot

Rule

Path

Disallow

threatview.app

Rule

Path

Disallow

gaisbot/3.0

Rule

Path

Disallow

bingbot/2.0

Rule

Path

Disallow

semrushbot/7~bl

Rule

Path

Disallow

semrushbot/7

Rule

Path

Disallow

semrushbot-ba

Rule

Path

Disallow

semrushbot

Rule

Path

Disallow

semrushbot

Rule

Path

Disallow

youbot

Rule

Path

Disallow

wpbot/1.0

Rule

Path

Disallow

seolyt/1.0

Rule

Path

Disallow

claudebot

Rule

Path

Disallow

curl/7.61.1

Rule

Path

Disallow

ddg_android/5.169.0

Rule

Path

Disallow

curl/7.54.0

Rule

Path

Disallow

axios/1.6.5

Rule

Path

Disallow

linux mozilla

Rule

Path

Disallow

go-http-client/2.0

Rule

Path

Disallow

got

Rule

Path

Disallow

mozilla/5.0

Rule

Path

Disallow

*

Rule

Path

Disallow

/admin/

Disallow

/auth/

Disallow

/assets/browser-update*.js

Disallow

/email/

Disallow

/session

Disallow

/user-api-key

Disallow

/*?api_key*

Disallow

/*?*api_key*

Disallow

/badges

Disallow

/my

Disallow

/search

Disallow

/tag/*/l

Disallow

/t/*/*.rss

Disallow

/c/*.rss

googlebot

Rule

Path

Disallow

/admin/

Disallow

/auth/

Disallow

/assets/browser-update*.js

Disallow

/email/

Disallow

/session

Disallow

/user-api-key

Disallow

/*?api_key*

Disallow

/*?*api_key*

Other Records

Field

Value

sitemap

https://publishers.academy/sitemap.xml

Comments

See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file

Warnings

8 invalid lines.

publishers.academyrobots.txt

Resource Scan

Scan Details

Last Scan

Groups

mauibot

blexbot

seo spider

yandexbot

alexawebsearchplatform

baiduspider

betabot

petalbot

dotbot

screaming frog seo spider

crawl

exabot

gigabot

infoseek sidewinder

linkchecker

netsongbot

dataforseobot

dataforseo-bot

mj12bot/

linkdexbot

adsbot

awariosmartbot

feedspot

sogou web spider

feedspot/1.0

keybot

zoominfobot

feedspot

checkhost

orbbot

yandeximages

wellknownbot

surdotlybot

gdnplus.com

online-webceo-bot

scrapy

rogerbot

mj12bot

dotbot

alexibot

surveybot

xenu

exabot

gigabot

blekkobot

mecrawler

turnitinbot

python-requests

python-urllib/2.7

curl/7.35.0

wp_is_mobile

java/11.0.10

photon

photon/1.0

serendeputybot

webpage-inspector.com

twingly recon

bidtellect/0.0.958.0

slackbot-linkexpanding 1.0

df bot 1.0

who.is bot

sem rush bot

yandex bot

sogou bot

majestic bot

exalead bot

baidu bot

mediamathbot/1.0

trendictionbot0

omgili/0.5

iframely

cortex/1.0

trendictionbot0.5.0

domains project/1.3.7

trendictionbot

publishers.academy
robots.txt