abouttheartists.com
robots.txt

Groups

mediapartners-google

No rules defined. All paths allowed.

Other Records

Field	Value
crawl-delay	4

googlebot
googlebot-image
adsbot-google
adsbot-google-mobile-apps
bingbot
duckduckbot
msnbot
twitterbo
yandex
yeti
slurp
applebot
baiduspider
facebot
facebookexternalhit

Product	Comment
twitterbo	t

Rule	Path
Disallow	/*/edit
Disallow	/*/new
Disallow	/*/destroy
Disallow	/*/update
Disallow	/*/sort
Disallow	/movie
Disallow	/*/set_default_img
Disallow	/*/claim
Disallow	/session
Disallow	/user
Disallow	/articles
Disallow	/artists/has_worked_with/*
Disallow	/location/set_location
Disallow	/production_companies/*/pcps
Disallow	/production_works?div_id=*
Disallow	/production_engagements
Disallow	/venue_metros
Disallow	/contribution_characters
Disallow	/contributions
Disallow	/descriptions
Disallow	/jobs
Disallow	/job_categories
Disallow	/awards
Disallow	/award_recipients
Disallow	/award_categories/sidebar
Disallow	/productions?search
Disallow	/production_companies?search
Disallow	/plays?search
Disallow	/books?search
Disallow	/venues?search
Disallow	/artists?search
Disallow	/artists/same_name

Other Records

Field	Value
crawl-delay	4

*

Rule	Path
Disallow	/*/edit
Disallow	/*/new
Disallow	/*/destroy
Disallow	/*/update
Disallow	/*/sort
Disallow	/movie
Disallow	/*/set_default_img
Disallow	/*/claim
Disallow	/session
Disallow	/user
Disallow	/articles
Disallow	/*/search*
Disallow	/artists/has_worked_with/*
Disallow	/contributions/this_just_in
Disallow	/location/set_location
Disallow	/production_companies/*/pcps
Disallow	/production_works?div_id=*
Disallow	/production_engagements
Disallow	/venue_metros
Disallow	/contribution_characters
Disallow	/contributions
Disallow	/descriptions
Disallow	/jobs
Disallow	/job_categories
Disallow	/awards
Disallow	/award_recipients
Disallow	/award_categories/sidebar
Disallow	/productions?search
Disallow	/production_companies?search
Disallow	/plays?search
Disallow	/books?search
Disallow	/venues?search
Disallow	/artists?search
Disallow	/artists/same_name

Other Records

Field	Value
crawl-delay	30

openindexspider
ip-web-crawler.com
ronzoobot
turnitinbot
ahrefsbot
mj12bot
baiduspider
discobot
discoverybot
camontspider
sitebot
findshare bot
sheenbot
ezooms
sosospider
sosospider+
dotbot
panscient.com
sistrix
psbot
wbsearchbot
careerbot
jikespider
seokicks-robot
seokicks
exabot
blexbot
zumbot
yyspider
unisterbot
ccbot
riddler
spbot
xovibot
crazywebcrawler-spider
semrushbot
semrushbot-sa
tineye-bot
tineye-bot-live
memorybot
wotbox
wikido
yeti
maxpointcrawler
addthis.com robot tech.support@clearspring.com
addthis.com
smtbot
sogou web spider
sogou spider
seznambot
megaindex
ltx71
expo9
nextgensearchbot
experibot_v1
wesee_bot
omnibot
ia_archiver
istellabot
siteexplorer
dataprovider
garlikcrawler
dispatch
bubing
cliqzbot
gigabot
g-i-g-a-b-o-t
piplbot
idbot
deusu
contacts-crawler
mozilla/5.0+(compatible;+piplbot;+http://www.pipl.com/bot/)
alphaseobot
barkrowler
daum
mauibot
zoominfobot
a6-indexer
the knowledge ai
alphaseobot
alphaseobot-sa
semanticscholarbot
earwigbot
mojolicious
yandex
archive.org_bot
special_archiver
pinterestbot
googlebot-image
macocu
gptbot
megaindex.ru
dataforseobot
blexbot

Rule	Path
Disallow	/

*

Rule	Path
Disallow	/

Back to top

Other Records

Field	Value
sitemap	https://www.abouttheartists.com/sitemap_index.xml

Back to top

Comments

• See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file
• Please note that any unautorized copying or duplication of material from abouttheartists.com is expressly prohibited
• Republishing any of our data without express written permission from abouttheartists.com is prohibitted and will be prosecuted
• Accessing abouttheartist.com from behind a VPN, masking your identity behind IP addresses or User Agents of another service,
• or in any way attempting to disguise or conceal your identity while accessing our site, is prohibited.
• If you operate a search engine that will significantly increase our traffic, please contact us for permission to crawl the site.
• Otherwise, automated access to the site is prohibited.

Back to top

Warnings

• 1 invalid line.

Back to top