playshakespeare.com
robots.txt

Robots Exclusion Standard data for playshakespeare.com

Resource Scan

Scan Details

Site Domain playshakespeare.com
Base Domain playshakespeare.com
Scan Status Ok
Last Scan2024-06-21T09:53:17+00:00
Next Scan 2024-06-28T09:53:17+00:00

Last Scan

Scanned2024-06-21T09:53:17+00:00
URL https://playshakespeare.com/robots.txt
Redirect http://www.playshakespeare.com/robots.txt
Redirect Domain www.playshakespeare.com
Redirect Base playshakespeare.com
Domain IPs 69.48.180.100
Redirect IPs 69.48.180.100
Response IP 69.48.180.100
Found Yes
Hash b611e1475e612f0a35e0ea2b54b36d790aca09827c5c68610dba77567172e8bb
SimHash ba56db754971

Groups

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

slurp

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

linkdexbot

Rule Path
Disallow /

storygizebot

Rule Path
Disallow /

elefent

Rule Path
Disallow /

coccocbot-web

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

proximic

Rule Path
Disallow /

grapeshot

Rule Path
Disallow /

barkrowler/0.9

Rule Path
Disallow /

dotbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 9999

scoutjet

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 9999

ccbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 9999

bleriot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

ttd-content

Rule Path
Disallow /

blackboard safeassign

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

gumgum-bot

Rule Path
Disallow /

googlebot

Rule Path
Allow /*.js*
Allow /*.css*
Allow /*.png*
Allow /*.jpg*
Allow /*.gif*

*

Rule Path
Disallow /administrator/
Disallow /bin/
Disallow /cache/
Disallow /cli/
Disallow /includes/
Disallow /installation/
Disallow /language/
Disallow /layouts/
Disallow /libraries/
Disallow /logs/
Disallow /tmp/
Disallow /login
Disallow /reviews-extra
Disallow /jreviews
Disallow /jreviews/upload
Disallow /community/search/
Disallow /users/
Disallow /edward-ii/
Disallow /tags/
Disallow /directory/search-by/
Disallow /content/
Disallow /component/
Disallow /community
Disallow /community/apps/browse
Disallow /community/people
Disallow /community/photos/app
Disallow /community/groups/adddiscussion
Disallow /community/groups/display
Disallow /community/groups/discussapp
Disallow /community/search/
Disallow /community/profile/
Disallow /directory/favourites/
Disallow /directory/list-alpha/
Disallow /directory/owner/
Disallow /directory/review/
Disallow /directory/reviews/
Disallow /directory/listings/
Disallow /community/profile/
Disallow /community/stream/
Disallow /index.php/
Disallow /badbots/
Allow /components/*.js
Allow /components/*.css
Allow /components/*.png
Allow /components/*.jpg
Allow /components/*.gif
Allow /components/*.woff
Allow /components/*.svg
Allow /components/*.eot
Allow /components/*.ttf
Allow /templates/*.js
Allow /templates/*.png
Allow /templates/*.jpg
Allow /templates/*.gif
Allow /templates/*.css
Allow /templates/*.woff
Allow /templates/*.svg
Allow /templates/*.eot
Allow /templates/*.ttf
Allow /media/*.js
Allow /media/*.css
Allow /media/*.png
Allow /media/*.jpg
Allow /media/*.gif
Allow /media/*.woff
Allow /media/*.svg
Allow /media/*.eot
Allow /media/*.ttf
Allow /media/reviews/photos
Allow /media/reviews/photos/thumbnails
Allow /images/*.png
Allow /images/*.jpg
Allow /images/*.gif
Allow /templates/playshakespeare/css/*.css
Allow /media/system/css/*.css
Allow /components/com_easyblog/assets/css/*.css
Allow /modules/mod_pagescrollbuttons/libraries/styles/*.css
Allow /plugins/content/xml_content/css/*.css

Other Records

Field Value
sitemap https://www.playshakespeare.com/index.php?option=com_jmap&view=sitemap&format=xml
sitemap https://www.playshakespeare.com/index.php?option=com_jmap&view=sitemap&format=gnews

Comments

  • For syntax checking, see:
  • http://tool.motoricerca.info/robots-checker.phtml
  • User-agent: bingbot
  • Disallow: /
  • User-agent: Bingbot
  • Disallow: /
  • User-agent: SeznamBot
  • Request-rate: 1/100m
  • Seznambot: access everything, rate 1 URL per 100 minutes
  • Google keeps going here for bad URLs
  • Get the bad bots
  • Google recommends this
  • Direct paths don't work for Google. Let's try wildcards. This seems to work.
  • JSitemap entries