Log In Sign Up

www.articles.studio9xb.com
robots.txt

Robots Exclusion Standard data for www.articles.studio9xb.com

Resource Scan

Scan Details

Site Domain www.articles.studio9xb.com
Base Domain studio9xb.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-04-26T09:13:43+00:00
Next Scan 2025-07-25T09:13:43+00:00

Last Successful Scan

Scanned2024-09-06T06:15:09+00:00
URL https://www.articles.studio9xb.com/robots.txt
Domain IPs 162.241.85.74
Response IP 162.241.85.74
Found Yes
Hash b4dc81ff1a9fc004badaace1b200fb62fd40b4037ca093b23160d3f0d8a32841
SimHash 70949d01cf58

Groups

*

Rule Path
Allow /misc/*.css
Allow /misc/*.js
Allow /misc/*.gif
Allow /misc/*.jpg
Allow /misc/*.jpeg
Allow /misc/*.png
Allow /modules/*.css
Allow /modules/*.js
Allow /modules/*.gif
Allow /modules/*.jpg
Allow /modules/*.jpeg
Allow /modules/*.png
Allow /profiles/*.css
Allow /profiles/*.js
Allow /profiles/*.gif
Allow /profiles/*.jpg
Allow /profiles/*.jpeg
Allow /profiles/*.png
Allow /themes/*.css
Allow /themes/*.js
Allow /themes/*.gif
Allow /themes/*.jpg
Allow /themes/*.jpeg
Allow /themes/*.png
Disallow /includes/
Disallow /misc/
Disallow /modules/
Disallow /profiles/
Disallow /scripts/
Disallow /themes/
Disallow /CHANGELOG.txt
Disallow /cron.php
Disallow /INSTALL.mysql.txt
Disallow /INSTALL.pgsql.txt
Disallow /INSTALL.sqlite.txt
Disallow /install.php
Disallow /INSTALL.txt
Disallow /LICENSE.txt
Disallow /MAINTAINERS.txt
Disallow /update.php
Disallow /UPGRADE.txt
Disallow /xmlrpc.php
Disallow /admin/
Disallow /comment/reply/
Disallow /filter/tips/
Disallow /node/add/
Disallow /node/preview/
Disallow /search/
Disallow /user/register/
Disallow /user/password/
Disallow /user/login/
Disallow /user/logout/
Disallow /?q=admin%2F
Disallow /?q=comment%2Freply%2F
Disallow /?q=filter%2Ftips%2F
Disallow /?q=node%2Fadd%2F
Disallow /?q=search%2F
Disallow /?q=user%2Fpassword%2F
Disallow /?q=user%2Fregister%2F
Disallow /?q=user%2Flogin%2F
Disallow /?q=user%2Flogout%2F
Disallow /admin
Disallow /comment/reply
Disallow /filter/tips
Disallow /node/add
Disallow /node/preview
Disallow /search
Disallow /user/register
Disallow /user/password
Disallow /user/login
Disallow /user/logout
Disallow /?q=admin
Disallow /?q=comment%2Freply
Disallow /?q=filter%2Ftips
Disallow /?q=node%2Fadd
Disallow /?q=search
Disallow /?q=user%2Fpassword
Disallow /?q=user%2Fregister
Disallow /?q=user%2Flogin
Disallow /?q=user%2Flogout

Other Records

Field Value
crawl-delay 10

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

mj12bot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

semrushbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

dotbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

blexbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30
Back to top

Comments

  • robots.txt
  • This file is to prevent the crawling and indexing of certain parts
  • of your site by web crawlers and spiders run by sites like Yahoo!
  • and Google. By telling these "robots" where not to go on your site,
  • you save bandwidth and server resources.
  • This file will be ignored unless it is at the root of your host:
  • Used: http://example.com/robots.txt
  • Ignored: http://example.com/site/robots.txt
  • For more information about the robots.txt standard, see:
  • http://www.robotstxt.org/robotstxt.html
  • Allow CSS, JS, Images
  • Directories to disallow
  • Files to disallow
  • Paths (clean URLs)
  • Paths (no clean URLs)
  • Paths (clean URLs) - No trailing slash
  • Paths (no clean URLs) - No trailing slash
  • Specific crawl delays for known problematic bots
Back to top