Log In Sign Up

wbjournal.com
robots.txt

Robots Exclusion Standard data for wbjournal.com

Resource Scan

Scan Details

Site Domain wbjournal.com
Base Domain wbjournal.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-12-07T19:09:33+00:00
Next Scan 2026-03-07T19:09:33+00:00

Last Successful Scan

Scanned2024-04-24T03:08:31+00:00
URL https://wbjournal.com/robots.txt
Domain IPs 23.185.0.4, 2620:12a:8000::4, 2620:12a:8001::4
Response IP 23.185.0.4
Found Yes
Hash 890b98510825d46246fca403e6fddfda852f2d9454d4f0ae40bbb24522e7145d
SimHash 3896bd1bc74c

Groups

*

Rule Path
Allow /core/*.css$
Allow /core/*.css?
Allow /core/*.js$
Allow /core/*.js?
Allow /core/*.gif
Allow /core/*.jpg
Allow /core/*.jpeg
Allow /core/*.png
Allow /core/*.svg
Allow /profiles/*.css$
Allow /profiles/*.css?
Allow /profiles/*.js$
Allow /profiles/*.js?
Allow /profiles/*.gif
Allow /profiles/*.jpg
Allow /profiles/*.jpeg
Allow /profiles/*.png
Allow /profiles/*.svg
Allow /user/register
Allow /user/subscribe
Allow /subscribe
Disallow /core/
Disallow /profiles/
Disallow /README.txt
Disallow /web.config
Disallow /admin/
Disallow /comment/reply/
Disallow /filter/tips/
Disallow /node/add/
Disallow /search/
Disallow /user/register/
Disallow /user/password/
Disallow /user/login/
Disallow /user/logout/
Disallow /user/logout
Disallow /order_pdf
Disallow /order_pdf/*
Disallow /user/*
Disallow /sites/default/files/inline-files/*
Disallow /sites/default/files/saxotech-pdf/*
Disallow /index.php/admin/
Disallow /index.php/comment/reply/
Disallow /index.php/filter/tips/
Disallow /index.php/node/add/
Disallow /index.php/search/
Disallow /index.php/user/password/
Disallow /index.php/user/register/
Disallow /index.php/user/login/
Disallow /index.php/user/logout/
Disallow /e-newsletter/*

Other Records

Field Value
crawl-delay 20

seznambot

Rule Path
Disallow /

metajobbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

qwantify

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

bitvorebot

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

fuelbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /
Back to top

Comments

  • robots.txt
  • This file is to prevent the crawling and indexing of certain parts
  • of your site by web crawlers and spiders run by sites like Yahoo!
  • and Google. By telling these "robots" where not to go on your site,
  • you save bandwidth and server resources.
  • This file will be ignored unless it is at the root of your host:
  • Used: http://example.com/robots.txt
  • Ignored: http://example.com/site/robots.txt
  • For more information about the robots.txt standard, see:
  • http://www.robotstxt.org/robotstxt.html
  • CSS, JS, Images
  • Directories
  • Files
  • Paths (clean URLs)
  • Paths (no clean URLs)
  • Site Paths
  • disallow unwanted bots
Back to top