Log In Sign Up

qajourney.net
robots.txt

Robots Exclusion Standard data for qajourney.net

Resource Scan

Scan Details

Site Domain qajourney.net
Base Domain qajourney.net
Scan Status Ok
Last Scan2025-03-30T09:41:47+00:00
Next Scan 2025-04-06T09:41:47+00:00

Last Scan

Scanned2025-03-30T09:41:47+00:00
URL https://qajourney.net/robots.txt
Domain IPs 66.29.153.11
Response IP 66.29.153.11
Found Yes
Hash 95d30810e5b3f8c5e2b0ff34baa8413d7a79a31830c94c4f005cabb463479b93
SimHash c26e4ed0a619

Groups

*

Rule Path
Disallow /wp-admin/
Disallow /blogarama.com/

blogarama

Rule Path
Disallow /
Disallow /wp-includes/
Disallow /cgi-bin/
Disallow /search/
Disallow /?s=
Disallow /*?s=
Allow /category/
Allow /tag/
Allow /wp-content/themes/
Allow /wp-content/plugins/

mj12bot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

yandex

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

applebot

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

spider

Rule Path
Disallow /

crawl

Rule Path
Disallow /

survey

Rule Path
Disallow /

go-http-client

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

incutio xml

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://qajourney.net/sitemap_index.xml
Back to top

Comments

  • START YOAST BLOCK
  • ---------------------------
  • Block admin area
  • Block unnecessary WordPress directories
  • Block internal search pages to prevent spam indexing
  • Allow bots to crawl categories & tags (Previously blocked)
  • Allow bots to crawl CSS/JS for proper rendering
  • Sitemap location
  • ---------------------------
  • END YOAST BLOCK
  • 👇 CUSTOM BLOCKLIST: BAD BOTS
Back to top

Warnings

  • 2 invalid lines.
Back to top