Log In Sign Up

manwara.online
robots.txt

Robots Exclusion Standard data for manwara.online

Resource Scan

Scan Details

Site Domain manwara.online
Base Domain manwara.online
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2025-10-29T16:05:54+00:00
Next Scan 2025-12-28T16:05:54+00:00

Last Successful Scan

Scanned2025-08-07T14:43:31+00:00
URL https://manwara.online/robots.txt
Redirect https://www.manwara.org/robots.txt
Redirect Domain www.manwara.org
Redirect Base manwara.org
Domain IPs 104.21.112.1, 104.21.16.1, 104.21.32.1, 104.21.48.1, 104.21.64.1, 104.21.80.1, 104.21.96.1, 2606:4700:3030::6815:1001, 2606:4700:3030::6815:2001, 2606:4700:3030::6815:3001, 2606:4700:3030::6815:4001, 2606:4700:3030::6815:5001, 2606:4700:3030::6815:6001, 2606:4700:3030::6815:7001
Redirect IPs 104.21.23.185, 172.67.212.189, 2606:4700:3031::6815:17b9, 2606:4700:3037::ac43:d4bd
Response IP 104.21.23.185
Found Yes
Hash f7d7b5999e6067a68b9e6627ade083303f9a0f79b73ea1ecdb0e239538489596
SimHash 63284e53a2b0

Groups

*

Rule Path
Disallow /wp-admin/
Disallow /wp-includes/
Disallow /wp-login.php
Disallow /wp-register.php
Disallow /xmlrpc.php
Disallow */feed/
Disallow */trackback/
Disallow /search/
Disallow /?s=
Disallow /*?*
Allow /*.js
Allow /*.css
Allow /wp-content/uploads/

gptbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://manwara.org/sitemap_index.xml
Back to top

Comments

  • =============================================================
  • Optimized robots.txt file for manga website
  • Website: manwara.org
  • Last updated: 03/08/2025
  • =============================================================
  • 1. Sitemap location
  • =============================================================
  • 2. General rules for good bots (Google, Bing, etc.)
  • =============================================================
  • Block unnecessary WordPress folders and files
  • Block internal search results and unnecessary parameterized pages
  • Helps avoid duplicate content
  • Allow bots to access CSS and JS files for proper rendering
  • Important for mobile-friendly indexing
  • =============================================================
  • 3. Block AI data crawlers and spam bots
  • Helps protect content (especially manga images) and save bandwidth
  • =============================================================
Back to top