allfreenovel.cc
robots.txt

Robots Exclusion Standard data for allfreenovel.cc

Resource Scan

Scan Details

Site Domain allfreenovel.cc
Base Domain allfreenovel.cc
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-09-18T20:20:25+00:00
Next Scan 2025-10-02T20:20:25+00:00

Last Successful Scan

Scanned2025-09-03T18:55:04+00:00
URL https://allfreenovel.cc/robots.txt
Domain IPs 104.21.2.151, 172.67.129.85, 2606:4700:3031::6815:297, 2606:4700:3032::ac43:8155
Response IP 104.21.2.151
Found Yes
Hash 2de9c48ed28b96ffded96499a295a624a5cd2c9e432cc7480a36a6ded0516519
SimHash 63098842ee52

Groups

*

Rule Path
Allow /
Disallow /story/*/
Disallow /story/*/*/
Disallow /wp-admin/
Disallow /wp-includes/
Disallow /wp-content/plugins/
Disallow /wp-json/
Disallow /cart/
Disallow /checkout/
Disallow /my-account/
Disallow /private/
Disallow /members/

gptbot
claude-web
anthropic-ai
perplexitybot
googleother
bingbot
duckassistbot

Rule Path
Allow /llms.txt
Disallow /wp-admin/
Disallow /wp-includes/
Disallow /wp-content/plugins/
Disallow /wp-json/
Disallow /cart/
Disallow /checkout/
Disallow /my-account/
Disallow /private/
Disallow /members/

Other Records

Field Value
sitemap https://allfreenovel.cc/sitemap_index.xml

Comments

  • robots.txt for allfreenovel.cc
  • AI Crawlers