webnovel.io
robots.txt

Robots Exclusion Standard data for webnovel.io

Resource Scan

Scan Details

Site Domain webnovel.io
Base Domain webnovel.io
Scan Status Ok
Last Scan2024-09-23T13:55:51+00:00
Next Scan 2024-09-30T13:55:51+00:00

Last Scan

Scanned2024-09-23T13:55:51+00:00
URL https://webnovel.io/robots.txt
Domain IPs 104.21.88.204, 172.67.153.24, 2606:4700:3034::ac43:9918, 2606:4700:3036::6815:58cc
Response IP 172.67.153.24
Found Yes
Hash 76ed3cce3ddbc455335a0d873a61b1b846b9a7312ed1928ec65dccebfa33ffbf
SimHash 480c49604993

Groups

*

Rule Path
Disallow /api
Disallow /logs/

*

Rule Path
Allow /

semrushbot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://webnovel.io/sitemap-index.xml