Log In Sign Up

novel5s.com
robots.txt

Robots Exclusion Standard data for novel5s.com

Resource Scan

Scan Details

Site Domain novel5s.com
Base Domain novel5s.com
Scan Status Ok
Last Scan2024-09-26T01:00:12+00:00
Next Scan 2024-10-03T01:00:12+00:00

Last Scan

Scanned2024-09-26T01:00:12+00:00
URL https://novel5s.com/robots.txt
Domain IPs 104.21.81.130, 172.67.160.232, 2606:4700:3031::ac43:a0e8, 2606:4700:3037::6815:5182
Response IP 104.21.81.130
Found Yes
Hash 64f5ead745f6ec631668d4c23e9bcd3a3335791f12a3052d9e52d4df3b2ae2c2
SimHash 301501d04f8b

Groups

*

Rule Path
Allow /
Disallow /auth/*

proximic

Rule Path
Disallow /

obot

Rule Path
Disallow /

archive.org

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /

nutch

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

python-request

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

grapeshotcrawler

Rule Path
Disallow /

brute-force login bot

Rule Path
Disallow /

admantx

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

sitebot

Rule Path
Disallow /

spbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://novel5s.com/sitemap.xml
Back to top