scilit.net
robots.txt

Robots Exclusion Standard data for scilit.net

Resource Scan

Scan Details

Site Domain scilit.net
Base Domain scilit.net
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-09-08T19:39:00+00:00
Next Scan 2024-11-07T19:39:00+00:00

Last Successful Scan

Scanned2024-07-04T19:37:35+00:00
URL https://scilit.net/robots.txt
Redirect https://www.scilit.net/robots.txt
Redirect Domain www.scilit.net
Redirect Base scilit.net
Domain IPs 195.65.194.220
Redirect IPs 104.18.4.27, 104.18.5.27, 2606:4700::6812:41b, 2606:4700::6812:51b
Response IP 104.18.5.27
Found Yes
Hash 10fa74f90c61fdab3738fc6f9ccb9b9cba5943d4a9f0b2f24dbeab53553adf66
SimHash cf66cc209c91

Groups

*

Rule Path
Disallow

nuclei
wikido
riddler
petalbot
zoominfobot
go-http-client
node/simplecrawler
cazoodlebot
dotbot/1.0
gigabot
barkrowler
blexbot
magpie-crawler

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.scilit.net/sitemap.xml

Comments

  • Block bots that don't benefit us.