file.net
robots.txt

Robots Exclusion Standard data for file.net

Resource Scan

Scan Details

Site Domain file.net
Base Domain file.net
Scan Status Ok
Last Scan2024-04-19T21:44:45+00:00
Next Scan 2024-04-26T21:44:45+00:00

Last Scan

Scanned2024-04-19T21:44:45+00:00
URL https://file.net/robots.txt
Redirect https://www.file.net/robots.txt
Redirect Domain www.file.net
Redirect Base file.net
Domain IPs 104.26.6.175, 104.26.7.175, 172.67.69.104, 2606:4700:20::681a:6af, 2606:4700:20::681a:7af, 2606:4700:20::ac43:4568
Redirect IPs 104.26.6.175, 104.26.7.175, 172.67.69.104, 2606:4700:20::681a:6af, 2606:4700:20::681a:7af, 2606:4700:20::ac43:4568
Response IP 104.26.7.175
Found Yes
Hash e52ab3288097bfefbaebb29d3a9883510efe0ea54bc6bbb224ab5b2392a599d8
SimHash 4b7548f4e4b0

Groups

*

Rule Path
Disallow /cgi-bin
Disallow /cgi-sys

ahrefsbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

mediatoolkit

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

linkdexbot

Rule Path
Disallow /

grapeshotcrawler

Rule Path
Disallow /

proximic

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

extlinksbot

Rule Path
Disallow /

weborama-fetcher

Rule Path
Disallow /

expo9

Rule Path
Disallow /

spbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

mangoway

Rule Path
Disallow /

wotbox

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.file.net/google_sitemap.gz

Comments

  • Crawl-Delay: 20