Log In Sign Up

awnetenterprises.com
robots.txt

Robots Exclusion Standard data for awnetenterprises.com

Resource Scan

Scan Details

Site Domain awnetenterprises.com
Base Domain awnetenterprises.com
Scan Status Ok
Last Scan2024-06-13T02:17:31+00:00
Next Scan 2024-06-20T02:17:31+00:00

Last Scan

Scanned2024-06-13T02:17:31+00:00
URL https://awnetenterprises.com/robots.txt
Domain IPs 116.202.175.241
Response IP 116.202.175.241
Found Yes
Hash 80fb8df5ba7b6efc9f98e1a38160d23203231f90a23b3a594d0786c415f4400c
SimHash 40964b307688

Groups

googlebot

Rule Path
Allow /

unknown robot identified by bot\*

Rule Path
Disallow /

robot

Rule Path
Disallow /

velen

Rule Path
Disallow /

cfnetwork

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

empty user agent string

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

baidu ( catchall )

Rule Path
Disallow /

phantomjs

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

crawl

Rule Path
Disallow /

nbot

Rule Path
Disallow /

firefox version 10 and lower - various robots

Rule Path
Disallow /

curl

Rule Path
Disallow /

checker

Rule Path
Disallow /

mojeekbot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

yak-linkfluence

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

dalvik

Rule Path
Disallow /

a php script

Rule Path
Disallow /

scanner

Rule Path
Disallow /

netestate ne crawler

Rule Path
Disallow /

proximic

Rule Path
Disallow /

aihitbot

Rule Path
Disallow /

feed

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /
Back to top