Log In Sign Up

av01.org
robots.txt

Robots Exclusion Standard data for av01.org

Resource Scan

Scan Details

Site Domain av01.org
Base Domain av01.org
Scan Status Ok
Last Scan2025-10-18T02:56:29+00:00
Next Scan 2025-11-17T02:56:29+00:00

Last Scan

Scanned2025-10-18T02:56:29+00:00
URL https://av01.org/robots.txt
Domain IPs 104.21.16.232, 172.67.216.210, 2606:4700:3030::6815:10e8, 2606:4700:3030::ac43:d8d2
Response IP 172.67.216.210
Found Yes
Hash 7d75a5f781791fd3a542838bbfb262ee3d1bc7476cefe4a51ff3f1f9cf1182f4
SimHash 6d505601dbf6

Groups

duckduckbot

Rule Path
Disallow /_
Disallow */node/hit
Disallow */site/stat

applebot

Rule Path
Disallow /_
Disallow */node/hit
Disallow */site/stat

googlebot

Rule Path
Disallow /_
Disallow */node/hit
Disallow */site/stat

bingbot

Rule Path
Disallow /_
Disallow */node/hit
Disallow */site/stat

baiduspider

Rule Path
Disallow /_
Disallow */node/hit
Disallow */site/stat

msnbot

Rule Path
Disallow /_
Disallow */node/hit
Disallow */site/stat

yandexbot

Rule Path
Disallow /_
Disallow */node/hit
Disallow */site/stat

sogou

Rule Path
Disallow /_
Disallow */node/hit
Disallow */site/stat

facebookexternalhit

Rule Path
Disallow /_
Disallow */node/hit
Disallow */site/stat

twitterbot

Rule Path
Disallow /_
Disallow */node/hit
Disallow */site/stat

linkedinbot

Rule Path
Disallow /_
Disallow */node/hit
Disallow */site/stat

embedly

Rule Path
Disallow /_
Disallow */node/hit
Disallow */site/stat

quora link preview

Rule Path
Disallow /_
Disallow */node/hit
Disallow */site/stat

showyoubot

Rule Path
Disallow /_
Disallow */node/hit
Disallow */site/stat

outbrain

Rule Path
Disallow /_
Disallow */node/hit
Disallow */site/stat

pinterest

Rule Path
Disallow /_
Disallow */node/hit
Disallow */site/stat

developers.google.com

Rule Path
Disallow /_
Disallow */node/hit
Disallow */site/stat

slackbot

Rule Path
Disallow /_
Disallow */node/hit
Disallow */site/stat

vkshare

Rule Path
Disallow /_
Disallow */node/hit
Disallow */site/stat

w3c_validator

Rule Path
Disallow /_
Disallow */node/hit
Disallow */site/stat

whatsapp

Rule Path
Disallow /_
Disallow */node/hit
Disallow */site/stat

flipboard

Rule Path
Disallow /_
Disallow */node/hit
Disallow */site/stat

tumblr

Rule Path
Disallow /_
Disallow */node/hit
Disallow */site/stat

bitlybot

Rule Path
Disallow /_
Disallow */node/hit
Disallow */site/stat

skypeuripreview

Rule Path
Disallow /_
Disallow */node/hit
Disallow */site/stat

nuzzel

Rule Path
Disallow /_
Disallow */node/hit
Disallow */site/stat

discordbot

Rule Path
Disallow /_
Disallow */node/hit
Disallow */site/stat

google page speed

Rule Path
Disallow /_
Disallow */node/hit
Disallow */site/stat

google web preview

Rule Path
Disallow /_
Disallow */node/hit
Disallow */site/stat

google-read-aloud

Rule Path
Disallow /_
Disallow */node/hit
Disallow */site/stat

google favicon

Rule Path
Disallow /_
Disallow */node/hit
Disallow */site/stat

google search console

Rule Path
Disallow /_
Disallow */node/hit
Disallow */site/stat

google-structured-data-testing-tool

Rule Path
Disallow /_
Disallow */node/hit
Disallow */site/stat

google-amp-cache

Rule Path
Disallow /_
Disallow */node/hit
Disallow */site/stat

google-adwords

Rule Path
Disallow /_
Disallow */node/hit
Disallow */site/stat

google-adwords-keyword-planner

Rule Path
Disallow /_
Disallow */node/hit
Disallow */site/stat

mediapartners-google

Rule Path
Disallow /_
Disallow */node/hit
Disallow */site/stat

adsbot-google

Rule Path
Disallow /_
Disallow */node/hit
Disallow */site/stat

*

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://av01.org/sitemap/index
Back to top

Warnings

  • 4 invalid lines.
Back to top