Log In Sign Up

areawise.app
robots.txt

Robots Exclusion Standard data for areawise.app

Resource Scan

Scan Details

Site Domain areawise.app
Base Domain areawise.app
Scan Status Ok
Last Scan2026-03-10T13:16:41+00:00
Next Scan 2026-03-24T13:16:41+00:00

Last Scan

Scanned2026-03-10T13:16:41+00:00
URL https://areawise.app/robots.txt
Redirect https://www.areawise.app/robots.txt
Redirect Domain www.areawise.app
Redirect Base areawise.app
Domain IPs 75.2.60.5, 99.83.231.61
Redirect IPs 13.215.239.219, 2406:da18:b3d:e201::258, 2406:da18:b3d:e201::259, 52.74.6.109
Response IP 13.215.239.219
Found Yes
Hash a629c1bb228c3625fd26b324c59db2d72247fd8dc5419a5550022eb18013ce4c
SimHash 5866d3e0fb13

Groups

imagesiftbot

Rule Path
Disallow /

mediapartners-google

Rule Path
Disallow /

moodlebot

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

piplbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

applebot

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

youbot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

builtwith

Rule Path
Disallow /

cloudfind

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

megaindex

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

xobibot

Rule Path
Disallow /

proximic

Rule Path
Disallow /path/

baiduspider

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

psbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

sogou inst spider

Rule Path
Disallow /

yandex

Rule Path
Disallow /

yisouspider

Rule Path
Disallow /

*

Rule Path
Disallow /hero-dashboard/
Disallow /member/
Back to top

Comments

  • MoodleBot is Moodl's web crawler
  • magpie-crawler is Brandwatch.com's web crawler
Back to top

Warnings

  • 2 invalid lines.
Back to top