askthecrowd.app
robots.txt

Robots Exclusion Standard data for askthecrowd.app

Resource Scan

Scan Details

Site Domain askthecrowd.app
Base Domain askthecrowd.app
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2025-10-09T15:33:31+00:00
Next Scan 2026-01-07T15:33:31+00:00

Last Successful Scan

Scanned2022-12-01T17:36:46+00:00
URL https://askthecrowd.app/robots.txt
Domain IPs 104.21.46.201, 172.67.141.173
Response IP 172.67.141.173
Found Yes
Hash e470b61e83b001fb5351a037bea3afa1720d1e25acf8895bfbc6b70fc6a8b0cb
SimHash 49440420c391

Groups

*

Rule Path
Allow /$
Disallow /r/*

twitterbot

Rule Path
Allow /$
Allow /r/*

telegrambot (like twitterbot)

Rule Path
Allow /$
Allow /r/*

slackbot

Rule Path
Allow /$
Allow /r/*

facebookexternalhit

Rule Path
Allow /$
Allow /r/*