surveys.unitetheunion.org
robots.txt

Robots Exclusion Standard data for surveys.unitetheunion.org

Resource Scan

Scan Details

Site Domain surveys.unitetheunion.org
Base Domain unitetheunion.org
Scan Status Ok
Last Scan2025-11-26T08:14:53+00:00
Next Scan 2025-12-10T08:14:53+00:00

Last Scan

Scanned2025-11-26T08:14:53+00:00
URL https://surveys.unitetheunion.org/robots.txt
Domain IPs 34.111.192.176
Response IP 34.111.192.176
Found Yes
Hash 3d8398a58aa47b5d0e4e5b18bc220b5f1ba5415480c32eead39357374552fdf5
SimHash a318494a87f7

Groups

*

Rule Path
Disallow /
Allow /form/
Allow /0
Allow /1
Allow /2
Allow /3
Allow /4
Allow /5
Allow /6
Allow /7
Allow /8
Allow /9

bingbot

Rule Path
Disallow /
Allow /form/
Allow /0
Allow /1
Allow /2
Allow /3
Allow /4
Allow /5
Allow /6
Allow /7
Allow /8
Allow /9

gptbot
claudebot
claude-web
ccbot
applebot-extended
perplexitybot
meta-externalagent
meta-externalfetcher
anthropic-ai
cohere-ai
diffbot
facebookbot
bytespider

Rule Path
Disallow /
Allow /form/
Allow /0
Allow /1
Allow /2
Allow /3
Allow /4
Allow /5
Allow /6
Allow /7
Allow /8
Allow /9

linkedinbot
twitterbot
facebookbot
facebookexternalhit
meta-externalagent
meta-externalfetcher

Rule Path
Allow /agent/

Comments

  • surveys.unitetheunion.org Robots.txt