viralsweep.com
robots.txt

Robots Exclusion Standard data for viralsweep.com

Resource Scan

Scan Details

Site Domain viralsweep.com
Base Domain viralsweep.com
Scan Status Ok
Last Scan2025-10-26T22:27:02+00:00
Next Scan 2025-11-25T22:27:02+00:00

Last Scan

Scanned2025-10-26T22:27:02+00:00
URL https://viralsweep.com/robots.txt
Redirect https://www.viralsweep.com/robots.txt
Redirect Domain www.viralsweep.com
Redirect Base viralsweep.com
Domain IPs 104.18.41.5, 172.64.146.251, 2606:4700:4403::6812:2905, 2a06:98c1:310a::ac40:92fb
Redirect IPs 104.18.41.5, 172.64.146.251, 2606:4700:4403::6812:2905, 2a06:98c1:310a::ac40:92fb
Response IP 172.64.146.251
Found Yes
Hash 7d7dc3750c8b7711e879fbfa062161867e5580a3fc4054b6cfcfd04616166dfb
SimHash caa03015fbd0

Groups

*

Rule Path
Disallow /app-demo
Disallow /amazon-webstore
Disallow /features/management
Disallow /features/mobile-giveaways
Disallow /updates
Disallow /voices/interviews/hot
Disallow /voices/interviews/featured
Disallow /voices/interviews/question/*

twitterbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

Other Records

Field Value
sitemap https://viralsweep.com/sitemap.xml

Comments

  • Limit Bots' crawl rate to no more than 1 page every 5 seconds