8051projects.net
robots.txt

Robots Exclusion Standard data for 8051projects.net

Resource Scan

Scan Details

Site Domain 8051projects.net
Base Domain 8051projects.net
Scan Status Ok
Last Scan2025-12-10T02:51:38+00:00
Next Scan 2026-01-09T02:51:38+00:00

Last Scan

Scanned2025-12-10T02:51:38+00:00
URL https://8051projects.net/robots.txt
Redirect https://www.8051projects.net//robots.txt
Redirect Domain www.8051projects.net
Redirect Base 8051projects.net
Domain IPs 104.21.2.241, 172.67.129.212, 2606:4700:3035::ac43:81d4, 2606:4700:3037::6815:2f1
Redirect IPs 104.21.2.241, 172.67.129.212, 2606:4700:3035::ac43:81d4, 2606:4700:3037::6815:2f1
Response IP 172.67.129.212
Found Yes
Hash 37ede5d1954cdfef2a6668db9769c0f4f2a9578d19c113c679be49d167d515eb
SimHash bd148820eb11

Groups

*

Rule Path
Disallow /admin/
Disallow /handlers/
Disallow /e107_files/cache/
Disallow /e107_files/public/avatars/
Disallow /temp/

Other Records

Field Value
crawl-delay 5

Comments

  • User-agent: Baiduspider
  • User-agent: MJ12bot
  • User-agent: AhrefsBot
  • User-agent: YandexBot
  • User-agent: YandexImages
  • User-agent: Linguee Bot
  • User-agent: Seekport Crawler
  • User-agent: GrapeshotCrawler
  • User-agent: istellabot
  • User-agent: SeznamBot
  • User-agent: Amazonbot
  • Disallow: /