Log In Sign Up

allstars.com.sg
robots.txt

Robots Exclusion Standard data for allstars.com.sg

Resource Scan

Scan Details

Site Domain allstars.com.sg
Base Domain allstars.com.sg
Scan Status Ok
Last Scan2026-03-25T04:35:35+00:00
Next Scan 2026-04-24T04:35:35+00:00

Last Scan

Scanned2026-03-25T04:35:35+00:00
URL https://allstars.com.sg/robots.txt
Domain IPs 104.21.3.23, 172.67.130.16, 2606:4700:3034::ac43:8210, 2606:4700:3035::6815:317
Response IP 172.67.130.16
Found Yes
Hash 33e52cb33212d4402d3d9e4a4c27c7d0ed7a7b8a3ec2ecf7f106fd57134818a3
SimHash b00865c0c3f7

Groups

*

Rule Path
Disallow /wp-admin/
Disallow
Allow /privacy-policy-v1

Other Records

Field Value
crawl-delay 30

*

Rule Path
Disallow /*blackhole
Disallow /?blackhole

facebookexternalhit

Rule Path
Disallow

Other Records

Field Value
crawl-delay 30

facebot

Rule Path
Disallow

Other Records

Field Value
crawl-delay 30

mediapartners-google*

Rule Path
Disallow

googlebot-image

Rule Path
Allow /uploads.allstars.com.sg/
Disallow

googlebot-mobile

Rule Path
Disallow

freshpingbot/1.0 (+https://freshping.io/)

Rule Path
Disallow
Back to top

Comments

  • Blackhole
  • Adding Facebook bots -- for Social Login
  • Adding Google Bots - Allow
  • Adding Freshping Bot - Allow
Back to top

Warnings

  • `https` is not a known field.
Back to top