Log In Sign Up

fisip-umw.ac.id
robots.txt

Robots Exclusion Standard data for fisip-umw.ac.id

Resource Scan

Scan Details

Site Domain fisip-umw.ac.id
Base Domain fisip-umw.ac.id
Scan Status Ok
Last Scan2025-11-07T09:08:15+00:00
Next Scan 2025-12-07T09:08:15+00:00

Last Scan

Scanned2025-11-07T09:08:15+00:00
URL https://fisip-umw.ac.id/robots.txt
Domain IPs 104.21.69.132, 172.67.208.184, 2606:4700:3030::6815:4584, 2606:4700:3033::ac43:d0b8
Response IP 104.21.69.132
Found Yes
Hash f05910847e1701a272489d98b3bf6fac13927c71e2abcb9fcb4ce5b2fb9e543f
SimHash 071c4de2f623

Groups

*

Rule Path
Allow /
Disallow /admin/
Disallow /private/
Disallow /*.json$
Disallow /api/
Allow /about
Allow /programs
Allow /contact
Allow /teachers

ahrefsbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 0

semrushbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 0
Back to top

Other Records

Field Value
sitemap https://fisip-umw.ac.id/sitemap.xml
Back to top

Comments

  • Sitemaps
  • No crawl delay - allow fast crawling
  • Crawl-delay: 0
  • Disallow admin areas (if any)
  • Allow important pages
  • Encourage frequent crawling
Back to top