Log In Sign Up

resultbiselahore.netlify.app
robots.txt

Robots Exclusion Standard data for resultbiselahore.netlify.app

Resource Scan

Scan Details

Site Domain resultbiselahore.netlify.app
Base Domain resultbiselahore.netlify.app
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a server error.
Last Scan2025-11-08T08:42:57+00:00
Next Scan 2025-11-22T08:42:57+00:00

Last Successful Scan

Scanned2025-10-22T22:10:04+00:00
URL https://resultbiselahore.netlify.app/robots.txt
Domain IPs 13.215.239.219, 2406:da18:b3d:e201::258, 2406:da18:b3d:e201::259, 52.74.6.109
Response IP 13.215.239.219
Found Yes
Hash 049a58e748a98e4e95c95bdae925d524869592e41d4c7745013a5a11f1fae17b
SimHash 614e5a71f7f2

Groups

*

Rule Path
Allow /
Disallow /temp/
Disallow /*.tmp$
Disallow /*?*
Allow /results/
Allow /images/
Allow /css/
Allow /js/

Other Records

Field Value
crawl-delay 1

googlebot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 0.5

bingbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

semrushbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10
Back to top

Other Records

Field Value
sitemap https://resultbiselahore.netlify.app/sitemap.xml
sitemap https://resultbiselahore.netlify.app/sitemap_additional.xml
Back to top

Comments

  • Main sitemap
  • Additional sitemap
  • Optimize crawl rate for search engines
  • Disallow temporary or duplicate content
  • Allow important directories explicitly
  • Special instructions for Google
  • Special instructions for Bing
  • Block specific bots that might overload the server
Back to top