burlingtondowntown.ca
robots.txt

Robots Exclusion Standard data for burlingtondowntown.ca

Resource Scan

Scan Details

Site Domain burlingtondowntown.ca
Base Domain burlingtondowntown.ca
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-11-17T23:57:35+00:00
Next Scan 2026-01-16T23:57:35+00:00

Last Successful Scan

Scanned2025-08-27T14:36:44+00:00
URL https://burlingtondowntown.ca/robots.txt
Domain IPs 51.79.18.178
Response IP 51.79.18.178
Found Yes
Hash dd1f123f1a4af8500ae20d01882c33bddf2a73aacbe36c8c97f98f8a4e203318
SimHash 488a0850e9f1

Groups

barkrowler

Rule Path
Disallow /

qwantbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

googlebot

Rule Path
Disallow

Other Records

Field Value
crawl-delay 5

bingbot

Rule Path
Disallow

Other Records

Field Value
crawl-delay 10

meta-externalagent

Rule Path
Disallow

Other Records

Field Value
crawl-delay 10

*

Rule Path
Disallow /wp-admin/
Disallow /wp-includes/
Disallow /wp-content/plugins/
Disallow /wp-content/themes/
Allow /wp-admin/admin-ajax.php

*

Rule Path
Disallow /*?post_type=tribe_events
Disallow /?post_type=tribe_events&eventDisplay=day&paged=
Disallow /page/*/?post_type=tribe_events&eventDisplay=day

Comments

  • Allow Googlebot to access everything, but limit its crawl rate
  • Allow Bingbot to access everything, but limit its crawl rate
  • Disallow unnecessary areas for all other bots