costumes-au.com
robots.txt

Robots Exclusion Standard data for costumes-au.com

Resource Scan

Scan Details

Site Domain costumes-au.com
Base Domain costumes-au.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a server error.
Last Scan2026-02-28T00:28:26+00:00
Next Scan 2026-03-30T00:28:26+00:00

Last Successful Scan

Scanned2026-01-05T22:54:55+00:00
URL https://costumes-au.com/robots.txt
Redirect https://www.costumes-au.com/robots.txt
Redirect Domain www.costumes-au.com
Redirect Base costumes-au.com
Domain IPs 104.26.12.155, 104.26.13.155, 172.67.73.69, 2606:4700:20::681a:c9b, 2606:4700:20::681a:d9b, 2606:4700:20::ac43:4945
Redirect IPs 104.26.12.155, 104.26.13.155, 172.67.73.69, 2606:4700:20::681a:c9b, 2606:4700:20::681a:d9b, 2606:4700:20::ac43:4945
Response IP 104.26.13.155
Found Yes
Hash b70110d41d2915770428f49aa43637ad6ebf04cca247c9591662e1aa1e9e67a8
SimHash c00dd570c201

Groups

*

Rule Path
Disallow /search.html
Disallow /404.htm
Disallow /checkdelivery
Disallow /currency/
Disallow /cart/
Disallow /account/
Disallow /checkout/
Disallow /login/
Disallow /contacts/
Disallow /customer-service
Disallow /index.php/
Disallow /ca/

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /
Disallow /*?dir=
Disallow /*?order=
Disallow /*?limit=
Disallow /*?mode=
Disallow /*?p=
Disallow /*?___SID=
Disallow /*?SID=
Disallow /*?*

googlebot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

yandex

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

Other Records

Field Value
sitemap https://costumes-au.com/sitemap.xml

Comments

  • Prevent crawl of layered navigation, sorting, and pagination
  • Crawl rate limits for known bots
  • Sitemap location