Log In Sign Up

dressfinder.ca
robots.txt

Robots Exclusion Standard data for dressfinder.ca

Resource Scan

Scan Details

Site Domain dressfinder.ca
Base Domain dressfinder.ca
Scan Status Ok
Last Scan2025-04-29T10:07:39+00:00
Next Scan 2025-05-06T10:07:39+00:00

Last Scan

Scanned2025-04-29T10:07:39+00:00
URL https://dressfinder.ca/robots.txt
Domain IPs 104.21.2.73, 172.67.128.225, 2606:4700:3034::ac43:80e1, 2606:4700:3036::6815:249
Response IP 172.67.128.225
Found Yes
Hash b40e12e4c9e7b5028a13f75417bb50a8a3e9fe79ef9aa89316d3e31417621e96
SimHash 62165d58de13

Groups

*

Rule Path
Disallow /_Util
Disallow /Menu
Disallow /Search
Disallow /User

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

baiduspider

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

applebot

Rule Path
Disallow /

slurp

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

twiceler

Rule Path
Disallow /

serpstatbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

brightbot

Rule Path
Disallow /

grapeshot

Rule Path
Disallow /

proximic

Rule Path
Disallow /

spinn3r

Rule Path
Disallow /

riddler

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

worldbrewbot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

linkdexbot

Rule Path
Disallow /

wesee:ads/pagebot

Rule Path
Disallow /

wesee

Rule Path
Disallow /

pageanalyzer

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

orangebot

Rule Path
Disallow /

linkwalker

Rule Path
Disallow /

obot

Rule Path
Disallow /

nutch

Rule Path
Disallow /

yandex

Rule Path
Disallow /

linkwalker

Rule Path
Disallow /

femtosearchbot

Rule Path
Disallow *

serpstatbot

Rule Path
Disallow /

mauibot

Rule Path
Disallow *

dotbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

zoominfobot

Rule Path
Disallow /

adsbot

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

awariorssbot
awariosmartbot

Rule Path
Disallow /
Back to top

Comments

  • robots.txt for dressfinder.bride.ca
  • and bride.canada live! sites
  • Baiduspider
  • Petal Bot (Huwei)
  • Yahoo
  • AhrefsBot
  • AhrefsBot
  • http://www.grapeshot.co.uk/crawler.php
  • http://www.proximic.com/spider.html
  • http://spinn3r.com/robot
  • an online research project which investigates algorithms for mapping the topology of the Internet
  • SemrushBot/0.98~bl; +http://www.semrush.com/bot.html
  • http://www.marketbrew.com
  • http://help.yandex.com/search/robots/
  • http://www.linkdex.com/m/bots/
  • http://www.wesee.com/bot/
  • http://webmeup-crawler.com
  • support.orangebot@orange.com
  • http://www.brandprotect.com
  • http://filterdb.iss.net/crawler/
  • http://nutch.apache.org/bot.html
  • Chinese search engine spider so.360.cn
  • Browser: Scrapy/1.5.0 (+https://scrapy.org)
Back to top

Warnings

  • 2 invalid lines.
Back to top