Log In Sign Up

bride.ca
robots.txt

Robots Exclusion Standard data for bride.ca

Resource Scan

Scan Details

Site Domain bride.ca
Base Domain bride.ca
Scan Status Ok
Last Scan2025-04-11T01:35:53+00:00
Next Scan 2025-04-18T01:35:53+00:00

Last Scan

Scanned2025-04-11T01:35:53+00:00
URL https://bride.ca/robots.txt
Domain IPs 104.21.52.195, 172.67.203.57, 2606:4700:3032::6815:34c3, 2606:4700:3037::ac43:cb39
Response IP 104.21.52.195
Found Yes
Hash 0d3bfc8fea0f27b4f5582d65e23efc582b3859a16ed7eaf3eb69081393bb692c
SimHash 46165d40a303

Groups

*

Rule Path
Disallow /images
Disallow /error
Disallow /includes
Disallow /tags
Disallow /wedding-classifieds
Disallow /wedding-directory

baiduspider

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

slurp

Rule Path
Disallow /

applebot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

twiceler

Rule Path
Disallow /

bingbot

Rule Path
Disallow /

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

mj12bot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

grapeshot

Rule Path
Disallow /

proximic

Rule Path
Disallow /

spinn3r

Rule Path
Disallow /

riddler

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

worldbrewbot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

linkdexbot

Rule Path
Disallow /

wesee:ads/pagebot

Rule Path
Disallow /

wesee

Rule Path
Disallow /

pageanalyzer

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

orangebot

Rule Path
Disallow /

linkwalker

Rule Path
Disallow /

obot

Rule Path
Disallow /

nutch

Rule Path
Disallow /

yandex

Rule Path
Disallow /

linkwalker

Rule Path
Disallow /

femtosearchbot

Rule Path
Disallow *

serpstatbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

zoominfobot

Rule Path
Disallow /

seekport

Rule Path
Disallow /

barkrowler/0.9

Rule Path
Disallow /

scrapy

Rule Path
Disallow /
Back to top

Comments

  • robots.txt for http://www.bride.ca
  • Baiduspider
  • Petal Bot (Huwei)
  • Yahoo
  • Bing
  • AhrefsBot
  • http://www.grapeshot.co.uk/crawler.php
  • http://www.proximic.com/spider.html
  • http://spinn3r.com/robot
  • an online research project which investigates algorithms for mapping the topology of the Internet
  • SemrushBot/0.98~bl; +http://www.semrush.com/bot.html
  • http://www.marketbrew.com
  • http://help.yandex.com/search/robots/
  • http://www.linkdex.com/m/bots/
  • http://www.wesee.com/bot/
  • http://webmeup-crawler.com
  • support.orangebot@orange.com
  • http://www.brandprotect.com
  • http://filterdb.iss.net/crawler/
  • http://nutch.apache.org/bot.html
  • Chinese search engine spider so.360.cn
  • Browser: Scrapy/1.5.0 (+https://scrapy.org)
Back to top

Warnings

  • 2 invalid lines.
Back to top