Log In Sign Up

zellers.thebay.com
robots.txt

Robots Exclusion Standard data for zellers.thebay.com

Resource Scan

Scan Details

Site Domain zellers.thebay.com
Base Domain thebay.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-05-23T03:52:05+00:00
Next Scan 2024-06-06T03:52:05+00:00

Last Successful Scan

Scanned2024-04-15T02:25:46+00:00
URL https://zellers.thebay.com/robots.txt
Domain IPs 23.32.29.90, 23.32.29.99
Response IP 23.32.29.90
Found Yes
Hash 9fac64ff55552066282d4a6e538749b640f2c99e9b3dd56a9e07907f823d7ebb
SimHash 2c444b586eb6

Groups

*

Rule Path
Allow /account/login
Disallow /account/
Disallow /search
Disallow /cart
Disallow /checkout
Disallow /orderconfirm
Disallow /wishlist
Disallow /on/demandware.store/
Disallow /c/*_*
Disallow /*_*?cgid
Disallow /*cgid*
Disallow /*pmin*
Disallow /*pmax*
Disallow /*prefn2*
Disallow /*prefn3*
Disallow /*prefn4*
Disallow /*prefv2*
Disallow /*prefv3*
Disallow /*prefv4*
Disallow /*srule*

ahrefsbot/6.1

Rule Path
Disallow /

ahrefsbot/6.1 news

Rule Path
Disallow /

ahrefsbot/6.1 sa

Rule Path
Disallow /

alexa site audit/1.0

Rule Path
Disallow /

special_archiver/3.1.1

Rule Path
Disallow /

awariosmartbot/1.0

Rule Path
Disallow /

bitlybot/3.0

Rule Path
Disallow /

blexbot/1.0

Rule Path
Disallow /

brandverity/1.0

Rule Path
Disallow /

ccbot/2.0

Rule Path
Disallow /

coccocbot-web/1.0

Rule Path
Disallow /

researchscan/t13rl

Rule Path
Disallow /

researchscan/t12ca

Rule Path
Disallow /

researchscan/t12sns

Rule Path
Disallow /

curious george

Rule Path
Disallow /

datagnionbot

Rule Path
Disallow /

datanyze

Rule Path
Disallow /

daum/4.1

Rule Path
Disallow /

diffbot/0.1

Rule Path
Disallow /

discordbot/2.0

Rule Path
Disallow /

domainstatsbot/1.0

Rule Path
Disallow /

dotbot/1.1

Rule Path
Disallow /

dow jones searchbot

Rule Path
Disallow /

easybib autocite

Rule Path
Disallow /

embedly/0.2

Rule Path
Disallow /

evc-batch/2.0

Rule Path
Disallow /

exabot/3.0

Rule Path
Disallow /

exabot-thumbnails

Rule Path
Disallow /

arquivo-web-crawler

Rule Path
Disallow /

flipboardproxy/1.2

Rule Path
Disallow /

garlikcrawler/1.2

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

grapeshotcrawler/2.0

Rule Path
Disallow /

houzzbot

Rule Path
Disallow /

hubspot marketing grader

Rule Path
Disallow /

adstxtcrawler/1.0

Rule Path
Disallow /

iframely/1.3.1

Rule Path
Disallow /

jooblebot/2.0

Rule Path
Disallow /

linkedinbot/1.0

Rule Path
Disallow /

ltx71

Rule Path
Disallow /

lumtelbot/1.0

Rule Path
Disallow /

magibot/1.0.0

Rule Path
Disallow /

mail.ru_bot/2.0

Rule Path
Disallow /

mail.ru_bot/img/2.0

Rule Path
Disallow /

mediatoolkitbot

Rule Path
Disallow /

mj12bot/v1.4.8

Rule Path
Disallow /

yeti/1.1

Rule Path
Disallow /

panscient.com

Rule Path
Disallow /

paperlibot/2.1

Rule Path
Disallow /

pingdom.com_bot 1.4

Rule Path
Disallow /

pingdompagespeed/1.0

Rule Path
Disallow /

piplbot

Rule Path
Disallow /

pocketparser/2.0

Rule Path
Disallow /

proximic

Rule Path
Disallow /

pubmatic crawler bot

Rule Path
Disallow /

ravencrawler/2.0

Rule Path
Disallow /

rival iq

Rule Path
Disallow /

scrapy/1.5.1

Rule Path
Disallow /

semrushbot/6~bl

Rule Path
Disallow /

semrushbot-sa/0.97

Rule Path
Disallow /

semrushbot-si/0.97

Rule Path
Disallow /

semrushbot/1.0~bm

Rule Path
Disallow /

seznambot/3.2

Rule Path
Disallow /

linkcheck

Rule Path
Disallow /

skypeuripreview0.5

Rule Path
Disallow /

slackbot-linkexpanding 1.0

Rule Path
Disallow /

smtbot/1.0

Rule Path
Disallow /

sogou web spider/4.0

Rule Path
Disallow /

superpagesurlverifybot/1.0

Rule Path
Disallow /

traackr.com

Rule Path
Disallow /

trendictionbot 0.5.0

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

tweetmemebot/4.0

Rule Path
Disallow /

twitterbot/1.0

Rule Path
Disallow /

um-ln/1.0

Rule Path
Disallow /

uptimerobot/2.0

Rule Path
Disallow /

velenpublicwebcrawler/1.0

Rule Path
Disallow /

virustotalcloud

Rule Path
Disallow /

weborama-fetcher

Rule Path
Disallow /

wordpress.com mshots

Rule Path
Disallow /

linkbot 1.0

Rule Path
Disallow /

zoominfo bot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.thebay.com/sitemap_index.xml
Back to top

Comments

  • Search refinement URL Parameters
Back to top

Warnings

  • 2 invalid lines.
Back to top