Log In Sign Up

thebay.com
robots.txt

Robots Exclusion Standard data for thebay.com

Resource Scan

Scan Details

Site Domain thebay.com
Base Domain thebay.com
Scan Status Failed
Failure ReasonScan timed out.
Last Scan2024-09-17T03:57:35+00:00
Next Scan 2024-12-16T03:57:35+00:00

Last Successful Scan

Scanned2024-03-31T00:50:52+00:00
URL https://www.thebay.com/robots.txt
Domain IPs 23.32.29.105, 23.32.29.107
Response IP 23.32.29.105
Found Yes
Hash 46b2e4769ebf41b5eef5ec6c80c2d41a661f6057204f5267cdd808358744323f
SimHash 2c444b586eb6

Groups

*

Rule Path
Allow /account/login
Disallow /account/
Disallow /search
Disallow /cart
Disallow /checkout
Disallow /orderconfirm
Disallow /wishlist
Disallow /on/demandware.store/
Disallow /c/*_*
Disallow /*_*?cgid
Disallow /*cgid*
Disallow /*pmin*
Disallow /*pmax*
Disallow /*prefn2*
Disallow /*prefn3*
Disallow /*prefn4*
Disallow /*prefv2*
Disallow /*prefv3*
Disallow /*prefv4*

ahrefsbot/6.1

Rule Path
Disallow /

ahrefsbot/6.1 news

Rule Path
Disallow /

ahrefsbot/6.1 sa

Rule Path
Disallow /

alexa site audit/1.0

Rule Path
Disallow /

special_archiver/3.1.1

Rule Path
Disallow /

awariosmartbot/1.0

Rule Path
Disallow /

bitlybot/3.0

Rule Path
Disallow /

blexbot/1.0

Rule Path
Disallow /

brandverity/1.0

Rule Path
Disallow /

ccbot/2.0

Rule Path
Disallow /

coccocbot-web/1.0

Rule Path
Disallow /

researchscan/t13rl

Rule Path
Disallow /

researchscan/t12ca

Rule Path
Disallow /

researchscan/t12sns

Rule Path
Disallow /

curious george

Rule Path
Disallow /

datagnionbot

Rule Path
Disallow /

datanyze

Rule Path
Disallow /

daum/4.1

Rule Path
Disallow /

diffbot/0.1

Rule Path
Disallow /

discordbot/2.0

Rule Path
Disallow /

domainstatsbot/1.0

Rule Path
Disallow /

dotbot/1.1

Rule Path
Disallow /

dow jones searchbot

Rule Path
Disallow /

easybib autocite

Rule Path
Disallow /

embedly/0.2

Rule Path
Disallow /

evc-batch/2.0

Rule Path
Disallow /

exabot/3.0

Rule Path
Disallow /

exabot-thumbnails

Rule Path
Disallow /

arquivo-web-crawler

Rule Path
Disallow /

flipboardproxy/1.2

Rule Path
Disallow /

garlikcrawler/1.2

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

grapeshotcrawler/2.0

Rule Path
Disallow /

houzzbot

Rule Path
Disallow /

hubspot marketing grader

Rule Path
Disallow /

adstxtcrawler/1.0

Rule Path
Disallow /

iframely/1.3.1

Rule Path
Disallow /

jooblebot/2.0

Rule Path
Disallow /

linkedinbot/1.0

Rule Path
Disallow /

ltx71

Rule Path
Disallow /

lumtelbot/1.0

Rule Path
Disallow /

magibot/1.0.0

Rule Path
Disallow /

mail.ru_bot/2.0

Rule Path
Disallow /

mail.ru_bot/img/2.0

Rule Path
Disallow /

mediatoolkitbot

Rule Path
Disallow /

mj12bot/v1.4.8

Rule Path
Disallow /

yeti/1.1

Rule Path
Disallow /

panscient.com

Rule Path
Disallow /

paperlibot/2.1

Rule Path
Disallow /

pingdom.com_bot 1.4

Rule Path
Disallow /

pingdompagespeed/1.0

Rule Path
Disallow /

piplbot

Rule Path
Disallow /

pocketparser/2.0

Rule Path
Disallow /

proximic

Rule Path
Disallow /

pubmatic crawler bot

Rule Path
Disallow /

ravencrawler/2.0

Rule Path
Disallow /

rival iq

Rule Path
Disallow /

scrapy/1.5.1

Rule Path
Disallow /

semrushbot/6~bl

Rule Path
Disallow /

semrushbot-sa/0.97

Rule Path
Disallow /

semrushbot-si/0.97

Rule Path
Disallow /

semrushbot/1.0~bm

Rule Path
Disallow /

seznambot/3.2

Rule Path
Disallow /

linkcheck

Rule Path
Disallow /

skypeuripreview0.5

Rule Path
Disallow /

slackbot-linkexpanding 1.0

Rule Path
Disallow /

smtbot/1.0

Rule Path
Disallow /

sogou web spider/4.0

Rule Path
Disallow /

superpagesurlverifybot/1.0

Rule Path
Disallow /

traackr.com

Rule Path
Disallow /

trendictionbot 0.5.0

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

tweetmemebot/4.0

Rule Path
Disallow /

twitterbot/1.0

Rule Path
Disallow /

um-ln/1.0

Rule Path
Disallow /

uptimerobot/2.0

Rule Path
Disallow /

velenpublicwebcrawler/1.0

Rule Path
Disallow /

virustotalcloud

Rule Path
Disallow /

weborama-fetcher

Rule Path
Disallow /

wordpress.com mshots

Rule Path
Disallow /

linkbot 1.0

Rule Path
Disallow /

zoominfo bot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.thebay.com/sitemap_index.xml
Back to top

Comments

  • Search refinement URL Parameters
Back to top

Warnings

  • 2 invalid lines.
Back to top