webstore.com
robots.txt

Robots Exclusion Standard data for webstore.com

Resource Scan

Scan Details

Site Domain webstore.com
Base Domain webstore.com
Scan Status Failed
Failure ReasonScan timed out.
Last Scan2024-10-01T19:48:40+00:00
Next Scan 2024-10-08T19:48:40+00:00

Last Successful Scan

Scanned2024-09-23T19:47:24+00:00
URL https://webstore.com/robots.txt
Domain IPs 54.86.125.223
Response IP 54.86.125.223
Found Yes
Hash 7f833986fc76eda1f48b5ec332cd49de8a687281d1baba23a6d98e269dbe632e
SimHash d08c554aad0a

Groups

mediapartners-google

Rule Path
Disallow

googlebot

Rule Path
Disallow

*

Rule Path
Disallow /login.php
Disallow /register.php
Disallow /user_reputation.php
Disallow /*user_reputation
Disallow /contact_us
Disallow /buy_out
Disallow /*buy_out
Disallow /bid_history
Disallow /*bid_history
Disallow /*item_watch*
Disallow /*help$
Disallow /support.php
Disallow /*members_area
Disallow /*make_offer*
Disallow /*edit_item
Disallow /*easy_post
Disallow /*sell_item
Disallow /*message_board
Disallow /*search.php

mj12bot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

yandex

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

rankactivelinkbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

keybot

Rule Path
Disallow /

bdcbot

Rule Path
Disallow /

clickagy

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

sogou blog

Rule Path
Disallow /

sogou inst spider

Rule Path
Disallow /

sogou news spider

Rule Path
Disallow /

sogou orion spider

Rule Path
Disallow /

sogou spider2

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

sogou inst spider

Rule Path
Disallow /

yisouspider

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

megaindex.com

Rule Path
Disallow /

megaindex.ru/2.0

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

nutch

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

facebookexternalhit

Rule Path
Disallow /

facebookcatalog

Rule Path
Disallow /

meta-externalagent

Rule Path
Disallow /

meta-externalfetcher

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

youbot

Rule Path
Disallow /

Other Records

Field Value
sitemap http://www.webstore.com/sitemap.xml

Warnings

  • 2 invalid lines.