diasporarecords.com
robots.txt
Robots Exclusion Standard data for diasporarecords.com
Resource Scan
Scan Details
| Site Domain | diasporarecords.com |
| Base Domain | diasporarecords.com |
| Scan Status | Ok |
| Last Scan | 2024-10-17T20:16:28+00:00 |
| Next Scan | 2024-11-16T20:16:28+00:00 |
Last Scan
| Scanned | 2024-10-17T20:16:28+00:00 |
| URL | http://diasporarecords.com/robots.txt |
| Domain IPs | 195.167.226.21, 195.167.226.28 |
| Response IP | 195.167.226.28 |
| Found | Yes |
| Hash | 48dbb95c8fe5a71ae00a87d0a72687dfba14918d0b065cb4ec9f5dabd5b39b77 |
| SimHash | 72776070c281 |
Groups
bingbot
adidxbot
bingpreview
microsoftpreview
gptbot
chatgpt-user
oai-searchbot
applebot
semrushbot
pinterestbot
ia_archiver
No rules defined. All paths allowed.
Other Records
| Field | Value |
|---|---|
| crawl-delay | 120 |
facebookexternalhit/1.1
facebookexternalhit/*
meta-externalagent/1.1
meta-externalagent/*
No rules defined. All paths allowed.
Other Records
| Field | Value |
|---|---|
| crawl-delay | 120 |
twiceler
pompos
semanticdiscovery
gaisbot
qihoobot
gigabot
shopwiki
ngbot/4.5
sosoimagespider
voilabot
httrack
httrack 3.0
winhttrack
voyager
dotbot
turnitinbot
turnitinbot/2.1
ahrefsbot
auto spider 1.0
exabot
blexbot
exabot-thumbnails
megaindex
petalbot
serpstatbot
linespider
ccbot/1.0
mj12bot
webalta crawler/2.0
boitho.com-dc
fatbot
discobot
sosospider
baiduspider
mlbot
bot/1.0
cazoodlebot
seekbot
becomebot
asterias
snapbot
ichiro
irlbot
searchdaimon.com-dc
yanga worldsearch bot
sapphirewebcrawler/1.0
yeti/1.0
twengabot
speedy
yandex
dataforseobot
bytespider
bytedance
y!j-bru/vsidx
y!j-brw
y!j-asr
y!j-mmp/dscv
y!j-wsc
panscient.com
| Rule | Path |
|---|---|
| Disallow | / |
*
| Rule | Path |
|---|---|
| Disallow | /admin/ |
| Disallow | /CFIDE/ |
| Disallow | /caddie/ |
| Disallow | /commande/ |
| Disallow | /checkout/ |
| Disallow | /cart/ |
| Disallow | /gestion/ |
| Disallow | /moteur_de_rech/ |
| Disallow | /signup/ |
| Disallow | /login/ |
| Disallow | /vendre/ |
| Disallow | /*/get_picture_http_to_https.cfm |
| Disallow | /*/search_others_items_others_sellers_bottom_page.cfm |
| Disallow | /*/search_others_items_and_artists_same_seller_new.cfm |
| Disallow | /*/search_others_items_same_seller.cfm |
| Disallow | /*/search_others_items_others_sellers.cfm |
| Disallow | /*/search_other_picture_from_other_sellers.cfm |
| Disallow | /*/affiliatebanners.cfm |
| Disallow | /*?t= |
| Disallow | /*%26t%3D |
| Disallow | /*?l= |
| Disallow | /*%26l%3D |