Log In Sign Up

dihuni.com
robots.txt

Robots Exclusion Standard data for dihuni.com

Resource Scan

Scan Details

Site Domain dihuni.com
Base Domain dihuni.com
Scan Status Ok
Last Scan2024-09-05T11:01:01+00:00
Next Scan 2024-10-05T11:01:01+00:00

Last Scan

Scanned2024-09-05T11:01:01+00:00
URL https://dihuni.com/robots.txt
Domain IPs 192.124.249.14
Response IP 192.124.249.14
Found Yes
Hash 5d772551f21e769d485473880fcecf96b6ab9d8d1a0bc51f2ef65f3b83375465
SimHash d60fde7e8a93

Groups

*

Rule Path
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php
Disallow /*?filter_
Disallow /page/*/
Disallow /shop/page/*/
Disallow /shop/page/
Disallow /shop/page
Disallow /*wc-ajax%3Dget_refreshed_fragments
Disallow /cart/
Disallow /checkout/
Disallow /*?min_price=
Disallow /*?max_price=
Disallow /wp-content/plugins/
Disallow /products/corsair/
Disallow /brand/
Disallow /product-type/
Disallow /product-family/
Disallow /cpu-platform/
Disallow /wp-json/
Disallow /*add-to-cart%3D*
Disallow /my-account/
Disallow /*sitemap.txt
Disallow /?add-to-cart=*
Disallow /*?add-to-cart=*
Disallow /?add-to-cart=
Disallow /*?add-to-cart=

stress-agent

Rule Path
Disallow /

dataforseobot/1.0

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

msiecrawler

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

microsoft url control

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

seekportbot

Rule Path
Disallow /

baiduspider

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

baiduspider

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

yandex

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

mj12bot

Rule Path
Disallow /

yandexbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

yandex*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

etaospider

Rule Path
Disallow /

psbot

Rule Path
Disallow /

linkdexbot

Rule Path
Disallow /

seznambot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

shopwiki

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 15

sosospider

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 300

baiduspider

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

easouspider

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

etaospider

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

nuclei

Rule Path
Disallow /

wikido

Rule Path
Disallow /

riddler

Rule Path
Disallow /

go-http-client

Rule Path
Disallow /

node/simplecrawler

Rule Path
Disallow /

cazoodlebot

Rule Path
Disallow /

dotbot/1.0

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

brightedge

Rule Path
Disallow /

brightedge

Rule Path
Disallow /

brightedge crawler/1.0 (crawler@brightedge.com)

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.dihuni.com/shop/feed/atom/
Back to top