Log In Sign Up

cultfurniture.com
robots.txt

Robots Exclusion Standard data for cultfurniture.com

Resource Scan

Scan Details

Site Domain cultfurniture.com
Base Domain cultfurniture.com
Scan Status Ok
Last Scan2024-10-22T02:23:13+00:00
Next Scan 2024-11-21T02:23:13+00:00

Last Scan

Scanned2024-10-22T02:23:13+00:00
URL https://cultfurniture.com/robots.txt
Redirect https://www.cultfurniture.com/robots.txt
Redirect Domain www.cultfurniture.com
Redirect Base cultfurniture.com
Domain IPs 104.22.50.143, 104.22.51.143, 172.67.12.179, 2606:4700:10::6816:328f, 2606:4700:10::6816:338f, 2606:4700:10::ac43:cb3
Redirect IPs 104.22.50.143, 104.22.51.143, 172.67.12.179, 2606:4700:10::6816:328f, 2606:4700:10::6816:338f, 2606:4700:10::ac43:cb3
Response IP 104.22.50.143
Found Yes
Hash b7c453dc4bfcfcf08f7291caaa3954f536e74322edcf11e69d7b89cbb071f75e
SimHash 700f9670261e

Groups

etaospider

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

brandverity

Rule Path
Disallow /

seekport crawler

Rule Path
Disallow /

seekportbot

Rule Path
Disallow /

blexbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

mj12bot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 60

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

ahrefssiteaudit

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

semrushbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

dataforseobot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

pricebot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

dotbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

charlotte

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

pinterestbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

seznambot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

*

Rule Path
Allow /
Disallow /ajax/removefrombasket/
Disallow /basket$
Disallow /checkout/
Disallow /errors/
Disallow /quickbuy/
Disallow /product_personalisation/
Disallow /register/
Disallow /register
Disallow /vs/
Disallow /images/products/yofla/
Disallow /customer_groups/choose_group/
Disallow /formbuilder/
Disallow /callbacks/
Disallow /ajax/formatprice/
Disallow /products/
Disallow /*under-25-gbp-b1$
Disallow /*25-to-50-gbp-b2$
Disallow /*50-to-75-gbp-b3$
Disallow /*75-to-100-gbp-b4$
Disallow /*100-to-200-gbp-b5
Disallow /*200-to-300-gbp-b6$
Disallow /*300-to-400-gbp-b7$
Disallow /upselling/show/
Disallow /search/

Other Records

Field Value
crawl-delay 2

twitterbot
googlebot
googlebot-image

Rule Path
Allow /
Back to top

Other Records

Field Value
sitemap https://www.cultfurniture.com/sitemap-index.xml
Back to top

Warnings

  • `request-rate` is not a known field.
Back to top