Log In Sign Up

axartoner.com
robots.txt

Robots Exclusion Standard data for axartoner.com

Resource Scan

Scan Details

Site Domain axartoner.com
Base Domain axartoner.com
Scan Status Ok
Last Scan2024-06-29T08:46:02+00:00
Next Scan 2024-07-29T08:46:02+00:00

Last Scan

Scanned2024-06-29T08:46:02+00:00
URL https://axartoner.com/robots.txt
Redirect https://www.axartoner.com/robots.txt
Redirect Domain www.axartoner.com
Redirect Base axartoner.com
Domain IPs 146.59.208.75
Redirect IPs 146.59.208.75
Response IP 146.59.208.75
Found Yes
Hash 73715a1b02f3f75bea0b73b3e5a7b4460b4193bba6276ab44fe51aeead8dd2d1
SimHash 915ad9104e53

Groups

*

Rule Path
Allow /
Disallow /includes/
Disallow /cgi-bin/
Disallow /ext/
Disallow /googlesitemap/
Disallow /upload/
Disallow /feed/
Disallow /404.php
Disallow /popup_image.php
Disallow /popup_search_help.php
Disallow /account.php
Disallow /account_edit.php
Disallow /account_history.php
Disallow /account_history_info.php
Disallow /account_password.php
Disallow /advanced_search.php*
Disallow /add_checkout_success.php
Disallow /address_book.php
Disallow /address_book_process.php
Disallow /advanced_search.php
Disallow /advanced_search_result.php
Disallow /advanced_search_result.php*
Disallow /comparador.php
Disallow /checkout_confirmation.php
Disallow /checkout_payment.php
Disallow /checkout_payment_address.php
Disallow /checkout_process.php
Disallow /checkout_shipping.php
Disallow /checkout_shipping_address.php
Disallow /checkout_success.php
Disallow /contact_bean.php
Disallow /cookie_usage.php
Disallow /create_account.php
Disallow /create_account_success.php
Disallow /help_form.php
Disallow /login.php
Disallow /notify.php
Disallow /password_forgotten.php
Disallow /popup_image.php
Disallow /popup_categories.php
Disallow /popup_categories.php?*
Disallow /product_compats.php
Disallow /shopping_cart.php
Disallow /tellamamos.php
Disallow /product_reviews_write.php
Disallow /product_compats.php
Disallow /product_compats.php?*
Disallow /index.php*
Disallow /ficha-duda.php*
Disallow /ficha-barato.php*
Disallow /*action%3D*

msiecrawler

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

httrack

Rule Path
Disallow /

microsoft.url.control

Rule Path
Disallow /

libwww

Rule Path
Disallow /

linguee

Rule Path
Disallow /

archive.org

Rule Path
Disallow /

kalooga

Rule Path
Disallow /

spbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

sbider

Rule Path
Disallow /

sitebot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

noxtrumbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 50

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 50

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

baiduspider

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.axartoner.com/sitemapindex.xml
Back to top

Comments

  • Lista de bots que suelen respetar el robots.txt pero rara
  • vez hacen un buen uso del sitio y abusan bastante
  • Añadir al gusto del consumidor
  • Slurp (Yahoo!), Noxtrum y el bot de MSN a veces tienen
  • idas de pinza, toca decirles que reduzcan la marcha.
  • El valor es en segundos y podeis dejarlo bajo e ir
  • subiendo hasta el punto optimo.
  • SEO-related bots
Back to top