Log In Sign Up

wakeupbodynutrition.com
robots.txt

Robots Exclusion Standard data for wakeupbodynutrition.com

Resource Scan

Scan Details

Site Domain wakeupbodynutrition.com
Base Domain wakeupbodynutrition.com
Scan Status Ok
Last Scan2024-09-10T18:52:00+00:00
Next Scan 2024-10-10T18:52:00+00:00

Last Scan

Scanned2024-09-10T18:52:00+00:00
URL https://wakeupbodynutrition.com/robots.txt
Domain IPs 104.26.12.54, 104.26.13.54, 172.67.68.157, 2606:4700:20::681a:c36, 2606:4700:20::681a:d36, 2606:4700:20::ac43:449d
Response IP 172.67.68.157
Found Yes
Hash 4724fffa168daf23f3280e83af490cf8557488c24c3df158a8e59000eb3ed28c
SimHash 6019eb12c2d1

Groups

*

Rule Path
Disallow /wp-content/uploads/wc-logs/
Disallow /wp-content/uploads/woocommerce_transient_files/
Disallow /wp-content/uploads/woocommerce_uploads/
Disallow /*add-to-cart%3D*
Disallow /*add_to_wishlist%3D*
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php
Disallow /thanks/
Disallow /*page/
Disallow /author/
Disallow /category/
Disallow /cart/
Disallow /checkout/
Disallow /my-account/
Disallow /portfolio_page/
Disallow /orders/
Disallow /search
Disallow /wishlist/

googlebot

Rule Path
Allow /

googlebot-mobile

Rule Path
Allow /

googlebot-news

Rule Path
Allow /

googlebot-video

Rule Path
Allow /

storebot-google

Rule Path
Allow /

adsbot-google

Rule Path
Allow /

bingbot

Rule Path
Allow /

msnbot

Rule Path
Allow /

msnbot-media

Rule Path
Allow /

adldxbot

Rule Path
Allow /

googlebot-image

Rule Path
Allow /*

ia_archiver

Rule Path
Disallow /

duggmirror

Rule Path
Disallow /

googlebot

Rule Path
Disallow /*.inc$
Disallow /*.cgi$
Disallow /*.wmv$
Disallow /*.ico$
Disallow /*.opml$
Disallow /*.shtml$
Disallow /*.jpg$
Disallow /*.cgi$
Disallow /*.xhtml$

mediapartners-google*

Rule Path
Disallow
Allow /*

google plus share

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

yandex

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

yandexbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

yandex bot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

duckduckbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

baiduspider

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

baiduspider-image

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

baiduspider-mobile

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

baiduspider-news

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

baiduspider-video

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

sogou

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

soso spider

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

exabot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

sogou spider

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

seekportbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

facebook external hit

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

google feedfetcher

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

pinterest

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

dotbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

ahrefs

Rule Path
Disallow /

majestic

Rule Path
Disallow /
Back to top

Comments

  • allow google image bot to search all images
  • disallow archiving site
  • disable duggmirror
  • disallow all files ending with these extensions
  • Google AdSense
  • Slow down a bit
  • And block bots that keep hitting us too hard / do not obey the disallow/nofollows:
Back to top