Log In Sign Up

ncfitnessgear.com.au
robots.txt

Robots Exclusion Standard data for ncfitnessgear.com.au

Resource Scan

Scan Details

Site Domain ncfitnessgear.com.au
Base Domain ncfitnessgear.com.au
Scan Status Ok
Last Scan2024-06-09T06:51:52+00:00
Next Scan 2024-07-09T06:51:52+00:00

Last Scan

Scanned2024-06-09T06:51:52+00:00
URL https://ncfitnessgear.com.au/robots.txt
Domain IPs 172.66.40.146, 172.66.43.110, 2606:4700:3108::ac42:2892, 2606:4700:3108::ac42:2b6e
Response IP 172.66.40.146
Found Yes
Hash b6b6360fe580aea8cd6a7bfd9b2420a8c3e359195a01a5b86d8e2864e3d7b4b9
SimHash 0a765c63579b

Groups

*

Rule Path
Disallow /wp-admin
Disallow /checkout/
Disallow /cart/
Allow /wp-content/uploads/
Allow /*.js*
Allow /*.css*
Allow /*.JS*
Allow /*.CSS*

adsbot-google

Rule Path
Disallow /checkout
Disallow /cart
Allow /*.js*
Allow /*.css*
Allow /*.JS*
Allow /*.CSS*

nutch

Rule Path
Disallow /

ahrefssiteaudit

Rule Path
Disallow /wp-admin
Disallow /cart
Disallow /checkout

Other Records

Field Value
crawl-delay 10

ahrefsbot

Rule Path
Disallow /wp-admin
Disallow /cart
Disallow /checkout

Other Records

Field Value
crawl-delay 10

mj12bot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

pinterest

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

a6-indexer

Rule Path
Disallow /

alphaseobot

Rule Path
Disallow /

alphaseobot-sa

Rule Path
Disallow /

aspiegelbot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

blackboard safeassign

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

crawler4j

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

liebaofast

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

mauibot (crawler.feedback+wc@gmail.com)

Rule Path
Disallow /

megaindex.ru/2.0

Rule Path
Disallow /

mqqbrowser

Rule Path
Disallow /

nimbostratus-bot/v1.3.2

Rule Path
Disallow /

qwant-news

Rule Path
Disallow /

qwantify

Rule Path
Disallow /

seekport crawler

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

sputnikbot/2.3

Rule Path
Disallow /

the knowledge ai

Rule Path
Disallow /

timpibot/0.8

Rule Path
Disallow /

tinytestbot

Rule Path
Disallow /

ucbrowser

Rule Path
Disallow /

yacybot

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

yandexbot/3.0

Rule Path
Disallow /

yeti

Rule Path
Disallow /

yisouspider

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.ncfitnessgear.com.au/sitemap_index.xml
Back to top

Comments

  • Google adsbot ignores robots.txt unless specifically named!
  • Block bots
  • RDH, 03/11/22:
  • Comment this out for JOT, who applied for a Crossref Similiarty Check account with TurnitIn;
  • User-agent: TurnitinBot
  • Disallow: /
Back to top