Log In Sign Up

truckertotrucker.com
robots.txt

Robots Exclusion Standard data for truckertotrucker.com

Resource Scan

Scan Details

Site Domain truckertotrucker.com
Base Domain truckertotrucker.com
Scan Status Ok
Last Scan2024-06-01T16:52:06+00:00
Next Scan 2024-06-08T16:52:06+00:00

Last Scan

Scanned2024-06-01T16:52:06+00:00
URL https://truckertotrucker.com/robots.txt
Domain IPs 104.26.6.219, 104.26.7.219, 172.67.72.146, 2606:4700:20::681a:6db, 2606:4700:20::681a:7db, 2606:4700:20::ac43:4892
Response IP 172.67.72.146
Found Yes
Hash ae76f0d7f34dcf947a15e54ee21b58474e2f43346b007489dca7bcb9323e23cb
SimHash 60c2106ac1a2

Groups

adbeat_bot

Rule Path
Disallow /

agentlinkspammer

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

ahrefsbot/4.0

Rule Path
Disallow /

aihitbot

Rule Path
Disallow /

aihitbot/1.0

Rule Path
Disallow /

aihitbot/1.1

Rule Path
Disallow /

acoon

Rule Path
Disallow /

arachmo

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

baiduspider+

Rule Path
Disallow /

baiduspider+(+http://www.baidu.com/search/spider.htm)

Rule Path
Disallow /

baiduspider/2.0;+http://www.baidu.com/search/spider.html

Rule Path
Disallow /

baiduspider/2.0

Rule Path
Disallow /

mozilla/5.0(compatible; baiduspider/2.0; +http://www.baidu.com/search/spider.html)

Rule Path
Disallow /

careerbot

Rule Path
Disallow /

comodospider/nutch-1.2

Rule Path
Disallow /

easouspider

Rule Path
Disallow /

exabot/3.0

Rule Path
Disallow /

exalead

Rule Path
Disallow /

exalead crawler

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

ezooms/1.0

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

mozilla/5.0 (compatible; ezooms/1.0; ezooms.bot[at]gmail[dot]com)

Rule Path
Disallow /

findlinks/2.6

Rule Path
Disallow /

java/1.6.0_04

Rule Path
Disallow /

jikespider

Rule Path
Disallow /

kaloogabot

Rule Path
Disallow /

mail.ru_bot/2.0

Rule Path
Disallow /

mail.ru

Rule Path
Disallow /

mail.ru

Rule Path
Disallow /

mail.ru_bot/2.0; +http://go.mail.ru/help/robots

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

mj12bot/v1.4.3

Rule Path
Disallow /

moget

Rule Path
Disallow /

ichiro

Rule Path
Disallow /

ichiro 3.0

Rule Path
Disallow /

naverbot

Rule Path
Disallow /

yeti

Rule Path
Disallow /

netcraftsurveyagent/1.0

Rule Path
Disallow /

openwebindex/nutch-1.6

Rule Path
Disallow /

openwebindex

Rule Path
Disallow /

panoptastudybot

Rule Path
Disallow /

checks.panopta.com

Rule Path
Disallow /

psbot

Rule Path
Disallow /

plukkie

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

seznambot/1.0

Rule Path
Disallow /

seznambot/1.1

Rule Path
Disallow /

seznambot/3.0

Rule Path
Disallow /

sistrixcrawler

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

sistrix crawler

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

sosospider+

Rule Path
Disallow /

sosospider

Rule Path
Disallow /

sosospider/2.0

Rule Path
Disallow /

surveybot

Rule Path
Disallow /

wada.vn

Rule Path
Disallow /

wada.vn vietnamese search

Rule Path
Disallow /

wada.vn vietnamese search/2.1

Rule Path
Disallow /

yandex

Rule Path
Disallow /

yandex/1.01.001

Rule Path
Disallow /

yandexbot/3.0-mirrordetector

Rule Path
Disallow /

yandeximages/3.0

Rule Path
Disallow /

yandexsomething/1.

Rule Path
Disallow /

yandex.com

Rule Path
Disallow /

yandexbot/3.0

Rule Path
Disallow /

yisouspider

Rule Path
Disallow /

youdaobot/1.0

Rule Path
Disallow /

youdaobot/1.0

Rule Path
Disallow /

zao

Rule Path
Disallow /

*

Rule Path
Disallow /adserver/
Disallow /affiliate/
Disallow /aspnet_client/
Disallow /cart/
Disallow /CFIDE/
Disallow /cgi-bin/
Disallow /com/
Disallow /console/
Disallow /forms/
Disallow /fuseguard/
Disallow /greybox/
Disallow /help/
Disallow /includes/
Disallow /js/
Disallow /listings/
Disallow /partners/
Disallow /pics/
Disallow /products/
Disallow /redirect/
Disallow /resources/
Disallow /search-results/
Disallow /sched/
Disallow /site/login.cfm
Disallow /styles/
Disallow /tags/
Disallow /templates/
Disallow /test/
Disallow /trucking-equipment/
Disallow /truckertotrucker/
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php
Disallow /ws/
Disallow /*.cfm
Back to top

Other Records

Field Value
sitemap https://truckertotrucker.com/sitemaps/index.xml
sitemap https://truckertotrucker.com/sitemaps/blog/index.xml
Back to top

Comments

  • Adbeat ads
  • AgentLinkSpammer
  • AhrefsBot ads
  • aiHitBot Ukraine or Russia
  • Acoon Germany
  • Arachmo Japan
  • Baiduspider China and Japan
  • careerbot Germany
  • COMODOSpider/Nutch-1.2 United Kingdom
  • EasouSpider - China
  • Exabot/3.0 - France proxy scraper
  • Exalead proxy scraper France
  • Ezooms and dotbot
  • findlinks/2.6 Germany http://wortschatz.uni-leipzig.de/findlinks
  • Java/1.6.0_04
  • JikeSpider China
  • KaloogaBot Netherlands contextual advertising
  • Mail.RU_Bot/2.0 Russia
  • Mail.RU Russia
  • Mail.Ru Russia
  • MJ12bot United Kingdom
  • MJ12bot/v1.4.3 United Kingdon
  • Ichiro Japan
  • Ichiro 3.0 Japan
  • NetcraftSurveyAgent/1.0
  • OpenWebIndex/Nutch-1.6 Germany
  • panoptaStudyBot checks.panopta.com monitor
  • panoptaStudyBot checks.panopta.com monitor
  • picsearch Sweden searches for pictures
  • plukkie Dutch (botje.nl)/Belgium (botje.be)/France (botje.fr)/United Kingdom (botje.co.uk) search engine
  • SeznamBot Czech Republic
  • SeznamBot/3.0
  • SistrixCrawler Germany DE
  • Sogou
  • Sosospider - China http://help.soso.com/webspider.htm
  • Sosospider - China
  • Sosospider/2.0 - China may not obey robots.txt
  • 360Spider China
  • SurveyBot
  • Wada.vn Vietnamese Search/2.1
  • Yandex
  • YisouSpider China
  • YoudaoBot/1.0 China
  • YoudaoBot China
  • Zao - Japan
  • Disallow: /images-inter/
  • Disallow: /images-secondary/
  • Disallow: /images-sell-truck/
  • Disallow: /images-truckerpass/
  • Disallow: /net0/
  • Disallow: /support/
  • Disallow: /trailers/
  • Disallow: /trucks/
  • Disallow: /wp-content/
  • Disallow: /wp-includes/
  • Remove cfm pages 12/15/23
Back to top

Warnings

  • 8 invalid lines.
Back to top