hotrodtime.com
robots.txt

Robots Exclusion Standard data for hotrodtime.com

Resource Scan

Scan Details

Site Domain hotrodtime.com
Base Domain hotrodtime.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2024-10-29T18:36:50+00:00
Next Scan 2025-01-27T18:36:50+00:00

Last Successful Scan

Scanned2023-10-06T17:36:37+00:00
URL https://hotrodtime.com/robots.txt
Domain IPs 50.31.98.241
Response IP 50.31.98.241
Found Yes
Hash 01fcb2e3be8c1c7083990bb3604108af41cbd8300c874d3931bdd55de2232666
SimHash e804c2f8eff4

Groups

*

Rule Path
Allow /*.js*
Allow /*.css*
Allow /*.png*
Allow /*.jpg*
Allow /*.gif*

special_archiver/3.1.1

Rule Path
Disallow /

awariosmartbot/1.0

Rule Path
Disallow /

bitlybot/3.0

Rule Path
Disallow /

blexbot/1.0

Rule Path
Disallow /

brandverity/1.0

Rule Path
Disallow /

ccbot/2.0

Rule Path
Disallow /

coccocbot-web/1.0

Rule Path
Disallow /

researchscan/t13rl

Rule Path
Disallow /

researchscan/t12ca

Rule Path
Disallow /

researchscan/t12sns

Rule Path
Disallow /

curious george

Rule Path
Disallow /

datagnionbot

Rule Path
Disallow /

datanyze

Rule Path
Disallow /

daum/4.1

Rule Path
Disallow /

diffbot/0.1

Rule Path
Disallow /

discordbot/2.0

Rule Path
Disallow /

domainstatsbot/1.0

Rule Path
Disallow /

dotbot/1.1

Rule Path
Disallow /

dow jones searchbot

Rule Path
Disallow /

easybib autocite

Rule Path
Disallow /

embedly/0.2

Rule Path
Disallow /

evc-batch/2.0

Rule Path
Disallow /

exabot/3.0

Rule Path
Disallow /

exabot-thumbnails

Rule Path
Disallow /

arquivo-web-crawler

Rule Path
Disallow /

flipboardproxy/1.2

Rule Path
Disallow /

garlikcrawler/1.2

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

grapeshotcrawler/2.0

Rule Path
Disallow /

houzzbot

Rule Path
Disallow /

hubspot marketing grader

Rule Path
Disallow /

adstxtcrawler/1.0

Rule Path
Disallow /

iframely/1.3.1

Rule Path
Disallow /

jooblebot/2.0

Rule Path
Disallow /

linkedinbot/1.0

Rule Path
Disallow /

ltx71

Rule Path
Disallow /

lumtelbot/1.0

Rule Path
Disallow /

magibot/1.0.0

Rule Path
Disallow /

mail.ru_bot/2.0

Rule Path
Disallow /

mail.ru_bot/img/2.0

Rule Path
Disallow /

mediatoolkitbot

Rule Path
Disallow /

mj12bot/v1.4.8

Rule Path
Disallow /

yeti/1.1

Rule Path
Disallow /

panscient.com

Rule Path
Disallow /

paperlibot/2.1

Rule Path
Disallow /

pingdom.com_bot 1.4

Rule Path
Disallow /

pingdompagespeed/1.0

Rule Path
Disallow /

piplbot

Rule Path
Disallow /

pocketparser/2.0

Rule Path
Disallow /

proximic

Rule Path
Disallow /

pubmatic crawler bot

Rule Path
Disallow /

ravencrawler/2.0

Rule Path
Disallow /

rival iq

Rule Path
Disallow /

scrapy/1.5.1

Rule Path
Disallow /

seznambot/3.2

Rule Path
Disallow /

linkcheck

Rule Path
Disallow /

skypeuripreview0.5

Rule Path
Disallow /

slackbot-linkexpanding 1.0

Rule Path
Disallow /

smtbot/1.0

Rule Path
Disallow /

sogou web spider/4.0

Rule Path
Disallow /

superpagesurlverifybot/1.0

Rule Path
Disallow /

traackr.com

Rule Path
Disallow /

trendictionbot 0.5.0

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

tweetmemebot/4.0

Rule Path
Disallow /

um-ln/1.0

Rule Path
Disallow /

uptimerobot/2.0

Rule Path
Disallow /

velenpublicwebcrawler/1.0

Rule Path
Disallow /

virustotalcloud

Rule Path
Disallow /

weborama-fetcher

Rule Path
Disallow /

wordpress.com mshots

Rule Path
Disallow /

linkbot 1.0

Rule Path
Disallow /

zoominfo bot

Rule Path
Disallow /
Disallow /administrator/
Disallow /bin/
Disallow /cache/
Disallow /cli/
Allow /components/*.css
Allow /components/*.js
Allow /includes/*.css
Allow /includes/*.js
Disallow /includes/
Disallow /installation/
Disallow /language/
Allow /layouts/*.css
Allow /layouts/*.js
Disallow /layouts/
Allow /libraries/*.css
Allow /libraries/*.js
Disallow /libraries/
Disallow /logs/
Allow /modules/*.css
Allow /modules/*.js
Allow /plugins/*.css
Allow /plugins/*.js
Disallow /tmp/

*

Rule Path
Allow /sitemap-4seo.xml

Other Records

Field Value
sitemap https://www.hotrodtime.com/sitemap.xml
sitemap https://www.hotrodtime.com/sitemap-4seo.xml

Comments

  • 4SEO-sitemap: sitemap - added on 2021-09-09 21:45:01 - Please do not modify manually.
  • /4SEO-sitemap

Warnings

  • 2 invalid lines.