Log In Sign Up

whatech.com
robots.txt

Robots Exclusion Standard data for whatech.com

Resource Scan

Scan Details

Site Domain whatech.com
Base Domain whatech.com
Scan Status Ok
Last Scan2024-09-23T18:08:32+00:00
Next Scan 2024-09-30T18:08:32+00:00

Last Scan

Scanned2024-09-23T18:08:32+00:00
URL https://whatech.com/robots.txt
Domain IPs 104.26.4.171, 104.26.5.171, 172.67.70.244, 2606:4700:20::681a:4ab, 2606:4700:20::681a:5ab, 2606:4700:20::ac43:46f4
Response IP 104.26.5.171
Found Yes
Hash 8ee6a5628fd266af6c0b47fcaeb7d8e3f9a7694ccc47644a033a240cc8796a66
SimHash 6a535640b4aa

Groups

*

Rule Path
Disallow /

googlebot

Rule Path
Disallow

googlebot-news

Rule Path
Disallow

bingbot

Rule Path
Disallow

slurp

Rule Path
Disallow

duckduckbot

Rule Path
Disallow

baiduspider

Rule Path
Disallow

yandexbot

Rule Path
Disallow

facebot

Rule Path
Disallow

facebookexternalhit

Rule Path
Disallow

chatgptbot

Rule Path
Disallow

applebot

Rule Path
Disallow

*

Rule Path
Allow /*.js$
Allow /*.css$
Allow /*.png$
Allow /*.jpg$
Allow /*.gif$

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

sistrix crawler

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

jobs.de-robot

Rule Path
Disallow /

unisterbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

searchmetricsbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

surveybot

Rule Path
Disallow /

seodiver

Rule Path
Disallow /

spbot

Rule Path
Disallow /

wotbox

Rule Path
Disallow /

meanpathbot

Rule Path
Disallow /

backlinkcrawler

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

obot

Rule Path
Disallow /

fr-crawler

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

megaindex.com

Rule Path
Disallow /

cloudservermarketspider

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

careerbot

Rule Path
Disallow /

lipperhey-kaus-australis

Rule Path
Disallow /

seoscanners.net

Rule Path
Disallow /

metajobbot

Rule Path
Disallow /

spiderbot

Rule Path
Disallow /

linkstats

Rule Path
Disallow /

jobboersebot

Rule Path
Disallow /

iccrawler

Rule Path
Disallow /

plista

Rule Path
Disallow /

domain re-animator bot

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

coccoc

Rule Path
Disallow /

um-ic

Rule Path
Disallow /

mindupbot

Rule Path
Disallow /

sg-orbiter

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

qwantify

Rule Path
Disallow /

kraken

Rule Path
Disallow /

plukkie

Rule Path
Disallow /

safednsbot

Rule Path
Disallow /

haosouspider

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

openhosebot

Rule Path
Disallow /

screaming frog seo spider

Rule Path
Disallow /

thumbsniper

Rule Path
Disallow /

r6_commentreader

Rule Path
Disallow /

implisensebot

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

aihitbot

Rule Path
Disallow /

wbsearchbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

yandex

Rule Path
Disallow /

bubing

Rule Path
Disallow /

cegbfeieh

Rule Path
Disallow /

panscient.com

Rule Path
Disallow /

alphabot

Rule Path
Disallow /

backlinkbot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

bdcbot

Rule Path
Disallow /

blackwidow

Rule Path
Disallow /

blekkobot

Rule Path
Disallow /

linkdexbot

Rule Path
Disallow /

siteexplorer.info

Rule Path
Disallow /

twengabot

Rule Path
Disallow /

wbsearchbot

Rule Path
Disallow /

linkfluence

Rule Path
Disallow /

proximic

Rule Path
Disallow /

pagepeeker

Rule Path
Disallow /

yisouspider

Rule Path
Disallow /

mail.ru_bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

xovibot

Rule Path
Disallow /

alphaseobot

Rule Path
Disallow /

seobilitybot

Rule Path
Disallow /

rankactivelinkbot

Rule Path
Disallow /

jamesbot

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

sentibot

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

jetbot

Rule Path
Disallow /

turingos

Rule Path
Disallow /

grapeshotcrawler

Rule Path
Disallow /

rainbot

Rule Path
Disallow /

admantx

Rule Path
Disallow /

trendiction

Rule Path
Disallow /

catchbot

Rule Path
Disallow /

netcraftsurveyagent

Rule Path
Disallow /

megaindex

Rule Path
Disallow /

paperlibot

Rule Path
Disallow /

showyoubot

Rule Path
Disallow /

zoominfobot

Rule Path
Disallow /

seekport

Rule Path
Disallow /

piplbot

Rule Path
Disallow /

adscanner

Rule Path
Disallow /

a6-indexer

Rule Path
Disallow /

zoombot

Rule Path
Disallow /

komodia

Rule Path
Disallow /

istellabot

Rule Path
Disallow /

webinator

Rule Path
Disallow /

openwebspider

Rule Path
Disallow /

linkwalker

Rule Path
Disallow /

capsabot

Rule Path
Disallow /

cegbot

Rule Path
Disallow /

cyclopsbot

Rule Path
Disallow /

heritrix

Rule Path
Disallow /

yeti

Rule Path
Disallow /

nerdybot

Rule Path
Disallow /

buck

Rule Path
Disallow /

advisorbot

Rule Path
Disallow /

obot

Rule Path
Disallow /

academic

Rule Path
Disallow /

dot

Rule Path
Disallow /

extractbot

Rule Path
Disallow /

re-re studio

Rule Path
Disallow /

proximic

Rule Path
Disallow /

atomic_email_hunter

Rule Path
Disallow /

backlinktest

Rule Path
Disallow /

ccminer

Rule Path
Disallow /

contentcrawler

Rule Path
Disallow /

crawlbot

Rule Path
Disallow /

cyberpatrol

Rule Path
Disallow /

datacha0s

Rule Path
Disallow /

dataprovider

Rule Path
Disallow /

dc-mediatorbot

Rule Path
Disallow /

findxbot

Rule Path
Disallow /

htdig

Rule Path
Disallow /

ichiro

Rule Path
Disallow /

linkpadbot

Rule Path
Disallow /

moreoverbot

Rule Path
Disallow /

niki-bot

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

w3c_validator

Rule Path
Disallow /

yacy

Rule Path
Disallow /

*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30
Back to top

Other Records

Field Value
sitemap https://www.whatech.com/?format=feed&type=rss
sitemap https://www.whatech.com/og/markets-research?format=feed&type=rss
Back to top

Comments

  • Allow specific bots
  • Allow essential resources
  • Disallow known malicious and unwanted bots
  • Sitemap entries
  • Slow down bots
Back to top

Warnings

  • 4 invalid lines.
Back to top