Log In Sign Up

tech.mydiv.net
robots.txt

Robots Exclusion Standard data for tech.mydiv.net

Resource Scan

Scan Details

Site Domain tech.mydiv.net
Base Domain mydiv.net
Scan Status Ok
Last Scan2025-07-04T15:38:20+00:00
Next Scan 2025-08-03T15:38:20+00:00

Last Scan

Scanned2025-07-04T15:38:20+00:00
URL https://tech.mydiv.net/robots.txt
Domain IPs 51.91.80.68
Response IP 51.91.80.68
Found Yes
Hash 51cd9dd3b45c3e949ac349d4cf74030cf8ced31c65d0eec77f1eac8b8a6936bb
SimHash 327651e2cee9

Groups

mediapartners-google*

Rule Path
Disallow

google-site-verification

Rule Path
Disallow

adsbot-google

Rule Path
Disallow

googleother

Rule Path
Disallow

chatgpt

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

bard

Rule Path
Disallow /

gemini

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

anthropic

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

sogou-test-spider

Rule Path
Disallow /

openai user-agent

Rule Path
Disallow /

youbot

Rule Path
Disallow /

openai api

Rule Path
Disallow /

openai-rewriter

Rule Path
Disallow /

openai-content-extractor

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

anthropicai

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

perplexity

Rule Path
Disallow /

quora-bot

Rule Path
Disallow /

aicontentdetector

Rule Path
Disallow /

ai writer bot

Rule Path
Disallow /

ai-copywriter

Rule Path
Disallow /

neurobot

Rule Path
Disallow /

copyleaksbot

Rule Path
Disallow /

writerai

Rule Path
Disallow /

aiseo-bot

Rule Path
Disallow /

ink-bot

Rule Path
Disallow /

contentbot

Rule Path
Disallow /

hypotenuse-bot

Rule Path
Disallow /

jasperbot

Rule Path
Disallow /

writesonic

Rule Path
Disallow /

rytrbot

Rule Path
Disallow /

copyai

Rule Path
Disallow /

kafkaibot

Rule Path
Disallow /

scalenutbot

Rule Path
Disallow /

copymatic

Rule Path
Disallow /

growthbar-ai

Rule Path
Disallow /

longshotai

Rule Path
Disallow /

frasebot

Rule Path
Disallow /

surferbot

Rule Path
Disallow /

neuraltextbot

Rule Path
Disallow /

inkforall

Rule Path
Disallow /

scalenutai

Rule Path
Disallow /

zyrobot

Rule Path
Disallow /

outrankingbot

Rule Path
Disallow /

ink-ai

Rule Path
Disallow /

simplifiedai

Rule Path
Disallow /

anywordbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

screaming frog seo spider

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

wotbox

Rule Path
Disallow /

yisouspider

Rule Path
Disallow /

yandeximages

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

sogou orion spider

Rule Path
Disallow /

exabot

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

spbot

Rule Path
Disallow /

dotcrawler

Rule Path
Disallow /

seobilitybot

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

baiduspider-image

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

cocolyzebot

Rule Path
Disallow /

mojeekbot

Rule Path
Disallow /

seekport crawler

Rule Path
Disallow /

nimbostratus-bot

Rule Path
Disallow /

crawler4j

Rule Path
Disallow /

linkpadbot

Rule Path
Disallow /

boardreader favicon fetcher

Rule Path
Disallow /

domain re-animator bot

Rule Path
Disallow /

headlesschrome

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

python-requests

Rule Path
Disallow /

node.js

Rule Path
Disallow /

libwww-perl

Rule Path
Disallow /

axios

Rule Path
Disallow /

go-http-client

Rule Path
Disallow /

java/1.8.0

Rule Path
Disallow /

curl

Rule Path
Disallow /

*

Rule Path
Disallow /search
Disallow /exports/
Disallow /imgs/core/captcha/
Disallow /gf/
Disallow /cgi-bin/
Disallow /clicklink
Disallow /clicksite
Disallow /commentsredir
Disallow /postinfo
Disallow /buy
Disallow /ddl
Disallow /informers
Disallow /sf
Disallow /ajax
Disallow /subsc
Disallow /comments.a
Disallow /reviews.a
Disallow /reviews.ra
Disallow /qans.stmp
Disallow /qans.addq
Disallow /qans.t0
Disallow /qans.r
Disallow /qans.a
Disallow /addscreens
Disallow /dfile
Disallow /dlfile
Disallow /adload
Disallow /tags
Disallow /viewtags
Disallow /buylink
Disallow /collections/show.

yandex

Rule Path
Disallow /search
Disallow /exports/
Disallow /imgs/core/captcha/
Disallow /gf/
Disallow /cgi-bin/
Disallow /clicklink
Disallow /clicksite
Disallow /commentsredir
Disallow /postinfo
Disallow /buy
Disallow /ddl
Disallow /informers
Disallow /sf
Disallow /ajax
Disallow /subsc
Disallow /comments.a
Disallow /reviews.a
Disallow /reviews.ra
Disallow /qans.stmp
Disallow /qans.addq
Disallow /qans.t0
Disallow /qans.r
Disallow /qans.a
Disallow /addscreens
Disallow /dfile
Disallow /dlfile
Disallow /adload
Disallow /tags
Disallow /viewtags
Disallow /buylink
Disallow /collections/show.
Back to top

Comments

  • tech.mydiv.net
  • Disallow AI bots
  • Disallow SEO bots
Back to top

Warnings

  • 2 invalid lines.
  • `clean-param` is not a known field.
Back to top