Log In Sign Up

comsoc.org
robots.txt

Robots Exclusion Standard data for comsoc.org

Resource Scan

Scan Details

Site Domain comsoc.org
Base Domain comsoc.org
Scan Status Ok
Last Scan2024-10-21T16:00:00+00:00
Next Scan 2024-11-20T16:00:00+00:00

Last Scan

Scanned2024-10-21T16:00:00+00:00
URL https://comsoc.org/robots.txt
Redirect https://www.comsoc.org/robots.txt
Redirect Domain www.comsoc.org
Redirect Base comsoc.org
Domain IPs 18.210.113.138
Redirect IPs 18.210.113.138
Response IP 18.210.113.138
Found Yes
Hash 014636c4a4739569220aee9f3764ac41c9a62d209c52de830beef29f7d3fc567
SimHash 3d94bf50866c

Groups

*

Rule Path
Allow /core/*.css$
Allow /core/*.css?
Allow /core/*.js$
Allow /core/*.js?
Allow /core/*.gif
Allow /core/*.jpg
Allow /core/*.jpeg
Allow /core/*.png
Allow /core/*.svg
Allow /profiles/*.css$
Allow /profiles/*.css?
Allow /profiles/*.js$
Allow /profiles/*.js?
Allow /profiles/*.gif
Allow /profiles/*.jpg
Allow /profiles/*.jpeg
Allow /profiles/*.png
Allow /profiles/*.svg
Disallow /core/
Disallow /profiles/
Disallow /README.md
Disallow /composer/Metapackage/README.txt
Disallow /composer/Plugin/ProjectMessage/README.md
Disallow /composer/Plugin/Scaffold/README.md
Disallow /composer/Plugin/VendorHardening/README.txt
Disallow /composer/Template/README.txt
Disallow /modules/README.txt
Disallow /sites/README.txt
Disallow /themes/README.txt
Disallow /web.config
Disallow /admin/
Disallow /comment/reply/
Disallow /filter/tips
Disallow /node/add/
Disallow /search/
Disallow /user/register
Disallow /user/password
Disallow /user/login
Disallow /user/logout
Disallow /media/oembed
Disallow /*/media/oembed
Disallow /index.php/admin/
Disallow /index.php/comment/reply/
Disallow /index.php/filter/tips
Disallow /index.php/node/add/
Disallow /index.php/search/
Disallow /index.php/user/password
Disallow /index.php/user/register
Disallow /index.php/user/login
Disallow /index.php/user/logout
Disallow /index.php/media/oembed
Disallow /index.php/*/media/oembed

gptbot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

chatgpt

Rule Path
Disallow /

openai

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

applebot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

youbot

Rule Path
Disallow /

zumbot

Rule Path
Disallow /

wotbox

Rule Path
Disallow /

spbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

x-ms-client-application:

Rule Path
Disallow /

python-requests/2.24.0

Rule Path
Disallow /

nuclei

Rule Path
Disallow /

dotbot/1.1

Rule Path
Disallow /

aria2/1.34.0

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

dotbot

Rule Path
Disallow /
Disallow /

golden

Rule Path
Disallow /

gulper

Rule Path
Disallow /

magus

Rule Path
Disallow /

miralinks

Rule Path
Disallow /

piepmatz

Rule Path
Disallow /

seobilitybot

Rule Path
Disallow /

snap

Rule Path
Disallow /

space

Rule Path
Disallow /

superfeedr

Rule Path
Disallow /

xing

Rule Path
Disallow /

mediacloud

Rule Path
Disallow /

rss2tg

Rule Path
Disallow /

wp.com

Rule Path
Disallow /

yacybot

Rule Path
Disallow /

bomborabot

Rule Path
Disallow /

rytebot

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

_zbot

Rule Path
Disallow /

adsbot/3.1

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

anderspinkbot

Rule Path
Disallow /

atomseobot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

cltbot

Rule Path
Disallow /

chatgpt

Rule Path
Disallow /

checkbot

Rule Path
Disallow /

cincraw

Rule Path
Disallow /

citation_bot

Rule Path
Disallow /

codegetterbot

Rule Path
Disallow /

crowdtanglebot

Rule Path
Disallow /

crsspxlbot

Rule Path
Disallow /

cuberssreader

Rule Path
Disallow /

danibot

Rule Path
Disallow /

dark_matter_bot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

deskyobot

Rule Path
Disallow /

diffeobot

Rule Path
Disallow /

dingtalkbot-linkservice

Rule Path
Disallow /

discordbot

Rule Path
Disallow /

dubbotbot

Rule Path
Disallow /

duckduckbot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

folkbot

Rule Path
Disallow /

fuze_bot

Rule Path
Disallow /

gaisbot

Rule Path
Disallow /

getdigestbot

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

girafabot

Rule Path
Disallow /

gnowitnewsbot

Rule Path
Disallow /

go-http-client

Rule Path
Disallow /

hatenablog-bot

Rule Path
Disallow /

hotjava

Rule Path
Disallow /

infotigerbot

Rule Path
Disallow /

java-http-client

Rule Path
Disallow /

linkisbot

Rule Path
Disallow /

linkpadbot

Rule Path
Disallow /

livelapbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

magibot

Rule Path
Disallow /

mail.ru_bot

Rule Path
Disallow /

mediumbot

Rule Path
Disallow /

mojeekbot

Rule Path
Disallow /

monsidobot

Rule Path
Disallow /

moodlebot

Rule Path
Disallow /

neevabot

Rule Path
Disallow /

netpeakcheckerbot

Rule Path
Disallow /

nimbostratus-bot

Rule Path
Disallow /

onbbot

Rule Path
Disallow /

onalyticabot

Rule Path
Disallow /

paperlibot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

phractusbot

Rule Path
Disallow /

pleroma

Rule Path
Disallow /

quora-bot

Rule Path
Disallow /

rasabot

Rule Path
Disallow /

redirectbot

Rule Path
Disallow /

refindbot

Rule Path
Disallow /

researchbot

Rule Path
Disallow /

rightintelbot

Rule Path
Disallow /

riverbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

serptimizerbot

Rule Path
Disallow /

smtbot

Rule Path
Disallow /

solofield

Rule Path
Disallow /

sabsimbot

Rule Path
Disallow /

scribbr-citation-bot

Rule Path
Disallow /

semanticscholarbot

Rule Path
Disallow /

semanticbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

serendeputybot

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

sirdatabot

Rule Path
Disallow /

siteanalyzerbot

Rule Path
Disallow /

siteauditbot

Rule Path
Disallow /

sitecheckerbotcrawler

Rule Path
Disallow /

slack-imgproxy

Rule Path
Disallow /

slackbot

Rule Path
Disallow /

sottopop

Rule Path
Disallow /

superbot

Rule Path
Disallow /

surdotlybot

Rule Path
Disallow /

synologychatbot

Rule Path
Disallow /

tmmbot

Rule Path
Disallow /

tsmbot

Rule Path
Disallow /

telegrambot

Rule Path
Disallow /

terkkobot

Rule Path
Disallow /

tineye-bot

Rule Path
Disallow /

tkbot

Rule Path
Disallow /

triplecheckerrobot

Rule Path
Disallow /

tulipchain

Rule Path
Disallow /

uptimerobot

Rule Path
Disallow /

v-bot

Rule Path
Disallow /

webbot

Rule Path
Disallow /

wellknownbot

Rule Path
Disallow /

xlinkbotreverter

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

yandexrenderresourcesbot

Rule Path
Disallow /

yandexnews

Rule Path
Disallow /

yandexrca

Rule Path
Disallow /

yellowbrandprotectionbot

Rule Path
Disallow /

zoombot

Rule Path
Disallow /

zoominfobot

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /

awesome_bot

Rule Path
Disallow /

badbot

Rule Path
Disallow /

bl.uk_lddc_bot

Rule Path
Disallow /

bnf.fr_bot

Rule Path
Disallow /

coccocbot

Rule Path
Disallow /

cubebot

Rule Path
Disallow /

curl-all

Rule Path
Disallow /

digitalshadowsbot

Rule Path
Disallow /

equellaurlbot

Rule Path
Disallow /

harsilbot

Rule Path
Disallow /

i2kconnect

Rule Path
Disallow /

isec_bot

Rule Path
Disallow /

libcurl

Rule Path
Disallow /

linkdexbot

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

moatbot

Rule Path
Disallow /

monitorapp-robot-parser

Rule Path
Disallow /

notestock

Rule Path
Disallow /

online-webceo-bot

Rule Path
Disallow /

opengraph-bot

Rule Path
Disallow /

pagepeeker

Rule Path
Disallow /

prophy-bot

Rule Path
Disallow /

python-requests

Rule Path
Disallow /

repology-linkchecker

Rule Path
Disallow /

sc_bot

Rule Path
Disallow /

seo-audit-check-bot

Rule Path
Disallow /

startmebot

Rule Path
Disallow /

strutbot

Rule Path
Disallow /

trendictionbot0

Rule Path
Disallow /

undefined

Rule Path
Disallow /

unirest-java

Rule Path
Disallow /

urlchecker

Rule Path
Disallow /

yandexaccessibilitybot

Rule Path
Disallow /

yandeximages

Rule Path
Disallow /

python

Rule Path
Disallow /
Back to top

Comments

  • robots.txt
  • This file is to prevent the crawling and indexing of certain parts
  • of your site by web crawlers and spiders run by sites like Yahoo!
  • and Google. By telling these "robots" where not to go on your site,
  • you save bandwidth and server resources.
  • This file will be ignored unless it is at the root of your host:
  • Used: http://example.com/robots.txt
  • Ignored: http://example.com/site/robots.txt
  • For more information about the robots.txt standard, see:
  • http://www.robotstxt.org/robotstxt.html
  • CSS, JS, Images
  • Directories
  • Files
  • Paths (clean URLs)
  • Paths (no clean URLs)
  • Additional unwanted bots
  • Custombot sigs
Back to top

Warnings

  • 1 invalid line.
Back to top