gspvhcute24.contest.codeforces.com
robots.txt

Robots Exclusion Standard data for gspvhcute24.contest.codeforces.com

Resource Scan

Scan Details

Site Domain gspvhcute24.contest.codeforces.com
Base Domain codeforces.com
Scan Status Ok
Last Scan2025-06-02T18:52:28+00:00
Next Scan 2025-07-02T18:52:28+00:00

Last Scan

Scanned2025-06-02T18:52:28+00:00
URL https://gspvhcute24.contest.codeforces.com/robots.txt
Domain IPs 95.163.252.67
Response IP 95.163.252.67
Found Yes
Hash 7d930df0b686fb0b52a5a41273832543e0cd5e87d67d19e5766c20be847e8e71
SimHash b1390119ef75

Groups

mj12bot

Rule Path
Disallow /

mediapartners-google

Rule Path
Disallow /

israbot

Rule Path
Disallow /

orthogaffe

Rule Path
Disallow /

ubicrawler

Rule Path
Disallow /

doc

Rule Path
Disallow /

zao

Rule Path
Disallow /

sitecheck.internetseer.com

Rule Path
Disallow /

zealbot

Rule Path
Disallow /

msiecrawler

Rule Path
Disallow /

sitesnagger

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

fetch

Rule Path
Disallow /

offline explorer

Rule Path
Disallow /

teleport

Rule Path
Disallow /

teleportpro

Rule Path
Disallow /

webzip

Rule Path
Disallow /

linko

Rule Path
Disallow /

httrack

Rule Path
Disallow /

microsoft.url.control

Rule Path
Disallow /

xenu

Rule Path
Disallow /

larbin

Rule Path
Disallow /

libwww

Rule Path
Disallow /

zyborg

Rule Path
Disallow /

download ninja

Rule Path
Disallow /

fast

Rule Path
Disallow /

grub-client

Rule Path
Disallow /

k2spider

Rule Path
Disallow /

npbot

Rule Path
Disallow /

webreaper

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

openai-crawler

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

youbot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

applebot

Rule Path
Disallow /

facebookexternalhit

Rule Path
Disallow /

metaai-bot

Rule Path
Disallow /

huggingfacebot

Rule Path
Disallow /

scrapy

Rule Path
Disallow /
Allow /api/

python-requests

Rule Path
Disallow /
Allow /api/

curl

Rule Path
Disallow /
Allow /api/

wget

Rule Path
Disallow /
Allow /api/

*

Rule Path
Disallow /admin
Disallow /alpha
Disallow /attachment
Disallow /comments/with
Disallow /contest
Disallow /contestInvitation
Disallow /contestRegistrants
Disallow /contestRegistration
Disallow /contests/with
Disallow /css
Disallow /data
Disallow /enter
Disallow /gifts
Disallow /trustList
Disallow /group
Disallow /gym/edit
Disallow /images
Disallow /js
Disallow /list/view
Disallow /lists
Disallow /lists/edit
Disallow /lists/new
Disallow /logout
Disallow /markitup
Disallow /mashups
Disallow /modules
Disallow /printEntry
Disallow /problemset/standings
Disallow /problemset/status
Disallow /problemset/submit
Disallow /proposals
Disallow /ratings
Disallow /ratings/page
Disallow /recent-actions
Disallow /register
Disallow /renderer
Disallow /site
Disallow /scripts
Disallow /search
Disallow /settings
Disallow /society
Disallow /spectator
Disallow /st.codeforces.com
Disallow /submissions
Disallow */submission/*
Disallow /support
Disallow /system
Disallow /teams/invite
Disallow /teams/with
Disallow /top-contributed
Disallow /u
Disallow /unsubscribe
Disallow /uploadfile
Disallow /user
Disallow /userForm
Disallow /userFormFieldValue/download
Disallow /userform
Disallow /userlist
Disallow /users
Disallow /usertalk
Disallow /verifyme

Comments

  • Block known bad bots and aggressive scrapers
  • AI and LLM crawlers
  • Common scraping tools (only allow /api)
  • General restrictions for all bots