codeberg.org
robots.txt
Robots Exclusion Standard data for codeberg.org
Resource Scan
Scan Details
| Site Domain | codeberg.org |
| Base Domain | codeberg.org |
| Scan Status | Ok |
| Last Scan | 2024-11-12T16:57:08+00:00 |
| Next Scan | 2024-11-26T16:57:08+00:00 |
Last Scan
| Scanned | 2024-11-12T16:57:08+00:00 |
| URL | https://codeberg.org/robots.txt |
| Domain IPs | 2001:67c:1401:20f0::1, 217.197.91.145 |
| Response IP | 217.197.91.145 |
| Found | Yes |
| Hash | 078e461b70ba7ba5232c5217dcb18dc44ea54ca1136b17b36c37360edf72da1e |
| SimHash | 52084993a746 |
Groups
*
| Rule | Path | Comment |
|---|---|---|
| Disallow | /api/* | - |
| Disallow | /avatars | - |
| Disallow | /user/* | - |
| Disallow | /*/*/src/commit/* | - |
| Disallow | /*/*/commit/* | - |
| Disallow | /*/*/*/refs/* | - |
| Disallow | /*/*/*/star | - |
| Disallow | /*/*/*/watch | - |
| Disallow | /*/*/labels | - |
| Disallow | /*/*/activity/* | - |
| Disallow | /vendor/* | - |
| Disallow | /swagger.*.json | - |
| Disallow | /explore/*?* | - |
| Disallow | /repo/create | - |
| Disallow | /repo/migrate | - |
| Disallow | /org/create | - |
| Disallow | /*/*/fork | - |
| Disallow | /*/*/watchers | - |
| Disallow | /*/*/stargazers | - |
| Disallow | /*/*/forks | - |
| Disallow | /*/*/activity | - |
| Disallow | /*/*/projects | - |
| Disallow | /*/*/commits/ | - |
| Disallow | /*/*/branches | - |
| Disallow | /*/*/tags | - |
| Disallow | /*/*/compare | - |
| Disallow | /*/*/lastcommit/* | - |
| Disallow | /*/*/issues/new | - |
| Disallow | /*/*/issues/?* | - |
| Disallow | /*/*/issues?* | - |
| Disallow | /*/*/pulls/?* | - |
| Disallow | /*/*/pulls?* | - |
| Disallow | /*/*/pulls/*/files | - |
| Disallow | /*/tree/ | - |
| Disallow | /*/download | - |
| Disallow | /*/revisions | - |
| Disallow | /*/commits/*?author | - |
| Disallow | /*/commits/*?path | - |
| Disallow | /*/comments | - |
| Disallow | /*/blame/ | - |
| Disallow | /*/raw/ | - |
| Disallow | /*/cache/ | - |
| Disallow | /.git/ | - |
| Disallow | */.git/ | - |
| Disallow | /*.git | - |
| Disallow | /*.atom | - |
| Disallow | /*.rss | - |
| Disallow | /*/*/archive/ | - |
| Disallow | *.bundle | - |
| Disallow | */commit/*.patch | - |
| Disallow | */commit/*.diff | - |
| Disallow | /*lang%3D* | - |
| Disallow | /*source%3D* | - |
| Disallow | /*ref_cta%3D* | - |
| Disallow | /*plan%3D* | - |
| Disallow | /*return_to%3D* | - |
| Disallow | /*ref_loc%3D* | - |
| Disallow | /*setup_organization%3D* | - |
| Disallow | /*source_repo%3D* | - |
| Disallow | /*ref_page%3D* | - |
| Disallow | /*source%3D* | - |
| Disallow | /*referrer%3D* | - |
| Disallow | /*report%3D* | - |
| Disallow | /*author%3D* | - |
| Disallow | /*since%3D* | - |
| Disallow | /*until%3D* | - |
| Disallow | /*commits?author=* | - |
| Disallow | /*tab%3D* | - |
| Disallow | /*q%3D* | - |
| Disallow | /*repo-search-archived%3D* | - |
| Disallow | /Codeberg/*/*/Imprint.md | - |
| Disallow | /mirror | huge linux mirror, pointless to index |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 2 |
amazonbot
anthropic-ai
applebot-extended
bytespider
ccbot
chatgpt-user
claudebot
claude-web
cohere-ai
diffbot
facebookbot
facebookexternalhit
friendlycrawler
google-extended
gptbot
icc-crawler
imagesiftbot
img2dataset
meta-externalagent
oai-searchbot
omgili
omgilibot
perplexitybot
petalbot
scrapy
timpibot
velenpublicwebcrawler
youbot
| Rule | Path |
|---|---|
| Disallow | / |
Comments