gitea.osmocom.org
robots.txt
Robots Exclusion Standard data for gitea.osmocom.org
Resource Scan
Scan Details
| Site Domain | gitea.osmocom.org |
| Base Domain | osmocom.org |
| Scan Status | Ok |
| Last Scan | 2024-10-08T00:46:15+00:00 |
| Next Scan | 2024-10-22T00:46:15+00:00 |
Last Scan
| Scanned | 2024-10-08T00:46:15+00:00 |
| URL | https://gitea.osmocom.org/robots.txt |
| Domain IPs | 2a01:4f8:120:8470::2, 78.46.96.155 |
| Response IP | 78.46.96.155 |
| Found | Yes |
| Hash | d9b4c40c0f241c02b00a77450e39994896a6fe170982a88e102eb92ced9f7312 |
| SimHash | 521a4993a7c6 |
Groups
amazonbot
applebot
applebot-extended
bytespider
ccbot
chatgpt-user
claude-web
claudebot
diffbot
facebookbot
friendlycrawler
gptbot
google-extended
googleother
googleother-image
googleother-video
icc-crawler
imagesiftbot
meta-externalagent
meta-externalfetcher
oai-searchbot
perplexitybot
petalbot
scrapy
timpibot
velenpublicwebcrawler
webzio-extended
youbot
anthropic-ai
cohere-ai
facebookexternalhit
img2dataset
omgili
omgilibot
| Rule | Path |
|---|---|
| Disallow | / |
*
| Rule | Path | Comment |
|---|---|---|
| Disallow | /api/* | - |
| Disallow | /avatars | - |
| Disallow | /user/* | - |
| Disallow | /*/*/src/commit/* | - |
| Disallow | /*/*/commit/* | - |
| Disallow | /*/*/*/refs/* | - |
| Disallow | /*/*/*/star | - |
| Disallow | /*/*/*/watch | - |
| Disallow | /*/*/labels | - |
| Disallow | /*/*/activity/* | - |
| Disallow | /vendor/* | - |
| Disallow | /swagger.*.json | - |
| Disallow | /explore/*?* | - |
| Disallow | /repo/create | - |
| Disallow | /repo/migrate | - |
| Disallow | /org/create | - |
| Disallow | /*/*/fork | - |
| Disallow | /*/*/watchers | - |
| Disallow | /*/*/stargazers | - |
| Disallow | /*/*/forks | - |
| Disallow | /*/*/activity | - |
| Disallow | /*/*/projects | - |
| Disallow | /*/*/commits/ | - |
| Disallow | /*/*/branches | - |
| Disallow | /*/*/tags | - |
| Disallow | /*/*/compare | - |
| Disallow | /*/*/lastcommit/* | - |
| Disallow | /*/*/issues/new | - |
| Disallow | /*/*/issues/?* | - |
| Disallow | /*/*/issues?* | - |
| Disallow | /*/*/pulls/?* | - |
| Disallow | /*/*/pulls?* | - |
| Disallow | /*/*/pulls/*/files | - |
| Disallow | /*/tree/ | - |
| Disallow | /*/download | - |
| Disallow | /*/revisions | - |
| Disallow | /*/commits/*?author | - |
| Disallow | /*/commits/*?path | - |
| Disallow | /*/comments | - |
| Disallow | /*/blame/ | - |
| Disallow | /*/raw/ | - |
| Disallow | /*/cache/ | - |
| Disallow | /.git/ | - |
| Disallow | */.git/ | - |
| Disallow | /*.git | - |
| Disallow | /*.atom | - |
| Disallow | /*.rss | - |
| Disallow | /*/*/archive/ | - |
| Disallow | *.bundle | - |
| Disallow | */commit/*.patch | - |
| Disallow | */commit/*.diff | - |
| Disallow | /*lang%3D* | - |
| Disallow | /*source%3D* | - |
| Disallow | /*ref_cta%3D* | - |
| Disallow | /*plan%3D* | - |
| Disallow | /*return_to%3D* | - |
| Disallow | /*ref_loc%3D* | - |
| Disallow | /*setup_organization%3D* | - |
| Disallow | /*source_repo%3D* | - |
| Disallow | /*ref_page%3D* | - |
| Disallow | /*source%3D* | - |
| Disallow | /*referrer%3D* | - |
| Disallow | /*report%3D* | - |
| Disallow | /*author%3D* | - |
| Disallow | /*since%3D* | - |
| Disallow | /*until%3D* | - |
| Disallow | /*commits?author=* | - |
| Disallow | /*tab%3D* | - |
| Disallow | /*q%3D* | - |
| Disallow | /*repo-search-archived%3D* | - |
| Disallow | /Codeberg/*/*/Imprint.md | - |
| Disallow | /mirror | huge linux mirror, pointless to index |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 2 |
amazonbot
anthropic-ai
applebot-extended
bytespider
ccbot
chatgpt-user
claudebot
claude-web
cohere-ai
diffbot
facebookbot
facebookexternalhit
friendlycrawler
google-extended
gptbot
icc-crawler
imagesiftbot
img2dataset
meta-externalagent
oai-searchbot
omgili
omgilibot
perplexitybot
petalbot
scrapy
timpibot
velenpublicwebcrawler
youbot
| Rule | Path |
|---|---|
| Disallow | / |
Comments