forge.mikegerwitz.com
robots.txt
Robots Exclusion Standard data for forge.mikegerwitz.com
Resource Scan
Scan Details
| Site Domain | forge.mikegerwitz.com |
| Base Domain | mikegerwitz.com |
| Scan Status | Ok |
| Last Scan | 2025-12-19T05:51:20+00:00 |
| Next Scan | 2025-12-20T05:51:20+00:00 |
Last Scan
| Scanned | 2025-12-19T05:51:20+00:00 |
| URL | https://forge.mikegerwitz.com/robots.txt |
| Domain IPs | 149.56.108.225 |
| Response IP | 149.56.108.225 |
| Found | Yes |
| Hash | 74726909567c2d45662b7057ada3fd32a076ac86b98c121ac1e93ba36a402e98 |
| SimHash | 520c5991a7c4 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /api/* |
| Disallow | /avatars |
| Disallow | /user/* |
| Disallow | /*/*/src/commit/* |
| Disallow | /*/*/commit/* |
| Disallow | /*/*/*/refs/* |
| Disallow | /*/*/*/star |
| Disallow | /*/*/*/watch |
| Disallow | /*/*/labels |
| Disallow | /*/*/activity/* |
| Disallow | /vendor/* |
| Disallow | /swagger.*.json |
| Disallow | /explore/*?* |
| Disallow | /repo/create |
| Disallow | /repo/migrate |
| Disallow | /org/create |
| Disallow | /*/*/fork |
| Disallow | /*/*/watchers |
| Disallow | /*/*/stargazers |
| Disallow | /*/*/forks |
| Disallow | /*/*/activity |
| Disallow | /*/*/projects |
| Disallow | /*/*/commits/ |
| Disallow | /*/*/branches |
| Disallow | /*/*/tags |
| Disallow | /*/*/compare |
| Disallow | /*/*/lastcommit/* |
| Disallow | /*/*/issues/new |
| Disallow | /*/*/issues/?* |
| Disallow | /*/*/issues?* |
| Disallow | /*/*/pulls/?* |
| Disallow | /*/*/pulls?* |
| Disallow | /*/*/pulls/*/files |
| Disallow | /*/tree/ |
| Disallow | /*/download |
| Disallow | /*/revisions |
| Disallow | /*/commits/*?author |
| Disallow | /*/commits/*?path |
| Disallow | /*/comments |
| Disallow | /*/blame/ |
| Disallow | /*/raw/ |
| Disallow | /*/cache/ |
| Disallow | /.git/ |
| Disallow | */.git/ |
| Disallow | /*.git |
| Disallow | /*.atom |
| Disallow | /*.rss |
| Disallow | /*/*/archive/ |
| Disallow | *.bundle |
| Disallow | */commit/*.patch |
| Disallow | */commit/*.diff |
| Disallow | /*lang%3D* |
| Disallow | /*source%3D* |
| Disallow | /*ref_cta%3D* |
| Disallow | /*plan%3D* |
| Disallow | /*return_to%3D* |
| Disallow | /*ref_loc%3D* |
| Disallow | /*setup_organization%3D* |
| Disallow | /*source_repo%3D* |
| Disallow | /*ref_page%3D* |
| Disallow | /*source%3D* |
| Disallow | /*referrer%3D* |
| Disallow | /*report%3D* |
| Disallow | /*author%3D* |
| Disallow | /*since%3D* |
| Disallow | /*until%3D* |
| Disallow | /*commits?author=* |
| Disallow | /*tab%3D* |
| Disallow | /*q%3D* |
| Disallow | /*repo-search-archived%3D* |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 2 |
amazonbot
anthropic-ai
applebot-extended
bytespider
ccbot
chatgpt-user
claudebot
claude-web
cohere-ai
diffbot
facebookbot
facebookexternalhit
friendlycrawler
google-extended
gptbot
icc-crawler
imagesiftbot
img2dataset
meta-externalagent
oai-searchbot
omgili
omgilibot
perplexitybot
petalbot
scrapy
timpibot
velenpublicwebcrawler
youbot
| Rule | Path |
|---|---|
| Disallow | / |
Comments