gitdab.com
robots.txt

Robots Exclusion Standard data for gitdab.com

Resource Scan

Scan Details

Site Domain gitdab.com
Base Domain gitdab.com
Scan Status Ok
Last Scan2025-10-28T02:20:16+00:00
Next Scan 2025-11-11T02:20:16+00:00

Last Scan

Scanned2025-10-28T02:20:16+00:00
URL https://gitdab.com/robots.txt
Domain IPs 79.110.170.24
Response IP 79.110.170.24
Found Yes
Hash 75a46b85269fc8b37ebe39e56e70838e3d8cefad799b87ca1109a989995b854e
SimHash df2dd5e0ff90

Groups

*

Rule Path
Disallow /*/*/archive/
Disallow /*/*/src/commit/
Disallow /*/*/raw/commit/

blexbot

Rule Path
Disallow /*/*/archive/
Disallow /*/*/src/commit/
Disallow /*/*/raw/commit/

ahrefsbot

Rule Path
Disallow /*/*
Disallow /*/*/archive/

semrushbot

Rule Path
Disallow /*/*
Disallow /*/*/archive/

barkrowler

Rule Path
Disallow /*/*
Disallow /*/*/archive/

amazonbot

Rule Path
Disallow /*/*
Disallow /*/*/archive/