indiedb.com
robots.txt
Robots Exclusion Standard data for indiedb.com
Resource Scan
Scan Details
| Site Domain | indiedb.com |
| Base Domain | indiedb.com |
| Scan Status | Ok |
| Last Scan | 2024-09-20T12:24:26+00:00 |
| Next Scan | 2024-09-27T12:24:26+00:00 |
Last Scan
| Scanned | 2024-09-20T12:24:26+00:00 |
| URL | https://indiedb.com/robots.txt |
| Redirect | https://www.indiedb.com/robots.txt |
| Redirect Domain | www.indiedb.com |
| Redirect Base | indiedb.com |
| Domain IPs | 104.26.12.36, 104.26.13.36, 172.67.69.154, 2606:4700:20::681a:c24, 2606:4700:20::681a:d24, 2606:4700:20::ac43:459a |
| Redirect IPs | 104.26.12.36, 104.26.13.36, 172.67.69.154, 2606:4700:20::681a:c24, 2606:4700:20::681a:d24, 2606:4700:20::ac43:459a |
| Response IP | 172.67.69.154 |
| Found | Yes |
| Hash | b469b9f207c0566126380fc4ced227dc208a4b3e33f5ead50162b7647bffe212 |
| SimHash | 2f58f67a6ed0 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /*? |
| Disallow | /*/ajax/ |
| Disallow | /*/blog/ |
| Disallow | /*/blogs/ |
| Disallow | /*/comments/ |
| Disallow | /*/add/ |
| Disallow | /*/edit/ |
| Disallow | /*/images/archive |
| Disallow | /*/page/ |
| Disallow | /*/stats/ |
| Disallow | /*/videos/archive |
| Disallow | /addons/start/ |
| Disallow | /admin |
| Disallow | /articles/ajax/ |
| Disallow | /blog |
| Disallow | /blogs |
| Disallow | /cart |
| Disallow | /comment |
| Disallow | /company/ajax/ |
| Disallow | /cutoff/html/ |
| Disallow | /downloads/ajax/ |
| Disallow | /downloads/mirror/ |
| Disallow | /downloads/start/ |
| Disallow | /downloads/timeout/ |
| Disallow | /external |
| Disallow | /features/ajax/ |
| Disallow | /forum/ajax/ |
| Disallow | /forum/shortcut/ |
| Disallow | /forum/search |
| Disallow | /games/ajax/ |
| Disallow | /groups/ajax/ |
| Disallow | /groups/gamefront/ |
| Disallow | /groups/indiedb/ |
| Disallow | /groups/moddb/ |
| Disallow | /groups/slidedb/ |
| Disallow | /groups/vrdb/ |
| Disallow | /home |
| Disallow | /hardware |
| Disallow | /media |
| Disallow | /members/ajax/ |
| Disallow | /members/logout/ |
| Disallow | /members/password/ |
| Disallow | /members/permission/ |
| Disallow | /messages |
| Disallow | /mods |
| Disallow | /newsletter/ajax/ |
| Disallow | /platforms |
| Disallow | /profile |
| Disallow | /reviews/ajax/ |
| Disallow | /search |
| Disallow | /software |
| Disallow | /staff |
| Disallow | /statistics |
| Disallow | /tags/ajax/ |
| Disallow | /tutorials/ajax/ |
mj12bot
ahrefsbot
semrushbot
dotbot
mauibot
petalbot
| Rule | Path |
|---|---|
| Disallow | /*? |
| Disallow | /*/ajax/ |
| Disallow | /*/blog/ |
| Disallow | /*/blogs/ |
| Disallow | /*/comments/ |
| Disallow | /*/add/ |
| Disallow | /*/edit/ |
| Disallow | /*/images/archive |
| Disallow | /*/page/ |
| Disallow | /*/stats/ |
| Disallow | /*/videos/archive |
| Disallow | /addons/start/ |
| Disallow | /admin |
| Disallow | /articles/ajax/ |
| Disallow | /blog |
| Disallow | /blogs |
| Disallow | /cart |
| Disallow | /comment |
| Disallow | /company/ajax/ |
| Disallow | /cutoff/html/ |
| Disallow | /downloads/ajax/ |
| Disallow | /downloads/mirror/ |
| Disallow | /downloads/start/ |
| Disallow | /downloads/timeout/ |
| Disallow | /external |
| Disallow | /features/ajax/ |
| Disallow | /forum/ajax/ |
| Disallow | /forum/shortcut/ |
| Disallow | /forum/search |
| Disallow | /games/ajax/ |
| Disallow | /groups/ajax/ |
| Disallow | /groups/gamefront/ |
| Disallow | /groups/indiedb/ |
| Disallow | /groups/moddb/ |
| Disallow | /groups/slidedb/ |
| Disallow | /groups/vrdb/ |
| Disallow | /home |
| Disallow | /hardware |
| Disallow | /media |
| Disallow | /members |
| Disallow | /messages |
| Disallow | /mods |
| Disallow | /newsletter/ajax/ |
| Disallow | /platforms |
| Disallow | /profile |
| Disallow | /reviews/ajax/ |
| Disallow | /search |
| Disallow | /software |
| Disallow | /staff |
| Disallow | /statistics |
| Disallow | /tags/ajax/ |
| Disallow | /tutorials/ajax/ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.indiedb.com/api/siteindex |
| sitemap | https://www.indiedb.com/api/siteindex |