Log In Sign Up

indiegames.news
robots.txt

Robots Exclusion Standard data for indiegames.news

Resource Scan

Scan Details

Site Domain indiegames.news
Base Domain indiegames.news
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-10-30T06:18:25+00:00
Next Scan 2024-11-06T06:18:25+00:00

Last Successful Scan

Scanned2024-10-28T06:18:08+00:00
URL http://indiegames.news/robots.txt
Redirect https://gamedev.net:443/robots.txt
Redirect Domain gamedev.net
Redirect Base gamedev.net
Domain IPs 130.211.24.203
Redirect IPs 130.211.24.203
Response IP 130.211.24.203
Found Yes
Hash 12c0544db46c6fb7bb81af86af31856870ae49d836a2b3459df90c79917d4585
SimHash 690f846828f3

Groups

*

Rule Path
Disallow /api/
Disallow /*admin/
Disallow /cgi-bin/
Disallow /datastore/
Disallow /system/
Disallow /login/
Disallow /logout/
Disallow /profile/*

Other Records

Field Value
crawl-delay 10
Back to top

Other Records

Field Value
sitemap https://gamedev.net/sitemap.xml
Back to top

Comments

  • Disallow: /applications/
  • Disallow: /plugins/
  • Disallow files
  • Disallow: /403error.php
  • Disallow: /404error.php
  • Disallow: /500error.php
  • Disallow: /Credits.txt
  • Disallow: /error.php
  • Disallow: /upgrading.html
  • Querystring
  • Disallow: /?tab=*
  • Disallow: /*?app=*
  • Disallow: /*sortby=*
  • Disallow: /*/?do=download
  • Disallow: /profile/*/?do=*
  • Disallow: /profile/*/content/
  • Disallow: /*?do=add
  • Disallow: /*?do=email
  • Disallow: /*?do=getNewComment
  • Disallow: /*?do=getLastComment
  • Disallow: /*?do=getLastComment
  • Disallow: /*?do=reportComment
  • Allow specific parts
  • Allow: /applications/core/interface/imageproxy/imageproxy.php?img=*
Back to top