Log In Sign Up

gildor.org
robots.txt

Robots Exclusion Standard data for gildor.org

Resource Scan

Scan Details

Site Domain gildor.org
Base Domain gildor.org
Scan Status Ok
Last Scan2025-10-08T05:51:45+00:00
Next Scan 2025-11-07T05:51:45+00:00

Last Scan

Scanned2025-10-08T05:51:45+00:00
URL https://gildor.org/robots.txt
Domain IPs 87.236.16.99
Response IP 87.236.16.99
Found Yes
Hash 4ef70c157e2f0c2abee26e66ffcb4fdbed702b5f74947a7055292ca173e211e4
SimHash bef6d11b4b55

Groups

*

Rule Path
Disallow /scripts/
Disallow /updates/
Disallow /profiles/
Disallow /xmlrpc.php
Disallow /search/
Disallow /down/
Disallow /donate
Disallow /en/donate
Disallow /smf/index.php?action=search
Disallow /smf/index.php?action=stats
Disallow /smf/*type%3Drss*
Disallow /smf/*imode*

serpstatbot

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

zumbot

Rule Path
Disallow /

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

yandex

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

qwantify

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

google-youtube-links

Rule Path
Disallow /
Back to top

Comments

  • This file is to prevent the crawling and indexing of certain parts
  • of your site by web crawlers and spiders run by sites like Yahoo!
  • and Google. By telling these "robots" where not to go on your site,
  • you save bandwidth and server resources.
  • For more information about the robots.txt standard, see:
  • http://www.robotstxt.org/robotstxt.html
  • Directories
  • comment "disallow" for some theme-related things, otherwise google thinks that that site is not mobile-friendly
  • Files
  • Paths (clean URLs)
  • Custom paths
  • Forum
  • Disallow: /smf/index.php?action=printpage* - invisible for bots
  • Disallow: /smf/index.php?action=profile* - ...
  • Disallow: /smf/index.php?action=login - ...
  • Disallow: /smf/index.php?action=url
  • Disallow: /smf/index.php?action=help
  • Disallow: /smf/index.php?action=recent - disabled with meta noindex
  • Disallow: /smf/index.php?topic*new - now has canonical URL, so doesn't matter
  • Mobile version of forum, terrible one
  • Disallow: /smf/*wap* -- redirecting to no-wap2 page with 301 code
  • Heavy site loading
  • this bot has nearly the same IP as serpstatbot, seems same group, and doing same heavy load of the site
  • made 6.5k requests during night
  • Youtube link scanner - generates 404 errors when someone uploads a video with site refs
  • https://sudofox.hatenablog.com/entry/google-is-scanning-for-and-crawling-urls-in-your-private-youtube-videos
Back to top

Warnings

  • `host` is not a known field.
Back to top