datnet.org
robots.txt

Robots Exclusion Standard data for datnet.org

Resource Scan

Scan Details

Site Domain datnet.org
Base Domain datnet.org
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2025-10-06T19:12:11+00:00
Next Scan 2026-01-04T19:12:11+00:00

Last Successful Scan

Scanned2023-02-27T17:48:24+00:00
URL https://datnet.org/robots.txt
Domain IPs 104.21.18.91, 172.67.181.135, 2606:4700:3033::6815:125b, 2606:4700:3037::ac43:b587
Response IP 104.21.18.91
Found Yes
Hash 5ca0ad2885c63e23597d0b56b522607d62c29692cdced1c0918e8d1e0e481360
SimHash 38304d034c98

Groups

*

Rule Path
Disallow /startTopic/
Disallow /discover/unread/
Disallow /markallread/
Disallow /staff/
Disallow /online/
Disallow /discover/
Disallow /leaderboard/
Disallow /search/
Disallow /*?advancedSearchForm=
Disallow /register/
Disallow /lostpassword/
Disallow /login/
Disallow /*?sortby=
Disallow /*?filter=
Disallow /*?tab=
Disallow /*?do=
Disallow /*ref%3D
Disallow /*?forumId*
Disallow /profile/

Other Records

Field Value
sitemap http://datnet.org/sitemap.php

Comments

  • Rules for Invision Community (https://invisioncommunity.com)
  • Block pages with no unique content
  • Block faceted pages and 301 redirect pages
  • Block profile pages as these have little unique value, consume a lot of crawl time and contain hundreds of 301 links
  • Sitemap URL