Log In Sign Up

turkdlsi.xyz
robots.txt

Robots Exclusion Standard data for turkdlsi.xyz

Resource Scan

Scan Details

Site Domain turkdlsi.xyz
Base Domain turkdlsi.xyz
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2025-09-09T12:18:22+00:00
Next Scan 2025-12-08T12:18:22+00:00

Last Successful Scan

Scanned2021-12-08T19:00:02+00:00
URL https://turkdlsi.xyz/robots.txt
Redirect https://turk-dli.xyz/robots.txt
Redirect Domain turk-dli.xyz
Redirect Base turk-dli.xyz
Response IP 104.21.65.244
Found Yes
Hash ed58c588e49802b6e7a86bf76e9869cc789d8e7d9b428c7e2c7ffb177ee0af15
SimHash 4a7d4d0a00f0

Groups

*

Rule Path
Disallow /wp-content/
Disallow /wp-admin/
Disallow /wp-includes/
Disallow /wp-
Disallow /tag/
Disallow /cgi-bin/
Disallow /feed/
Disallow /trackback/
Disallow */trackback*
Disallow /stats*
Disallow /about/legal-notice/
Disallow /about/copyright-policy/
Disallow /about/terms-and-conditions/
Disallow /tag
Disallow /docs*
Disallow /manual*
Disallow /category/uncategorized*

googlebot

Rule Path
Disallow /*.php$
Disallow /*.js$
Disallow /*.inc$
Disallow /*.css$
Disallow /*.gz$
Disallow /*.cgi$
Disallow /*.wmv$
Disallow /*.php*
Disallow /*.gz$
Allow /wp-content/uploads/

ia_archiver

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://tdlnew.xyz/sitemap.xml
Back to top

Comments

  • disallow all files in these WordPress directories
  • disallow all files in these directories
  • disallow robots from parsing individual post feeds and trackbacks
  • disallow any files that are stats related
  • disallow files ending with the following extensions
  • disallow WayBack archiving site
Back to top