learn-wordpress.tk
robots.txt

Robots Exclusion Standard data for learn-wordpress.tk

Resource Scan

Scan Details

Site Domain learn-wordpress.tk
Base Domain learn-wordpress.tk
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2024-10-11T17:57:39+00:00
Next Scan 2025-01-09T17:57:39+00:00

Last Successful Scan

Scanned2023-06-20T12:09:08+00:00
URL https://learn-wordpress.tk/robots.txt
Domain IPs 76.76.21.21
Response IP 76.76.21.21
Found Yes
Hash 466d3f3e10f37156e2dbb87b6d199b722a68225b337f8001484ccf2cb37e5d0c
SimHash 530dd8524531

Groups

*

Rule Path
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php
Allow /wp-admin/images/*
Disallow /wp-includes/
Allow /wp-includes/js
Allow /wp-includes/css
Disallow /trackback/
Disallow /xmlrpc.php
Disallow /feed/
Disallow /*/feed/*
Disallow /*/*?s=*
Disallow /*/*.inc$
Disallow /transfer/
Disallow /refer/
Disallow /*/cgi-bin/*
Disallow /*/blackhole/*
Disallow /*/trackback/*
Disallow /*/xmlrpc.php
Disallow /suggest/?*
Disallow /readme.html
Disallow /*?hpp_next=*
Disallow /tool/

easouspider
ezooms
mj12bot
sitesucker
httrack
httrack website copier
teleport
teleportpro
emailcollector
emailsiphon
webbandit
webzip
webreaper
webstripper
web downloader
webcopier
offline explorer pro
offline commander
leech
websnake
blackwidow
http weazel

Rule Path
Disallow /

nutch

Rule Path
Disallow /

mj12bot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

pinerest

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

ia_archiver

Rule Path
Disallow /

Comments

  • Disallow: /wp-content/plugins/
  • Disallow: /wp-content/themes/
  • protect my site from HTTrack or other software's ripping?
  • https://example.com/sitemap.xml