Log In Sign Up

tastymoney.hk
robots.txt

Robots Exclusion Standard data for tastymoney.hk

Resource Scan

Scan Details

Site Domain tastymoney.hk
Base Domain tastymoney.hk
Scan Status Ok
Last Scan2025-12-10T18:12:37+00:00
Next Scan 2025-12-17T18:12:37+00:00

Last Scan

Scanned2025-12-10T18:12:37+00:00
URL https://tastymoney.hk/robots.txt
Domain IPs 104.21.45.39, 172.67.209.103, 2606:4700:3033::6815:2d27, 2606:4700:3034::ac43:d167
Response IP 172.67.209.103
Found Yes
Hash e4a4c75e198b677898e5970db48253c9204072b5378231195ca863607114bbc7
SimHash 28ae985b0d35

Groups

*

Rule Path
Allow /
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php
Disallow /uploads/wpo-plugins-tables-list.json
Disallow /feed/

brightbot

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

bot

Rule Path
Disallow /sharebox/
Back to top

Comments

  • This file instructs web robots (crawlers) on what parts of the site they should not access.
  • It does NOT enforce access restrictions; it's a guideline.
  • For enforcement, refer to server-side rules (e.g., .htaccess, firewall).
  • --- Default Rules for All Well-Behaved Bots ---
  • By default, allow all well-behaved bots to crawl everything unless specifically disallowed below.
  • Standard WordPress Disallows:
  • Keep bots out of the main admin area, but allow specific AJAX file for functionality.
  • Disallow specific JSON file that might contain sensitive plugin data
  • Disallow Brightbot from the entire site
  • Disallow Scrapy (often used for scraping)
  • Disallow lowercase 'scrapy' user-agent (case variation)
  • Disallow GPTBot (OpenAI's crawler) from the entire site
Back to top