weather-in.by
robots.txt

Robots Exclusion Standard data for weather-in.by

Resource Scan

Scan Details

Site Domain weather-in.by
Base Domain weather-in.by
Scan Status Ok
Last Scan2024-05-28T02:54:22+00:00
Next Scan 2024-06-27T02:54:22+00:00

Last Scan

Scanned2024-05-28T02:54:22+00:00
URL https://weather-in.by/robots.txt
Domain IPs 104.21.87.198, 172.67.145.202, 2606:4700:3031::ac43:91ca, 2606:4700:3032::6815:57c6
Response IP 172.67.145.202
Found Yes
Hash 4dc21e68468a99a8a114dfc6348e3803a8c9c264684b3d32f9f7728fa481d076
SimHash 2a44815b6433

Groups

*

Rule Path
Disallow /wp-admin/
Disallow /wp-includes/
Disallow /wp-content/plugins/
Disallow /wp-content/cache/
Disallow /wp-json/
Disallow /xmlrpc.php
Disallow /readme.html
Disallow /*?
Disallow /?s=
Allow /*.css$
Allow /*.js$

Comments

  • General rules for all user-agents
  • Directories and files disallowed for crawling
  • Query-based rules
  • Allowed file extensions