Log In Sign Up

romanarmytalk.com
robots.txt

Robots Exclusion Standard data for romanarmytalk.com

Resource Scan

Scan Details

Site Domain romanarmytalk.com
Base Domain romanarmytalk.com
Scan Status Ok
Last Scan2024-05-28T07:08:21+00:00
Next Scan 2024-06-27T07:08:21+00:00

Last Scan

Scanned2024-05-28T07:08:21+00:00
URL https://romanarmytalk.com/robots.txt
Redirect https://www.romanarmytalk.com/rat/robots.txt
Redirect Domain www.romanarmytalk.com
Redirect Base romanarmytalk.com
Domain IPs 104.21.1.51, 172.67.128.139, 2606:4700:3030::ac43:808b, 2606:4700:3036::6815:133
Redirect IPs 104.21.1.51, 172.67.128.139, 2606:4700:3030::ac43:808b, 2606:4700:3036::6815:133
Response IP 104.21.1.51
Found Yes
Hash 2ac22d61b7715fdf075fa780ce6123a8df136e2e128a2b41855626b5b64f52e8
SimHash a21f1d1ac3e6

Groups

*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 3
Back to top

Comments

  • If the Joomla site is installed within a folder such as at
  • e.g. www.example.com/joomla/ the robots.txt file MUST be
  • moved to the site root at e.g. www.example.com/robots.txt
  • AND the joomla folder name MUST be prefixed to the disallowed
  • path, e.g. the Disallow rule for the /administrator/ folder
  • MUST be changed to read Disallow: /joomla/administrator/
  • For more information about the robots.txt standard, see:
  • http://www.robotstxt.org/orig.html
  • For syntax checking, see:
  • http://www.sxw.org.uk/computing/robots/check.html
Back to top