cruisinggays.com
robots.txt

Robots Exclusion Standard data for cruisinggays.com

Resource Scan

Scan Details

Site Domain cruisinggays.com
Base Domain cruisinggays.com
Scan Status Ok
Last Scan2024-10-03T15:02:30+00:00
Next Scan 2024-10-10T15:02:30+00:00

Last Scan

Scanned2024-10-03T15:02:30+00:00
URL https://www.cruisinggays.com/robots.txt
Domain IPs 23.99.0.12
Response IP 23.99.0.12
Found Yes
Hash 2789b1ba1bd663cb24613572dd52186000f60b061ac96a38ec9ef3bbade11545
SimHash 11d7cb0205d4

Groups

scrapy

Rule Path
Disallow /

blogslive

Rule Path
Disallow /

topicblogs

Rule Path
Disallow /

*

Rule Path
Disallow /Services/
Disallow /Personals/
Disallow /members.aspx
Disallow /Members.aspx
Disallow /Profile.aspx
Disallow /GeoServices
Disallow /Account
Allow /

Comments

  • Robots.txt file from http://www.cruisinggays.com
  • All robots will spider the domain