rugbyontario.com
robots.txt

Robots Exclusion Standard data for rugbyontario.com

Resource Scan

Scan Details

Site Domain rugbyontario.com
Base Domain rugbyontario.com
Scan Status Ok
Last Scan2024-05-18T12:43:23+00:00
Next Scan 2024-06-17T12:43:23+00:00

Last Scan

Scanned2024-05-18T12:43:23+00:00
URL https://rugbyontario.com/robots.txt
Redirect https://www.rugbyontario.com/robots.txt
Redirect Domain www.rugbyontario.com
Redirect Base rugbyontario.com
Domain IPs 18.155.192.107, 18.155.192.5, 18.155.192.91, 18.155.192.96
Redirect IPs 18.164.154.118, 18.164.154.34, 18.164.154.53, 18.164.154.87, 2600:9000:2024:4a00:0:efc4:edc0:93a1, 2600:9000:2024:5000:0:efc4:edc0:93a1, 2600:9000:2024:6600:0:efc4:edc0:93a1, 2600:9000:2024:7c00:0:efc4:edc0:93a1, 2600:9000:2024:8800:0:efc4:edc0:93a1, 2600:9000:2024:ac00:0:efc4:edc0:93a1, 2600:9000:2024:bc00:0:efc4:edc0:93a1, 2600:9000:2024:c00:0:efc4:edc0:93a1
Response IP 108.157.52.21
Found Yes
Hash 5e1ce8e2339181656ffb82f300d1e498055d7977146890dbac9811a7a7665213
SimHash 69019a000793

Groups

scrapy

Rule Path
Allow /

*

Rule Path
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php

Other Records

Field Value
sitemap https://www.rugbyontario.com/wp-sitemap.xml