Log In Sign Up

pulteneybusinessdirectory.com.au
robots.txt

Robots Exclusion Standard data for pulteneybusinessdirectory.com.au

Resource Scan

Scan Details

Site Domain pulteneybusinessdirectory.com.au
Base Domain pulteneybusinessdirectory.com.au
Scan Status Ok
Last Scan2024-09-25T00:16:08+00:00
Next Scan 2024-10-02T00:16:08+00:00

Last Scan

Scanned2024-09-25T00:16:08+00:00
URL https://pulteneybusinessdirectory.com.au/robots.txt
Domain IPs 104.21.25.120, 172.67.134.54, 2606:4700:3033::6815:1978, 2606:4700:3035::ac43:8636
Response IP 104.21.25.120
Found Yes
Hash c6fcc7330fa5c23df4c761af48b81d449c1cd29941b0e95aab87b6d0ad9d7070
SimHash 65a0387064b7

Groups

*

Rule Path
Disallow /search
Disallow /update/
Disallow /directions/
Disallow /add-review/

bingbot

Rule Path
Disallow /search
Disallow /update/
Disallow /directions/
Disallow /add-review/

Other Records

Field Value
crawl-delay 20

bingpreview

Rule Path
Disallow /search
Disallow /update/
Disallow /directions/
Disallow /add-review/

Other Records

Field Value
crawl-delay 20
Back to top

Other Records

Field Value
sitemap https://pulteneybusinessdirectory.com.au/sitemap.xml
Back to top

Comments

  • www.robotstxt.org/
  • Allow crawling of all content
Back to top