itm.com.au
robots.txt

Robots Exclusion Standard data for itm.com.au

Resource Scan

Scan Details

Site Domain itm.com.au
Base Domain itm.com.au
Scan Status Ok
Last Scan2024-09-07T04:05:04+00:00
Next Scan 2024-10-07T04:05:04+00:00

Last Scan

Scanned2024-09-07T04:05:04+00:00
URL https://itm.com.au/robots.txt
Redirect https://www.itm.com.au/robots.txt
Redirect Domain www.itm.com.au
Redirect Base itm.com.au
Domain IPs 103.51.112.151
Redirect IPs 103.51.112.151
Response IP 103.51.112.151
Found Yes
Hash a95a40b0e435dca5e39652c0ccbb81a532b076e7339f6ed7dd2b4b4874caa4f0
SimHash 06145c00c7f3

Groups

teoma

Rule Path
Disallow /

twiceler

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

scrubby

Rule Path
Disallow /

nutch

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

naverbot

Rule Path
Disallow /

yeti

Rule Path
Disallow /

psbot

Rule Path
Disallow /

asterias

Rule Path
Disallow /

yahoo-blogs

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

sosospider

Rule Path
Disallow /

*

Rule Path
Disallow /Enquiry/
Disallow /Admin
Disallow /Areas

Other Records

Field Value
crawl-delay 2

Other Records

Field Value
sitemap https://www.itm.com.au/XMLSitemap

Comments

  • robots.txt
  • beweb robotos
  • these are generally useless bots