Log In Sign Up

industrysearch.com.au
robots.txt

Robots Exclusion Standard data for industrysearch.com.au

Resource Scan

Scan Details

Site Domain industrysearch.com.au
Base Domain industrysearch.com.au
Scan Status Ok
Last Scan2026-03-06T17:35:10+00:00
Next Scan 2026-04-05T17:35:10+00:00

Last Scan

Scanned2026-03-06T17:35:10+00:00
URL https://industrysearch.com.au/robots.txt
Redirect https://www.industrysearch.com.au/robots.txt
Redirect Domain www.industrysearch.com.au
Redirect Base industrysearch.com.au
Domain IPs 104.26.8.193, 104.26.9.193, 172.67.75.178, 2606:4700:20::681a:8c1, 2606:4700:20::681a:9c1, 2606:4700:20::ac43:4bb2
Redirect IPs 104.26.8.193, 104.26.9.193, 172.67.75.178, 2606:4700:20::681a:8c1, 2606:4700:20::681a:9c1, 2606:4700:20::ac43:4bb2
Response IP 172.67.75.178
Found Yes
Hash f851aaee4c47099dd21e7778cdaf6bbe4041a208a11fb1f04066d5dbf519afc7
SimHash ca87c30246a3

Groups

*

Rule Path
Disallow /ScriptResource.axd
Disallow /WebResource.axd
Disallow /cgi-bin/
Disallow /Search/
Disallow /search/
Disallow /search?q=
Disallow /sma/
Disallow /oma/
Disallow /*img0.
Disallow *page%3D*

baiduspider
baiduspider-video
baiduspider-image

Rule Path
Disallow /

grapeshot

Rule Path
Disallow /

jikespider

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

naverbot
yeti

Rule Path
Disallow /

proximic

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

sosospider

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

youdaobot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.industrysearch.com.au/restsitemap.xml
sitemap https://www.industrysearch.com.au/productssitemap.xml?pg=1
sitemap https://www.industrysearch.com.au/productssitemap.xml?pg=2
Back to top

Comments

  • Rest Sitemap
  • Product1 Sitemap
  • Product2 Sitemap
  • Baidu (CN)
  • Info: http://www.baidu.com/search/spider.htm
  • Required robots.txt code:
  • SoGou (CN)
  • Info: http://www.sogou.com/docs/help/webmasters.htm#07
  • Required robots.txt code:
Back to top