Log In Sign Up

seek.com
robots.txt

Robots Exclusion Standard data for seek.com

Resource Scan

Scan Details

Site Domain seek.com
Base Domain seek.com
Scan Status Ok
Last Scan2024-11-13T17:24:43+00:00
Next Scan 2024-11-20T17:24:43+00:00

Last Scan

Scanned2024-11-13T17:24:43+00:00
URL https://seek.com/robots.txt
Redirect https://www.seek.com.au/robots.txt
Redirect Domain www.seek.com.au
Redirect Base seek.com.au
Domain IPs 13.55.41.93, 52.64.134.99
Redirect IPs 104.18.42.193, 172.64.145.63, 2606:4700:4400::6812:2ac1, 2606:4700:4400::ac40:913f
Response IP 172.64.145.63
Found Yes
Hash 36fdd72990c2d36ffdb81417656a43aebb257deeab0789720b14e1596a6422f9
SimHash 8016c9c0de95

Groups

mediapartners-google
adidxbot

Rule Path
Disallow

*

Rule Path
Disallow */job/
Disallow *?returnUrl=
Disallow *?page=

linkedinbot
baiduspider
petalbot

Rule Path
Disallow /

anthropic-ai
bytespider
ccbot
diffbot
google-extended
omgili
gptbot

Rule Path
Disallow /companies
Disallow */job/

linkedinbot

Rule Path
Allow */job/
Back to top

Comments

  • robots.txt file for www.seek.com.au
  • Unrestricted access
  • Default directives
  • Disallowed bots
  • Exceptions
Back to top