job-con.jp
robots.txt

Robots Exclusion Standard data for job-con.jp

Resource Scan

Scan Details

Site Domain job-con.jp
Base Domain job-con.jp
Scan Status Ok
Last Scan2024-07-01T11:12:22+00:00
Next Scan 2024-07-08T11:12:22+00:00

Last Scan

Scanned2024-07-01T11:12:22+00:00
URL https://job-con.jp/robots.txt
Domain IPs 104.20.240.67, 104.20.241.67, 2606:4700:10::6814:f043, 2606:4700:10::6814:f143
Response IP 104.20.240.67
Found Yes
Hash f151d9867144a9e54fa0948d1f54b9eee5a1d01b108af40511d683bfaec4007f
SimHash 755388d4e2bb

Groups

*

Rule Path
Disallow /special/driver/search/
Disallow /special/driver/*?
Disallow /special/security/search/
Disallow /special/security/*?

trovitbot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

baiduspider+

Rule Path
Disallow /

baidumobaider

Rule Path
Disallow /

deusu

Rule Path
Disallow /

dataprovider

Rule Path
Disallow /

haosouspider

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

proximic

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

Other Records

Field Value
sitemap https://job-con.jp/sitemap.xml
sitemap https://job-con.jp/special/driver/sitemap.xml