djc.com
robots.txt
Robots Exclusion Standard data for djc.com
Resource Scan
Scan Details
| Site Domain | djc.com |
| Base Domain | djc.com |
| Scan Status | Failed |
| Failure Reason | Scan timed out. |
| Last Scan | 2024-09-19T21:11:10+00:00 |
| Next Scan | 2024-12-18T21:11:10+00:00 |
Last Successful Scan
| Scanned | 2022-05-04T05:13:37+00:00 |
| URL | https://www.djc.com/robots.txt |
| Response IP | 65.57.72.51 |
| Found | Yes |
| Hash | 1e10ca9d8e103226c896153b5517bbaa1be7e698b0b61e1ae05847015409a297 |
| SimHash | aebc43d242bb |
Groups
boardreader
boardtracker
gigabot
twiceler
dotbot
baidu
baiduspider
baiduspider-video
baiduspider-image
naverbot
sosospider
yandex
youdaobot
yeti
proximic
sogou
heretrix
sogou spider
sogou web spider
mj12bot
| Rule | Path |
|---|---|
| Disallow | / |
*
| Rule | Path |
|---|---|
| Disallow | /Ads |
| Disallow | /ppv |
| Disallow | /cust/openx/ |
| Disallow | /credit/ |
| Disallow | /no/ |
| Disallow | /notices/ |
| Disallow | /func/ |
| Disallow | /cust/ |
| Disallow | /*/firms.html |
| Disallow | /*/wanted.html |
| Disallow | /firms/ |
| Disallow | /co/wanted.html |
| Disallow | /bu/wanted.html |
| Disallow | /ae/wanted.html |
| Disallow | /re/wanted.html |
| Disallow | /tech/wanted.html |
| Disallow | /en/wanted.html |
| Disallow | /le/wanted.html |
| Disallow | /co/firms.html |
| Disallow | /bu/firms.html |
| Disallow | /ae/firms.html |
| Disallow | /re/firms.html |
| Disallow | /tech/firms.html |
| Disallow | /en/firms.html |
| Disallow | /le/firms.html |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 5 |