york.crimewatchpa.com
robots.txt
Robots Exclusion Standard data for york.crimewatchpa.com
Resource Scan
Scan Details
| Site Domain | york.crimewatchpa.com |
| Base Domain | crimewatchpa.com |
| Scan Status | Ok |
| Last Scan | 2/24/2025, 12:23:21 AM |
| Next Scan | 3/26/2025, 12:23:21 AM |
Last Scan
| Scanned | 2/24/2025, 12:23:21 AM |
| URL | https://york.crimewatchpa.com/robots.txt |
| Domain IPs | 45.33.65.102 |
| Response IP | 45.33.65.102 |
| Found | Yes |
| Hash | 627011f0b9c21b9d8ca087de1b5113bbf51dc578f407d6f5b472dca3dc3208c4 |
| SimHash | 425cd51884f1 |
Groups
googlebot
| Rule | Path |
|---|---|
| Disallow | /includes/ |
| Disallow | /misc/ |
| Disallow | /modules/ |
| Disallow | /profiles/ |
| Disallow | /scripts/ |
| Disallow | /themes/ |
| Disallow | /cron.php |
| Disallow | /update.php |
| Disallow | /xmlrpc.php |
| Disallow | /admin/ |
| Disallow | /comment/reply/ |
| Disallow | /filter/tips/ |
| Disallow | /node/add/ |
| Disallow | /search/ |
| Disallow | /taxonomy/ |
| Disallow | /user/ |
| Disallow | /users/ |
| Disallow | /?q=admin%2F |
| Disallow | /?q=comment%2Freply%2F |
| Disallow | /?q=filter%2Ftips%2F |
| Disallow | /?q=node%2Fadd%2F |
| Disallow | /?q=search%2F |
| Disallow | /?q=taxonomy%2F |
| Disallow | /?q=user%2F |
| Disallow | /?q=users%2F |
| Allow | /sites/default/files/styles/group_logo/public/*/group/logos/ |
| Disallow | /sites/default/files/ |
| Disallow | */charges/ |
| Disallow | */case-region/ |
| Disallow | */case-status/ |
| Disallow | */incident-type/ |
| Disallow | /*crimewatch_toolbar* |
| Disallow | /*?quicktabs_* |
| Disallow | /*%26quicktabs_* |
| Disallow | /*?qt-* |
| Disallow | /*%26qt-* |
| Disallow | *?page=* |
| Disallow | *%26page%3D* |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 20 |
googlebot-mobile
| Rule | Path |
|---|---|
| Disallow | /includes/ |
| Disallow | /misc/ |
| Disallow | /modules/ |
| Disallow | /profiles/ |
| Disallow | /scripts/ |
| Disallow | /themes/ |
| Disallow | /cron.php |
| Disallow | /update.php |
| Disallow | /xmlrpc.php |
| Disallow | /admin/ |
| Disallow | /comment/reply/ |
| Disallow | /filter/tips/ |
| Disallow | /node/add/ |
| Disallow | /search/ |
| Disallow | /taxonomy/ |
| Disallow | /user/ |
| Disallow | /users/ |
| Disallow | /?q=admin%2F |
| Disallow | /?q=comment%2Freply%2F |
| Disallow | /?q=filter%2Ftips%2F |
| Disallow | /?q=node%2Fadd%2F |
| Disallow | /?q=search%2F |
| Disallow | /?q=taxonomy%2F |
| Disallow | /?q=user%2F |
| Disallow | /?q=users%2F |
| Allow | /sites/default/files/styles/group_logo/public/*/group/logos/ |
| Disallow | /sites/default/files/ |
| Disallow | */charges/ |
| Disallow | */case-region/ |
| Disallow | */case-status/ |
| Disallow | */incident-type/ |
| Disallow | /*crimewatch_toolbar* |
| Disallow | /*?quicktabs_* |
| Disallow | /*%26quicktabs_* |
| Disallow | /*?qt-* |
| Disallow | /*%26qt-* |
| Disallow | *?page=* |
| Disallow | *%26page%3D* |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 20 |
googlebot-image
| Rule | Path |
|---|---|
| Disallow | /includes/ |
| Disallow | /misc/ |
| Disallow | /modules/ |
| Disallow | /profiles/ |
| Disallow | /scripts/ |
| Disallow | /themes/ |
| Disallow | /cron.php |
| Disallow | /update.php |
| Disallow | /xmlrpc.php |
| Disallow | /admin/ |
| Disallow | /comment/reply/ |
| Disallow | /filter/tips/ |
| Disallow | /node/add/ |
| Disallow | /search/ |
| Disallow | /taxonomy/ |
| Disallow | /user/ |
| Disallow | /users/ |
| Disallow | /?q=admin%2F |
| Disallow | /?q=comment%2Freply%2F |
| Disallow | /?q=filter%2Ftips%2F |
| Disallow | /?q=node%2Fadd%2F |
| Disallow | /?q=search%2F |
| Disallow | /?q=taxonomy%2F |
| Disallow | /?q=user%2F |
| Disallow | /?q=users%2F |
| Allow | /sites/default/files/styles/group_logo/public/*/group/logos/ |
| Disallow | /sites/default/files/ |
| Disallow | */charges/ |
| Disallow | */case-region/ |
| Disallow | */case-status/ |
| Disallow | */incident-type/ |
| Disallow | /*crimewatch_toolbar* |
| Disallow | /*?quicktabs_* |
| Disallow | /*%26quicktabs_* |
| Disallow | /*?qt-* |
| Disallow | /*%26qt-* |
| Disallow | *?page=* |
| Disallow | *%26page%3D* |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 20 |
googlebot-news
| Rule | Path |
|---|---|
| Disallow | /includes/ |
| Disallow | /misc/ |
| Disallow | /modules/ |
| Disallow | /profiles/ |
| Disallow | /scripts/ |
| Disallow | /themes/ |
| Disallow | /cron.php |
| Disallow | /update.php |
| Disallow | /xmlrpc.php |
| Disallow | /admin/ |
| Disallow | /comment/reply/ |
| Disallow | /filter/tips/ |
| Disallow | /node/add/ |
| Disallow | /search/ |
| Disallow | /taxonomy/ |
| Disallow | /user/ |
| Disallow | /users/ |
| Disallow | /?q=admin%2F |
| Disallow | /?q=comment%2Freply%2F |
| Disallow | /?q=filter%2Ftips%2F |
| Disallow | /?q=node%2Fadd%2F |
| Disallow | /?q=search%2F |
| Disallow | /?q=taxonomy%2F |
| Disallow | /?q=user%2F |
| Disallow | /?q=users%2F |
| Allow | /sites/default/files/styles/group_logo/public/*/group/logos/ |
| Disallow | /sites/default/files/ |
| Disallow | */charges/ |
| Disallow | */case-region/ |
| Disallow | */case-status/ |
| Disallow | */incident-type/ |
| Disallow | /*crimewatch_toolbar* |
| Disallow | /*?quicktabs_* |
| Disallow | /*%26quicktabs_* |
| Disallow | /*?qt-* |
| Disallow | /*%26qt-* |
| Disallow | *?page=* |
| Disallow | *%26page%3D* |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 20 |
bingbot
| Rule | Path |
|---|---|
| Disallow | /includes/ |
| Disallow | /misc/ |
| Disallow | /modules/ |
| Disallow | /profiles/ |
| Disallow | /scripts/ |
| Disallow | /themes/ |
| Disallow | /cron.php |
| Disallow | /update.php |
| Disallow | /xmlrpc.php |
| Disallow | /admin/ |
| Disallow | /comment/reply/ |
| Disallow | /filter/tips/ |
| Disallow | /node/add/ |
| Disallow | /search/ |
| Disallow | /taxonomy/ |
| Disallow | /user/ |
| Disallow | /users/ |
| Disallow | /?q=admin%2F |
| Disallow | /?q=comment%2Freply%2F |
| Disallow | /?q=filter%2Ftips%2F |
| Disallow | /?q=node%2Fadd%2F |
| Disallow | /?q=search%2F |
| Disallow | /?q=taxonomy%2F |
| Disallow | /?q=user%2F |
| Disallow | /?q=users%2F |
| Allow | /sites/default/files/styles/group_logo/public/*/group/logos/ |
| Disallow | /sites/default/files/ |
| Disallow | */charges/ |
| Disallow | */case-region/ |
| Disallow | */case-status/ |
| Disallow | */incident-type/ |
| Disallow | /*crimewatch_toolbar* |
| Disallow | /*?quicktabs_* |
| Disallow | /*%26quicktabs_* |
| Disallow | /*?qt-* |
| Disallow | /*%26qt-* |
| Disallow | *?page=* |
| Disallow | *%26page%3D* |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 20 |
msnbot-media
| Rule | Path |
|---|---|
| Disallow | /includes/ |
| Disallow | /misc/ |
| Disallow | /modules/ |
| Disallow | /profiles/ |
| Disallow | /scripts/ |
| Disallow | /themes/ |
| Disallow | /cron.php |
| Disallow | /update.php |
| Disallow | /xmlrpc.php |
| Disallow | /admin/ |
| Disallow | /comment/reply/ |
| Disallow | /filter/tips/ |
| Disallow | /node/add/ |
| Disallow | /search/ |
| Disallow | /taxonomy/ |
| Disallow | /user/ |
| Disallow | /users/ |
| Disallow | /?q=admin%2F |
| Disallow | /?q=comment%2Freply%2F |
| Disallow | /?q=filter%2Ftips%2F |
| Disallow | /?q=node%2Fadd%2F |
| Disallow | /?q=search%2F |
| Disallow | /?q=taxonomy%2F |
| Disallow | /?q=user%2F |
| Disallow | /?q=users%2F |
| Allow | /sites/default/files/styles/group_logo/public/*/group/logos/ |
| Disallow | /sites/default/files/ |
| Disallow | */charges/ |
| Disallow | */case-region/ |
| Disallow | */case-status/ |
| Disallow | */incident-type/ |
| Disallow | /*crimewatch_toolbar* |
| Disallow | /*?quicktabs_* |
| Disallow | /*%26quicktabs_* |
| Disallow | /*?qt-* |
| Disallow | /*%26qt-* |
| Disallow | *?page=* |
| Disallow | *%26page%3D* |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 20 |
slurp
| Rule | Path |
|---|---|
| Disallow | /includes/ |
| Disallow | /misc/ |
| Disallow | /modules/ |
| Disallow | /profiles/ |
| Disallow | /scripts/ |
| Disallow | /themes/ |
| Disallow | /cron.php |
| Disallow | /update.php |
| Disallow | /xmlrpc.php |
| Disallow | /admin/ |
| Disallow | /comment/reply/ |
| Disallow | /filter/tips/ |
| Disallow | /node/add/ |
| Disallow | /search/ |
| Disallow | /taxonomy/ |
| Disallow | /user/ |
| Disallow | /users/ |
| Disallow | /?q=admin%2F |
| Disallow | /?q=comment%2Freply%2F |
| Disallow | /?q=filter%2Ftips%2F |
| Disallow | /?q=node%2Fadd%2F |
| Disallow | /?q=search%2F |
| Disallow | /?q=taxonomy%2F |
| Disallow | /?q=user%2F |
| Disallow | /?q=users%2F |
| Allow | /sites/default/files/styles/group_logo/public/*/group/logos/ |
| Disallow | /sites/default/files/ |
| Disallow | */charges/ |
| Disallow | */case-region/ |
| Disallow | */case-status/ |
| Disallow | */incident-type/ |
| Disallow | /*crimewatch_toolbar* |
| Disallow | /*?quicktabs_* |
| Disallow | /*%26quicktabs_* |
| Disallow | /*?qt-* |
| Disallow | /*%26qt-* |
| Disallow | *?page=* |
| Disallow | *%26page%3D* |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 20 |
yahoo-mmcrawler
| Rule | Path |
|---|---|
| Disallow | /includes/ |
| Disallow | /misc/ |
| Disallow | /modules/ |
| Disallow | /profiles/ |
| Disallow | /scripts/ |
| Disallow | /themes/ |
| Disallow | /cron.php |
| Disallow | /update.php |
| Disallow | /xmlrpc.php |
| Disallow | /admin/ |
| Disallow | /comment/reply/ |
| Disallow | /filter/tips/ |
| Disallow | /node/add/ |
| Disallow | /search/ |
| Disallow | /taxonomy/ |
| Disallow | /user/ |
| Disallow | /users/ |
| Disallow | /?q=admin%2F |
| Disallow | /?q=comment%2Freply%2F |
| Disallow | /?q=filter%2Ftips%2F |
| Disallow | /?q=node%2Fadd%2F |
| Disallow | /?q=search%2F |
| Disallow | /?q=taxonomy%2F |
| Disallow | /?q=user%2F |
| Disallow | /?q=users%2F |
| Allow | /sites/default/files/styles/group_logo/public/*/group/logos/ |
| Disallow | /sites/default/files/ |
| Disallow | */charges/ |
| Disallow | */case-region/ |
| Disallow | */case-status/ |
| Disallow | */incident-type/ |
| Disallow | /*crimewatch_toolbar* |
| Disallow | /*?quicktabs_* |
| Disallow | /*%26quicktabs_* |
| Disallow | /*?qt-* |
| Disallow | /*%26qt-* |
| Disallow | *?page=* |
| Disallow | *%26page%3D* |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 20 |
yandex
| Rule | Path |
|---|---|
| Disallow | /includes/ |
| Disallow | /misc/ |
| Disallow | /modules/ |
| Disallow | /profiles/ |
| Disallow | /scripts/ |
| Disallow | /themes/ |
| Disallow | /cron.php |
| Disallow | /update.php |
| Disallow | /xmlrpc.php |
| Disallow | /admin/ |
| Disallow | /comment/reply/ |
| Disallow | /filter/tips/ |
| Disallow | /node/add/ |
| Disallow | /search/ |
| Disallow | /taxonomy/ |
| Disallow | /user/ |
| Disallow | /users/ |
| Disallow | /?q=admin%2F |
| Disallow | /?q=comment%2Freply%2F |
| Disallow | /?q=filter%2Ftips%2F |
| Disallow | /?q=node%2Fadd%2F |
| Disallow | /?q=search%2F |
| Disallow | /?q=taxonomy%2F |
| Disallow | /?q=user%2F |
| Disallow | /?q=users%2F |
| Allow | /sites/default/files/styles/group_logo/public/*/group/logos/ |
| Disallow | /sites/default/files/ |
| Disallow | */charges/ |
| Disallow | */case-region/ |
| Disallow | */case-status/ |
| Disallow | */incident-type/ |
| Disallow | /*crimewatch_toolbar* |
| Disallow | /*?quicktabs_* |
| Disallow | /*%26quicktabs_* |
| Disallow | /*?qt-* |
| Disallow | /*%26qt-* |
| Disallow | *?page=* |
| Disallow | *%26page%3D* |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 20 |
baiduspider
| Rule | Path |
|---|---|
| Disallow | /includes/ |
| Disallow | /misc/ |
| Disallow | /modules/ |
| Disallow | /profiles/ |
| Disallow | /scripts/ |
| Disallow | /themes/ |
| Disallow | /cron.php |
| Disallow | /update.php |
| Disallow | /xmlrpc.php |
| Disallow | /admin/ |
| Disallow | /comment/reply/ |
| Disallow | /filter/tips/ |
| Disallow | /node/add/ |
| Disallow | /search/ |
| Disallow | /taxonomy/ |
| Disallow | /user/ |
| Disallow | /users/ |
| Disallow | /?q=admin%2F |
| Disallow | /?q=comment%2Freply%2F |
| Disallow | /?q=filter%2Ftips%2F |
| Disallow | /?q=node%2Fadd%2F |
| Disallow | /?q=search%2F |
| Disallow | /?q=taxonomy%2F |
| Disallow | /?q=user%2F |
| Disallow | /?q=users%2F |
| Allow | /sites/default/files/styles/group_logo/public/*/group/logos/ |
| Disallow | /sites/default/files/ |
| Disallow | */charges/ |
| Disallow | */case-region/ |
| Disallow | */case-status/ |
| Disallow | */incident-type/ |
| Disallow | /*crimewatch_toolbar* |
| Disallow | /*?quicktabs_* |
| Disallow | /*%26quicktabs_* |
| Disallow | /*?qt-* |
| Disallow | /*%26qt-* |
| Disallow | *?page=* |
| Disallow | *%26page%3D* |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 20 |
psbot
| Rule | Path |
|---|---|
| Disallow | /includes/ |
| Disallow | /misc/ |
| Disallow | /modules/ |
| Disallow | /profiles/ |
| Disallow | /scripts/ |
| Disallow | /themes/ |
| Disallow | /cron.php |
| Disallow | /update.php |
| Disallow | /xmlrpc.php |
| Disallow | /admin/ |
| Disallow | /comment/reply/ |
| Disallow | /filter/tips/ |
| Disallow | /node/add/ |
| Disallow | /search/ |
| Disallow | /taxonomy/ |
| Disallow | /user/ |
| Disallow | /users/ |
| Disallow | /?q=admin%2F |
| Disallow | /?q=comment%2Freply%2F |
| Disallow | /?q=filter%2Ftips%2F |
| Disallow | /?q=node%2Fadd%2F |
| Disallow | /?q=search%2F |
| Disallow | /?q=taxonomy%2F |
| Disallow | /?q=user%2F |
| Disallow | /?q=users%2F |
| Allow | /sites/default/files/styles/group_logo/public/*/group/logos/ |
| Disallow | /sites/default/files/ |
| Disallow | */charges/ |
| Disallow | */case-region/ |
| Disallow | */case-status/ |
| Disallow | */incident-type/ |
| Disallow | /*crimewatch_toolbar* |
| Disallow | /*?quicktabs_* |
| Disallow | /*%26quicktabs_* |
| Disallow | /*?qt-* |
| Disallow | /*%26qt-* |
| Disallow | *?page=* |
| Disallow | *%26page%3D* |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 20 |
nutch
| Rule | Path |
|---|---|
| Disallow | /includes/ |
| Disallow | /misc/ |
| Disallow | /modules/ |
| Disallow | /profiles/ |
| Disallow | /scripts/ |
| Disallow | /themes/ |
| Disallow | /cron.php |
| Disallow | /update.php |
| Disallow | /xmlrpc.php |
| Disallow | /admin/ |
| Disallow | /comment/reply/ |
| Disallow | /filter/tips/ |
| Disallow | /node/add/ |
| Disallow | /search/ |
| Disallow | /taxonomy/ |
| Disallow | /user/ |
| Disallow | /users/ |
| Disallow | /?q=admin%2F |
| Disallow | /?q=comment%2Freply%2F |
| Disallow | /?q=filter%2Ftips%2F |
| Disallow | /?q=node%2Fadd%2F |
| Disallow | /?q=search%2F |
| Disallow | /?q=taxonomy%2F |
| Disallow | /?q=user%2F |
| Disallow | /?q=users%2F |
| Allow | /sites/default/files/styles/group_logo/public/*/group/logos/ |
| Disallow | /sites/default/files/ |
| Disallow | */charges/ |
| Disallow | */case-region/ |
| Disallow | */case-status/ |
| Disallow | */incident-type/ |
| Disallow | /*crimewatch_toolbar* |
| Disallow | /*?quicktabs_* |
| Disallow | /*%26quicktabs_* |
| Disallow | /*?qt-* |
| Disallow | /*%26qt-* |
| Disallow | *?page=* |
| Disallow | *%26page%3D* |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 20 |
java
| Rule | Path |
|---|---|
| Disallow | /includes/ |
| Disallow | /misc/ |
| Disallow | /modules/ |
| Disallow | /profiles/ |
| Disallow | /scripts/ |
| Disallow | /themes/ |
| Disallow | /cron.php |
| Disallow | /update.php |
| Disallow | /xmlrpc.php |
| Disallow | /admin/ |
| Disallow | /comment/reply/ |
| Disallow | /filter/tips/ |
| Disallow | /node/add/ |
| Disallow | /search/ |
| Disallow | /taxonomy/ |
| Disallow | /user/ |
| Disallow | /users/ |
| Disallow | /?q=admin%2F |
| Disallow | /?q=comment%2Freply%2F |
| Disallow | /?q=filter%2Ftips%2F |
| Disallow | /?q=node%2Fadd%2F |
| Disallow | /?q=search%2F |
| Disallow | /?q=taxonomy%2F |
| Disallow | /?q=user%2F |
| Disallow | /?q=users%2F |
| Allow | /sites/default/files/styles/group_logo/public/*/group/logos/ |
| Disallow | /sites/default/files/ |
| Disallow | */charges/ |
| Disallow | */case-region/ |
| Disallow | */case-status/ |
| Disallow | */incident-type/ |
| Disallow | /*crimewatch_toolbar* |
| Disallow | /*?quicktabs_* |
| Disallow | /*%26quicktabs_* |
| Disallow | /*?qt-* |
| Disallow | /*%26qt-* |
| Disallow | *?page=* |
| Disallow | *%26page%3D* |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 20 |
wget
| Rule | Path |
|---|---|
| Disallow | /includes/ |
| Disallow | /misc/ |
| Disallow | /modules/ |
| Disallow | /profiles/ |
| Disallow | /scripts/ |
| Disallow | /themes/ |
| Disallow | /cron.php |
| Disallow | /update.php |
| Disallow | /xmlrpc.php |
| Disallow | /admin/ |
| Disallow | /comment/reply/ |
| Disallow | /filter/tips/ |
| Disallow | /node/add/ |
| Disallow | /search/ |
| Disallow | /taxonomy/ |
| Disallow | /user/ |
| Disallow | /users/ |
| Disallow | /?q=admin%2F |
| Disallow | /?q=comment%2Freply%2F |
| Disallow | /?q=filter%2Ftips%2F |
| Disallow | /?q=node%2Fadd%2F |
| Disallow | /?q=search%2F |
| Disallow | /?q=taxonomy%2F |
| Disallow | /?q=user%2F |
| Disallow | /?q=users%2F |
| Allow | /sites/default/files/styles/group_logo/public/*/group/logos/ |
| Disallow | /sites/default/files/ |
| Disallow | */charges/ |
| Disallow | */case-region/ |
| Disallow | */case-status/ |
| Disallow | */incident-type/ |
| Disallow | /*crimewatch_toolbar* |
| Disallow | /*?quicktabs_* |
| Disallow | /*%26quicktabs_* |
| Disallow | /*?qt-* |
| Disallow | /*%26qt-* |
| Disallow | *?page=* |
| Disallow | *%26page%3D* |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 20 |
curl
| Rule | Path |
|---|---|
| Disallow | /includes/ |
| Disallow | /misc/ |
| Disallow | /modules/ |
| Disallow | /profiles/ |
| Disallow | /scripts/ |
| Disallow | /themes/ |
| Disallow | /cron.php |
| Disallow | /update.php |
| Disallow | /xmlrpc.php |
| Disallow | /admin/ |
| Disallow | /comment/reply/ |
| Disallow | /filter/tips/ |
| Disallow | /node/add/ |
| Disallow | /search/ |
| Disallow | /taxonomy/ |
| Disallow | /user/ |
| Disallow | /users/ |
| Disallow | /?q=admin%2F |
| Disallow | /?q=comment%2Freply%2F |
| Disallow | /?q=filter%2Ftips%2F |
| Disallow | /?q=node%2Fadd%2F |
| Disallow | /?q=search%2F |
| Disallow | /?q=taxonomy%2F |
| Disallow | /?q=user%2F |
| Disallow | /?q=users%2F |
| Allow | /sites/default/files/styles/group_logo/public/*/group/logos/ |
| Disallow | /sites/default/files/ |
| Disallow | */charges/ |
| Disallow | */case-region/ |
| Disallow | */case-status/ |
| Disallow | */incident-type/ |
| Disallow | /*crimewatch_toolbar* |
| Disallow | /*?quicktabs_* |
| Disallow | /*%26quicktabs_* |
| Disallow | /*?qt-* |
| Disallow | /*%26qt-* |
| Disallow | *?page=* |
| Disallow | *%26page%3D* |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 20 |
python-urllib
| Rule | Path |
|---|---|
| Disallow | /includes/ |
| Disallow | /misc/ |
| Disallow | /modules/ |
| Disallow | /profiles/ |
| Disallow | /scripts/ |
| Disallow | /themes/ |
| Disallow | /cron.php |
| Disallow | /update.php |
| Disallow | /xmlrpc.php |
| Disallow | /admin/ |
| Disallow | /comment/reply/ |
| Disallow | /filter/tips/ |
| Disallow | /node/add/ |
| Disallow | /search/ |
| Disallow | /taxonomy/ |
| Disallow | /user/ |
| Disallow | /users/ |
| Disallow | /?q=admin%2F |
| Disallow | /?q=comment%2Freply%2F |
| Disallow | /?q=filter%2Ftips%2F |
| Disallow | /?q=node%2Fadd%2F |
| Disallow | /?q=search%2F |
| Disallow | /?q=taxonomy%2F |
| Disallow | /?q=user%2F |
| Disallow | /?q=users%2F |
| Allow | /sites/default/files/styles/group_logo/public/*/group/logos/ |
| Disallow | /sites/default/files/ |
| Disallow | */charges/ |
| Disallow | */case-region/ |
| Disallow | */case-status/ |
| Disallow | */incident-type/ |
| Disallow | /*crimewatch_toolbar* |
| Disallow | /*?quicktabs_* |
| Disallow | /*%26quicktabs_* |
| Disallow | /*?qt-* |
| Disallow | /*%26qt-* |
| Disallow | *?page=* |
| Disallow | *%26page%3D* |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 20 |
commons-httpclient
| Rule | Path |
|---|---|
| Disallow | /includes/ |
| Disallow | /misc/ |
| Disallow | /modules/ |
| Disallow | /profiles/ |
| Disallow | /scripts/ |
| Disallow | /themes/ |
| Disallow | /cron.php |
| Disallow | /update.php |
| Disallow | /xmlrpc.php |
| Disallow | /admin/ |
| Disallow | /comment/reply/ |
| Disallow | /filter/tips/ |
| Disallow | /node/add/ |
| Disallow | /search/ |
| Disallow | /taxonomy/ |
| Disallow | /user/ |
| Disallow | /users/ |
| Disallow | /?q=admin%2F |
| Disallow | /?q=comment%2Freply%2F |
| Disallow | /?q=filter%2Ftips%2F |
| Disallow | /?q=node%2Fadd%2F |
| Disallow | /?q=search%2F |
| Disallow | /?q=taxonomy%2F |
| Disallow | /?q=user%2F |
| Disallow | /?q=users%2F |
| Allow | /sites/default/files/styles/group_logo/public/*/group/logos/ |
| Disallow | /sites/default/files/ |
| Disallow | */charges/ |
| Disallow | */case-region/ |
| Disallow | */case-status/ |
| Disallow | */incident-type/ |
| Disallow | /*crimewatch_toolbar* |
| Disallow | /*?quicktabs_* |
| Disallow | /*%26quicktabs_* |
| Disallow | /*?qt-* |
| Disallow | /*%26qt-* |
| Disallow | *?page=* |
| Disallow | *%26page%3D* |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 20 |
libwww
| Rule | Path |
|---|---|
| Disallow | /includes/ |
| Disallow | /misc/ |
| Disallow | /modules/ |
| Disallow | /profiles/ |
| Disallow | /scripts/ |
| Disallow | /themes/ |
| Disallow | /cron.php |
| Disallow | /update.php |
| Disallow | /xmlrpc.php |
| Disallow | /admin/ |
| Disallow | /comment/reply/ |
| Disallow | /filter/tips/ |
| Disallow | /node/add/ |
| Disallow | /search/ |
| Disallow | /taxonomy/ |
| Disallow | /user/ |
| Disallow | /users/ |
| Disallow | /?q=admin%2F |
| Disallow | /?q=comment%2Freply%2F |
| Disallow | /?q=filter%2Ftips%2F |
| Disallow | /?q=node%2Fadd%2F |
| Disallow | /?q=search%2F |
| Disallow | /?q=taxonomy%2F |
| Disallow | /?q=user%2F |
| Disallow | /?q=users%2F |
| Allow | /sites/default/files/styles/group_logo/public/*/group/logos/ |
| Disallow | /sites/default/files/ |
| Disallow | */charges/ |
| Disallow | */case-region/ |
| Disallow | */case-status/ |
| Disallow | */incident-type/ |
| Disallow | /*crimewatch_toolbar* |
| Disallow | /*?quicktabs_* |
| Disallow | /*%26quicktabs_* |
| Disallow | /*?qt-* |
| Disallow | /*%26qt-* |
| Disallow | *?page=* |
| Disallow | *%26page%3D* |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 20 |
httpunit
| Rule | Path |
|---|---|
| Disallow | /includes/ |
| Disallow | /misc/ |
| Disallow | /modules/ |
| Disallow | /profiles/ |
| Disallow | /scripts/ |
| Disallow | /themes/ |
| Disallow | /cron.php |
| Disallow | /update.php |
| Disallow | /xmlrpc.php |
| Disallow | /admin/ |
| Disallow | /comment/reply/ |
| Disallow | /filter/tips/ |
| Disallow | /node/add/ |
| Disallow | /search/ |
| Disallow | /taxonomy/ |
| Disallow | /user/ |
| Disallow | /users/ |
| Disallow | /?q=admin%2F |
| Disallow | /?q=comment%2Freply%2F |
| Disallow | /?q=filter%2Ftips%2F |
| Disallow | /?q=node%2Fadd%2F |
| Disallow | /?q=search%2F |
| Disallow | /?q=taxonomy%2F |
| Disallow | /?q=user%2F |
| Disallow | /?q=users%2F |
| Allow | /sites/default/files/styles/group_logo/public/*/group/logos/ |
| Disallow | /sites/default/files/ |
| Disallow | */charges/ |
| Disallow | */case-region/ |
| Disallow | */case-status/ |
| Disallow | */incident-type/ |
| Disallow | /*crimewatch_toolbar* |
| Disallow | /*?quicktabs_* |
| Disallow | /*%26quicktabs_* |
| Disallow | /*?qt-* |
| Disallow | /*%26qt-* |
| Disallow | *?page=* |
| Disallow | *%26page%3D* |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 20 |
phpcrawl
| Rule | Path |
|---|---|
| Disallow | /includes/ |
| Disallow | /misc/ |
| Disallow | /modules/ |
| Disallow | /profiles/ |
| Disallow | /scripts/ |
| Disallow | /themes/ |
| Disallow | /cron.php |
| Disallow | /update.php |
| Disallow | /xmlrpc.php |
| Disallow | /admin/ |
| Disallow | /comment/reply/ |
| Disallow | /filter/tips/ |
| Disallow | /node/add/ |
| Disallow | /search/ |
| Disallow | /taxonomy/ |
| Disallow | /user/ |
| Disallow | /users/ |
| Disallow | /?q=admin%2F |
| Disallow | /?q=comment%2Freply%2F |
| Disallow | /?q=filter%2Ftips%2F |
| Disallow | /?q=node%2Fadd%2F |
| Disallow | /?q=search%2F |
| Disallow | /?q=taxonomy%2F |
| Disallow | /?q=user%2F |
| Disallow | /?q=users%2F |
| Allow | /sites/default/files/styles/group_logo/public/*/group/logos/ |
| Disallow | /sites/default/files/ |
| Disallow | */charges/ |
| Disallow | */case-region/ |
| Disallow | */case-status/ |
| Disallow | */incident-type/ |
| Disallow | /*crimewatch_toolbar* |
| Disallow | /*?quicktabs_* |
| Disallow | /*%26quicktabs_* |
| Disallow | /*?qt-* |
| Disallow | /*%26qt-* |
| Disallow | *?page=* |
| Disallow | *%26page%3D* |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 20 |
facebookexternalhit
| Rule | Path |
|---|---|
| Allow | /sites/default/files/ |
| Disallow | /includes/ |
| Disallow | /misc/ |
| Disallow | /modules/ |
| Disallow | /profiles/ |
| Disallow | /scripts/ |
| Disallow | /themes/ |
| Disallow | /cron.php |
| Disallow | /update.php |
| Disallow | /xmlrpc.php |
| Disallow | /admin/ |
| Disallow | /comment/reply/ |
| Disallow | /filter/tips/ |
| Disallow | /node/add/ |
| Disallow | /search/ |
| Disallow | /taxonomy/ |
| Disallow | /user/ |
| Disallow | /users/ |
| Disallow | /?q=admin%2F |
| Disallow | /?q=comment%2Freply%2F |
| Disallow | /?q=filter%2Ftips%2F |
| Disallow | /?q=node%2Fadd%2F |
| Disallow | /?q=search%2F |
| Disallow | /?q=taxonomy%2F |
| Disallow | /?q=user%2F |
| Disallow | /?q=users%2F |
| Disallow | */charges/ |
| Disallow | */case-region/ |
| Disallow | */case-status/ |
| Disallow | */incident-type/ |
| Disallow | /*crimewatch_toolbar* |
| Disallow | /*?quicktabs_* |
| Disallow | /*%26quicktabs_* |
| Disallow | /*?qt-* |
| Disallow | /*%26qt-* |
| Disallow | *?page=* |
| Disallow | *%26page%3D* |
twitterbot
| Rule | Path |
|---|---|
| Disallow | /includes/ |
| Disallow | /misc/ |
| Disallow | /modules/ |
| Disallow | /profiles/ |
| Disallow | /scripts/ |
| Disallow | /themes/ |
| Disallow | /cron.php |
| Disallow | /update.php |
| Disallow | /xmlrpc.php |
| Disallow | /admin/ |
| Disallow | /comment/reply/ |
| Disallow | /filter/tips/ |
| Disallow | /node/add/ |
| Disallow | /search/ |
| Disallow | /taxonomy/ |
| Disallow | /user/ |
| Disallow | /users/ |
| Disallow | /?q=admin%2F |
| Disallow | /?q=comment%2Freply%2F |
| Disallow | /?q=filter%2Ftips%2F |
| Disallow | /?q=node%2Fadd%2F |
| Disallow | /?q=search%2F |
| Disallow | /?q=taxonomy%2F |
| Disallow | /?q=user%2F |
| Disallow | /?q=users%2F |
| Allow | /sites/default/files/styles/group_logo/public/*/group/logos/ |
| Allow | /sites/default/files/styles/social/* |
| Disallow | /sites/default/files/ |
| Disallow | */charges/ |
| Disallow | */case-region/ |
| Disallow | */case-status/ |
| Disallow | */incident-type/ |
| Disallow | /*crimewatch_toolbar* |
| Disallow | /*?quicktabs_* |
| Disallow | /*%26quicktabs_* |
| Disallow | /*?qt-* |
| Disallow | /*%26qt-* |
| Disallow | *?page=* |
| Disallow | *%26page%3D* |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 20 |
twitterbot/1.0
| Rule | Path |
|---|---|
| Disallow | /includes/ |
| Disallow | /misc/ |
| Disallow | /modules/ |
| Disallow | /profiles/ |
| Disallow | /scripts/ |
| Disallow | /themes/ |
| Disallow | /cron.php |
| Disallow | /update.php |
| Disallow | /xmlrpc.php |
| Disallow | /admin/ |
| Disallow | /comment/reply/ |
| Disallow | /filter/tips/ |
| Disallow | /node/add/ |
| Disallow | /search/ |
| Disallow | /taxonomy/ |
| Disallow | /user/ |
| Disallow | /users/ |
| Disallow | /?q=admin%2F |
| Disallow | /?q=comment%2Freply%2F |
| Disallow | /?q=filter%2Ftips%2F |
| Disallow | /?q=node%2Fadd%2F |
| Disallow | /?q=search%2F |
| Disallow | /?q=taxonomy%2F |
| Disallow | /?q=user%2F |
| Disallow | /?q=users%2F |
| Allow | /sites/default/files/ |
| Disallow | */charges/ |
| Disallow | */case-region/ |
| Disallow | */case-status/ |
| Disallow | */incident-type/ |
| Disallow | /*crimewatch_toolbar* |
| Disallow | /*?quicktabs_* |
| Disallow | /*%26quicktabs_* |
| Disallow | /*?qt-* |
| Disallow | /*%26qt-* |
| Disallow | *?page=* |
| Disallow | *%26page%3D* |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 20 |
exabot
| Rule | Path |
|---|---|
| Disallow | /includes/ |
| Disallow | /misc/ |
| Disallow | /modules/ |
| Disallow | /profiles/ |
| Disallow | /scripts/ |
| Disallow | /themes/ |
| Disallow | /cron.php |
| Disallow | /update.php |
| Disallow | /xmlrpc.php |
| Disallow | /admin/ |
| Disallow | /comment/reply/ |
| Disallow | /filter/tips/ |
| Disallow | /node/add/ |
| Disallow | /search/ |
| Disallow | /taxonomy/ |
| Disallow | /user/ |
| Disallow | /users/ |
| Disallow | /?q=admin%2F |
| Disallow | /?q=comment%2Freply%2F |
| Disallow | /?q=filter%2Ftips%2F |
| Disallow | /?q=node%2Fadd%2F |
| Disallow | /?q=search%2F |
| Disallow | /?q=taxonomy%2F |
| Disallow | /?q=user%2F |
| Disallow | /?q=users%2F |
| Allow | /sites/default/files/styles/group_logo/public/*/group/logos/ |
| Disallow | /sites/default/files/ |
| Disallow | */charges/ |
| Disallow | */case-region/ |
| Disallow | */case-status/ |
| Disallow | */incident-type/ |
| Disallow | /*crimewatch_toolbar* |
| Disallow | /*?quicktabs_* |
| Disallow | /*%26quicktabs_* |
| Disallow | /*?qt-* |
| Disallow | /*%26qt-* |
| Disallow | *?page=* |
| Disallow | *%26page%3D* |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 20 |
daumoa
| Rule | Path |
|---|---|
| Disallow | /includes/ |
| Disallow | /misc/ |
| Disallow | /modules/ |
| Disallow | /profiles/ |
| Disallow | /scripts/ |
| Disallow | /themes/ |
| Disallow | /cron.php |
| Disallow | /update.php |
| Disallow | /xmlrpc.php |
| Disallow | /admin/ |
| Disallow | /comment/reply/ |
| Disallow | /filter/tips/ |
| Disallow | /node/add/ |
| Disallow | /search/ |
| Disallow | /taxonomy/ |
| Disallow | /user/ |
| Disallow | /users/ |
| Disallow | /?q=admin%2F |
| Disallow | /?q=comment%2Freply%2F |
| Disallow | /?q=filter%2Ftips%2F |
| Disallow | /?q=node%2Fadd%2F |
| Disallow | /?q=search%2F |
| Disallow | /?q=taxonomy%2F |
| Disallow | /?q=user%2F |
| Disallow | /?q=users%2F |
| Allow | /sites/default/files/styles/group_logo/public/*/group/logos/ |
| Disallow | /sites/default/files/ |
| Disallow | */charges/ |
| Disallow | */case-region/ |
| Disallow | */case-status/ |
| Disallow | */incident-type/ |
| Disallow | /*crimewatch_toolbar* |
| Disallow | /*?quicktabs_* |
| Disallow | /*%26quicktabs_* |
| Disallow | /*?qt-* |
| Disallow | /*%26qt-* |
| Disallow | *?page=* |
| Disallow | *%26page%3D* |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 20 |
*
| Rule | Path |
|---|---|
| Disallow | /includes/ |
| Disallow | /misc/ |
| Disallow | /modules/ |
| Disallow | /profiles/ |
| Disallow | /scripts/ |
| Disallow | /themes/ |
| Disallow | /cron.php |
| Disallow | /update.php |
| Disallow | /xmlrpc.php |
| Disallow | /admin/ |
| Disallow | /comment/reply/ |
| Disallow | /filter/tips/ |
| Disallow | /node/add/ |
| Disallow | /search/ |
| Disallow | /taxonomy/ |
| Disallow | /user/ |
| Disallow | /users/ |
| Disallow | /?q=admin%2F |
| Disallow | /?q=comment%2Freply%2F |
| Disallow | /?q=filter%2Ftips%2F |
| Disallow | /?q=node%2Fadd%2F |
| Disallow | /?q=search%2F |
| Disallow | /?q=taxonomy%2F |
| Disallow | /?q=user%2F |
| Disallow | /?q=users%2F |
| Allow | /sites/default/files/styles/group_logo/public/*/group/logos/ |
| Disallow | /sites/default/files/ |
| Disallow | */charges/ |
| Disallow | */case-region/ |
| Disallow | */case-status/ |
| Disallow | */incident-type/ |
| Disallow | /*crimewatch_toolbar* |
| Disallow | /*?quicktabs_* |
| Disallow | /*%26quicktabs_* |
| Disallow | /*?qt-* |
| Disallow | /*%26qt-* |
| Disallow | *?page=* |
| Disallow | *%26page%3D* |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 30 |
Comments