du.ae
robots.txt
Robots Exclusion Standard data for du.ae
Resource Scan
Scan Details
| Site Domain | du.ae |
| Base Domain | du.ae |
| Scan Status | Ok |
| Last Scan | 2024-11-12T06:22:30+00:00 |
| Next Scan | 2024-11-19T06:22:30+00:00 |
Last Scan
| Scanned | 2024-11-12T06:22:30+00:00 |
| URL | https://du.ae/robots.txt |
| Redirect | https://www.du.ae/robots.txt |
| Redirect Domain | www.du.ae |
| Redirect Base | du.ae |
| Domain IPs | 5.32.4.225 |
| Redirect IPs | 5.32.4.225 |
| Response IP | 5.32.4.225 |
| Found | Yes |
| Hash | 97098d637d85784cd255ac1e3b9f9f1aaf66d47e9b3828d153d0e67ca40a2700 |
| SimHash | 4915d884c691 |
Groups
googlebot
| Rule | Path |
|---|---|
| Disallow | /servlet/Xcelerate |
| Disallow | /servlet/ContentServer |
| Disallow | /servlet/Satellite |
googlebot-image
| Rule | Path |
|---|---|
| Disallow | /servlet/Xcelerate |
| Disallow | /servlet/ContentServer |
| Disallow | /servlet/Satellite |
googlebot-news
| Rule | Path |
|---|---|
| Disallow | /servlet/Xcelerate |
| Disallow | /servlet/ContentServer |
| Disallow | /servlet/Satellite |
googlebot-video
| Rule | Path |
|---|---|
| Disallow | /servlet/Xcelerate |
| Disallow | /servlet/ContentServer |
| Disallow | /servlet/Satellite |
mediapartners-google
| Rule | Path |
|---|---|
| Disallow | /servlet/Xcelerate |
| Disallow | /servlet/ContentServer |
| Disallow | /servlet/Satellite |
mediapartners
| Rule | Path |
|---|---|
| Disallow | /servlet/Xcelerate |
| Disallow | /servlet/ContentServer |
| Disallow | /servlet/Satellite |
adsbot-google
| Rule | Path |
|---|---|
| Disallow | /servlet/Xcelerate |
| Disallow | /servlet/ContentServer |
| Disallow | /servlet/Satellite |
adsbot-google-mobile
| Rule | Path |
|---|---|
| Disallow | /servlet/Xcelerate |
| Disallow | /servlet/ContentServer |
| Disallow | /servlet/Satellite |
apis-google
| Rule | Path |
|---|---|
| Disallow | /servlet/Xcelerate |
| Disallow | /servlet/ContentServer |
| Disallow | /servlet/Satellite |
adsbot-google-mobile-apps
| Rule | Path |
|---|---|
| Disallow | /servlet/Xcelerate |
| Disallow | /servlet/ContentServer |
| Disallow | /servlet/Satellite |
bingbot
| Rule | Path |
|---|---|
| Disallow | /servlet/Xcelerate |
| Disallow | /servlet/ContentServer |
| Disallow | /servlet/Satellite |
yahoo! slurp
| Rule | Path |
|---|---|
| Disallow | /servlet/Xcelerate |
| Disallow | /servlet/ContentServer |
| Disallow | /servlet/Satellite |
robozilla
| Rule | Path |
|---|---|
| Disallow | /servlet/Xcelerate |
| Disallow | /servlet/ContentServer |
| Disallow | /servlet/Satellite |
nutch
| Rule | Path |
|---|---|
| Disallow | /servlet/Xcelerate |
| Disallow | /servlet/ContentServer |
| Disallow | /servlet/Satellite |
*
| Rule | Path |
|---|---|
| Disallow | / |