thebrew.ae
robots.txt
Robots Exclusion Standard data for thebrew.ae
Resource Scan
Scan Details
| Site Domain | thebrew.ae |
| Base Domain | thebrew.ae |
| Scan Status | Ok |
| Last Scan | 5/18/2025, 3:54:35 AM |
| Next Scan | 5/25/2025, 3:54:35 AM |
Last Scan
| Scanned | 5/18/2025, 3:54:35 AM |
| URL | https://thebrew.ae/robots.txt |
| Redirect | https://thebrewnews.com/robots.txt |
| Redirect Domain | thebrewnews.com |
| Redirect Base | thebrewnews.com |
| Domain IPs | 104.21.112.1, 104.21.16.1, 104.21.32.1, 104.21.48.1, 104.21.64.1, 104.21.80.1, 104.21.96.1, 2606:4700:3030::6815:1001, 2606:4700:3030::6815:2001, 2606:4700:3030::6815:3001, 2606:4700:3030::6815:4001, 2606:4700:3030::6815:5001, 2606:4700:3030::6815:6001, 2606:4700:3030::6815:7001 |
| Redirect IPs | 104.21.55.101, 172.67.147.117, 2606:4700:3030::ac43:9375, 2606:4700:3032::6815:3765 |
| Response IP | 104.21.55.101 |
| Found | Yes |
| Hash | ee24b5b32299e7f546e3e3c21c100f8d4d5eba365be2b6cf55c8d458725fb335 |
| SimHash | 211109106fd4 |
Groups
googlebot
| Rule | Path |
|---|---|
| Disallow | /test/ |
| Disallow | /uae-names-street-after-doctor-george-matthew-of-indian-origin/ |
| Disallow | /jobs/ |
| Disallow | /submit-your-creative/ |
| Disallow | /log-in/ |
| Disallow | /password-reset/ |
| Disallow | /account/ |
| Disallow | /profile/ |
| Disallow | /submit-a-post/ |
| Disallow | /home-ads/ |
adsbot-google
| Rule | Path |
|---|---|
| Disallow | /test/ |
| Disallow | /uae-names-street-after-doctor-george-matthew-of-indian-origin/ |
| Disallow | /jobs/ |
| Disallow | /submit-your-creative/ |
| Disallow | /log-in/ |
| Disallow | /password-reset/ |
| Disallow | /account/ |
| Disallow | /profile/ |
| Disallow | /submit-a-post/ |
| Disallow | /home-ads/ |
googlebot-image
| Rule | Path |
|---|---|
| Disallow | /log-in/ |
| Disallow | /account/ |
| Disallow | /password-reset/ |
| Disallow | /profile/ |
bingbot
| Rule | Path |
|---|---|
| Disallow | /test/ |
| Disallow | /uae-names-street-after-doctor-george-matthew-of-indian-origin/ |
| Disallow | /jobs/ |
| Disallow | /submit-your-creative/ |
| Disallow | /log-in/ |
| Disallow | /password-reset/ |
| Disallow | /account/ |
| Disallow | /profile/ |
| Disallow | /submit-a-post/ |
| Disallow | /home-ads/ |
*
| Rule | Path |
|---|---|
| Disallow | /test/ |
| Disallow | /uae-names-street-after-doctor-george-matthew-of-indian-origin/ |
| Disallow | /jobs/ |
| Disallow | /submit-your-creative/ |
| Disallow | /log-in/ |
| Disallow | /password-reset/ |
| Disallow | /account/ |
| Disallow | /profile/ |
| Disallow | /submit-a-post/ |
| Disallow | /home-ads/ |
| Disallow | /wp-admin/ |
| Allow | /wp-admin/admin-ajax.php |
Other Records
| Field | Value |
|---|---|
| sitemap | https://thebrewnews.com/sitemap_index.xml |