Log In Sign Up

ud.ac.ae
robots.txt

Robots Exclusion Standard data for ud.ac.ae

Resource Scan

Scan Details

Site Domain ud.ac.ae
Base Domain ud.ac.ae
Scan Status Ok
Last Scan2025-02-12T18:31:03+00:00
Next Scan 2025-03-14T18:31:03+00:00

Last Scan

Scanned2025-02-12T18:31:03+00:00
URL https://www.ud.ac.ae/robots.txt
Redirect https://ud.ac.ae/robots.txt
Redirect Domain ud.ac.ae
Redirect Base ud.ac.ae
Domain IPs 2001:19f0:4401:3b9:5400:3ff:fe9c:1a18, 66.22.97.148
Redirect IPs 2001:19f0:4401:3b9:5400:3ff:fe9c:1a18, 66.22.97.148
Response IP 66.22.97.148
Found Yes
Hash 7886acb45bb3c61c11ac691744ff381786fb14433915f2bd447d1f0f021bec6d
SimHash 68bcc810f2f2

Groups

*

Rule Path
Disallow /wp-json/
Disallow /?rest_route=

*

Rule Path
Disallow /wp-admin/
Disallow /wp-includes/
Disallow /wp-content/uploads/formidable/
Disallow /cgi-bin/
Disallow /tmp/
Disallow /private/
Disallow /search
Disallow /*.pdf$
Disallow /*.doc$
Disallow /*.docx$
Disallow /*.xls$
Disallow /*.ppt$
Disallow /*.pptx$

badbot

Rule Path
Disallow /

evilscraper

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /
Disallow /360-Virtual-Tour/?id=

googlebot

Rule Path
Disallow

bingbot

Rule Path
Disallow
Back to top

Other Records

Field Value
sitemap https://ud.ac.ae/sitemap_index.xml
Back to top

Comments

  • START YOAST BLOCK
  • ---------------------------
  • Block specific bots
  • Allow specific bots (like Google and Bing) to crawl everything
  • ---------------------------
  • END YOAST BLOCK
Back to top