Log In Sign Up

theurbanlist.com
robots.txt

Robots Exclusion Standard data for theurbanlist.com

Resource Scan

Scan Details

Site Domain theurbanlist.com
Base Domain theurbanlist.com
Scan Status Ok
Last Scan2024-06-11T08:46:44+00:00
Next Scan 2024-06-18T08:46:44+00:00

Last Scan

Scanned2024-06-11T08:46:44+00:00
URL https://theurbanlist.com/robots.txt
Redirect https://www.theurbanlist.com/robots.txt
Redirect Domain www.theurbanlist.com
Redirect Base theurbanlist.com
Domain IPs 104.21.81.124, 172.67.161.28, 2606:4700:3032::ac43:a11c, 2606:4700:3034::6815:517c
Redirect IPs 104.21.81.124, 172.67.161.28, 2606:4700:3032::ac43:a11c, 2606:4700:3034::6815:517c
Response IP 172.67.161.28
Found Yes
Hash 81f7cfd774cd9c81f7b02073d3e313c13bc82bae600b18c895d1cf81c3cd80c0
SimHash 70850840a243

Groups

*

Rule Path
Disallow /system/*

rogerbot

Rule Path
Disallow /user/save/*
Disallow /user/unsave/*
Back to top

Other Records

Field Value
sitemap https://www.theurbanlist.com/sydney/sitemap
sitemap https://www.theurbanlist.com/brisbane/sitemap
sitemap https://www.theurbanlist.com/melbourne/sitemap
sitemap https://www.theurbanlist.com/perth/sitemap
sitemap https://www.theurbanlist.com/goldcoast/sitemap
sitemap https://www.theurbanlist.com/sunshinecoast/sitemap
sitemap https://www.theurbanlist.com/singapore/sitemap
sitemap https://www.theurbanlist.com/nz/sitemap
Back to top

Comments

  • www.robotstxt.org/
  • Allow crawling of all content
  • Stop SEO Moz from picking up all the empty bookmarking pages.
Back to top