origin.streetdirectory.com
robots.txt

Robots Exclusion Standard data for origin.streetdirectory.com

Resource Scan

Scan Details

Site Domain origin.streetdirectory.com
Base Domain streetdirectory.com
Scan Status Ok
Last Scan2024-05-24T17:58:04+00:00
Next Scan 2024-05-31T17:58:04+00:00

Last Scan

Scanned2024-05-24T17:58:04+00:00
URL https://origin.streetdirectory.com/robots.txt
Domain IPs 18.140.46.169, 3.1.61.210
Response IP 3.1.61.210
Found Yes
Hash 7847b7bff1d084a52e41a396d8a6e6b35e1a5a27f984ffc1a9e3b3488cb97534
SimHash e1530aae5bab

Groups

petalbot

Rule Path
Disallow /

*

Rule Path
Disallow /account/login/
Disallow /asia_travel/search/
Disallow /businessfinder/uk/
Disallow /businessfinder/usa/
Disallow /businessfinder/australia/
Disallow /businessfinder/philippines/
Disallow /businessfinder/canada/
Disallow /businessfinder/hongkong/
Disallow /tutors/singapore/tutor_details/
Disallow /u/in/
Disallow /u/hk/
Disallow /u/ca/
Disallow /u/us/
Disallow /u/au/
Disallow /u/my/
Disallow /u/uk/
Disallow /australia/
Disallow /api/merge_layer.php
Disallow /asia_travel/search/searchredirect.php
Disallow /asia_travel/travel_sites/whats_nearby/
Disallow /asia_travel/travel/resize_images.php
Disallow /routing/
Disallow /api/map/world.cgi
Disallow /sd_mobile/direction/
Disallow /lyricadvisor/nice_song.php
Disallow /etoday/send_article.php
Disallow /food_editorials/topics/
Disallow /travel_guide/editorials/other/
Disallow /resize_images.php
Disallow /feedback.php
Disallow /indonesia/jakarta/explain_property_error.php
Disallow /malaysia/kl/explain_property_error.php
Disallow /stock_images/travel/show_resize_image.php
Disallow /indonesia/jakarta/nearby_enlarge.php
Disallow /indonesia/jakarta/nearby_list.php
Disallow /malaysia/kl/nearby_list.php
Disallow /asiarooms/kl/travel.php
Disallow /web/
Disallow /api/map/world.cgi
Disallow /world.cgi
Disallow /jabodetabek.cgi
Disallow /hotels
Disallow /hotel_reservations
Disallow /email.php