streetdirectory.com
robots.txt

Robots Exclusion Standard data for streetdirectory.com

Resource Scan

Scan Details

Site Domain streetdirectory.com
Base Domain streetdirectory.com
Scan Status Ok
Last Scan2024-05-09T20:22:11+00:00
Next Scan 2024-05-16T20:22:11+00:00

Last Scan

Scanned2024-05-09T20:22:11+00:00
URL https://www.streetdirectory.com/robots.txt
Domain IPs 13.33.88.10, 13.33.88.71, 13.33.88.73, 13.33.88.84, 2600:9000:223b:3c00:16:1ca8:e000:93a1, 2600:9000:223b:4200:16:1ca8:e000:93a1, 2600:9000:223b:8e00:16:1ca8:e000:93a1, 2600:9000:223b:a000:16:1ca8:e000:93a1, 2600:9000:223b:ac00:16:1ca8:e000:93a1, 2600:9000:223b:ae00:16:1ca8:e000:93a1, 2600:9000:223b:b200:16:1ca8:e000:93a1, 2600:9000:223b:fc00:16:1ca8:e000:93a1
Response IP 13.33.88.84
Found Yes
Hash 7847b7bff1d084a52e41a396d8a6e6b35e1a5a27f984ffc1a9e3b3488cb97534
SimHash e1530aae5bab

Groups

petalbot

Rule Path
Disallow /

*

Rule Path
Disallow /account/login/
Disallow /asia_travel/search/
Disallow /businessfinder/uk/
Disallow /businessfinder/usa/
Disallow /businessfinder/australia/
Disallow /businessfinder/philippines/
Disallow /businessfinder/canada/
Disallow /businessfinder/hongkong/
Disallow /tutors/singapore/tutor_details/
Disallow /u/in/
Disallow /u/hk/
Disallow /u/ca/
Disallow /u/us/
Disallow /u/au/
Disallow /u/my/
Disallow /u/uk/
Disallow /australia/
Disallow /api/merge_layer.php
Disallow /asia_travel/search/searchredirect.php
Disallow /asia_travel/travel_sites/whats_nearby/
Disallow /asia_travel/travel/resize_images.php
Disallow /routing/
Disallow /api/map/world.cgi
Disallow /sd_mobile/direction/
Disallow /lyricadvisor/nice_song.php
Disallow /etoday/send_article.php
Disallow /food_editorials/topics/
Disallow /travel_guide/editorials/other/
Disallow /resize_images.php
Disallow /feedback.php
Disallow /indonesia/jakarta/explain_property_error.php
Disallow /malaysia/kl/explain_property_error.php
Disallow /stock_images/travel/show_resize_image.php
Disallow /indonesia/jakarta/nearby_enlarge.php
Disallow /indonesia/jakarta/nearby_list.php
Disallow /malaysia/kl/nearby_list.php
Disallow /asiarooms/kl/travel.php
Disallow /web/
Disallow /api/map/world.cgi
Disallow /world.cgi
Disallow /jabodetabek.cgi
Disallow /hotels
Disallow /hotel_reservations
Disallow /email.php