webpages.streetdirectory.com
robots.txt

Robots Exclusion Standard data for webpages.streetdirectory.com

Resource Scan

Scan Details

Site Domain webpages.streetdirectory.com
Base Domain streetdirectory.com
Scan Status Ok
Last Scan2024-06-10T22:37:42+00:00
Next Scan 2024-06-17T22:37:42+00:00

Last Scan

Scanned2024-06-10T22:37:42+00:00
URL http://webpages.streetdirectory.com/robots.txt
Redirect https://www.streetdirectory.com/robots.txt
Redirect Domain www.streetdirectory.com
Redirect Base streetdirectory.com
Domain IPs 54.169.90.138
Redirect IPs 13.33.88.10, 13.33.88.71, 13.33.88.73, 13.33.88.84, 2600:9000:223b:2000:16:1ca8:e000:93a1, 2600:9000:223b:3200:16:1ca8:e000:93a1, 2600:9000:223b:5600:16:1ca8:e000:93a1, 2600:9000:223b:7a00:16:1ca8:e000:93a1, 2600:9000:223b:c200:16:1ca8:e000:93a1, 2600:9000:223b:d000:16:1ca8:e000:93a1, 2600:9000:223b:ea00:16:1ca8:e000:93a1, 2600:9000:223b:fa00:16:1ca8:e000:93a1
Response IP 13.33.88.71
Found Yes
Hash 7847b7bff1d084a52e41a396d8a6e6b35e1a5a27f984ffc1a9e3b3488cb97534
SimHash e1530aae5bab

Groups

petalbot

Rule Path
Disallow /

*

Rule Path
Disallow /account/login/
Disallow /asia_travel/search/
Disallow /businessfinder/uk/
Disallow /businessfinder/usa/
Disallow /businessfinder/australia/
Disallow /businessfinder/philippines/
Disallow /businessfinder/canada/
Disallow /businessfinder/hongkong/
Disallow /tutors/singapore/tutor_details/
Disallow /u/in/
Disallow /u/hk/
Disallow /u/ca/
Disallow /u/us/
Disallow /u/au/
Disallow /u/my/
Disallow /u/uk/
Disallow /australia/
Disallow /api/merge_layer.php
Disallow /asia_travel/search/searchredirect.php
Disallow /asia_travel/travel_sites/whats_nearby/
Disallow /asia_travel/travel/resize_images.php
Disallow /routing/
Disallow /api/map/world.cgi
Disallow /sd_mobile/direction/
Disallow /lyricadvisor/nice_song.php
Disallow /etoday/send_article.php
Disallow /food_editorials/topics/
Disallow /travel_guide/editorials/other/
Disallow /resize_images.php
Disallow /feedback.php
Disallow /indonesia/jakarta/explain_property_error.php
Disallow /malaysia/kl/explain_property_error.php
Disallow /stock_images/travel/show_resize_image.php
Disallow /indonesia/jakarta/nearby_enlarge.php
Disallow /indonesia/jakarta/nearby_list.php
Disallow /malaysia/kl/nearby_list.php
Disallow /asiarooms/kl/travel.php
Disallow /web/
Disallow /api/map/world.cgi
Disallow /world.cgi
Disallow /jabodetabek.cgi
Disallow /hotels
Disallow /hotel_reservations
Disallow /email.php