webpages.streetdirectory.com
robots.txt

Robots Exclusion Standard data for webpages.streetdirectory.com

Resource Scan

Scan Details

Site Domain webpages.streetdirectory.com
Base Domain streetdirectory.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a server error.
Last Scan2024-10-31T23:15:35+00:00
Next Scan 2024-11-14T23:15:35+00:00

Last Successful Scan

Scanned2024-10-16T23:14:34+00:00
URL http://webpages.streetdirectory.com/robots.txt
Redirect https://www.streetdirectory.com/robots.txt
Redirect Domain www.streetdirectory.com
Redirect Base streetdirectory.com
Domain IPs 54.169.90.138
Redirect IPs 13.33.88.10, 13.33.88.71, 13.33.88.73, 13.33.88.84, 2600:9000:223b:4000:16:1ca8:e000:93a1, 2600:9000:223b:6000:16:1ca8:e000:93a1, 2600:9000:223b:7200:16:1ca8:e000:93a1, 2600:9000:223b:7400:16:1ca8:e000:93a1, 2600:9000:223b:8c00:16:1ca8:e000:93a1, 2600:9000:223b:be00:16:1ca8:e000:93a1, 2600:9000:223b:e200:16:1ca8:e000:93a1, 2600:9000:223b:fc00:16:1ca8:e000:93a1
Response IP 13.33.88.10
Found Yes
Hash df050c7921cdd1f1686127c6db98ca1ac2a5997b54218955efd7ffc38e424e9b
SimHash f11319364bab

Groups

petalbot

Rule Path
Disallow /

friendlycrawler

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

qwantify

Rule Path
Disallow /

senutobot

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

awariorssbot

Rule Path
Disallow /

awariosmartbot

Rule Path
Disallow /

*

Rule Path
Disallow /account/login/
Disallow /asia_travel/search/
Disallow /businessfinder/uk/
Disallow /businessfinder/usa/
Disallow /businessfinder/australia/
Disallow /businessfinder/philippines/
Disallow /businessfinder/canada/
Disallow /businessfinder/hongkong/
Disallow /tutors/singapore/tutor_details/
Disallow /u/in/
Disallow /u/hk/
Disallow /u/ca/
Disallow /u/us/
Disallow /u/au/
Disallow /u/my/
Disallow /u/uk/
Disallow /australia/
Disallow /api/merge_layer.php
Disallow /asia_travel/search/searchredirect.php
Disallow /asia_travel/travel_sites/whats_nearby/
Disallow /asia_travel/travel/resize_images.php
Disallow /routing/
Disallow /api/map/world.cgi
Disallow /sd_mobile/direction/
Disallow /lyricadvisor/nice_song.php
Disallow /etoday/send_article.php
Disallow /food_editorials/topics/
Disallow /travel_guide/editorials/other/
Disallow /resize_images.php
Disallow /feedback.php
Disallow /indonesia/jakarta/explain_property_error.php
Disallow /malaysia/kl/explain_property_error.php
Disallow /stock_images/travel/show_resize_image.php
Disallow /indonesia/jakarta/nearby_enlarge.php
Disallow /indonesia/jakarta/nearby_list.php
Disallow /malaysia/kl/nearby_list.php
Disallow /asiarooms/kl/travel.php
Disallow /web/
Disallow /api/map/world.cgi
Disallow /world.cgi
Disallow /jabodetabek.cgi
Disallow /hotels
Disallow /hotel_reservations
Disallow /email.php