yellowpages.com.au
robots.txt

Robots Exclusion Standard data for yellowpages.com.au

Resource Scan

Scan Details

Site Domain yellowpages.com.au
Base Domain yellowpages.com.au
Scan Status Ok
Last Scan2024-10-26T19:51:28+00:00
Next Scan 2024-11-02T19:51:28+00:00

Last Scan

Scanned2024-10-26T19:51:28+00:00
URL https://yellowpages.com.au/robots.txt
Redirect https://www.yellowpages.com.au/robots.txt
Redirect Domain www.yellowpages.com.au
Redirect Base yellowpages.com.au
Domain IPs 23.32.29.105, 23.32.29.98
Redirect IPs 23.32.29.105, 23.32.29.98
Response IP 23.32.29.98
Found Yes
Hash dc2dcc8f936c547213ce2ab8aaae3849338c6e265c97e18fce008108de7eb10e
SimHash 6f192120c51c

Groups

*

Rule Path
Disallow /auth
Disallow /autosuggest
Disallow /performSendToMobile
Disallow /renderSendToMobile
Disallow /getDirections
Disallow /login
Disallow /onlineSolution
Disallow /Orp
Disallow /Orp/
Disallow /facebook/status
Disallow /reviews
Disallow /static
Disallow /choose
Disallow /review/*.html
Disallow /bi/
Disallow /yp/
Disallow /widget/
Disallow /login/
Disallow /map/
Disallow /profile/
Disallow /shop/
Disallow /dataprotection*
Disallow /xmlrpc
Disallow /toprated
Disallow /online-staging
Disallow /find/b/
Disallow /*?display=

yahooseeker/m1a1-r2d2

Rule Path
Disallow /

slurp

Rule Path
Disallow /

msnbot_mobile

Rule Path
Disallow /

jumpbot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

baiduspider/2.0

Rule Path
Disallow /

baiduspider-image

Rule Path
Disallow /

baiduspider-video

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

Other Records

Field Value
sitemap https://www.yellowpages.com.au/sitemap.xml.gz
sitemap https://www.yellowpages.com.au/YellowAtom.xml.gz
sitemap https://www.yellowpages.com.au/YellowArticleAtom.xml.gz