allrecipes.com.au
robots.txt
Robots Exclusion Standard data for allrecipes.com.au
Resource Scan
Scan Details
Site Domain | allrecipes.com.au |
Base Domain | allrecipes.com.au |
Scan Status | Failed |
Failure Reason | Scan timed out. |
Last Scan | 2024-06-08T15:19:28+00:00 |
Next Scan | 2024-06-15T15:19:28+00:00 |
Last Successful Scan
Scanned | 2022-04-05T02:36:39+00:00 |
URL | http://allrecipes.com.au/robots.txt |
Redirect | https://www.allrecipes.com/robots.txt |
Redirect Domain | www.allrecipes.com |
Redirect Base | allrecipes.com |
Response IP | 52.85.3.125 |
Found | Yes |
Hash | 5ab117abe39a2f2e8a7d055d48f3393d7148dacb0df9a2d27db50c5a84d9076b |
SimHash | 8f14eab6c411 |
Groups
*
Rule | Path |
---|---|
Disallow | /account/ |
Disallow | /admin/ |
Disallow | /personal-recipe/ |
Disallow | /recipe-tools/ |
Disallow | /search/ |
Disallow | /custom-recipe/ |
Disallow | /content/partner/ |
Disallow | /featured/ |
Allow | /featured/Thanksgiving2020 |
Disallow | /cook/my/ |
Disallow | /cook/* |
Disallow | /*fullrecipenutrition |
Disallow | /dyn/ |
Disallow | /home/ |
Disallow | /labs/ |
Disallow | /order/ |
Disallow | /child/ |
Disallow | /photos/ |
Disallow | /common/ |
Disallow | /content/ |
Disallow | /private/ |
Disallow | /premium/ |
Disallow | /templates/ |
Disallow | /templatedata/ |
Disallow | /videos/theme/ |
Disallow | /videos/widgets/ |
Disallow | /*.jhtml |
Disallow | /*___itemUrl___ |
Disallow | /embed |
Disallow | /oembed |
Disallow | /sweepstakes/rules/ |
Disallow | /sweepstakes/facts/ |
Disallow | /go? |
Other Records
Field | Value |
---|---|
crawl-delay | 1 |
slurp
Rule | Path |
---|---|
Disallow | /account/ |
Disallow | /admin/ |
Disallow | /personal-recipe/ |
Disallow | /recipe-tools/ |
Disallow | /search/ |
Disallow | /custom-recipe/ |
Disallow | /new-this-month/ |
Disallow | /content/partner/ |
Disallow | /featured/ |
Disallow | /cook/my/ |
Disallow | /cook/* |
Disallow | /*fullrecipenutrition |
Disallow | /dyn/ |
Disallow | /home/ |
Disallow | /labs/ |
Disallow | /order/ |
Disallow | /child/ |
Disallow | /photos/ |
Disallow | /common/ |
Disallow | /content/ |
Disallow | /private/ |
Disallow | /premium/ |
Disallow | /templates/ |
Disallow | /templatedata/ |
Disallow | /videos/theme/ |
Disallow | /videos/widgets/ |
Disallow | /*.jhtml |
Disallow | /*___itemUrl___ |
Disallow | /embed |
Disallow | /oembed |
Disallow | /sweepstakes/rules/ |
Disallow | /sweepstakes/facts/ |
Disallow | /go? |
Other Records
Field | Value |
---|---|
crawl-delay | 0.05 |
googlebot
Rule | Path |
---|---|
Disallow | /account/ |
Disallow | /admin/ |
Disallow | /personal-recipe/ |
Disallow | /recipe-tools/ |
Disallow | /search* |
Disallow | /custom-recipe/ |
Disallow | /new-this-month/ |
Disallow | /content/partner/ |
Disallow | /cook/my/ |
Disallow | /cook/* |
Disallow | /*fullrecipenutrition |
Disallow | /dyn/ |
Disallow | /home/ |
Disallow | /labs/ |
Disallow | /order/ |
Disallow | /child/ |
Disallow | /photos/ |
Disallow | /common/ |
Disallow | /content/ |
Disallow | /private/ |
Disallow | /premium/ |
Disallow | /templates/ |
Disallow | /templatedata/ |
Disallow | /videos/theme/ |
Disallow | /videos/widgets/ |
Disallow | /*.jhtml |
Disallow | /*___itemUrl___ |
Disallow | /embed |
Disallow | /oembed |
Disallow | /syndication/ |
Disallow | /sweepstakes/rules/ |
Disallow | /sweepstakes/facts/ |
Disallow | /go? |
mediapartners-google
Rule | Path |
---|---|
Disallow | /account/ |
Disallow | /admin/ |
Disallow | /personal-recipe/ |
Disallow | /recipe-tools/ |
Disallow | /search* |
Disallow | /custom-recipe/ |
Disallow | /new-this-month/ |
Disallow | /content/partner/ |
Disallow | /featured/ |
Disallow | /cook/my/ |
Disallow | /cook/* |
Disallow | /*fullrecipenutrition |
Disallow | /dyn/ |
Disallow | /home/ |
Disallow | /labs/ |
Disallow | /order/ |
Disallow | /child/ |
Disallow | /photos/ |
Disallow | /common/ |
Disallow | /content/ |
Disallow | /private/ |
Disallow | /premium/ |
Disallow | /templates/ |
Disallow | /templatedata/ |
Disallow | /videos/theme/ |
Disallow | /videos/widgets/ |
Disallow | /*.jhtml |
Disallow | /*___itemUrl___ |
Disallow | /embed |
Disallow | /oembed |
Disallow | /sweepstakes/rules/ |
Disallow | /sweepstakes/facts/ |
Disallow | /go? |
bingbot
Rule | Path |
---|---|
Disallow | /account/ |
Disallow | /admin/ |
Disallow | /personal-recipe/ |
Disallow | /recipe-tools/ |
Disallow | /search/ |
Disallow | /custom-recipe/ |
Disallow | /new-this-month/ |
Disallow | /content/partner/ |
Disallow | /featured/ |
Disallow | /cook/my/ |
Disallow | /cook/* |
Disallow | /*fullrecipenutrition |
Disallow | /dyn/ |
Disallow | /home/ |
Disallow | /labs/ |
Disallow | /order/ |
Disallow | /child/ |
Disallow | /photos/ |
Disallow | /common/ |
Disallow | /content/ |
Disallow | /private/ |
Disallow | /premium/ |
Disallow | /templates/ |
Disallow | /templatedata/ |
Disallow | /videos/theme/ |
Disallow | /videos/widgets/ |
Disallow | /*.jhtml |
Disallow | /*___itemUrl___ |
Disallow | /embed |
Disallow | /oembed |
Disallow | /sweepstakes/rules/ |
Disallow | /sweepstakes/facts/ |
Disallow | /go? |
Other Records
Field | Value |
---|---|
crawl-delay | 0.05 |
msnbot
Rule | Path |
---|---|
Disallow | /account/ |
Disallow | /admin/ |
Disallow | /personal-recipe/ |
Disallow | /recipe-tools/ |
Disallow | /search/ |
Disallow | /custom-recipe/ |
Disallow | /new-this-month/ |
Disallow | /content/partner/ |
Disallow | /featured/ |
Disallow | /cook/my/ |
Disallow | /cook/* |
Disallow | /*fullrecipenutrition |
Disallow | /dyn/ |
Disallow | /home/ |
Disallow | /labs/ |
Disallow | /order/ |
Disallow | /child/ |
Disallow | /photos/ |
Disallow | /common/ |
Disallow | /content/ |
Disallow | /private/ |
Disallow | /premium/ |
Disallow | /templates/ |
Disallow | /templatedata/ |
Disallow | /videos/theme/ |
Disallow | /videos/widgets/ |
Disallow | /*.jhtml |
Disallow | /*___itemUrl___ |
Disallow | /embed |
Disallow | /oembed |
Disallow | /sweepstakes/rules/ |
Disallow | /sweepstakes/facts/ |
Disallow | /go? |
Other Records
Field | Value |
---|---|
crawl-delay | 0.05 |
duckduckbot
Rule | Path |
---|---|
Disallow | /account/ |
Disallow | /admin/ |
Disallow | /personal-recipe/ |
Disallow | /recipe-tools/ |
Disallow | /search/ |
Disallow | /custom-recipe/ |
Disallow | /new-this-month/ |
Disallow | /content/partner/ |
Disallow | /featured/ |
Disallow | /cook/my/ |
Disallow | /cook/* |
Disallow | /*fullrecipenutrition |
Disallow | /dyn/ |
Disallow | /home/ |
Disallow | /labs/ |
Disallow | /order/ |
Disallow | /child/ |
Disallow | /photos/ |
Disallow | /common/ |
Disallow | /content/ |
Disallow | /private/ |
Disallow | /premium/ |
Disallow | /templates/ |
Disallow | /templatedata/ |
Disallow | /videos/theme/ |
Disallow | /videos/widgets/ |
Disallow | /*.jhtml |
Disallow | /*___itemUrl___ |
Disallow | /embed |
Disallow | /oembed |
Disallow | /sweepstakes/rules/ |
Disallow | /sweepstakes/facts/ |
Disallow | /go? |
Other Records
Field | Value |
---|---|
crawl-delay | 0.05 |
baiduspider
Rule | Path |
---|---|
Disallow | /account/ |
Disallow | /admin/ |
Disallow | /personal-recipe/ |
Disallow | /recipe-tools/ |
Disallow | /search/ |
Disallow | /custom-recipe/ |
Disallow | /new-this-month/ |
Disallow | /content/partner/ |
Disallow | /featured/ |
Disallow | /cook/my/ |
Disallow | /cook/* |
Disallow | /*fullrecipenutrition |
Disallow | /dyn/ |
Disallow | /home/ |
Disallow | /labs/ |
Disallow | /order/ |
Disallow | /child/ |
Disallow | /photos/ |
Disallow | /common/ |
Disallow | /content/ |
Disallow | /private/ |
Disallow | /premium/ |
Disallow | /templates/ |
Disallow | /templatedata/ |
Disallow | /videos/theme/ |
Disallow | /videos/widgets/ |
Disallow | /*.jhtml |
Disallow | /*___itemUrl___ |
Disallow | /embed |
Disallow | /oembed |
Disallow | /sweepstakes/rules/ |
Disallow | /sweepstakes/facts/ |
Disallow | /go? |
Other Records
Field | Value |
---|---|
crawl-delay | 0.05 |
yandexbot
Rule | Path |
---|---|
Disallow | /account/ |
Disallow | /admin/ |
Disallow | /personal-recipe/ |
Disallow | /recipe-tools/ |
Disallow | /search/ |
Disallow | /custom-recipe/ |
Disallow | /new-this-month/ |
Disallow | /content/partner/ |
Disallow | /featured/ |
Disallow | /cook/my/ |
Disallow | /cook/* |
Disallow | /*fullrecipenutrition |
Disallow | /dyn/ |
Disallow | /home/ |
Disallow | /labs/ |
Disallow | /order/ |
Disallow | /child/ |
Disallow | /photos/ |
Disallow | /common/ |
Disallow | /content/ |
Disallow | /private/ |
Disallow | /premium/ |
Disallow | /templates/ |
Disallow | /templatedata/ |
Disallow | /videos/theme/ |
Disallow | /videos/widgets/ |
Disallow | /*.jhtml |
Disallow | /*___itemUrl___ |
Disallow | /embed |
Disallow | /oembed |
Disallow | /sweepstakes/rules/ |
Disallow | /sweepstakes/facts/ |
Disallow | /go? |
Other Records
Field | Value |
---|---|
crawl-delay | 0.05 |
naverbot
Rule | Path |
---|---|
Disallow | /account/ |
Disallow | /admin/ |
Disallow | /personal-recipe/ |
Disallow | /recipe-tools/ |
Disallow | /search/ |
Disallow | /custom-recipe/ |
Disallow | /new-this-month/ |
Disallow | /content/partner/ |
Disallow | /featured/ |
Disallow | /cook/my/ |
Disallow | /cook/* |
Disallow | /*fullrecipenutrition |
Disallow | /dyn/ |
Disallow | /home/ |
Disallow | /labs/ |
Disallow | /order/ |
Disallow | /child/ |
Disallow | /photos/ |
Disallow | /common/ |
Disallow | /content/ |
Disallow | /private/ |
Disallow | /premium/ |
Disallow | /templates/ |
Disallow | /templatedata/ |
Disallow | /videos/theme/ |
Disallow | /videos/widgets/ |
Disallow | /*.jhtml |
Disallow | /*___itemUrl___ |
Disallow | /embed |
Disallow | /oembed |
Disallow | /sweepstakes/rules/ |
Disallow | /sweepstakes/facts/ |
Disallow | /go? |
Other Records
Field | Value |
---|---|
crawl-delay | 0.05 |
Other Records
Field | Value |
---|---|
sitemap | https://www.allrecipes.com/sitemap.xml |
sitemap | https://www.allrecipes.com/news-sitemap.xml |
Warnings
- 2 invalid lines.