allrecipes.asia
robots.txt
Robots Exclusion Standard data for allrecipes.asia
Resource Scan
Scan Details
| Site Domain | allrecipes.asia |
| Base Domain | allrecipes.asia |
| Scan Status | Failed |
| Failure Reason | Scan timed out. |
| Last Scan | 2024-06-08T15:27:58+00:00 |
| Next Scan | 2024-06-15T15:27:58+00:00 |
Last Successful Scan
| Scanned | 2022-04-09T21:52:09+00:00 |
| URL | http://allrecipes.asia/robots.txt |
| Redirect | https://www.allrecipes.com/robots.txt |
| Redirect Domain | www.allrecipes.com |
| Redirect Base | allrecipes.com |
| Response IP | 52.84.45.13 |
| Found | Yes |
| Hash | 5ab117abe39a2f2e8a7d055d48f3393d7148dacb0df9a2d27db50c5a84d9076b |
| SimHash | 8f14eab6c411 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /account/ |
| Disallow | /admin/ |
| Disallow | /personal-recipe/ |
| Disallow | /recipe-tools/ |
| Disallow | /search/ |
| Disallow | /custom-recipe/ |
| Disallow | /content/partner/ |
| Disallow | /featured/ |
| Allow | /featured/Thanksgiving2020 |
| Disallow | /cook/my/ |
| Disallow | /cook/* |
| Disallow | /*fullrecipenutrition |
| Disallow | /dyn/ |
| Disallow | /home/ |
| Disallow | /labs/ |
| Disallow | /order/ |
| Disallow | /child/ |
| Disallow | /photos/ |
| Disallow | /common/ |
| Disallow | /content/ |
| Disallow | /private/ |
| Disallow | /premium/ |
| Disallow | /templates/ |
| Disallow | /templatedata/ |
| Disallow | /videos/theme/ |
| Disallow | /videos/widgets/ |
| Disallow | /*.jhtml |
| Disallow | /*___itemUrl___ |
| Disallow | /embed |
| Disallow | /oembed |
| Disallow | /sweepstakes/rules/ |
| Disallow | /sweepstakes/facts/ |
| Disallow | /go? |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 1 |
slurp
| Rule | Path |
|---|---|
| Disallow | /account/ |
| Disallow | /admin/ |
| Disallow | /personal-recipe/ |
| Disallow | /recipe-tools/ |
| Disallow | /search/ |
| Disallow | /custom-recipe/ |
| Disallow | /new-this-month/ |
| Disallow | /content/partner/ |
| Disallow | /featured/ |
| Disallow | /cook/my/ |
| Disallow | /cook/* |
| Disallow | /*fullrecipenutrition |
| Disallow | /dyn/ |
| Disallow | /home/ |
| Disallow | /labs/ |
| Disallow | /order/ |
| Disallow | /child/ |
| Disallow | /photos/ |
| Disallow | /common/ |
| Disallow | /content/ |
| Disallow | /private/ |
| Disallow | /premium/ |
| Disallow | /templates/ |
| Disallow | /templatedata/ |
| Disallow | /videos/theme/ |
| Disallow | /videos/widgets/ |
| Disallow | /*.jhtml |
| Disallow | /*___itemUrl___ |
| Disallow | /embed |
| Disallow | /oembed |
| Disallow | /sweepstakes/rules/ |
| Disallow | /sweepstakes/facts/ |
| Disallow | /go? |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 0.05 |
googlebot
| Rule | Path |
|---|---|
| Disallow | /account/ |
| Disallow | /admin/ |
| Disallow | /personal-recipe/ |
| Disallow | /recipe-tools/ |
| Disallow | /search* |
| Disallow | /custom-recipe/ |
| Disallow | /new-this-month/ |
| Disallow | /content/partner/ |
| Disallow | /cook/my/ |
| Disallow | /cook/* |
| Disallow | /*fullrecipenutrition |
| Disallow | /dyn/ |
| Disallow | /home/ |
| Disallow | /labs/ |
| Disallow | /order/ |
| Disallow | /child/ |
| Disallow | /photos/ |
| Disallow | /common/ |
| Disallow | /content/ |
| Disallow | /private/ |
| Disallow | /premium/ |
| Disallow | /templates/ |
| Disallow | /templatedata/ |
| Disallow | /videos/theme/ |
| Disallow | /videos/widgets/ |
| Disallow | /*.jhtml |
| Disallow | /*___itemUrl___ |
| Disallow | /embed |
| Disallow | /oembed |
| Disallow | /syndication/ |
| Disallow | /sweepstakes/rules/ |
| Disallow | /sweepstakes/facts/ |
| Disallow | /go? |
mediapartners-google
| Rule | Path |
|---|---|
| Disallow | /account/ |
| Disallow | /admin/ |
| Disallow | /personal-recipe/ |
| Disallow | /recipe-tools/ |
| Disallow | /search* |
| Disallow | /custom-recipe/ |
| Disallow | /new-this-month/ |
| Disallow | /content/partner/ |
| Disallow | /featured/ |
| Disallow | /cook/my/ |
| Disallow | /cook/* |
| Disallow | /*fullrecipenutrition |
| Disallow | /dyn/ |
| Disallow | /home/ |
| Disallow | /labs/ |
| Disallow | /order/ |
| Disallow | /child/ |
| Disallow | /photos/ |
| Disallow | /common/ |
| Disallow | /content/ |
| Disallow | /private/ |
| Disallow | /premium/ |
| Disallow | /templates/ |
| Disallow | /templatedata/ |
| Disallow | /videos/theme/ |
| Disallow | /videos/widgets/ |
| Disallow | /*.jhtml |
| Disallow | /*___itemUrl___ |
| Disallow | /embed |
| Disallow | /oembed |
| Disallow | /sweepstakes/rules/ |
| Disallow | /sweepstakes/facts/ |
| Disallow | /go? |
bingbot
| Rule | Path |
|---|---|
| Disallow | /account/ |
| Disallow | /admin/ |
| Disallow | /personal-recipe/ |
| Disallow | /recipe-tools/ |
| Disallow | /search/ |
| Disallow | /custom-recipe/ |
| Disallow | /new-this-month/ |
| Disallow | /content/partner/ |
| Disallow | /featured/ |
| Disallow | /cook/my/ |
| Disallow | /cook/* |
| Disallow | /*fullrecipenutrition |
| Disallow | /dyn/ |
| Disallow | /home/ |
| Disallow | /labs/ |
| Disallow | /order/ |
| Disallow | /child/ |
| Disallow | /photos/ |
| Disallow | /common/ |
| Disallow | /content/ |
| Disallow | /private/ |
| Disallow | /premium/ |
| Disallow | /templates/ |
| Disallow | /templatedata/ |
| Disallow | /videos/theme/ |
| Disallow | /videos/widgets/ |
| Disallow | /*.jhtml |
| Disallow | /*___itemUrl___ |
| Disallow | /embed |
| Disallow | /oembed |
| Disallow | /sweepstakes/rules/ |
| Disallow | /sweepstakes/facts/ |
| Disallow | /go? |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 0.05 |
msnbot
| Rule | Path |
|---|---|
| Disallow | /account/ |
| Disallow | /admin/ |
| Disallow | /personal-recipe/ |
| Disallow | /recipe-tools/ |
| Disallow | /search/ |
| Disallow | /custom-recipe/ |
| Disallow | /new-this-month/ |
| Disallow | /content/partner/ |
| Disallow | /featured/ |
| Disallow | /cook/my/ |
| Disallow | /cook/* |
| Disallow | /*fullrecipenutrition |
| Disallow | /dyn/ |
| Disallow | /home/ |
| Disallow | /labs/ |
| Disallow | /order/ |
| Disallow | /child/ |
| Disallow | /photos/ |
| Disallow | /common/ |
| Disallow | /content/ |
| Disallow | /private/ |
| Disallow | /premium/ |
| Disallow | /templates/ |
| Disallow | /templatedata/ |
| Disallow | /videos/theme/ |
| Disallow | /videos/widgets/ |
| Disallow | /*.jhtml |
| Disallow | /*___itemUrl___ |
| Disallow | /embed |
| Disallow | /oembed |
| Disallow | /sweepstakes/rules/ |
| Disallow | /sweepstakes/facts/ |
| Disallow | /go? |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 0.05 |
duckduckbot
| Rule | Path |
|---|---|
| Disallow | /account/ |
| Disallow | /admin/ |
| Disallow | /personal-recipe/ |
| Disallow | /recipe-tools/ |
| Disallow | /search/ |
| Disallow | /custom-recipe/ |
| Disallow | /new-this-month/ |
| Disallow | /content/partner/ |
| Disallow | /featured/ |
| Disallow | /cook/my/ |
| Disallow | /cook/* |
| Disallow | /*fullrecipenutrition |
| Disallow | /dyn/ |
| Disallow | /home/ |
| Disallow | /labs/ |
| Disallow | /order/ |
| Disallow | /child/ |
| Disallow | /photos/ |
| Disallow | /common/ |
| Disallow | /content/ |
| Disallow | /private/ |
| Disallow | /premium/ |
| Disallow | /templates/ |
| Disallow | /templatedata/ |
| Disallow | /videos/theme/ |
| Disallow | /videos/widgets/ |
| Disallow | /*.jhtml |
| Disallow | /*___itemUrl___ |
| Disallow | /embed |
| Disallow | /oembed |
| Disallow | /sweepstakes/rules/ |
| Disallow | /sweepstakes/facts/ |
| Disallow | /go? |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 0.05 |
baiduspider
| Rule | Path |
|---|---|
| Disallow | /account/ |
| Disallow | /admin/ |
| Disallow | /personal-recipe/ |
| Disallow | /recipe-tools/ |
| Disallow | /search/ |
| Disallow | /custom-recipe/ |
| Disallow | /new-this-month/ |
| Disallow | /content/partner/ |
| Disallow | /featured/ |
| Disallow | /cook/my/ |
| Disallow | /cook/* |
| Disallow | /*fullrecipenutrition |
| Disallow | /dyn/ |
| Disallow | /home/ |
| Disallow | /labs/ |
| Disallow | /order/ |
| Disallow | /child/ |
| Disallow | /photos/ |
| Disallow | /common/ |
| Disallow | /content/ |
| Disallow | /private/ |
| Disallow | /premium/ |
| Disallow | /templates/ |
| Disallow | /templatedata/ |
| Disallow | /videos/theme/ |
| Disallow | /videos/widgets/ |
| Disallow | /*.jhtml |
| Disallow | /*___itemUrl___ |
| Disallow | /embed |
| Disallow | /oembed |
| Disallow | /sweepstakes/rules/ |
| Disallow | /sweepstakes/facts/ |
| Disallow | /go? |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 0.05 |
yandexbot
| Rule | Path |
|---|---|
| Disallow | /account/ |
| Disallow | /admin/ |
| Disallow | /personal-recipe/ |
| Disallow | /recipe-tools/ |
| Disallow | /search/ |
| Disallow | /custom-recipe/ |
| Disallow | /new-this-month/ |
| Disallow | /content/partner/ |
| Disallow | /featured/ |
| Disallow | /cook/my/ |
| Disallow | /cook/* |
| Disallow | /*fullrecipenutrition |
| Disallow | /dyn/ |
| Disallow | /home/ |
| Disallow | /labs/ |
| Disallow | /order/ |
| Disallow | /child/ |
| Disallow | /photos/ |
| Disallow | /common/ |
| Disallow | /content/ |
| Disallow | /private/ |
| Disallow | /premium/ |
| Disallow | /templates/ |
| Disallow | /templatedata/ |
| Disallow | /videos/theme/ |
| Disallow | /videos/widgets/ |
| Disallow | /*.jhtml |
| Disallow | /*___itemUrl___ |
| Disallow | /embed |
| Disallow | /oembed |
| Disallow | /sweepstakes/rules/ |
| Disallow | /sweepstakes/facts/ |
| Disallow | /go? |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 0.05 |
naverbot
| Rule | Path |
|---|---|
| Disallow | /account/ |
| Disallow | /admin/ |
| Disallow | /personal-recipe/ |
| Disallow | /recipe-tools/ |
| Disallow | /search/ |
| Disallow | /custom-recipe/ |
| Disallow | /new-this-month/ |
| Disallow | /content/partner/ |
| Disallow | /featured/ |
| Disallow | /cook/my/ |
| Disallow | /cook/* |
| Disallow | /*fullrecipenutrition |
| Disallow | /dyn/ |
| Disallow | /home/ |
| Disallow | /labs/ |
| Disallow | /order/ |
| Disallow | /child/ |
| Disallow | /photos/ |
| Disallow | /common/ |
| Disallow | /content/ |
| Disallow | /private/ |
| Disallow | /premium/ |
| Disallow | /templates/ |
| Disallow | /templatedata/ |
| Disallow | /videos/theme/ |
| Disallow | /videos/widgets/ |
| Disallow | /*.jhtml |
| Disallow | /*___itemUrl___ |
| Disallow | /embed |
| Disallow | /oembed |
| Disallow | /sweepstakes/rules/ |
| Disallow | /sweepstakes/facts/ |
| Disallow | /go? |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 0.05 |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.allrecipes.com/sitemap.xml |
| sitemap | https://www.allrecipes.com/news-sitemap.xml |
Warnings
- 2 invalid lines.