Log In Sign Up

worldrecipes.eu
robots.txt

Robots Exclusion Standard data for worldrecipes.eu

Resource Scan

Scan Details

Site Domain worldrecipes.eu
Base Domain worldrecipes.eu
Scan Status Ok
Last Scan2024-11-09T12:21:34+00:00
Next Scan 2024-11-16T12:21:34+00:00

Last Scan

Scanned2024-11-09T12:21:34+00:00
URL https://worldrecipes.eu/robots.txt
Domain IPs 172.66.41.48, 172.66.42.208, 2606:4700:3108::ac42:2930, 2606:4700:3108::ac42:2ad0
Response IP 172.66.42.208
Found Yes
Hash 0ede29ddf2f998343e5d3598e6040263853d095fe527a5b584a45c6dcf5e45e0
SimHash 6e35f043d574

Groups

*
bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 25

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 15

duckduckbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 15

baiduspider

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 15

ia_archiver

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 15

yandex

Rule Path
Disallow /nova/
Disallow /index.php/

googlebot

Rule Path
Disallow /rel/topics

googlebot

Rule Path
Disallow /rel/topics/all-lt
Disallow /lt/*
Disallow /lv/*
Disallow /de/*
Disallow /pl/*
Disallow /es/*
Disallow /category/*
Disallow /search/*
Disallow /api/*
Back to top