osusume.mynavi.jp
robots.txt

Robots Exclusion Standard data for osusume.mynavi.jp

Resource Scan

Scan Details

Site Domain osusume.mynavi.jp
Base Domain mynavi.jp
Scan Status Ok
Last Scan2024-06-01T10:44:38+00:00
Next Scan 2024-07-01T10:44:38+00:00

Last Scan

Scanned2024-06-01T10:44:38+00:00
URL https://osusume.mynavi.jp/robots.txt
Domain IPs 34.149.181.249
Response IP 34.149.181.249
Found Yes
Hash c7fe19d5a46878a54202a20dc5dcdb3aedba48527278e47309ff65f9a1d013cc
SimHash 8b055e10ce83

Groups

megaindex.ru

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

icc-crawler

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 360

Other Records

Field Value
sitemap https://osusume.mynavi.jp/sitemapindex.xml
sitemap https://osusume.mynavi.jp/beauty/sitemapindex.xml