Log In Sign Up

green-japan.com
robots.txt

Robots Exclusion Standard data for green-japan.com

Resource Scan

Scan Details

Site Domain green-japan.com
Base Domain green-japan.com
Scan Status Ok
Last Scan2024-05-10T12:32:12+00:00
Next Scan 2024-06-09T12:32:12+00:00

Last Scan

Scanned2024-05-10T12:32:12+00:00
URL https://green-japan.com/robots.txt
Redirect https://www.green-japan.com:443/robots.txt
Redirect Domain www.green-japan.com
Redirect Base green-japan.com
Domain IPs 13.33.30.110, 13.33.30.119, 13.33.30.58, 13.33.30.90
Redirect IPs 13.33.30.110, 13.33.30.119, 13.33.30.58, 13.33.30.90
Response IP 13.33.30.110
Found Yes
Hash bc4da0c56453370a8b5e019190ea5c178ba421f280238d91ead3d907fb29596a
SimHash 281b9b85a3eb

Groups

*

Rule Path
Disallow /bridge/
Disallow /ks/
Disallow /admin/
Disallow /mypage0*
Disallow /contents/lp/
Disallow /registrations/
Disallow /favorites/
Disallow /messages/
Disallow /job_applies/
Disallow /profiles/
Disallow /user_searches/
Disallow /browsing_history*
Disallow /policies/faq
Disallow /contents/about
Disallow /contents/terms
Disallow /contents/privacy
Disallow /mobile/contents/lp/
Disallow /mobile/registrations/
Disallow /mobile/favorites/
Disallow /mobile/messages/
Disallow /mobile/job_applies/
Disallow /mobile/profiles
Disallow /mobile/browsing_history
Disallow /client/
Disallow /requests/
Disallow /pdf/accessibility_document.pdf
Disallow /pr/3861$
Disallow /company/646$
Disallow /company/647$
Disallow /company/648$
Disallow /company/174$
Disallow /company/791$
Disallow /company/2154$
Disallow /company/3861$
Disallow /company/1177$
Disallow /jobs/1177$
Disallow /job/9122$
Disallow /job/9123$
Disallow /job/9124$
Disallow /job/9125$
Disallow /job/9436$
Disallow /job/9437$
Disallow /job/1408$
Disallow /job/3358$
Disallow /job/3490$
Disallow /job/8050$
Disallow /job/11719$
Disallow /job/12352$
Disallow /job/12479$
Disallow /job/12503$
Disallow /job/12508$
Disallow /job/12509$
Disallow /job/12570$
Disallow /job/12747$
Disallow /job/12775$
Disallow /job/12855$
Disallow /job/13042$
Disallow /job/13480$
Disallow /job/13718$
Disallow /job/13726$
Disallow /job/15463$
Disallow /job/16088$
Disallow /job/16117$
Disallow /job/34396$
Disallow /job/34412$
Disallow /job/34632$
Disallow /job/34633$
Disallow /job/36090$
Disallow /job/38043$
Disallow /job/39721$
Disallow /job/39763$
Disallow /job/39764$
Disallow /job/40685$
Disallow /job/40694$
Disallow /job/40695$
Disallow /job/40696$
Disallow /job/42329$
Disallow /job/42333$
Disallow /job/42349$
Disallow /job/42350$
Disallow /job/42351$
Disallow /job/42354$
Disallow /job/42391$
Disallow /job/42347$

baiduspider

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

trovitbot

Rule Path
Disallow /

rankactivelinkbot

Rule Path
Disallow /

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 120

baiduspider+

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 120

yandexbot

Rule Path
Disallow /

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

y!j-brj/yats crawler

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

jooblebot

Rule Path
Disallow /

mj12bot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 300
Back to top

Other Records

Field Value
sitemap http://www.green-japan.com/sitemap.xml
Back to top

Comments

  • See http://www.robotstxt.org/wc/norobots for documentation on how to use the robots.txt file
Back to top