Log In Sign Up

www.city.ogi.lg.jp
robots.txt

Robots Exclusion Standard data for www.city.ogi.lg.jp

Resource Scan

Scan Details

Site Domain www.city.ogi.lg.jp
Base Domain ogi.lg.jp
Scan Status Ok
Last Scan2024-11-03T15:45:07+00:00
Next Scan 2024-12-03T15:45:07+00:00

Last Scan

Scanned2024-11-03T15:45:07+00:00
URL https://www.city.ogi.lg.jp/robots.txt
Domain IPs 96.17.96.11, 96.17.96.31
Response IP 23.44.5.40
Found Yes
Hash b826a082931b32ce8bb5e66b07542174296a33746def2fefe14061debecd62fa
SimHash 485c4490f21d

Groups

*

Rule Path
Disallow /manage/
Disallow /main/main/*
Disallow /main.php/main/*
Disallow /main//*
Disallow /hospital/manage/
Disallow /hospital/main/main/*
Disallow /hospital/main//*
Disallow /hospital/main.php/main/*

mj12bot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

mappy

Rule Path
Disallow /

coccocbot-web

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

slackbot

Rule Path
Disallow /

bbot

Rule Path
Disallow /

brands-bot

Rule Path
Disallow /

brands-bot-logo

Rule Path
Disallow /

seekport crawler

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

dnbcrawler-analytics

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

linespider

Rule Path
Disallow /

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 3600

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 3600

applebot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 3600

twitterbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 3600

ndl-japan

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 3600

twitterbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 3600
Back to top