Log In Sign Up

www.city.saga-kashima.lg.jp
robots.txt

Robots Exclusion Standard data for www.city.saga-kashima.lg.jp

Resource Scan

Scan Details

Site Domain www.city.saga-kashima.lg.jp
Base Domain saga-kashima.lg.jp
Scan Status Ok
Last Scan2024-11-03T15:23:44+00:00
Next Scan 2024-12-03T15:23:44+00:00

Last Scan

Scanned2024-11-03T15:23:44+00:00
URL https://www.city.saga-kashima.lg.jp/robots.txt
Domain IPs 96.17.96.11, 96.17.96.31
Response IP 23.44.5.40
Found Yes
Hash f8b82083ac1b59f35feab464bbb7dde4f990a62559faae2d508b4c052bf83e3f
SimHash 485cd080b699

Groups

*

Rule Path
Disallow /manage/
Disallow /rss/
Disallow /main/main/*
Disallow /main/kanko/*
Disallow /main/43.html?*
Disallow /main/223.html?*
Disallow /main/256.html?*

mj12bot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

mappy

Rule Path
Disallow /

coccocbot-web

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

slackbot

Rule Path
Disallow /

bbot

Rule Path
Disallow /

brands-bot

Rule Path
Disallow /

brands-bot-logo

Rule Path
Disallow /

seekport crawler

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

dnbcrawler-analytics

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

linespider

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

thesis-research-bot

Rule Path
Disallow /

fidget-spinner-bot

Rule Path
Disallow /

my-tiny-bot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

h-bot

Rule Path
Disallow /

bingbot

Rule Path
Disallow /main/223.html?*
Disallow /main/256.html?*

Other Records

Field Value
crawl-delay 10

icc-crawler

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

gptbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10
Back to top