award.tabelog.com
robots.txt

Robots Exclusion Standard data for award.tabelog.com

Resource Scan

Scan Details

Site Domain award.tabelog.com
Base Domain tabelog.com
Scan Status Ok
Last Scan2025-06-01T15:10:21+00:00
Next Scan 2025-07-01T15:10:21+00:00

Last Scan

Scanned2025-06-01T15:10:21+00:00
URL https://award.tabelog.com/robots.txt
Domain IPs 3.114.133.109, 54.168.169.53
Response IP 54.168.169.53
Found Yes
Hash 6080205b3a1c4711b8f18a8fe1254f8df116ba0fdd505a7164a3719b89901f9c
SimHash 6144d85083b3

Groups

twitterbot/1.0

Rule Path
Allow /

*

Rule Path
Disallow

baiduspider

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

baidumobaider

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

baiduimagespider

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

psbot

Rule Path
Disallow /

becomebot

Rule Path
Disallow /

teoma

Rule Path
Disallow /

ask jeeves

Rule Path
Disallow /

listran

Rule Path
Disallow /

gptbot

Rule Path
Disallow /