Log In Sign Up

ifoodie.tw
robots.txt

Robots Exclusion Standard data for ifoodie.tw

Resource Scan

Scan Details

Site Domain ifoodie.tw
Base Domain ifoodie.tw
Scan Status Ok
Last Scan2024-11-09T09:46:21+00:00
Next Scan 2024-11-16T09:46:21+00:00

Last Scan

Scanned2024-11-09T09:46:21+00:00
URL https://ifoodie.tw/robots.txt
Domain IPs 107.178.252.150
Response IP 107.178.252.150
Found Yes
Hash 6c7dcf9c3d05e1a91fee90b159746ad33283547455c951cd3234d893222ce5ba
SimHash f13e023067b2

Groups

googleother

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

petalbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 30

bytespider

Rule Path
Disallow /

velenpublicwebcrawler

Rule Path
Disallow /

dotbot

Rule Path
Disallow /explore/

applebot

Rule Path
Disallow /explore/

amazonbot

Rule Path
Disallow /

proximic

Rule Path
Disallow /

serpstatbot

Rule Path
Disallow /explore/

trendictionbot

Rule Path
Disallow /explore/

mj12bot

Rule Path
Disallow /explore/

admantx

Rule Path
Disallow /explore/

Other Records

Field Value
crawl-delay 30

admantx

Rule Path
Disallow /explore/

Other Records

Field Value
crawl-delay 30

ias_crawler

Rule Path
Disallow /explore/

criteobot/0.1

Rule Path
Disallow /explore/

Other Records

Field Value
crawl-delay 30

ahrefsbot

Rule Path
Disallow

Other Records

Field Value
crawl-delay 30

grapeshot

Rule Path
Disallow

Other Records

Field Value
crawl-delay 30

*

Rule Path
Disallow /explore/*page

scrapy

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

dotbot
dotbot

Rule Path
Disallow /

buck

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /
Back to top

Comments

  • Google Other
  • bingbot
  • Search engine of Huawei
  • Disallow: /explore/
  • TikTok
  • https://velen.io/
  • Moz dotbot
  • Applebot
  • Amazonbot
  • Disallow: /explore/
  • Block Comscore Web Crawler
  • serpstatbot
  • trendictionBot
  • obeys robots.txt
  • admantx does not care about robots.txt
  • Used for Teads
  • ==== Update at 2024/1/6 ====
  • https://dataforseo.com/dataforseo-bot
  • User-agent: DataForSeoBot
  • Disallow: /explore/
  • Crawl-delay: 30
  • https://www.semrush.com/bot/
  • http://www.opensiteexplorer.org/dotbot
  • https://app.hypefactors.com/media-monitoring/about.html
  • https://babbar.tech/crawler
Back to top