flirtbox.jp
robots.txt

Robots Exclusion Standard data for flirtbox.jp

Resource Scan

Scan Details

Site Domain flirtbox.jp
Base Domain flirtbox.jp
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2024-09-23T08:05:41+00:00
Next Scan 2024-12-22T08:05:41+00:00

Last Successful Scan

Scanned2024-02-27T07:40:23+00:00
URL https://flirtbox.jp/robots.txt
Domain IPs 104.21.24.216, 172.67.220.216, 2606:4700:3032::6815:18d8, 2606:4700:3032::ac43:dcd8
Response IP 104.21.24.216
Found Yes
Hash f7ba3f00f02f0e740ad51557f6f31e223146bef01e0fdb09e05f76ee1bb28cc3
SimHash 275999c0a7f4

Groups

*

Rule Path
Disallow /hotornot.php
Disallow /api/*
Disallow /registerNew.php
Disallow /rateAction.php
Disallow /login.php
Disallow /reportNew.php
Disallow /read.php
Disallow /reply.php

mediapartners-google

Rule Path
Disallow

mj12bot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

piplbot

Rule Path
Disallow /

searchmetricsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

grapeshotcrawler

Rule Path
Disallow /

xovibot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

stackrambler

Rule Path
Disallow /

googlebot

Rule Path
Disallow register.php

googlebot

Rule Path
Disallow report2.php

admantx

Rule Path
Disallow /

proximic

Rule Path
Disallow /

mediapartners-google*

Rule Path
Disallow

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 120

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 120

mj12bot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

teoma

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 120

sitesnagger

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

fetch

Rule Path
Disallow /

offline explorer

Rule Path
Disallow /

teleport

Rule Path
Disallow /

teleportpro

Rule Path
Disallow /

webzip

Rule Path
Disallow /

httrack

Rule Path
Disallow /

xenu

Rule Path
Disallow /

larbin

Rule Path
Disallow /

libwww

Rule Path
Disallow /

zyborg

Rule Path
Disallow /

download ninja

Rule Path
Disallow /

wgetx

Rule Path
Disallow /

npbot

Rule Path
Disallow /

crystalsemanticsbot

Rule Path
Disallow /

yisouspider

Rule Path
Disallow /

bl.uk_lddc_bot*

Rule Path
Disallow /

unisterbot*

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

ezooms/1.0

Rule Path
Disallow /

mozilla/5.0 (compatible; ezooms/1.0; ezooms.bot[at]gmail[dot]com)

Rule Path
Disallow /

sistrix crawler

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

*

Rule Path
Disallow /checkConfirmationCode.php
Disallow *report2.php*
Disallow /reportNew.php
Disallow /report2.php*
Disallow /usrimg.php
Disallow /profile.php
Disallow /admin.php
Disallow /admin/
Disallow /includes/
Disallow /themes/
Disallow /blocks/
Disallow /manual/
Disallow /modules/
Disallow /language/
Disallow /secureCode.php
Disallow /secureIDCheck.php
Disallow /rateAction.php
Disallow /rateAction.php*

Comments

  • disobeying robots.txt is charged with 5 Euros / 1000 requests + 1000 Euros
  • service charge + legal fees
  • Ezooms
  • Disallow: Sistrix
  • Disallow: Sistrix
  • Disallow: SEOkicks-Robot