flirtbox.co.za
robots.txt

Robots Exclusion Standard data for flirtbox.co.za

Resource Scan

Scan Details

Site Domain flirtbox.co.za
Base Domain flirtbox.co.za
Scan Status Ok
Last Scan2024-10-02T22:28:41+00:00
Next Scan 2024-10-09T22:28:41+00:00

Last Scan

Scanned2024-10-02T22:28:41+00:00
URL https://flirtbox.co.za/robots.txt
Domain IPs 104.21.20.174, 172.67.193.67, 2606:4700:3031::6815:14ae, 2606:4700:3033::ac43:c143
Response IP 172.67.193.67
Found Yes
Hash f7ba3f00f02f0e740ad51557f6f31e223146bef01e0fdb09e05f76ee1bb28cc3
SimHash 275999c0a7f4

Groups

*

Rule Path
Disallow /hotornot.php
Disallow /api/*
Disallow /registerNew.php
Disallow /rateAction.php
Disallow /login.php
Disallow /reportNew.php
Disallow /read.php
Disallow /reply.php

mediapartners-google

Rule Path
Disallow

mj12bot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

piplbot

Rule Path
Disallow /

searchmetricsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

grapeshotcrawler

Rule Path
Disallow /

xovibot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

stackrambler

Rule Path
Disallow /

googlebot

Rule Path
Disallow register.php

googlebot

Rule Path
Disallow report2.php

admantx

Rule Path
Disallow /

proximic

Rule Path
Disallow /

mediapartners-google*

Rule Path
Disallow

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 120

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 120

mj12bot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

teoma

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 120

sitesnagger

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

fetch

Rule Path
Disallow /

offline explorer

Rule Path
Disallow /

teleport

Rule Path
Disallow /

teleportpro

Rule Path
Disallow /

webzip

Rule Path
Disallow /

httrack

Rule Path
Disallow /

xenu

Rule Path
Disallow /

larbin

Rule Path
Disallow /

libwww

Rule Path
Disallow /

zyborg

Rule Path
Disallow /

download ninja

Rule Path
Disallow /

wgetx

Rule Path
Disallow /

npbot

Rule Path
Disallow /

crystalsemanticsbot

Rule Path
Disallow /

yisouspider

Rule Path
Disallow /

bl.uk_lddc_bot*

Rule Path
Disallow /

unisterbot*

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

ezooms/1.0

Rule Path
Disallow /

mozilla/5.0 (compatible; ezooms/1.0; ezooms.bot[at]gmail[dot]com)

Rule Path
Disallow /

sistrix crawler

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

*

Rule Path
Disallow /checkConfirmationCode.php
Disallow *report2.php*
Disallow /reportNew.php
Disallow /report2.php*
Disallow /usrimg.php
Disallow /profile.php
Disallow /admin.php
Disallow /admin/
Disallow /includes/
Disallow /themes/
Disallow /blocks/
Disallow /manual/
Disallow /modules/
Disallow /language/
Disallow /secureCode.php
Disallow /secureIDCheck.php
Disallow /rateAction.php
Disallow /rateAction.php*

Comments

  • disobeying robots.txt is charged with 5 Euros / 1000 requests + 1000 Euros
  • service charge + legal fees
  • Ezooms
  • Disallow: Sistrix
  • Disallow: Sistrix
  • Disallow: SEOkicks-Robot