refari.co
robots.txt

Robots Exclusion Standard data for refari.co

Resource Scan

Scan Details

Site Domain refari.co
Base Domain refari.co
Scan Status Ok
Last Scan2024-10-24T22:24:07+00:00
Next Scan 2024-11-23T22:24:07+00:00

Last Scan

Scanned2024-10-24T22:24:07+00:00
URL https://refari.co/robots.txt
Redirect https://www.refari.co/robots.txt
Redirect Domain www.refari.co
Redirect Base refari.co
Domain IPs 170.64.250.23, 2400:6180:10:200::7e:6000
Redirect IPs 170.64.250.23
Response IP 170.64.250.23
Found Yes
Hash 1352135e00eb55c3822f0e2db3c40d973d6a901d21bbcc3a17abd9407b1d6f9c
SimHash cc365823c9e8

Groups

*

Rule Path
Disallow
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php
Disallow /wp-includes/
Allow /wp-includes/js/
Allow /wp-includes/images/
Disallow /wp-config-sample.php
Disallow /wp-admin/install.php
Disallow /wp-login.php
Disallow /wp-register.php
Disallow /cgi-bin/
Disallow /junk/
Disallow /tmp/
Disallow /.tmp/
Disallow /cdn-cgi/bm/cv/
Disallow /cdn-cgi/challenge-platform/
Disallow /.well-known/acme-challenge/
Disallow /trackback/
Disallow /xmlrpc.php
Disallow /feed/
Disallow /readme.html
Disallow /readme.*.html
Disallow /licence.txt
Disallow /license.txt
Disallow /installer.php
Disallow /install.php
Disallow /phpinfo.php
Disallow /?s=*
Disallow /search/*
Disallow /site.webmanifest
Disallow /wp-json/

nuclei
wikido
riddler
petalbot
zoominfobot
go-http-client
node/simplecrawler
cazoodlebot
dotbot/1.0
gigabot
barkrowler
blexbot
magpie-crawler

Rule Path
Disallow /

Other Records

Field Value
sitemap https://refari.co/sitemap_index.xml

Comments

  • Allow All Bots
  • Link to sitemap
  • Block the admin area but allow ajax callbacks
  • Block WP Includes
  • Block WP Install
  • Block Regsiter and Login
  • Block cgi-bin & temp
  • Block SSL Challlenge
  • Block Trackback, Feed and XMLrpc
  • Disallow Common
  • Block Search
  • Disallow Bot Spam
  • Block wp-json API
  • Reject zero benefit bots