bgospel.com
robots.txt

Robots Exclusion Standard data for bgospel.com

Resource Scan

Scan Details

Site Domain bgospel.com
Base Domain bgospel.com
Scan Status Ok
Last Scan2024-09-18T20:47:24+00:00
Next Scan 2024-09-25T20:47:24+00:00

Last Scan

Scanned2024-09-18T20:47:24+00:00
URL https://bgospel.com/robots.txt
Domain IPs 104.21.16.28, 172.67.166.11, 2606:4700:3032::6815:101c, 2606:4700:3034::ac43:a60b
Response IP 104.21.16.28
Found Yes
Hash e35b095e21b1409de33e103e29921581fca08f36e31952ec0dca254d15e73292
SimHash 0c355488cef8

Groups

*

Rule Path
Disallow
Disallow /cgi-bin/
Disallow /wp-admin/
Disallow /scripts/
Disallow /wp-includes/
Disallow /js/
Disallow /themes/
Disallow /tmp/
Disallow /wp-login/
Disallow /wp-content/uploads/wpo-plugins-tables-list.json
Disallow /wp-json/wp/v2/users/1
Disallow /wp-login.php
Disallow /?q=admin%2F
Disallow /?q=comment%2Freply%2F
Disallow /?q=filter%2Ftips%2F
Disallow /?q=node%2Fadd%2F
Disallow /?q=search%2F
Disallow /?q=user%2Fpassword%2F
Disallow /?q=user%2Fregister%2F
Disallow /?q=user%2Flogin%2F
Disallow /?q=user%2Flogout%2F
Disallow /CHANGELOG.txt
Disallow /cron.php
Disallow /INSTALL.mysql.txt
Disallow /INSTALL.pgsql.txt
Disallow /INSTALL.sqlite.txt
Disallow /install.php
Disallow /INSTALL.txt
Disallow /LICENSE.txt
Disallow /MAINTAINERS.txt
Disallow /update.php
Disallow /UPGRADE.txt
Disallow /xmlrpc.php
Disallow /*promotion%40jubau.com$
Disallow /*jubaugroup$
Disallow /en/*
Disallow /blog/*
Disallow /*-2$
Disallow /*b2gospel$
Disallow /*?cat=*
Disallow /?p=*
Disallow /Category/uncategorized/*
Disallow /*feed$
Disallow /newlogin*
Disallow /*jubau$
Disallow /*?attachment_id$
Disallow /*feed$
Disallow /newlogin*
Disallow /*?attachment_id$
Disallow /*.js?gcb=*
Disallow /*.js?cb=*
Disallow /*/rss2$
Disallow /*?swcfpc=1$
Disallow /tdb_templates/
Disallow /feed/
Disallow *rss

nutch

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

naverbot

Rule Path
Disallow /

aspiegelbot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

blackboard safeassign

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

crawler4j

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

nutch

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

naverbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

liebaofast

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

mauibot (crawler.feedback+wc@gmail.com)

Rule Path
Disallow /

megaindex.ru/2.0

Rule Path
Disallow /

mqqbrowser

Rule Path
Disallow /

nimbostratus-bot/v1.3.2

Rule Path
Disallow /

qwantify

Rule Path
Disallow /

seekport crawler

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

sputnikbot/2.3

Rule Path
Disallow /

the knowledge ai

Rule Path
Disallow /

timpibot/0.8

Rule Path
Disallow /

tinytestbot

Rule Path
Disallow /

ucbrowser

Rule Path
Disallow /

yacybot

Rule Path
Disallow /

yeti

Rule Path
Disallow /

yisouspider

Rule Path
Disallow /

awariobot/1.0

Rule Path
Disallow /

awariosmartbot/1.0

Rule Path
Disallow /

awariorssbot/1.0

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.bgospel.com/sitemap_index.xml
sitemap https://www.bgospel.com/post-sitemap1.xml
sitemap https://www.bgospel.com/category-sitemap.xml
sitemap https://www.bgospel.com/page-sitemap.xml

Comments

  • Paths (no clean URLs)
  • Files
  • bad URL
  • Block bots