Log In Sign Up

apkonline.net
robots.txt

Robots Exclusion Standard data for apkonline.net

Resource Scan

Scan Details

Site Domain apkonline.net
Base Domain apkonline.net
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2024-04-30T02:21:17+00:00
Next Scan 2024-07-29T02:21:17+00:00

Last Successful Scan

Scanned2024-01-01T22:34:23+00:00
URL https://apkonline.net/robots.txt
Redirect https://www.apkonline.net/robots.txt
Redirect Domain www.apkonline.net
Redirect Base apkonline.net
Domain IPs 104.26.0.63, 104.26.1.63, 172.67.74.254, 2606:4700:20::681a:13f, 2606:4700:20::681a:3f, 2606:4700:20::ac43:4afe
Redirect IPs 104.26.0.63, 104.26.1.63, 172.67.74.254, 2606:4700:20::681a:13f, 2606:4700:20::681a:3f, 2606:4700:20::ac43:4afe
Response IP 104.26.1.63
Found Yes
Hash 01124d5702211d6d699a3a841c584ac2eafb47947d86ef2c87d5724d189731da
SimHash ca665561057f

Groups

*

Rule Path
Disallow /administrator/
Disallow /ngx_pagespeed_beacon
Disallow /runapk/start-androidapk.html
Disallow /myapkdownloader/*
Disallow /myapkdownloader/
Disallow /apkdownloader.php
Disallow /apkdownloader.php*
Allow /

sogou web spider

Rule Path
Disallow /

weborama-fetcher

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

addthis.com (http://support.addthis.com/)

Rule Path
Disallow /

weborama-fetcher

Rule Path
Disallow /

proximic

Rule Path
Disallow /

grapeshot

Rule Path
Disallow /

proximic

Rule Path
Disallow /

seokicks

Rule Path
Disallow /

criteobot

Rule Path
Disallow /

dataforseo.com

Rule Path
Disallow /

a6-indexer

Rule Path
Disallow /

alphaseobot

Rule Path
Disallow /

alphaseobot-sa

Rule Path
Disallow /

aspiegelbot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

blackboard safeassign

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

crawler4j

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

liebaofast

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

mauibot (crawler.feedback+wc@gmail.com)

Rule Path
Disallow /

megaindex.ru/2.0

Rule Path
Disallow /

mqqbrowser

Rule Path
Disallow /

nimbostratus-bot/v1.3.2

Rule Path
Disallow /

qwant-news

Rule Path
Disallow /

qwantify

Rule Path
Disallow /

seekport crawler

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

sputnikbot/2.3

Rule Path
Disallow /

the knowledge ai

Rule Path
Disallow /

timpibot/0.8

Rule Path
Disallow /

tinytestbot

Rule Path
Disallow /

ucbrowser

Rule Path
Disallow /

yacybot

Rule Path
Disallow /

yeti

Rule Path
Disallow /

yisouspider

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.apkonline.net/sitemap.xml
Back to top

Comments

  • Block bots
  • RDH, 08.19.19: I really don't want to block Applebot, but for now, I am. It is crawling us too much
  • User-agent: Applebot
  • Disallow: /
  • RDH, 05.13.20: I really don't want to block bing, but for now, I am. It is also already in htaccess rules
  • User-agent: bingbot/2.0
  • Disallow: /
  • RDH, 06.30.21: Very temporary to get some relief.
  • User-Agent: Googlebot
  • Disallow: /
  • User-agent: SemrushBot
  • Disallow: /
  • User-agent: SemrushBot-SA
  • Disallow: /
  • RDH, 03/11/22:
  • Comment this out for JOT, who applied for a Crossref Similiarty Check account with TurnitIn;
  • User-agent: TurnitinBot
  • Disallow: /
  • User-agent: YandexBot
  • Disallow: /
  • User-agent: YandexBot/3.0
  • Disallow: /
Back to top