Log In Sign Up

middleeastbasketball.com
robots.txt

Robots Exclusion Standard data for middleeastbasketball.com

Resource Scan

Scan Details

Site Domain middleeastbasketball.com
Base Domain middleeastbasketball.com
Scan Status Ok
Last Scan2024-11-10T18:31:36+00:00
Next Scan 2024-11-17T18:31:36+00:00

Last Scan

Scanned2024-11-10T18:31:36+00:00
URL https://www.middleeastbasketball.com/robots.txt
Domain IPs 64.150.180.207
Response IP 64.150.180.207
Found Yes
Hash 81a61f9943e9abc7f52a1c9483bcb44d72eacdf7bbf6a6007c9e3110f0e3b162
SimHash 02c4d44081c2

Groups

*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

mj12bot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 100

baiduspider

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1000

ahrefsbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

clickagy intelligence bot

Rule Path
Disallow /

clickagy intelligence bot v2

Rule Path
Disallow /

clickagy

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

rogerbot/1.1

Rule Path
Disallow /

rogerbot/1.2

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

baiduspider-video

Rule Path
Disallow /

baiduspider-image

Rule Path
Disallow /

baiduspider+

Rule Path
Disallow /

baiduspider+(+http://www.baidu.com/search/spider.htm)

Rule Path
Disallow /

baiduspider/2.0;+http://www.baidu.com/search/spider.html

Rule Path
Disallow /

baiduspider/2.0

Rule Path
Disallow /

mozilla/5.0 (compatible; baiduspider/2.0; +http://www.baidu.com/search/spider.html)

Rule Path
Disallow /

spider

Rule Path
Disallow /

yandex

Rule Path
Disallow /

yandex.com

Rule Path
Disallow /

yandexbot/3.0

Rule Path
Disallow /

sosospider+

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

ezooms/1.0

Rule Path
Disallow /

mozilla/5.0 (compatible; ezooms/1.0; ezooms.bot[at]gmail[dot]com)

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

wotbox/2.01

Rule Path
Disallow /

dwbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

seokicks

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

nutch

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

maxpointcrawler

Rule Path
Disallow /

getintent

Rule Path
Disallow /

admantx

Rule Path
Disallow /

admantx platform semantic analyzer - admantx inc. - www.admantx.com - support@admantx.com

Rule Path
Disallow /

admantx platform semantic analyzer us - turn - admantx inc. - www.admantx.com - support@admantx.com

Rule Path
Disallow /

getintent crawler

Rule Path
Disallow /

proximic

Rule Path
Disallow /

proximic

Rule Path
Disallow /

integralads

Rule Path
Disallow /

genieo

Rule Path
Disallow /

linkdexbot/

Rule Path
Disallow /

linkdexbot/2.0

Rule Path
Disallow /

linkdexbot/2.1

Rule Path
Disallow /

clickagy intelligence bot

Rule Path
Disallow /

clickagy intelligence bot v2

Rule Path
Disallow /

clickagy

Rule Path
Disallow /

searchie/1.0

Rule Path
Disallow /

searchie/1.0 (a storm-based crawler; https://www.searchie.org; admin@searchie.org)

Rule Path
Disallow /

searchie

Rule Path
Disallow /

www.searchie.org

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

mediatoolkitbot

Rule Path
Disallow /

ltx71

Rule Path
Disallow /

kraken

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

teeraidbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

spbot

Rule Path
Disallow /

dow jones searchbot

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

diffbot/0.1

Rule Path
Disallow /

mozilla/5.0 (windows; u; windows nt 5.1; en-us; rv:1.9.1.2) gecko/20090729 firefox/3.5.2 (.net clr 3.5.30729; diffbot/0.1; +http://www.diffbot.com)

Rule Path
Disallow /

pathdefender

Rule Path
Disallow /

pathdefender/1.0

Rule Path
Disallow /

pathdefender/1.0; +http://www.pathdefender.com/help/crawler

Rule Path
Disallow /

brokenlinkcheck.com/1.1

Rule Path
Disallow /

brokenlinkcheck.com/1.2

Rule Path
Disallow /

spbot

Rule Path
Disallow /
Back to top

Comments

  • Baiduspider
  • Yandex
  • Sosospider
  • Ezooms
  • Sogou
  • AhrefsBot
  • 80legs.com
  • ADmantX
  • linkdexbot
  • Diffbot
  • PathDefender
Back to top

Warnings

  • 8 invalid lines.
Back to top