Log In Sign Up

freearenas.com
robots.txt

Robots Exclusion Standard data for freearenas.com

Resource Scan

Scan Details

Site Domain freearenas.com
Base Domain freearenas.com
Scan Status Ok
Last Scan2024-10-09T12:16:04+00:00
Next Scan 2024-10-16T12:16:04+00:00

Last Scan

Scanned2024-10-09T12:16:04+00:00
URL https://freearenas.com/robots.txt
Domain IPs 2406:da18:9d0:143f:2124:4e9c:36a9:d9de, 52.221.42.138
Response IP 52.221.42.138
Found Yes
Hash fe40bf4db33dd3ac70983f18ea58ec4571dbce05d7f5628282fbc986c553f0b5
SimHash 7424d156b3b5

Groups

*

Rule Path
Allow /

*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 150

googlebot

Rule Path
Allow /

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 15

baiduspider

Rule Path
Disallow /

baiduspider+

Rule Path
Disallow /

baiduspider+ (+; http://www.baidu.com/search/spider.htm; ubuntu; compatible; version; platform; .net clr 2.0.50727; version)

Rule Path
Disallow /

baiduspider+(+http://www.baidu.com/search/spider.htm)

Rule Path
Disallow /

baiduspider+(+http://www.baidu.com/search/spider_jp.html)

Rule Path
Disallow /

fatbot

Rule Path
Disallow /

mozilla/5.0 (compatible; fatbot 2.0; http://www.thefind.com/main/crawlerfaqs.fhtml)

Rule Path
Disallow /

geniebot

Rule Path
Disallow /

geniebot (http://64.5.245.11/faq/faq.html)

Rule Path
Disallow /

geniebot enash@genieknows.com

Rule Path
Disallow /

geniebot wgao@genieknows.com

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

gigabot/1.0

Rule Path
Disallow /

gigabot/2.0

Rule Path
Disallow /

gigabot/2.0 (http://www.gigablast.com/spider.html)

Rule Path
Disallow /

gigabot/2.0/gigablast.com/spider.html

Rule Path
Disallow /

gigabot/2.0; http://www.gigablast.com/spider.html

Rule Path
Disallow /

gigabot/2.0att

Rule Path
Disallow /

gigabot/3.0 (http://www.gigablast.com/spider.html)

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

mj12bot/v1.0.0 (http://majestic12.co.uk/bot.php?+)

Rule Path
Disallow /

mj12bot/v1.0.1 (http://majestic12.co.uk/bot.php?+)

Rule Path
Disallow /

mj12bot/v1.0.2 (http://majestic12.co.uk/bot.php?+)

Rule Path
Disallow /

mj12bot/v1.0.3 (http://majestic12.co.uk/bot.php?+)

Rule Path
Disallow /

mj12bot/v1.0.4 (http://majestic12.co.uk/bot.php?+)

Rule Path
Disallow /

mj12bot/v1.0.5 (http://majestic12.co.uk/bot.php?+)

Rule Path
Disallow /

mj12bot/v1.0.6 (http://majestic12.co.uk/bot.php?+)

Rule Path
Disallow /

mj12bot/v1.0.7 (http://majestic12.co.uk/bot.php?+)

Rule Path
Disallow /

mj12bot/v1.0.8 (http://majestic12.co.uk/bot.php?+)

Rule Path
Disallow /

mj12bot/v1.1.0 (http://majestic12.co.uk/bot.php?+)

Rule Path
Disallow /

mj12bot/v1.1.1 (http://majestic12.co.uk/bot.php?+)

Rule Path
Disallow /

mj12bot/v1.1.2 (http://majestic12.co.uk/bot.php?+)

Rule Path
Disallow /

mj12bot/v1.2.0 (http://majestic12.co.uk/bot.php?+)

Rule Path
Disallow /

omniexplorer_bot

Rule Path
Disallow /

omniexplorer_bot/1.07 (+http://www.omni-explorer.com) internet categorizer

Rule Path
Disallow /

omniexplorer_bot/1.09 (+http://www.omni-explorer.com)

Rule Path
Disallow /

omniexplorer_bot/1.09 (+http://www.omni-explorer.com) cars crawler

Rule Path
Disallow /

omniexplorer_bot/1.09 (+http://www.omni-explorer.com) internet indexer

Rule Path
Disallow /

omniexplorer_bot/1.10 (+http://www.omni-explorer.com)

Rule Path
Disallow /

omniexplorer_bot/1.10 (+http://www.omni-explorer.com) jobs crawler

Rule Path
Disallow /

omniexplorer_bot/2.47 (+http://www.omni-explorer.com) worldindexer

Rule Path
Disallow /

omniexplorer_bot/2.93 (+http://www.omni-explorer.com) worldindexer

Rule Path
Disallow /

omniexplorer_bot/3.06c (+http://www.omni-explorer.com) worldindexer

Rule Path
Disallow /

omniexplorer_bot/3.06d (+http://www.omni-explorer.com) worldindexer

Rule Path
Disallow /

omniexplorer_bot/3.10 (+http://www.omni-explorer.com) worldindexer

Rule Path
Disallow /

omniexplorer_bot/3.10a (+http://www.omni-explorer.com) worldindexer

Rule Path
Disallow /

omniexplorer_bot/3.11a (+http://www.omni-explorer.com) worldindexer

Rule Path
Disallow /

omniexplorer_bot/3.11c (+http://www.omni-explorer.com) worldindexer

Rule Path
Disallow /

omniexplorer_bot/3.20b (+http://www.omni-explorer.com) worldindexer

Rule Path
Disallow /

omniexplorer_bot/3.21 (+http://www.omni-explorer.com) worldindexer

Rule Path
Disallow /

omniexplorer_bot/3.23 (+http://www.omni-explorer.com) worldindexer

Rule Path
Disallow /

omniexplorer_bot/3.24 (+http://www.omni-explorer.com) worldindexer

Rule Path
Disallow /

omniexplorer_bot/3.26b (+http://www.omni-explorer.com) worldindexer

Rule Path
Disallow /

omniexplorer_bot/3.26c (+http://www.omni-explorer.com) worldindexer

Rule Path
Disallow /

omniexplorer_bot/3.28 (+http://www.omni-explorer.com) worldindexer

Rule Path
Disallow /

omniexplorer_bot/3.94b (+http://www.omni-explorer.com) worldindexer

Rule Path
Disallow /

omniexplorer_bot/3.95 (+http://www.omni-explorer.com) worldindexer

Rule Path
Disallow /

omniexplorer_bot/3.98b (+http://www.omni-explorer.com) worldindexer

Rule Path
Disallow /

omniexplorer_bot/3.99 (+http://www.omni-explorer.com) worldindexer

Rule Path
Disallow /

omniexplorer_bot/4.32 (+http://www.omni-explorer.com) worldindexer

Rule Path
Disallow /

omniexplorer_bot/5.25 (+http://www.omni-explorer.com) worldindexer

Rule Path
Disallow /

omniexplorer_bot/5.73 (+http://www.omni-explorer.com) worldindexer

Rule Path
Disallow /

omniexplorer_bot/5.77e (+http://www.omni-explorer.com) worldindexer

Rule Path
Disallow /

omniexplorer_bot/6.10.0 (+http://www.omni-explorer.com) worldindexer

Rule Path
Disallow /

omniexplorer_bot/6.10.13 (+http://www.omni-explorer.com) worldindexer

Rule Path
Disallow /

omniexplorer_bot/6.10.7 (+http://www.omni-explorer.com) worldindexer

Rule Path
Disallow /

omniexplorer_bot/6.10.9 (+http://www.omni-explorer.com) worldindexer

Rule Path
Disallow /

omniexplorer_bot/6.70 (+http://www.omni-explorer.com) worldindexer

Rule Path
Disallow /

sbider

Rule Path
Disallow /

sbider/0.7 (sbider; http://www.sitesell.com/sbider.html; http://support.sitesell.com/contact-support.html)

Rule Path
Disallow /

sbider/0.8-dev (sbider; http://www.sitesell.com/sbider.html; http://support.sitesell.com/contact-support.html)

Rule Path
Disallow /

sbider/sbider-0.8.2-dev (http://www.sitesell.com/sbider.html)

Rule Path
Disallow /

shopwiki

Rule Path
Disallow /

shopwiki/1.0 ( +http://www.shopwiki.com/wiki/help:bot)

Rule Path
Disallow /

webalta crawler

Rule Path
Disallow /

webalta crawler/1.3.33 (http://www.webalta.net/ru/about_webmaster.html) (windows; u; windows nt 5.1; ru-ru)

Rule Path
Disallow /

webalta crawler/1.3.34 (http://www.webalta.net/ru/about_webmaster.html) (windows; u; windows nt 5.1; ru-ru)

Rule Path
Disallow /

ezooms/1.0

Rule Path
Disallow /

mozilla/3.0+(compatible;+indy+library)

Rule Path
Disallow /

*

Rule Path
Disallow /admin/
Back to top

Comments

  • list folders robots are not allowed to index
  • list specific files robots are not allowed to index
Back to top