finduslocal.net
robots.txt

Robots Exclusion Standard data for finduslocal.net

Resource Scan

Scan Details

Site Domain finduslocal.net
Base Domain finduslocal.net
Scan Status Failed
Failure ReasonScan timed out.
Last Scan2024-04-29T02:14:22+00:00
Next Scan 2024-06-28T02:14:22+00:00

Last Successful Scan

Scanned2021-10-02T20:04:29+00:00
URL http://finduslocal.net/robots.txt
Redirect https://www.finduslocal.com/robots.txt
Redirect Domain www.finduslocal.com
Redirect Base finduslocal.com
Found Yes
Hash d7e21e95ae3b4d600c26d6065e5cae35dfb910987d899dbcef56e0b45cb133c2
SimHash 2a1450d5d499

Groups

googlebot

Rule Path
Disallow

googlebot-mobile

Rule Path
Disallow

mediapartners-google

Rule Path
Disallow

apis-google

Rule Path
Disallow

google-structured-data-testing-tool

Rule Path
Disallow

googlebot-video

Rule Path
Disallow

googlebot-image

Rule Path
Disallow

adsbot-google

Rule Path
Disallow

adsbot-google-mobile

Rule Path
Disallow

adsbot-google-mobile-apps

Rule Path
Disallow

googlebot-news

Rule Path
Disallow

bingbot

Rule Path
Disallow /googlecache/

Other Records

Field Value
crawl-delay 10

msnbot

Rule Path
Disallow /googlecache/

Other Records

Field Value
crawl-delay 10

msnbot-media

Rule Path
Disallow /googlecache/

Other Records

Field Value
crawl-delay 10

adidxbot

Rule Path
Disallow /googlecache/

bingpreview

Rule Path
Disallow /googlecache/

slurp

Rule Path
Disallow /googlecache/

Other Records

Field Value
crawl-delay 5

yandexbot

Rule Path
Disallow /googlecache/

yandexmobilebot

Rule Path
Disallow /googlecache/

applebot

Rule Path
Disallow /googlecache/

duckduckbot

Rule Path
Disallow /googlecache/

seznambot

Rule Path
Disallow /googlecache/

teoma

Rule Path
Disallow /googlecache/

baidu

Rule Path
Disallow /googlecache/

facebookexternalhit

Rule Path
Disallow /googlecache/

twitterbot

Rule Path
Disallow /googlecache/

linkedinbot

Rule Path
Disallow /googlecache/

pinterest

Rule Path
Disallow /googlecache/

ahrefsbot

Rule Path
Disallow /googlecache/

mj12bot

Rule Path
Disallow /googlecache/

semrushbot

Rule Path
Disallow /googlecache/

ninjabot

Rule Path
Disallow /googlecache/

kingkevinbot

Rule Path
Disallow /googlecache/

scoutjet

Rule Path
Disallow /googlecache/

ia_archiver

Rule Path
Disallow /googlecache/

archive.org_bot

Rule Path
Disallow *?*
Disallow /googlecache/

spbot

Rule Path
Disallow /

shoemoneytoolsbot

Rule Path
Disallow /

*

Rule Path
Disallow /googlecache/

Other Records

Field Value
sitemap https://www.finduslocal.com/sitemap.xml

Comments

  • Updated on November the 4th, 2019
  • Updated on Jan 13, 2021- Provided by Webnauts

Warnings

  • 2 invalid lines.