Log In Sign Up

calcoastnews.com
robots.txt

Robots Exclusion Standard data for calcoastnews.com

Resource Scan

Scan Details

Site Domain calcoastnews.com
Base Domain calcoastnews.com
Scan Status Ok
Last Scan2024-09-24T13:08:04+00:00
Next Scan 2024-10-01T13:08:04+00:00

Last Scan

Scanned2024-09-24T13:08:04+00:00
URL https://calcoastnews.com/robots.txt
Domain IPs 159.135.28.65
Response IP 159.135.28.65
Found Yes
Hash 6e7192f16e8ca86ab544e982ee222d1482a888c2d1ed9e19d2ba847dce60dbbc
SimHash 06b6df735783

Groups

*

Rule Path
Disallow /wp-includes/
Disallow /tmp/
Disallow /images/2008/
Disallow /images/2009/
Disallow /images/2010/
Disallow /images/2011/
Disallow /images/2012/
Disallow /images/2013/
Disallow /images/2014/
Disallow /cgi-bin/
Disallow /wp-admin/
Disallow /search/
Disallow /author/
Disallow /calendar/admin/
Disallow /calendar/events/link/
Disallow /wp-content/
Disallow /ajax
Disallow /login/
Disallow /archives/
Disallow /login/
Disallow /random-event-images/
Disallow /*.css$
Disallow /*.js$
Disallow /wp-*
Disallow /trackback/
Disallow /*.inc$
Disallow /wp-comments-post.php
Disallow /admin-ajax.php
Allow /calendar/events/sitemap.php

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 25

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 25

flamingo_searchengine

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 25

gptbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

duggmirror

Rule Path
Disallow /

trapit

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

contextad

Rule Path
Disallow /

contextweb

Rule Path
Disallow /

linkdexbot

Rule Path
Disallow /

copyrightcheck

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

propowerbot/2.14

Rule Path
Disallow /

prowebwalker

Rule Path
Disallow /

scooperbot

Rule Path
Disallow /

psbot

Rule Path
Disallow /

riddler

Rule Path
Disallow /

infopath

Rule Path
Disallow /

maxpointcrawler

Rule Path
Disallow /

maxpointcrawler/nutch-1.1

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

ezooms.bot

Rule Path
Disallow /

buck

Rule Path
Disallow /

ias_crawler

Rule Path
Disallow /

istellabot

Rule Path
Disallow /

java 1.7 apache httpclient (linux x86_64) / gnowitnewsbot / contact information at http://www.gnowit.com

Rule Path
Disallow /

the knowledge ai

Rule Path
Disallow /

weborama-fetcher (+http://www.weborama.com)

Rule Path
Disallow /

velenpublicwebcrawler (velen.io)

Rule Path
Disallow /

sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm

Product Comment
sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm 07)
Rule Path
Disallow /

sc-downloader/2.0 (http://www.searchconcepts.ch)

Rule Path
Disallow /

owlin - https://www.owlin.com (contains googlebot for cookiewall)

Rule Path
Disallow /

omgili/0.5 +http://omgili.com

Rule Path
Disallow /

mozilla/5.0 (compatible; semrushbot/3~bl; +http://www.semrush.com/bot.html)

Rule Path
Disallow /

googlebot

Rule Path
Allow /
Allow /images/2012/

googlebot-news

Rule Path
Allow /
Disallow /calendar/events/sitemap.php

adsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 15
Back to top

Other Records

Field Value
sitemap http://calcoastnews.com/calendar/events/sitemap.php
sitemap http://calcoastnews.com/sitemap_index.xml
Back to top