static-particlenews-com-1654278173.us-west-2.elb.amazonaws.com
robots.txt

Resource Scan

Scan Details

Site Domain static-particlenews-com-1654278173.us-west-2.elb.amazonaws.com
Base Domain static-particlenews-com-1654278173.us-west-2.elb.amazonaws.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2024-08-12T05:00:31+00:00
Next Scan 2024-11-10T05:00:31+00:00

Last Successful Scan

Scanned2023-04-20T21:03:56+00:00
URL http://static-particlenews-com-1654278173.us-west-2.elb.amazonaws.com/robots.txt
Redirect https://www.newsbreakapp.com/robots.txt
Redirect Domain www.newsbreakapp.com
Redirect Base newsbreakapp.com
Domain IPs 34.223.147.128, 35.82.66.204
Redirect IPs 44.224.238.62, 44.236.132.154
Response IP 44.224.238.62
Found Yes
Hash abc0e48ededd548b05aa4cd41915dcae8d5e34a35d0b3e3f5a2f3c21954d5049
SimHash 004c529387c2

Groups

ccbot

Rule Path
Disallow /

ccbot/2.0

Rule Path
Disallow /

ccbot/2.0 (http://commoncrawl.org/faq/)

Rule Path
Disallow /

wikido

Rule Path
Disallow /

fr_crawler

Rule Path
Disallow /

yandex

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

baiduspider-image

Rule Path
Disallow /

baiduspider-video

Rule Path
Disallow /

baiduspider-favo

Rule Path
Disallow /

baiduspider-news

Rule Path
Disallow /

baiduspider-cpro

Rule Path
Disallow /

baiduspider-ads

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

bitvorebot

Rule Path
Disallow /

blp_bbot

Rule Path
Disallow /

heritrix

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

kraken

Rule Path
Disallow /

moatbot

Rule Path
Disallow /

bhcbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

synthesio

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

brandonbot

Rule Path
Disallow /

germcrawler

Rule Path
Disallow /

sogou

Rule Path
Disallow /

exabot

Rule Path
Disallow /

maxpointcrawler

Rule Path
Disallow /

admantx

Rule Path
Disallow /

*

Rule Path
Disallow /_api/
Disallow /n/
Disallow /v/
Disallow /s/

twitterbot

Rule Path
Allow /n/
Allow /v/
Allow /s/

facebookexternalhit

Rule Path
Allow /n/
Allow /v/
Allow /s/

Other Records

Field Value
sitemap https://www.newsbreak.com/sitemap.xml

Comments

  • New crawlers to block 2016