paneco.com.sg
robots.txt

Robots Exclusion Standard data for paneco.com.sg

Resource Scan

Scan Details

Site Domain paneco.com.sg
Base Domain paneco.com.sg
Scan Status Ok
Last Scan2024-05-30T09:18:16+00:00
Next Scan 2024-06-06T09:18:16+00:00

Last Scan

Scanned2024-05-30T09:18:16+00:00
URL https://paneco.com.sg/robots.txt
Redirect https://www.paneco.com.sg/robots.txt
Redirect Domain www.paneco.com.sg
Redirect Base paneco.com.sg
Domain IPs 151.101.1.132, 151.101.129.132, 151.101.193.132, 151.101.65.132
Redirect IPs 151.101.1.132, 151.101.129.132, 151.101.193.132, 151.101.65.132
Response IP 199.232.45.132
Found Yes
Hash d990068c25e9cb63d847199644e4b4ee5e9e8a24d4d23c1bde93be35ee4ad6aa
SimHash d01f5e00cc80

Groups

facebookexternalhit/*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

adbeat_bot

Rule Path
Disallow /

ahrefsbot
ahrefssiteaudit

Rule Path
Disallow /

backdoorbot/1.0

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

becomebot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

cloudfind

Rule Path
Disallow /

copyrightcheck

Rule Path
Disallow /

crawler4j

Rule Path
Disallow /

dataprovider.com

Rule Path
Disallow /

domaincrawler

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

etaospider

Rule Path
Disallow /

femtosearchbot

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

go-http-client

Rule Path
Disallow /

heritrix

Rule Path
Disallow /

httrack

Rule Path
Disallow /

jaunt

Rule Path
Disallow /

linkdexbot

Rule Path
Disallow /

libwww

Rule Path
Disallow /

lumtelbot

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

microsoft url control
microsoft url control - 5.01.4511
microsoft url control - 6.00.8169

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

mozilla/5.0 (compatible; msie 10.0; windows nt 6.1; trident/6.0) linkcheck by siteimprove.com
mozilla/5.0 (compatible; msie 10.0; windows nt 6.1; trident/6.0) sitecheck-sitecrawl by siteimprove.com

Rule Path
Disallow /

msiecrawler

Rule Path
Disallow /

nl-crawler

Rule Path
Disallow /

npbot-1/2.0

Rule Path
Disallow /

npbot

Rule Path
Disallow /

nutch

Rule Path
Disallow /

paperlibot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

phpcrawl

Rule Path
Disallow /

psbot

Rule Path
Disallow /

python-requests

Rule Path
Disallow /

python-urllib

Rule Path
Disallow /

python/3.8 aiohttp

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

sbsearch

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

semrushbot
semrushbot
semrushbot-sa

Rule Path
Disallow /

seo-audit-check-bot

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

sftlnr - (http://sftlnr.com/)

Rule Path
Disallow /

sistrix crawler

Rule Path
Disallow /

siteimprove

Rule Path
Disallow /

spankbot

Rule Path
Disallow /

stormcrawler

Rule Path
Disallow /

surveybot

Rule Path
Disallow /

surveybot_ignoreip

Rule Path
Disallow /

telesoft

Rule Path
Disallow /

tocrawl/urldispatcher

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

velenpublicwebcrawler

Rule Path
Disallow /

webbandit
webbandit/3.50

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

webenhancer

Rule Path
Disallow /

webreaper

Rule Path
Disallow /

webvac

Rule Path
Disallow /

wget
wget/1.5.3
wget/1.6

Rule Path
Disallow /

www-collector-e

Rule Path
Disallow /

xenu
xenu's link sleuth 1.1c

Rule Path
Disallow /

zoominfobot

Rule Path
Disallow /

*

Rule Path
Disallow /checkout
Disallow /cart
Disallow /wine/cart
Disallow /orders
Disallow /order_history
Disallow /wine/order_history
Disallow /user
Disallow /account
Disallow /wine/account
Disallow /api
Disallow /password
Disallow /admin
Disallow /business-registration
Disallow /*/reviews/new
Disallow /stock_notifications
Disallow /wine/stock_notifications
Disallow /offline.html
Disallow /rss
Disallow /feed
Disallow /feeds

Other Records

Field Value
sitemap http://paneco-sg-moonshine-production.s3.amazonaws.com/sitemaps/sitemap.xml.gz