Log In Sign Up

pxl.gv.com.sg
robots.txt

Robots Exclusion Standard data for pxl.gv.com.sg

Resource Scan

Scan Details

Site Domain pxl.gv.com.sg
Base Domain gv.com.sg
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2024-09-14T05:26:11+00:00
Next Scan 2024-12-13T05:26:11+00:00

Last Successful Scan

Scanned2021-10-19T08:14:10+00:00
URL https://pxl.gv.com.sg/robots.txt
Found Yes
Hash 6c06480aad20fc7494bc1dc3cac3c40dea0884243757d1268a55c50b723db309
SimHash 5b9ff54250d3

Groups

googlebot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 300

adsbot-google

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 300

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 300

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 300

twitterbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

yandex

Rule Path
Disallow /

proximic

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

aihitbot

Rule Path
Disallow /

infopath

Rule Path
Disallow /

infopath.2

Rule Path
Disallow /

swebot

Rule Path
Disallow /

ec2linkfinder

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

searchmetericsbot

Rule Path
Disallow /

wbsearchbot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

sosospider

Rule Path
Disallow /

ip-web-crawler.com

Rule Path
Disallow /

netestate ne crawler

Rule Path
Disallow /

aboundexbot

Rule Path
Disallow /

meanpathbot

Rule Path
Disallow /

mail.ru

Rule Path
Disallow /

spbot

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /

linkpadbot

Rule Path
Disallow /

easouspider

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

wotbox

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

aboundexbot

Rule Path
Disallow /

xovibot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

a6-indexer

Rule Path
Disallow /

riddler

Rule Path
Disallow /

loadtimebot

Rule Path
Disallow /

obot

Rule Path
Disallow /

mojeekbot

Rule Path
Disallow /

memorybot

Rule Path
Disallow /

ltx71

Rule Path
Disallow /

advbot

Rule Path
Disallow /

smtbot

Rule Path
Disallow /

yisouspider

Rule Path
Disallow /

lssrocketcrawler

Rule Path
Disallow /

panscient.com

Rule Path
Disallow /

vscooter

Rule Path
Disallow /

psbot

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

twiceler

Rule Path
Disallow /

yandex

Rule Path
Disallow /

taptubot

Rule Path
Disallow /

googlebot-image

Rule Path
Disallow /

twengabot

Rule Path
Disallow /

sitebot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

*

Rule Path
Disallow /
Back to top

Comments

  • Google Bot
  • Microsoft Bot
  • MSN Bot
  • Twitter Bot - Does not implement [Crawl-delay]
  • Bad Bots
  • Any Robot
Back to top