Log In Sign Up

apnarm.com.au
robots.txt

Robots Exclusion Standard data for apnarm.com.au

Resource Scan

Scan Details

Site Domain apnarm.com.au
Base Domain apnarm.com.au
Scan Status Ok
Last Scan2025-12-22T14:30:58+00:00
Next Scan 2026-01-21T14:30:58+00:00

Last Scan

Scanned2025-12-22T14:30:58+00:00
URL http://apnarm.com.au/robots.txt
Redirect https://www.newscorpaustralia.com/robots.txt
Redirect Domain www.newscorpaustralia.com
Redirect Base newscorpaustralia.com
Domain IPs 16.12.74.7, 3.5.164.244, 3.5.165.232, 3.5.165.242, 3.5.167.211, 3.5.168.225, 52.95.129.95, 52.95.135.23
Redirect IPs 23.49.8.165
Response IP 184.51.96.167
Found Yes
Hash 3c65912a378b14257aed10a69c4184ca3e8dcbb87977b929808538f9280867f6
SimHash 695457b086bf

Groups

screamingfrogseospider
comscorecrawler
moatbot
adbeat
integralads.com
expo9
outbrain
semasio.net
weborama-fetcher
yahooadmonitoring
netseer.com
semrushbot
omgili.com
facebookcrawler
facebookexternalhit
facebot
ias_crawler
ias_wombles
iasbot
identrics
meltwaternews
links.streem.com.au
webz-bot
linkedinbot
whatsapp
redditbot
mediapartners-google
google-inspectiontool
adsbot-google
storebot-google
adsbot-google-mobile
adsbot-google-mobile-apps
apis-google
google web light
google-ads-creatives-assistant
google-adwords
google-amphtml
google-cloudvertexbot
google-play-newsstand
google-read-aloud
google-safety
google-site-verification
google-structured-data-testing-tool
googleimageproxy
twitterbot
applebot
applebot-aggregator
applenewsbot
adidxbot
admantx
admantx-usbatch/3.1
ahc
ahc/2.1
ahrefsbot
amazonadbot
arquivo.pt
bidswitchbot/1.0
bnf.fr_bot
botify
checkmy
chrome-lighthouse
cognitiveseo
contentking
criteobot/0.1
datadog-synthetics
deepcrawl
dlvrit
echoboxbot/1.0
flipboard
grapeshotcrawler
gumgum-bot
hyscore/1.0
ia_archiver
iframely
jetoctopus
labne
leikibot
lumarbot
mj12bot
newrelicpinger
okta
parselybot
peer39_crawler/1.0
photon
pingdom
pinterestbot
proofpoint
proximic
pubmatic
rogerbot
rytebot
schema-validator
semantic-scholar
sentry
serankingbot
similarwebbot
sirdatabot
sistrix
site24x7
sitebulb
siteimprovebot
slack-imgproxy
slackbot
slackbot-linkexpanding
skypeuripreview
smartologybot
snap url preview service
snapchat
socialflow
statuscake
taboolabot
telegrambot
thetradedesk
uptimerobot
yahoomailproxy
yahoo-linkpreview

Rule Path
Disallow

googlebot
googlebot-image
googlebot-news
googlebot-video
googleother
googleother-image
googleother-video
bingbot
bingpreview
microsoftpreview
msnbot
duckduckbot
slurp
yahoo
y!j-bot
baiduspider
mojeekbot
petalbot
qwantbot
seznambot
sogou
yandex
yandexbot
yandeximages
yeti
yisouspider

Rule Path
Disallow

google-extended

Rule Path
Disallow /

*

Rule Path
Allow /ads.txt
Allow /app-ads.txt
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.newscorpaustralia.com/sitemap_index.xml
Back to top

Comments

  • NOTICE: Collection of content and other data on www.newscorpaustralia.com through automated means is prohibited unless you have express written permission from the publisher of the website and may only be conducted for the limited purpose contained in said permission.
  • Website Terms of Use may be found at https://www.newscorpaustralia.com/terms-conditions/
  • Agent Specific Allowed Section
  • ==========================
  • Web Search Engines
  • All remaining visitors
  • ==========================
  • ==========================
Back to top

Warnings

  • 2 invalid lines.
Back to top