Log In Sign Up

chainvalleygifts.com.au
robots.txt

Robots Exclusion Standard data for chainvalleygifts.com.au

Resource Scan

Scan Details

Site Domain chainvalleygifts.com.au
Base Domain chainvalleygifts.com.au
Scan Status Ok
Last Scan2024-11-08T10:55:06+00:00
Next Scan 2024-12-08T10:55:06+00:00

Last Scan

Scanned2024-11-08T10:55:06+00:00
URL https://chainvalleygifts.com.au/robots.txt
Domain IPs 103.42.110.27, 2400:b800:3:27::1
Response IP 103.42.110.27
Found Yes
Hash 3881b99b3fc5b348504af9a9f97504167a2840e4ce2ea63c3b467b972aa3c303
SimHash ce5cd850f098

Groups

*

Rule Path
Disallow

bytespider

Rule Path
Disallow /

sucuri uptime monitor

Rule Path
Disallow /

cpanel-http-client

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

coccocbot-web

Rule Path
Disallow /

coccocbot-image

Rule Path
Disallow /

photon

Rule Path
Disallow /

spawning-ai

Rule Path
Disallow /

rc-crawler

Rule Path
Disallow /

pandalytics

Rule Path
Disallow /

symfony browserkit

Rule Path
Disallow /

wappalyzer

Rule Path
Disallow /

go-http-client

Rule Path
Disallow /

mozlila/5.0 (linux; android 7.0; sm-g892a bulid/nrd90m; wv)

Rule Path
Disallow /

nuclei
wikido
riddler
petalbot
zoominfobot
go-http-client
node/simplecrawler
cazoodlebot
dotbot/1.0
gigabot
barkrowler
blexbot
magpie-crawler

Rule Path
Disallow /
Disallow /?s=*
Disallow /search/*
Disallow /cdn-cgi/bm/cv/
Disallow /cdn-cgi/challenge-platform/
Back to top

Other Records

Field Value
sitemap https://www.chainvalleygifts.com.au/sitemap_index.xml
Back to top

Comments

  • START YOAST BLOCK
  • ---------------------------
  • ---------------------------
  • END YOAST BLOCK
  • We're experimenting with blocking search results to prevent search result spam
  • Prevent crawling CF challenge URLs
Back to top