Log In Sign Up

findershopping.com.au
robots.txt

Robots Exclusion Standard data for findershopping.com.au

Resource Scan

Scan Details

Site Domain findershopping.com.au
Base Domain findershopping.com.au
Scan Status Ok
Last Scan2024-11-11T07:23:54+00:00
Next Scan 2024-11-18T07:23:54+00:00

Last Scan

Scanned2024-11-11T07:23:54+00:00
URL https://findershopping.com.au/robots.txt
Redirect https://www.findershopping.com.au/robots.txt
Redirect Domain www.findershopping.com.au
Redirect Base findershopping.com.au
Domain IPs 18.155.68.14, 18.155.68.26, 18.155.68.6, 18.155.68.62
Redirect IPs 18.155.68.14, 18.155.68.26, 18.155.68.6, 18.155.68.62
Response IP 18.155.68.6
Found Yes
Hash 53fcb5589090fac7680d8c5b442608460c19867991cfc776c59c40aedb6e30a3
SimHash 9455ca21c854

Groups

*

Rule Path
Disallow /wordpress/
Disallow /redirect/*
Disallow /wp-login*
Disallow /*?coupon
Disallow /*?pID
Disallow /*?pid
Disallow /*?link
Disallow /*.pdf
Disallow /users
Disallow /reply/*
Disallow /forum
Disallow /topic
Disallow /wp-api/1/user/me/notifications*
Disallow /user/*
Disallow */nbn-tracker/ext?redirect*
Disallow /search/*
Disallow *?s=*
Disallow *?search=*
Disallow /*/search/page/*
Disallow */feed
Disallow /wordpress/wp-admin/admin-ajax.php?action=get_geoip_country_modal_data_ajax
Disallow /wp-json/finder/v1/geoip/country-modal-data

adsbot-google

Rule Path
Disallow

twitterbot

Rule Path
Disallow

facebookexternalhit/1.1

Rule Path
Disallow
Disallow /*?via
Disallow /*?*purpose=
Disallow /credit-score?sitewide=credit-score
Allow /*?ver
Allow /wordpress/wp-admin/admin-ajax.php?action=*
Disallow /*?fsd=
Disallow /wp-content/mu-plugins/calculators/findercalcs/process/lead_*
Disallow /network?*
Back to top

Other Records

Field Value
sitemap https://www.findershopping.com.au/sitemap_index.xml
Back to top

Comments

  • Prevent crawling searches.
  • https://finder.atlassian.net/browse/CWS-452
  • Prevent crawling search url parameters
  • https://finder.atlassian.net/browse/CON-229
  • Prevent crawling additional searches.
  • https://finder.atlassian.net/browse/CWS-497
  • https://finder.atlassian.net/browse/ENG-481
  • Allow Google AdsBot to crawl anything
  • Allow Twitterbot to crawl anything
  • https://finder.atlassian.net/browse/OPS-915
  • Allow Facebook to crawl anything
  • https://finder.atlassian.net/browse/CC-389
  • Don't crawl twitter links with encoded URLs
  • https://finder.atlassian.net/browse/FD-6467
  • https://finder.atlassian.net/browse/GXUSR-37
  • Crawl image versions
  • https://finder.atlassian.net/browse/FD-9630
  • https://finder.atlassian.net/browse/OPS-498
  • https://finder.atlassian.net/browse/PROJ-174
  • Whitelabel endpoint
  • https://finder.atlassian.net/browse/CST-3340
  • Main sitemap
Back to top