Log In Sign Up

modeldns.com.au
robots.txt

Robots Exclusion Standard data for modeldns.com.au

Resource Scan

Scan Details

Site Domain modeldns.com.au
Base Domain modeldns.com.au
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2024-09-26T12:53:08+00:00
Next Scan 2024-10-03T12:53:08+00:00

Last Successful Scan

Scanned2024-08-26T12:52:43+00:00
URL https://modeldns.com.au/robots.txt
Domain IPs 104.21.1.4, 172.67.186.113, 2606:4700:3030::ac43:ba71, 2606:4700:3037::6815:104
Response IP 172.67.186.113
Found Yes
Hash ce021fb157264d2565e3581126f974c8117aeb6b94229ee7dec5de1135563f61
SimHash 615a7f9544d2

Groups

googlebot
slurp
bingbot
msnbot
msnbot-media
mediapartners-google
googlebot-image
yahoo-mmcrawler
baiduspider
facebookexternalhit
facebot
semrushbot

Rule Path
Disallow

semrushbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

ahrefsbot

Rule Path
Disallow /member-centre/*
Disallow /newsletter/
Disallow /video-library/
Disallow /video_gallery/
Disallow /CVS
Disallow /*.svn$
Disallow /*.idea$
Disallow /*.sql$
Disallow /*.tgz$
Disallow /cgi-bin/
Disallow /cleanup.php
Disallow /apc.php
Disallow /memcache.php
Disallow /phpinfo.php

Other Records

Field Value
crawl-delay 20

*

No rules defined. All paths allowed.

Back to top

Comments

  • robots.txt
  • GENERAL SETTINGS
  • Enable robots.txt rules for all crawlers
  • Crawl-delay parameter: number of seconds to wait between successive requests to the same server.
  • Set a custom crawl rate if you're experiencing traffic problems with your server.
  • Crawl-delay: 30
  • Magento sitemap: uncomment and replace the URL to your Magento sitemap file
  • Sitemap: http://www.example.com/sitemap/sitemap.xml
  • DEVELOPMENT RELATED SETTINGS
  • Do not crawl development files and folders: CVS, svn directories and dump files
  • SERVER SETTINGS
  • Do not crawl common server technical folders and files
  • IMAGE CRAWLERS SETTINGS
  • Extra: Uncomment if you do not wish Google and Bing to index your images
  • User-agent: Googlebot-Image
  • Disallow: /
  • User-agent: msnbot-media
  • Disallow: /
Back to top