volxweb.org
robots.txt

Groups

*

Other Records

Comments

• robots.txt
• This file is to prevent the crawling and indexing of certain parts
• of your site by web crawlers and spiders run by sites like Yahoo!
• and Google. By telling these "robots" where not to go on your site,
• you save bandwidth and server resources.
• This file will be ignored unless it is at the root of your host:
• Used: http://example.com/robots.txt
• Ignored: http://example.com/site/robots.txt
• For more information about the robots.txt standard, see:
• http://www.robotstxt.org/robotstxt.html
• X Edit ant then upload files robots.text === for DIRs: /dkk/ /dvv/ /fvv/ ===
• X -------------------------------------------------------------------------------------------------
• X (1) Edit dates of /m/v*/a*u/rob-modif.txt : S18724 ... S18806 ...
• X (2) Multiply: fffffffffff; cp -pfv /m/v*/a*u/rob-modif.txt /m/vox/dkk/robots.txt ; cp -pfv /m/v*/a*u/rob-modif.txt /m/v*/dvv/robots.txt ; cp -pfv /m/v*/a*u/rob-modif.txt /m/v*/fvv/robots.txt ;
• X (3) /a.../ftpx.pt : To search for *robots.txt - then to set to ON for upload.
• X INFO: robots.txt rules: ---
• X ----------------------------------------------
• X DELAY: S187 seen: Why re-inserted " Crawl-delay:" : S18720 this not convincing because not covers "high volume problem" with non-Google: : https://www.drupal.org/project/drupal/issues/2492191
• X DRUPAL: S187 seen: only for multi-domain: https://www.drupal.org/project/robotstxt
• X WILDDARD: S187 seen https://geoffkenyon.com/how-to-use-wildcards-robots-txt/
• X BASIC INFO / GOOGLE: https://support.google.com/webmasters/answer/6062608?hl=en
• X Major bots
• X --------------------------------------------
• X GOOGLE
• X Meaningg??? https://accounts.google.com/ServiceLogin/signinchooser?service=sitemaps&passive=1209600&continue=https%3A%2F%2Fwww.google.com%2Fwebmasters%2Ftools%2Frobots-testing-tool%3Fhl%3Den&followup=https%3A%2F%2Fwww.google.com%2Fwebmasters%2Ftools%2Frobots-testing-tool%3Fhl%3Den&hl=en&flowName=GlifWebSignIn&flowEntry=ServiceLogin
• X BING
• X AHREFS: for marketing, details: https://ahrefs.com/robot
• X delay (n seconds) : only in standard robots.txt fpr DRU7
• X S187 seen:fischerlaender.de/lexikon/crawl-delay : Crawl-Delay 60 Sek führt, dass pro Tag lediglich 1440 S indexiert werden können; site mit mehreren 10 000s zu wenig, um Site in Suchmaschinen aktuell zu halten.
• X CSS, JS, Images === S187 DRU8 pre-implemented
• X ------------------------------------------------------------------
• X This= in DRU8 standard robots.txt
• X in DRU7: not "core" but "misc - and lacking: *:svg
• X CSS, JS, Images === S187 DRU8 pre-implemented
• X ------------------------------------------------------------------
• X This= DRU7 standard robots.txt === DRU7 in vers S187: all identic. but in DRU7 lacking: *.svg
• X CSS, JS, Images === S187 this below ONLY in DRU8 pre-implemented - modules, thems
• X ------------------------------------------------------------------
• X Directories === S187 DRU8 pre-implemented
• X ------------------------------------------------------------------
• X "core" only here listed in DRU8 ; DIR exists in DRU7 but no files inside
• X "profiles" =in standard robots.txt for: DRU8 + DRI7
• X Directories === S187 DRU7 additonally pre-implemented
• X ------------------------------------------------------------------
• X *REM_S187 in fact all these DIRs in DRU7
• X --- in DRU8 only: "modules" "themes
• X Files === S187 DRU8 pre-implemented
• X ------------------------------------------------------------------
• Paths (clean URLs) === S187 DRU8 pre-implemented identical: DRU7
• X ------------------------------------------------------------------
• X added because seen in log for /dvv/ =DRU8 : S18720 and variant still in S1885 ===
• X ------------------------------------------------------------------
• X Paths (no clean URLs) === S187 only for DRU8 pre-implemented
• X ------------------------------------------------------------------
• X *REM_S187: not sure if this series can still occur
• X Paths (no clean URLs) === S187 DRU7 additonally pre-implemented
• X ------------------------------------------------------------------
• C *REM_S187 apparently these variants do not any more occur in DRU8
• Disallow: /?q=admin/
• Disallow: /?q=comment/reply/
• Disallow: /?q=filter/tips/
• Disallow: /?q=node/add/
• Disallow: /?q=search/
• Disallow: /?q=user/password/
• Disallow: /?q=user/register/
• Disallow: /?q=user/login/
• Disallow: /?q=user/logout/
• X Files - added S18720++
• X ----------------------------------
• CHANG... =only DRU7
• composer... =only DRU8
• COPY... =only DRU7
• cron... =only DRU7
• exam..git... =only DRU8
• === install... === INSTAL... === ALL=only DRU7
• MAINTAIN... =only DRU7
• php.ini : in S18720 only found in DRU8
• README : alreays abvoe in pre-implem.
• UPGRAD... =only DRU7
• web.config : already in standard list fpr DRU8 , same for DRU7
• xml... =only DRU7
• Disallow: /
• Disallow: /
• Disallow: /
• Disallow: /
• Disallow: /
• Disallow: /
• Disallow: /
• Disallow: /
• X Added S18720...S18795++
• X ----------------------------------------------------
• Node 061 164 http://www.infos7.org/node/165
• Node 061 164 http://www.infos7.org/node/165