Log In Sign Up

psychologytoday.com
robots.txt

Robots Exclusion Standard data for psychologytoday.com

Resource Scan

Scan Details

Site Domain psychologytoday.com
Base Domain psychologytoday.com
Scan Status Ok
Last Scan2024-10-28T18:28:58+00:00
Next Scan 2024-11-04T18:28:58+00:00

Last Scan

Scanned2024-10-28T18:28:58+00:00
URL https://psychologytoday.com/robots.txt
Redirect https://www.psychologytoday.com:443/robots.txt
Redirect Domain www.psychologytoday.com
Redirect Base psychologytoday.com
Domain IPs 34.192.101.34, 34.194.87.242
Redirect IPs 34.241.238.234, 52.209.91.157
Response IP 52.209.91.157
Found Yes
Hash be357783095a85701a3ea98b4c0a2d2b171e99ee5c99801286800af0b1dac31b
SimHash 180473da4a80

Groups

sosospider

Rule Path
Disallow /

yandex bot

Rule Path
Disallow /

larbin

Rule Path
Disallow /

bizwikibot

Rule Path
Disallow /

scoutjet

Rule Path
Disallow /

riddlerbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

shopwiki

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

*

Rule Path
Disallow /includes/
Disallow /misc/
Disallow /modules/
Disallow /profiles/
Disallow /scripts/
Disallow /themes/
Disallow /CHANGELOG.txt
Disallow /cron.php
Disallow /INSTALL.mysql.txt
Disallow /INSTALL.pgsql.txt
Disallow /INSTALL.sqlite.txt
Disallow /install.php
Disallow /INSTALL.txt
Disallow /LICENSE.txt
Disallow /MAINTAINERS.txt
Disallow /update.php
Disallow /UPGRADE.txt
Disallow /xmlrpc.php
Allow /misc/*.css$
Allow /misc/*.css?
Allow /misc/*.js$
Allow /misc/*.js?
Allow /misc/*.gif
Allow /misc/*.jpg
Allow /misc/*.jpeg
Allow /misc/*.png
Allow /modules/*.css$
Allow /modules/*.css?
Allow /modules/*.js$
Allow /modules/*.js?
Allow /modules/*.gif
Allow /modules/*.jpg
Allow /modules/*.jpeg
Allow /modules/*.png
Allow /profiles/*.css$
Allow /profiles/*.css?
Allow /profiles/*.js$
Allow /profiles/*.js?
Allow /profiles/*.gif
Allow /profiles/*.jpg
Allow /profiles/*.jpeg
Allow /profiles/*.png
Allow /themes/*.css$
Allow /themes/*.css?
Allow /themes/*.js$
Allow /themes/*.js?
Allow /themes/*.gif
Allow /themes/*.jpg
Allow /themes/*.jpeg
Allow /themes/*.png
Disallow /admin$
Disallow /admin/
Disallow /filter/tips/
Disallow /node/add
Disallow /search
Disallow /user$
Disallow /user/
Disallow /*/admin$
Disallow /*/admin/
Disallow /*/filter/tips/
Disallow /*/node/add/
Disallow /*/search/
Disallow /*/user$
Disallow /*/user/
Disallow /au/admin$
Disallow /au/admin/
Disallow /au/filter/tips/
Disallow /au/node/add/
Disallow /au/search/
Disallow /au/user$
Disallow /au/user/
Disallow /ca/admin$
Disallow /ca/admin/
Disallow /ca/filter/tips/
Disallow /ca/node/add/
Disallow /ca/search/
Disallow /ca/user$
Disallow /ca/user/
Disallow /gb/admin$
Disallow /gb/admin/
Disallow /gb/filter/tips/
Disallow /gb/node/add/
Disallow /gb/search/
Disallow /gb/user$
Disallow /gb/user/
Disallow /intl/admin$
Disallow /intl/admin/
Disallow /intl/filter/tips/
Disallow /intl/node/add/
Disallow /intl/search/
Disallow /intl/user$
Disallow /intl/user/
Disallow /us/admin$
Disallow /us/admin/
Disallow /us/filter/tips/
Disallow /us/node/add/
Disallow /us/search/
Disallow /us/user$
Disallow /us/user/
Disallow /?q=
Disallow /*?q=
Disallow /count.php
Disallow /es/admin$
Disallow /es/admin/
Disallow /es/filter/tips/
Disallow /es/node/add/
Disallow /es/search/
Disallow /es/user$
Disallow /es/user/
Disallow /mx/admin$
Disallow /mx/admin/
Disallow /mx/filter/tips/
Disallow /mx/node/add/
Disallow /mx/search/
Disallow /mx/user$
Disallow /mx/user/
Disallow /ar/admin$
Disallow /ar/admin/
Disallow /ar/filter/tips/
Disallow /ar/node/add/
Disallow /ar/search/
Disallow /ar/user$
Disallow /ar/user/
Disallow /ca/admin$
Disallow /ca/admin/
Disallow /ca/filter/tips/
Disallow /ca/node/add/
Disallow /ca/search/
Disallow /ca/user$
Disallow /ca/user/
Disallow /nz/admin$
Disallow /nz/admin/
Disallow /nz/filter/tips/
Disallow /nz/node/add/
Disallow /nz/search/
Disallow /nz/user$
Disallow /nz/user/
Disallow /za/admin$
Disallow /za/admin/
Disallow /za/filter/tips/
Disallow /za/node/add/
Disallow /za/search/
Disallow /za/user$
Disallow /za/user/
Disallow /ie/admin$
Disallow /ie/admin/
Disallow /ie/filter/tips/
Disallow /ie/node/add/
Disallow /ie/search/
Disallow /ie/user$
Disallow /ie/user/
Disallow /sg/admin$
Disallow /sg/admin/
Disallow /sg/filter/tips/
Disallow /sg/node/add/
Disallow /sg/search/
Disallow /sg/user$
Disallow /sg/user/
Disallow /*/*/content/verified-modal/*$
Disallow /*?quicktabs_5=
Disallow /*?topid=
Disallow /*?s=
Disallow /*?quote=
Disallow /*?psy=
Disallow /*?event=
Disallow /*?entity_type=
Disallow /*?entity_id=
Disallow /*?cost=
Disallow /*?bundle=
Disallow /*%26quicktabs_5%3D
Disallow /*%26topid%3D
Disallow /*%26s%3D
Disallow /*%26quote%3D
Disallow /*%26psy%3D
Disallow /*%26event%3D
Disallow /*%26entity_type%3D
Disallow /*%26entity_id%3D
Disallow /*%26cost%3D
Disallow /*%26bundle%3D

Other Records

Field Value
crawl-delay 10
Back to top

Comments

  • Resource Directories
  • Static Files
  • Static Drupal resources explicitly allowed
  • Drupal Paths
  • Disallow: /comment/
  • Drupal Paths, wildcard prefix
  • Disallow: /*/comment/
  • Drupal Paths, au prefix
  • Disallow: /au/comment/
  • Drupal Paths, ca prefix
  • Disallow: /ca/comment/
  • Drupal Paths, gb prefix
  • Disallow: /gb/comment/
  • Drupal Paths, intl prefix
  • Disallow: /intl/comment/
  • Drupal Paths, us prefix
  • Disallow: /us/comment/
  • Paths (no unclean URLs)
  • Count.php
  • es blocks
  • mx blocks
  • ar blocks
  • ca blocks
  • nz blocks
  • za blocks
  • ie blocks
  • sg blocks
  • Disallow verified-modal path
  • Disallow specific parameters -- SUPPT-1895
Back to top