Log In Sign Up

alvarezandmarsal.com
robots.txt

Robots Exclusion Standard data for alvarezandmarsal.com

Resource Scan

Scan Details

Site Domain alvarezandmarsal.com
Base Domain alvarezandmarsal.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-07-31T11:12:29+00:00
Next Scan 2024-10-29T11:12:29+00:00

Last Successful Scan

Scanned2023-06-15T10:42:05+00:00
URL https://www.alvarezandmarsal.com/robots.txt
Domain IPs 23.50.81.58
Response IP 184.26.211.10
Found Yes
Hash cb990f8afb91346ac9baaa22cdfc27376324584def583e1670eaed15686c1f21
SimHash 18349d984f7c

Groups

googlebot

Rule Path
Allow

googlebot-image

Rule Path
Allow

*

Rule Path
Disallow /includes/
Disallow /misc/
Disallow /modules/
Disallow /profiles/
Disallow /scripts/
Disallow /themes/
Disallow /CHANGELOG.txt
Disallow /cron.php
Disallow /INSTALL.mysql.txt
Disallow /INSTALL.pgsql.txt
Disallow /INSTALL.sqlite.txt
Disallow /install.php
Disallow /INSTALL.txt
Disallow /LICENSE.txt
Disallow /MAINTAINERS.txt
Disallow /update.php
Disallow /UPGRADE.txt
Disallow /xmlrpc.php
Disallow /ads.txt
Disallow /leasescre/robots.txt
Disallow /leasescre/thank-you/
Disallow /leasescre/sites/default/files/
Disallow /admin/
Disallow /comment/reply/
Disallow /filter/tips/
Disallow /node/add/
Disallow /search/
Disallow /user/register/
Disallow /user/password/
Disallow /user/login/
Disallow /user/logout/
Disallow /?q=admin%2F
Disallow /?q=comment%2Freply%2F
Disallow /?q=filter%2Ftips%2F
Disallow /?q=node%2Fadd%2F
Disallow /?q=search%2F
Disallow /?q=user%2Fpassword%2F
Disallow /?q=user%2Fregister%2F
Disallow /?q=user%2Flogin%2F
Disallow /?q=user%2Flogout%2F

Other Records

Field Value
crawl-delay 10

*

Rule Path
Disallow /*our-people/*?
Disallow /*our-people*?
Disallow /*?q=our-people$
Disallow /*?q=our-people%2F$
Disallow /*?q=our-people%2F*%3F
Disallow /*?q=our-people*%3F
Disallow /*our-people/search*
Disallow /*our-people/search/*
Disallow /*?q=our-people%2Fsearch*
Disallow /*?q=our-people%2Fsearch%2F*
Disallow /*?q=our-people%2Fsearch*%3F
Disallow /*?q=our-people%2Fsearch%2F*%3F
Disallow /*search/$
Disallow /*search$
Disallow /*search/*?
Disallow /*search*?
Disallow /*search/site$
Disallow /*search/site/$
Disallow /*search/site/*?
Disallow /*search/site*?
Disallow /*?q=search%2Fsite$
Disallow /*?q=search%2Fsite%2F$
Disallow /*?q=search%2Fsite%2F*%3F
Disallow /*?q=search%2Fsite*%3F
Disallow /*?q=search%2F*%3F
Disallow /*?q=search*%3F
Disallow /*?q=search$
Disallow /*?q=search%2F$
Disallow /*?sort*
Disallow /*%26sort*
Disallow /*?solrsort*
Disallow /*%26solrsort*
Disallow /*?f%5B*
Disallow /*?f%5B*
Disallow /*?solrsort

mj12bot
megaindex.ru
semrushbot
ahrefsbot
domaincrawler
blexbot
fast enterprise crawler
dotbot
weborama-fetcher
istellabot
bomborabot

Rule Path
Disallow /

femtosearchbot

Rule Path
Disallow /

yandexbot
baiduspider
oncrawl

Rule Path
Disallow /our-people
Disallow /our-people/
Disallow /our-people/search
Disallow /our-people/search/
Disallow /search/site
Disallow /search/site/
Back to top

Comments

  • robots.txt
  • This file is to prevent the crawling and indexing of certain parts
  • of your site by web crawlers and spiders run by sites like Yahoo!
  • and Google. By telling these "robots" where not to go on your site,
  • you save bandwidth and server resources.
  • This file will be ignored unless it is at the root of your host:
  • Used: http://example.com/robots.txt
  • Ignored: http://example.com/site/robots.txt
  • For more information about the robots.txt standard, see:
  • http://www.robotstxt.org/robotstxt.html
  • Directories
  • Files
  • Resquests from leasescre
  • Paths (clean URLs)
  • Paths (no clean URLs)
  • Search pages
  • Extras
  • no access for table sorting paths or any paths that have parameters.
Back to top