independentespanol.com
robots.txt

Robots Exclusion Standard data for independentespanol.com

Resource Scan

Scan Details

Site Domain independentespanol.com
Base Domain independentespanol.com
Scan Status Ok
Last Scan2024-06-28T14:12:43+00:00
Next Scan 2024-07-05T14:12:43+00:00

Last Scan

Scanned2024-06-28T14:12:43+00:00
URL https://independentespanol.com/robots.txt
Redirect https://www.independentespanol.com/robots.txt
Redirect Domain www.independentespanol.com
Redirect Base independentespanol.com
Domain IPs 151.101.130.49, 151.101.194.49, 151.101.2.49, 151.101.66.49
Redirect IPs 151.101.1.91, 151.101.129.91, 151.101.193.91, 151.101.65.91
Response IP 199.232.45.91
Found Yes
Hash a0a9b1a643a71a57db1cc2bf14256475705e55151f44617888d90b26f130fee8
SimHash 588c50088e53

Groups

*

Rule Path
Disallow /CHANGELOG.txt
Disallow /cron.php
Disallow /INSTALL.mysql.txt
Disallow /INSTALL.pgsql.txt
Disallow /INSTALL.sqlite.txt
Disallow /install.php
Disallow /INSTALL.txt
Disallow /LICENSE.txt
Disallow /MAINTAINERS.txt
Disallow /update.php
Disallow /UPGRADE.txt
Disallow /xmlrpc.php
Disallow /img/placeholder.gif
Disallow /admin/
Disallow /comment/reply/
Disallow /filter/tips/
Disallow /node/add/
Disallow /search/
Disallow /user/register/
Disallow /user/password/
Disallow /user/login/
Disallow /user/logout/
Disallow /internal-api/
Disallow /?q=admin%2F
Disallow /?q=comment%2Freply%2F
Disallow /?q=filter%2Ftips%2F
Disallow /?q=node%2Fadd%2F
Disallow /?q=search%2F
Disallow /?q=user%2Fpassword%2F
Disallow /?q=user%2Fregister%2F
Disallow /?q=user%2Flogin%2F
Disallow /?q=user%2Flogout%2F
Disallow /pugpig/
Disallow /videorequest
Disallow /api/
Disallow /api/html/
Disallow /api/user/
Disallow /71347885/
Disallow /*ILC-refresh

nutch

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.independentespanol.com/sitemaps/googlenews
sitemap https://www.independentespanol.com/sitemap.xml

Comments

  • Files
  • Paths (clean URLs)
  • Paths (no clean URLs)
  • Ignore refresh URLs