thealternativepress.com
robots.txt

Robots Exclusion Standard data for thealternativepress.com

Resource Scan

Scan Details

Site Domain thealternativepress.com
Base Domain thealternativepress.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-05-28T19:11:25+00:00
Next Scan 2024-08-26T19:11:25+00:00

Last Successful Scan

Scanned2023-05-05T18:54:05+00:00
URL http://thealternativepress.com/robots.txt
Redirect https://www.tapinto.net/robots.txt
Redirect Domain www.tapinto.net
Redirect Base tapinto.net
Domain IPs 35.173.112.78
Redirect IPs 3.208.171.244, 34.197.5.241, 34.200.196.84, 34.205.113.147, 35.171.196.162, 52.4.8.61
Response IP 3.208.171.244
Found Yes
Hash 9556e637420163a6925685aeb48ad18967ed6258d37156f2db0885c50d74ea76
SimHash e614fc155750

Groups

*

Rule Path
Disallow /admin
Disallow /users/sign_in
Disallow /users
Disallow /transactions
Disallow /article.asp
Disallow /column.asp
Disallow /photos
Disallow /tracked_urls
Disallow /search
Disallow /towns.json
Disallow /category.json
Disallow /*.php
Disallow /*.php$
Disallow /api/v1/masterhead*

googlebot

Rule Path
Disallow /admin
Disallow /users/sign_in
Disallow /users
Disallow /transactions
Disallow /photos
Disallow /tracked_urls
Disallow /article.asp*
Disallow /column.asp*
Disallow /admin
Disallow /users/sign_in
Disallow /transactions
Disallow /article.asp
Disallow /column.asp
Disallow /photos
Disallow /tracked_urls
Disallow /search*

googlebot-image

Rule Path
Allow /*
Disallow /admin
Disallow /users/sign_in
Disallow /users
Disallow /transactions
Disallow /article.asp*
Disallow /column.asp*
Disallow /search*

adsbot-google

Rule Path
Allow /*
Disallow /admin
Disallow /users/sign_in
Disallow /users
Disallow /transactions
Disallow /article.asp*
Disallow /column.asp*
Disallow /search*

linkedinbot/1.0

Rule Path
Allow /

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

ahrefsbot
ahrefssiteaudit
petalbot
semrushbot
semrushbot
semrushbot-sa

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.tapinto.net/sitemaps/sitemap-altpress-index.xml

Comments

  • See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file
  • To ban all spiders from the entire site uncomment the next two lines:
  • Disallow: /
  • slow down Yahoo
  • slow down Bing