talpiottalk.com
robots.txt

Robots Exclusion Standard data for talpiottalk.com

Resource Scan

Scan Details

Site Domain talpiottalk.com
Base Domain talpiottalk.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2024-09-06T10:35:01+00:00
Next Scan 2024-11-05T10:35:01+00:00

Last Successful Scan

Scanned2024-06-16T10:33:08+00:00
URL http://talpiottalk.com/robots.txt
Redirect http://www.talpiottalk.com/robots.txt
Redirect Domain www.talpiottalk.com
Redirect Base talpiottalk.com
Domain IPs 185.206.180.114, 2a00:1768:2001:63::46:104, 2a0b:1640:1:1:1:1:d38:c44d, 46.166.184.104
Redirect IPs 152.199.39.108, 2606:2800:247:1cb7:261b:1f9c:2074:3c
Response IP 152.199.39.108
Found Yes
Hash b5af3bf9dae0c9ac0bc3db33fa97813fbb554debe4eac2f4deb57ac0e35e5de9
SimHash b316da4c2ed7

Groups

*

Rule Path Comment
Disallow /wp-admin/ -
Allow /wp-admin/admin-ajax.php -
Disallow /wp-login.php -
Disallow /activate/ har har
Disallow /cgi-bin/ MT refugees
Disallow /mshots/v1/ -
Disallow /next/ -
Disallow /public.api/ -

irlbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 3600

Other Records

Field Value
sitemap http://talpiottalk.com/sitemap.xml
sitemap http://talpiottalk.com/news-sitemap.xml

Comments

  • If you are regularly crawling WordPress.com sites, please use our firehose to receive real-time push updates instead.
  • Please see https://developer.wordpress.com/docs/firehose/ for more details.
  • This file was generated on Sun, 15 Apr 2018 06:25:14 +0000