Log In Sign Up

tgs.co.il
robots.txt

Robots Exclusion Standard data for tgs.co.il

Resource Scan

Scan Details

Site Domain tgs.co.il
Base Domain tgs.co.il
Scan Status Ok
Last Scan2024-06-14T16:04:17+00:00
Next Scan 2024-06-21T16:04:17+00:00

Last Scan

Scanned2024-06-14T16:04:17+00:00
URL https://tgs.co.il/robots.txt
Redirect https://www.tgspot.co.il/robots.txt
Redirect Domain www.tgspot.co.il
Redirect Base tgspot.co.il
Domain IPs 104.21.16.26, 172.67.165.252, 2606:4700:3031::ac43:a5fc, 2606:4700:3035::6815:101a
Redirect IPs 104.21.22.40, 172.67.202.161, 2606:4700:3030::ac43:caa1, 2606:4700:3036::6815:1628
Response IP 104.21.22.40
Found Yes
Hash c2f5ab5e46f701b0e52fe4744cd6bebb242881e84264564e17dd7888d2c85a88
SimHash 020c5c7165e0

Groups

microsoft url control - 6.00.8169
*

Rule Path
Disallow /wp-admin/
Disallow /trackback/
Disallow /xmlrpc.php
Disallow /%3Cspan/
Disallow /%20class/
Disallow /%3Cspan%20class%3D
Disallow /2922790/
Disallow /comment-subscriptions*
Disallow /comment-subscriptions/
Disallow /community/
Disallow /7047/
Disallow /wp-content/uploads/comparison/
Disallow /wp-content/themes/tgspot/
Disallow *.php$
Disallow /marfeel_sw*
Disallow /?blackhole
Disallow /index?*

microsoft url control - 6.01.9782
simplepie

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

dataforseobot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

trendictionbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

yandex

Rule Path
Disallow /

googlebot-news

Rule Path
Allow /

mediapartners-google

Rule Path
Allow /

googlebot

Rule Path
Allow /

outbrain

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

baiduspider

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

duckduckbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

megaindex

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

megaindex.ru

Rule Path
Disallow /

megaindex.com

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

pinterestbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

gptbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10
Back to top

Other Records

Field Value
sitemap https://www.tgspot.co.il/sitemap_index.xml
Back to top

Comments

  • BEGIN XML-SITEMAP-PLUGIN
  • END XML-SITEMAP-PLUGIN
  • Disallow: /*?cat*
  • Disallow: /*?*
  • Disallow: /category/0/
  • Disallow: /category/operating-systems/
  • Disallow: */page/*
  • https://megaindex.com/crawler
Back to top