jasonfintips.com
robots.txt

Robots Exclusion Standard data for jasonfintips.com

Resource Scan

Scan Details

Site Domain jasonfintips.com
Base Domain jasonfintips.com
Scan Status Ok
Last Scan2024-11-13T20:50:17+00:00
Next Scan 2024-11-20T20:50:17+00:00

Last Scan

Scanned2024-11-13T20:50:17+00:00
URL https://jasonfintips.com/robots.txt
Domain IPs 104.21.47.215, 172.67.172.206, 2606:4700:3031::6815:2fd7, 2606:4700:3035::ac43:acce
Response IP 172.67.172.206
Found Yes
Hash a5966d4cdd0b334abd8206104145836d885f5843f85d5a32c0aba26db695afe8
SimHash 6b4252c32073

Groups

*

Rule Path Comment
Disallow /tag/ -
Disallow /staging/ -
Disallow /groups/*/search/ -
Disallow /?s= -
Disallow /members/ -
Disallow /activity/ -
Disallow /wp-content/plugins/ -
Disallow /wp-content/cache/ -
Disallow /wp-json/cleantalk-antispam/ -
Disallow /wp-includes/ -
Disallow /*ref%3D -
Disallow /*session%3D -
Disallow /*.php$ -
Disallow /*.cgi$ -
Disallow /*.asp$ -
Disallow /*?sessionid= -
Disallow /*utm_source%3D -
Disallow /*utm_medium%3D -
Disallow /*utm_campaign%3D -
Disallow /*gclid%3D -
Disallow /cdn-cgi/ Block Cloudflare scripts
Allow /wp-admin/admin-ajax.php -

googlebot-mobile

Rule Path
Allow /

mediapartners-google

Rule Path
Allow /

googlebot-image

Rule Path
Allow /

googlebot-video

Rule Path
Allow /

bingbot

Rule Path
Allow /

linkedinbot

Rule Path
Allow /

pinterestbot

Rule Path
Allow /

instagram

Rule Path
Allow /

facebookexternalhit

Rule Path
Allow /

cloudflare

Rule Path
Disallow /wp-admin/

Other Records

Field Value
sitemap https://jasonfintips.com/sitemap_index.xml

Comments

  • Sitemap location
  • Google Crawlers
  • Bingbot specific rules
  • Cloudflare can cache content but should not index admin pages