Log In Sign Up

frenchaccountants.com
robots.txt

Robots Exclusion Standard data for frenchaccountants.com

Resource Scan

Scan Details

Site Domain frenchaccountants.com
Base Domain frenchaccountants.com
Scan Status Ok
Last Scan2026-02-14T04:28:38+00:00
Next Scan 2026-02-21T04:28:38+00:00

Last Scan

Scanned2026-02-14T04:28:38+00:00
URL https://frenchaccountants.com/robots.txt
Domain IPs 104.21.71.132, 172.67.170.148, 2606:4700:3035::6815:4784, 2606:4700:3036::ac43:aa94
Response IP 104.21.71.132
Found Yes
Hash 19bddab2a1a24549c2827ad79dda8dd95241e347d3df96fc4f84812249de0dc0
SimHash 643657516af3

Groups

googlebot

Rule Path
Allow /

bingbot

Rule Path
Allow /

slurp

Rule Path
Allow /

duckduckbot

Rule Path
Allow /

ahrefsbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

semrushbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

screaming frog seo spider

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

dotbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

mj12bot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

gptbot

Rule Path
Allow /

chatgpt-user

Rule Path
Allow /

claudebot

Rule Path
Allow /

anthropic-ai

Rule Path
Allow /

ccbot

Rule Path
Allow /

perplexitybot

Rule Path
Allow /

amazonbot

Rule Path
Allow /

applebot

Rule Path
Allow /

facebookexternalhit

Rule Path
Allow /

twitterbot

Rule Path
Allow /

linkedinbot

Rule Path
Allow /

whatsapp

Rule Path
Allow /

pinterestbot

Rule Path
Allow /

snapbot

Rule Path
Allow /

redditbot

Rule Path
Allow /

discordbot

Rule Path
Allow /

slackbot

Rule Path
Allow /

telegrambot

Rule Path
Allow /

google-inspectiontool

Rule Path
Allow /

google-site-verification

Rule Path
Allow /

lighthouse

Rule Path
Allow /

google-pagespeed

Rule Path
Allow /

chrome-lighthouse

Rule Path
Allow /

uptimerobot

Rule Path
Allow /

baiduspider

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

serpstatbot

Rule Path
Disallow /

seokicks

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

tiktokspider

Rule Path
Disallow /

velenpublicwebcrawler

Rule Path
Disallow /

awariobot

Rule Path
Disallow /

mojeekbot

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

zealbot

Rule Path
Disallow /

aliyunsecbot

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

proximic

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

alphabot

Rule Path
Disallow /

sputnikbot

Rule Path
Disallow /

zoominfobot

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

*

Rule Path
Disallow /admin/
Disallow /members/
Disallow /login
Disallow /api/
Disallow /dashboard/
Disallow /lead/register-step-1
Disallow /ajax/
Disallow /api/v1/
Disallow /api/v2/
Disallow /lead/ajax/
Disallow /members/ajax/
Disallow /amember/
Disallow /application/
Disallow /library/
Disallow /vendor/
Disallow /*?search=
Disallow /*?filter=
Disallow /*?sort=
Disallow /*?page=
Disallow /*%26
Disallow /search/
Disallow /search?
Disallow /*?session=
Disallow /*?utm_
Disallow /*?ref=
Disallow /*?source=
Disallow /*.php$
Disallow /config/
Disallow /includes/
Disallow /cache/
Disallow /tmp/
Disallow /temp/
Disallow /logs/
Disallow /.git/
Disallow /.env
Disallow /composer.json
Disallow /package.json
Disallow /backup/
Disallow /backups/
Disallow /sql/
Disallow /.well-known/
Disallow /wp-admin/
Disallow /wp-content/
Allow /$
Allow /us/$
Allow /canada/$
Allow /us/alabama/$
Allow /us/alaska/$
Allow /us/arizona/$
Allow /us/arkansas/$
Allow /us/california/$
Allow /us/colorado/$
Allow /us/connecticut/$
Allow /us/delaware/$
Allow /us/district-of-columbia/$
Allow /us/florida/$
Allow /us/georgia/$
Allow /us/hawaii/$
Allow /us/idaho/$
Allow /us/illinois/$
Allow /us/indiana/$
Allow /us/iowa/$
Allow /us/kansas/$
Allow /us/kentucky/$
Allow /us/louisiana/$
Allow /us/maine/$
Allow /us/maryland/$
Allow /us/massachusetts/$
Allow /us/michigan/$
Allow /us/minnesota/$
Allow /us/mississippi/$
Allow /us/missouri/$
Allow /us/montana/$
Allow /us/nebraska/$
Allow /us/nevada/$
Allow /us/new-hampshire/$
Allow /us/new-jersey/$
Allow /us/new-mexico/$
Allow /us/new-york/$
Allow /us/north-carolina/$
Allow /us/north-dakota/$
Allow /us/ohio/$
Allow /us/oklahoma/$
Allow /us/oregon/$
Allow /us/pennsylvania/$
Allow /us/rhode-island/$
Allow /us/south-carolina/$
Allow /us/south-dakota/$
Allow /us/tennessee/$
Allow /us/texas/$
Allow /us/utah/$
Allow /us/vermont/$
Allow /us/virginia/$
Allow /us/washington/$
Allow /us/west-virginia/$
Allow /us/wisconsin/$
Allow /us/wyoming/$
Allow /us/new-york/new-york/$
Allow /us/california/los-angeles/$
Allow /us/illinois/chicago/$
Allow /us/texas/houston/$
Allow /us/arizona/phoenix/$
Allow /us/pennsylvania/philadelphia/$
Allow /us/texas/san-antonio/$
Allow /us/california/san-diego/$
Allow /us/texas/dallas/$
Allow /us/california/san-jose/$
Allow /us/texas/austin/$
Allow /us/florida/jacksonville/$
Allow /us/california/san-francisco/$
Allow /us/ohio/columbus/$
Allow /us/north-carolina/charlotte/$
Allow /us/indiana/indianapolis/$
Allow /us/washington/seattle/$
Allow /us/colorado/denver/$
Allow /us/washington-dc/washington/$
Allow /us/massachusetts/boston/$
Allow /us/texas/el-paso/$
Allow /us/tennessee/nashville/$
Allow /us/oklahoma/oklahoma-city/$
Allow /us/nevada/las-vegas/$
Allow /us/kentucky/louisville/$
Allow /us/oregon/portland/$
Allow /us/michigan/detroit/$
Allow /us/tennessee/memphis/$
Allow /us/maryland/baltimore/$
Allow /us/wisconsin/milwaukee/$
Allow /us/new-mexico/albuquerque/$
Allow /us/arizona/tucson/$
Allow /us/california/fresno/$
Allow /us/california/sacramento/$
Allow /us/missouri/kansas-city/$
Allow /us/arizona/mesa/$
Allow /us/georgia/atlanta/$
Allow /us/nebraska/omaha/$
Allow /us/colorado/colorado-springs/$
Allow /us/north-carolina/raleigh/$

*

Rule Path
Allow /$
Allow /us/$
Allow /canada/$
Disallow /admin/
Disallow /members/
Disallow /login
Disallow /api/
Disallow /dashboard/
Disallow /lead/register-step-1
Disallow /*?
Disallow /search/
Disallow /*.php$
Disallow /amember/
Disallow /application/
Disallow /vendor/

Other Records

Field Value
crawl-delay 5
Back to top

Other Records

Field Value
sitemap https://frenchaccountants.com/sitemap.xml
sitemap https://frenchaccountants.com/united-states-country-sitemap.xml
sitemap https://frenchaccountants.com/united-states-california-state-sitemap.xml
sitemap https://frenchaccountants.com/united-states-new-york-state-sitemap.xml
sitemap https://frenchaccountants.com/united-states-texas-state-sitemap.xml
sitemap https://frenchaccountants.com/united-states-illinois-state-sitemap.xml
sitemap https://frenchaccountants.com/united-states-florida-state-sitemap.xml
sitemap https://frenchaccountants.com/united-states-nevada-state-sitemap.xml
sitemap https://frenchaccountants.com/united-states-virginia-state-sitemap.xml
sitemap https://frenchaccountants.com/united-states-arizona-state-sitemap.xml
sitemap https://frenchaccountants.com/united-states-maryland-state-sitemap.xml
sitemap https://frenchaccountants.com/united-states-massachusetts-state-sitemap.xml
sitemap https://frenchaccountants.com/united-states-washington-state-sitemap.xml
sitemap https://frenchaccountants.com/united-states-pennsylvania-state-sitemap.xml
sitemap https://frenchaccountants.com/united-states-michigan-state-sitemap.xml
sitemap https://frenchaccountants.com/united-states-ohio-state-sitemap.xml
sitemap https://frenchaccountants.com/united-states-georgia-state-sitemap.xml
sitemap https://frenchaccountants.com/united-states-new-jersey-state-sitemap.xml
sitemap https://frenchaccountants.com/united-states-oklahoma-state-sitemap.xml
sitemap https://frenchaccountants.com/united-states-colorado-state-sitemap.xml
sitemap https://frenchaccountants.com/united-states-north-carolina-state-sitemap.xml
sitemap https://frenchaccountants.com/united-states-hawaii-state-sitemap.xml
sitemap https://frenchaccountants.com/united-states-indiana-state-sitemap.xml
sitemap https://frenchaccountants.com/united-states-tennessee-state-sitemap.xml
sitemap https://frenchaccountants.com/united-states-south-carolina-state-sitemap.xml
sitemap https://frenchaccountants.com/united-states-louisiana-state-sitemap.xml
sitemap https://frenchaccountants.com/articles-sitemap.xml
sitemap https://frenchaccountants.com/global-sitemap.xml
Back to top

Comments

  • Heritage Web Directories - robots.txt
  • Purpose: Control search engine crawling and protect resources
  • ====================================
  • SECTION 1: ALLOWED BOTS
  • ====================================
  • --- SEARCH ENGINES ---
  • Major search engines that drive organic traffic
  • --- SEO TOOLS ---
  • Reputable SEO tools for analysis and insights
  • --- AI BOTS (AMERICAN) ---
  • American AI companies - 1 second crawl delay
  • --- SOCIAL MEDIA CRAWLERS ---
  • Important for content sharing and previews
  • --- DEVELOPMENT & TESTING TOOLS ---
  • Google's official testing and verification bots
  • --- MONITORING SERVICES ---
  • ====================================
  • SECTION 2: BLOCKED BOTS
  • ====================================
  • Known bad actors and resource-intensive crawlers
  • --- FOREIGN SEARCH ENGINES ---
  • Chinese search engines
  • Russian search engines
  • --- AGGRESSIVE SEO BOTS ---
  • Less reputable or overly aggressive SEO crawlers
  • --- OTHER UNWANTED BOTS ---
  • ====================================
  • SECTION 3: PATH RULES FOR ALL BOTS
  • ====================================
  • These rules apply to all user agents
  • --- BLOCKED PATHS ---
  • Admin and member areas
  • Lead registration
  • AJAX and API endpoints
  • aMember specific paths
  • Search and filter pages (prevent duplicate content)
  • Session and tracking parameters
  • Technical/system files
  • Security-sensitive paths
  • --- EXPLICITLY ALLOWED PATHS ---
  • Homepage
  • Country pages
  • All US state pages
  • Top 40 US city pages (key for local SEO)
  • ====================================
  • SECTION 4: DEFAULT RULES
  • ====================================
  • Conservative rules for any unlisted bot
  • Critical paths for unknown crawlers
  • Repeat key disallow rules for safety
  • ====================================
  • SECTION 5: SITEMAP REFERENCE
  • ====================================
  • Point to XML sitemaps for better crawling
Back to top