Log In Sign Up

fextralife.com
robots.txt

Robots Exclusion Standard data for fextralife.com

Resource Scan

Scan Details

Site Domain fextralife.com
Base Domain fextralife.com
Scan Status Ok
Last Scan2025-09-24T23:07:58+00:00
Next Scan 2025-10-01T23:07:58+00:00

Last Scan

Scanned2025-09-24T23:07:58+00:00
URL https://fextralife.com/robots.txt
Domain IPs 138.199.46.65, 2400:52e0:1500::1272:1
Response IP 138.199.46.65
Found Yes
Hash 90e55472237c4aaf557de9ec2996889946f70c6b2aa62c888e6b87cfcc10d1c9
SimHash 7053135dc775

Groups

*

Rule Path
Disallow /pixel.png*
Disallow /search/
Disallow /profile/account/
Disallow /profile/following/
Disallow /profile/general/
Disallow /profile/mylists/
Disallow /profile/premium/
Disallow /strpixel.png*
Disallow /public/build/quill*
Disallow /fetch/tag/*
Disallow /comment-fetch/
Disallow /comment-reply-fetch/
Disallow /modules-fetch/
Disallow /fetch/quick-rating-widgets/
Disallow /tag/interaction-card/*
Disallow /tag/search/db-home/
Disallow /tag/search/listing-page/
Disallow /mymodule/tag/
Disallow /mymodule/vehicle-sub-model-year-tag/
Disallow /mymodule/vehicle-sub-model-year-tag-trim/compare/
Disallow /promo/next/mural
Disallow /promo/next/mural/*
Disallow /promo/next/mural?*

applebot
applebot-extended
bytespider
ccbot
claude-web
claudebot
diffbot
facebookbot
gptbot
httrack
nutch
offline explorer
scrapy
youbot
anthropic-ai
cohere-ai
omgili

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://fextralife.com/sitemap.xml
sitemap https://fextralife.com/post_google_news.xml
Back to top

Comments

  • Valnet content is made available for your non-commercial use subject to our
  • Terms of Use here: https://www.valnetinc.com/en/terms-of-use
  • Use of any robot, crawler, or other tool to scrape, harvest, extract, or retrieve any content on
  • this website using automated means is prohibited without written permission from Valnet.
  • Prohibited uses include but are not limited to:
  • (1) text and data mining under Art. 4 of the EU Directive on Copyright in the Digital Single
  • Market;
  • (2) development or operation of artificial intelligence or machine learning software or
  • databases, including by training, fine-tuning, embedding, and retrieval-augmented generation;
  • (3) creating data sets containing our content or sharing it with others; and
  • (4) any commercial purposes.
  • Contact info@valnetinc.com for assistance.
  • Block AI content scrapers
Back to top

Warnings

  • 1 invalid line.
Back to top