avtfinity.com
robots.txt

Robots Exclusion Standard data for avtfinity.com

Resource Scan

Scan Details

Site Domain avtfinity.com
Base Domain avtfinity.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-10-25T04:11:49+00:00
Next Scan 2025-10-26T04:11:49+00:00

Last Successful Scan

Scanned2025-10-18T03:35:59+00:00
URL https://avtfinity.com/robots.txt
Domain IPs 103.77.162.18
Response IP 103.77.162.18
Found Yes
Hash 449327dba23b1775d03020cc25700dabad407180551fe9fc6f83b423a2803b37
SimHash 62160ac3e5b5

Groups

*

Rule Path
Allow /
Allow /index.html
Allow /showcase.html
Allow /download.html
Allow /home.html
Allow /games.html
Allow /vfx.html
Allow /about.html
Allow /skills.html
Allow /contact.html
Allow /styles.css
Allow /script.js
Allow /game-data.json
Allow /showcase-data.json
Allow /*.jpg
Allow /*.jpeg
Allow /*.png
Allow /*.gif
Allow /*.webp
Allow /*.svg
Allow /*.css
Allow /*.js
Allow /*.json
Disallow /.htaccess
Disallow /.htpasswd
Disallow /config/
Disallow /logs/
Disallow /tmp/
Disallow /cache/

Other Records

Field Value
crawl-delay 1

googlebot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

bingbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

slurp

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

duckduckbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

baiduspider

Rule Path
Allow /

Other Records

Field Value
crawl-delay 2

yandexbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

Other Records

Field Value
sitemap https://avtfinity.com/sitemap.xml

Comments

  • Sitemap
  • Crawl-delay for respectful crawling
  • Allow all search engines to index all content
  • Allow access to images and assets
  • Disallow access to sensitive files
  • Specific instructions for major search engines