Log In Sign Up

hero-forge.com
robots.txt

Robots Exclusion Standard data for hero-forge.com

Resource Scan

Scan Details

Site Domain hero-forge.com
Base Domain hero-forge.com
Scan Status Ok
Last Scan2025-12-21T17:38:50+00:00
Next Scan 2025-12-28T17:38:50+00:00

Last Scan

Scanned2025-12-21T17:38:50+00:00
URL https://hero-forge.com/robots.txt
Domain IPs 199.36.158.100
Response IP 199.36.158.100
Found Yes
Hash 87cfc102a87a83e18f7199ca83838c72d4897c637f144b10127bffd53596f562
SimHash 160bfe70c5f2

Groups

*

Rule Path
Allow /

googlebot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

bingbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

slurp

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

duckduckbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

baiduspider

Rule Path
Allow /

Other Records

Field Value
crawl-delay 2

yandexbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 2

facebookexternalhit

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

twitterbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

linkedinbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

whatsapp

Rule Path
Allow /

telegrambot

Rule Path
Allow /

skypeuripreview

Rule Path
Allow /

applebot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

google-adsense

Rule Path
Allow /ads.txt
Allow /

google-ads-creatives-assistant

Rule Path
Allow /

ahrefsbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

semrushbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

mj12bot

Rule Path
Allow /
Disallow /api/
Disallow /.well-known/
Disallow /test-*
Disallow /*?source=twa
Disallow /proxy-server/
Disallow /node_modules/
Disallow /build/
Disallow /dist/
Disallow /.git/
Allow *.css$
Allow *.js$
Allow *.png$
Allow *.jpg$
Allow *.jpeg$
Allow *.gif$
Allow *.webp$
Allow *.svg$
Allow *.ico$
Allow *.woff$
Allow *.woff2$
Allow *.ttf$
Allow *.eot$
Allow *.json$
Allow *.xml$
Allow *.txt$
Allow *.pdf$
Allow /manifest.json
Allow /ads.txt
Allow /sitemap.xml
Allow /robots.txt
Allow /favicon.ico
Allow /googlee85c662c8a193daa.html
Allow /.well-known/assetlinks.json

Other Records

Field Value
crawl-delay 10

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

semrushbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

mj12bot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

dotbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

sitebot

Rule Path
Disallow /

webreaper

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

offline explorer

Rule Path
Disallow /

httrack

Rule Path
Disallow /

microsoft url control

Rule Path
Disallow /

emailcollector

Rule Path
Disallow /

emailsiphon

Rule Path
Disallow /

webbandit

Rule Path
Disallow /

emailwolf

Rule Path
Disallow /

extractorpro

Rule Path
Disallow /

copyrightcheck

Rule Path
Disallow /

crescent

Rule Path
Disallow /

sitesnagger

Rule Path
Disallow /

prowebwalker

Rule Path
Disallow /

cheesebot

Rule Path
Disallow /

lnspiderguy

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

ia_archiver/1.6

Rule Path
Disallow /

teleport

Rule Path
Disallow /

teleportpro

Rule Path
Disallow /

miixpc

Rule Path
Disallow /

telesoft

Rule Path
Disallow /

website quester

Rule Path
Disallow /

moget/2.1

Rule Path
Disallow /

webzip/4.0

Rule Path
Disallow /

stock photo downloader v1.0

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.hero-forge.com/sitemap.xml
Back to top

Comments

  • robots.txt for Hero Forge - AI Fantasy Character Generator
  • This file tells search engines how to crawl and index this website
  • Website: https://hero-forge-98367.web.app
  • Default rule for all crawlers
  • Sitemap locations
  • Major search engines - explicit permissions
  • Social media crawlers
  • Additional social media and messaging platforms
  • Ad network crawlers (for ads.txt verification)
  • SEO and analytics crawlers
  • Disallow paths that shouldn't be indexed
  • Explicitly allow important file types for SEO and functionality
  • Allow important files
  • Block aggressive crawlers that might impact performance
  • Block known bad bots
  • Additional information
  • Website: https://hero-forge-98367.web.app
  • Contact: jonslavey@hotmail.com
  • Description: AI-powered fantasy character generator for RPGs and D&D
  • Purpose: Create stunning fantasy characters for tabletop RPGs, D&D campaigns, and creative projects
  • Features: Character generation, artifact creation, banner design, animation
  • Last updated: 2025-10-06
Back to top