appsolves.dev
robots.txt

Robots Exclusion Standard data for appsolves.dev

Resource Scan

Scan Details

Site Domain appsolves.dev
Base Domain appsolves.dev
Scan Status Ok
Last Scan2025-12-14T21:04:25+00:00
Next Scan 2025-12-21T21:04:25+00:00

Last Scan

Scanned2025-12-14T21:04:25+00:00
URL https://appsolves.dev/robots.txt
Redirect https://www.appsolves.dev/robots.txt
Redirect Domain www.appsolves.dev
Redirect Base appsolves.dev
Domain IPs 198.185.159.145
Redirect IPs 185.199.108.153, 185.199.109.153, 185.199.110.153, 185.199.111.153, 2606:50c0:8000::153, 2606:50c0:8001::153, 2606:50c0:8002::153, 2606:50c0:8003::153
Response IP 185.199.109.153
Found Yes
Hash 02656af0c5043556387f0bf1e60cfc7ffadc887d002d03acfef8b09a368913b2
SimHash 511c5c536750

Groups

*

Rule Path
Disallow /admin/
Disallow /private/
Disallow /.git/
Disallow /src/
Disallow /node_modules/
Disallow /privacy_policy*
Disallow /terms_and_conditions*

googlebot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

google-adstxt

Rule Path
Allow /

mediapartners-google

Rule Path
Allow /

bingbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

semrushbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

wget

Rule Path
Disallow /

curl

Rule Path
Disallow /

httrack

Rule Path
Disallow /

webzip

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

Other Records

Field Value
sitemap https://appsolves.dev/sitemap.xml

Comments

  • Security-focused robots.txt
  • Allow legitimate crawlers for SEO
  • Block aggressive crawlers and scrapers
  • Block common scraping tools