Log In Sign Up

web.sellme.app
robots.txt

Robots Exclusion Standard data for web.sellme.app

Resource Scan

Scan Details

Site Domain web.sellme.app
Base Domain sellme.app
Scan Status Ok
Last Scan2025-12-23T22:14:07+00:00
Next Scan 2025-12-30T22:14:07+00:00

Last Scan

Scanned2025-12-23T22:14:07+00:00
URL https://web.sellme.app/robots.txt
Domain IPs 199.36.158.100, 2620:0:890::100
Response IP 199.36.158.100
Found Yes
Hash 3425b3fc49bfc2a879b1f9b719b8aa9d28dcae8bc2920bc128311a353e5818d1
SimHash 592cde736580

Groups

*

Rule Path
Allow /
Allow /general
Allow /cars
Allow /realestate
Allow /free
Allow /about
Disallow /admin
Disallow /private
Disallow /*.pdf$
Disallow /cgi-bin/
Disallow /tmp/
Disallow /backup/

Other Records

Field Value
crawl-delay 1

googlebot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 0

bingbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.sellme.app/sitemap.xml
Back to top

Comments

  • robots.txt for Sellme Web Application
  • This file helps search engines navigate and understand your site
  • Disallow admin and private areas
  • Disallow common spam patterns
  • Crawl delay for all bots (in seconds)
  • Sitemap location
  • Specific rules for Google
  • Specific rules for Bing
  • Block bad bots
Back to top