Log In Sign Up

re-clipped.com
robots.txt

Robots Exclusion Standard data for re-clipped.com

Resource Scan

Scan Details

Site Domain re-clipped.com
Base Domain re-clipped.com
Scan Status Failed
Failure ReasonScan timed out.
Last Scan2025-08-05T02:23:14+00:00
Next Scan 2025-08-19T02:23:14+00:00

Last Successful Scan

Scanned2025-07-14T01:45:46+00:00
URL https://re-clipped.com/robots.txt
Redirect https://www.re-clipped.com/robots.txt
Redirect Domain www.re-clipped.com
Redirect Base re-clipped.com
Domain IPs 216.198.79.193
Redirect IPs 216.198.79.193
Response IP 216.198.79.193
Found Yes
Hash f3596cce6c1472ab550a8aa8c74c1eff03f4af09716b1a16006c5138fda5444c
SimHash 64009a90e5b5

Groups

*

Rule Path
Allow /
Allow /assets/
Allow /src/
Allow /_next/static/
Disallow /admin/
Disallow /private/

Other Records

Field Value
crawl-delay 1

googlebot

Rule Path
Allow /

bingbot

Rule Path
Allow /

slurp

Rule Path
Allow /

gptbot

Rule Path
Allow /

chatgpt-user

Rule Path
Allow /

claude-web

Rule Path
Allow /

perplexitybot

Rule Path
Allow /
Back to top

Other Records

Field Value
sitemap https://www.re-clipped.com/sitemap.xml
Back to top

Comments

  • Allow all crawlers access to CSS and JS files
  • Block access to admin areas if any exist
  • Sitemap location
  • Crawl-delay for courtesy (optional)
  • Specific rules for major search engines
  • Allow ChatGPT and other AI crawlers
Back to top