Log In Sign Up

shortmeme.com
robots.txt

Robots Exclusion Standard data for shortmeme.com

Resource Scan

Scan Details

Site Domain shortmeme.com
Base Domain shortmeme.com
Scan Status Ok
Last Scan2026-01-11T18:35:04+00:00
Next Scan 2026-01-18T18:35:04+00:00

Last Scan

Scanned2026-01-11T18:35:04+00:00
URL https://shortmeme.com/robots.txt
Domain IPs 34.111.179.208
Response IP 34.111.179.208
Found Yes
Hash c61dde4026fe5ef62b569fb5ae3d083bc6557744e1ce6307dea6b3f17dd4e13f
SimHash 6e30cf422e31

Groups

*

Rule Path
Allow /
Disallow /api/
Allow /
Allow /partner-with-us
Allow /home
Allow /meme/
Allow /*.js
Allow /*.css

Other Records

Field Value
crawl-delay 1
Back to top

Other Records

Field Value
sitemap https://shortmeme.com/sitemap.xml
Back to top

Comments

  • Explicitly allow crawlers to all important pages
  • Important pages
  • Crawl delay directive to avoid overloading server
  • Sitemap
Back to top