Log In Sign Up

nerdyt.ag
robots.txt

Robots Exclusion Standard data for nerdyt.ag

Resource Scan

Scan Details

Site Domain nerdyt.ag
Base Domain nerdyt.ag
Scan Status Ok
Last Scan2026-03-29T04:11:36+00:00
Next Scan 2026-04-12T04:11:36+00:00

Last Scan

Scanned2026-03-29T04:11:36+00:00
URL https://nerdyt.ag/robots.txt
Domain IPs 104.21.63.34, 172.67.169.111, 2606:4700:3034::6815:3f22, 2606:4700:3036::ac43:a96f
Response IP 172.67.169.111
Found Yes
Hash c3fe2451905c6a4ac61637d800715047a7eb40c85ea5d54921e617f945c4ab0f
SimHash a600db72b663

Groups

*

Rule Path
Allow /
Allow /terms
Allow /privacy
Allow /login
Allow /register
Allow /directory
Disallow /dashboard/
Disallow /logout
Disallow /settings
Disallow /verify
Disallow /verify-email
Disallow /forgot-password
Disallow /reset-password
Disallow /change-password
Disallow /delete-account
Disallow /account-setup
Disallow /api/
Disallow /tracking
Allow /*/
Allow /privacy

Other Records

Field Value
crawl-delay 10
Back to top

Other Records

Field Value
sitemap https://nerdyt.ag/sitemap.xml
Back to top

Comments

  • robots.txt for NerdyTags (https://nerdyt.ag)
  • Allow crawling of public pages
  • Disallow admin and user-specific areas
  • Disallow API endpoints
  • Disallow tracking endpoints
  • Allow crawling of user profiles
  • Allow crawling of privacy page
  • Crawl delay to prevent server overload
Back to top