Log In Sign Up

aiartgen.net
robots.txt

Robots Exclusion Standard data for aiartgen.net

Resource Scan

Scan Details

Site Domain aiartgen.net
Base Domain aiartgen.net
Scan Status Ok
Last Scan2025-12-20T00:31:52+00:00
Next Scan 2026-01-03T00:31:52+00:00

Last Scan

Scanned2025-12-20T00:31:52+00:00
URL https://aiartgen.net/robots.txt
Redirect https://www.aiartgen.net/robots.txt
Redirect Domain www.aiartgen.net
Redirect Base aiartgen.net
Domain IPs 104.21.93.53, 172.67.204.154, 2606:4700:3033::6815:5d35, 2606:4700:3037::ac43:cc9a
Redirect IPs 104.21.93.53, 172.67.204.154, 2606:4700:3033::6815:5d35, 2606:4700:3037::ac43:cc9a
Response IP 172.67.204.154
Found Yes
Hash 86484e8e822a63f178b588559e1b5f265c54313c42250fe338cc050ca08dc121
SimHash 68158d3366f0

Groups

*

Rule Path
Allow /

googlebot

Rule Path
Allow /

bingbot

Rule Path
Allow /

slurp

Rule Path
Allow /

duckduckbot

Rule Path
Allow /
Disallow *.css$
Disallow *.js$
Disallow *.json$
Allow /screen_shot/
Allow /app_icon_new.jpg

Other Records

Field Value
crawl-delay 1
Back to top

Other Records

Field Value
sitemap https://aiartgen.net/sitemap.xml
Back to top

Comments

  • Sitemaps
  • Allow all search engines to crawl the site
  • Crawl delay for better server performance
  • Disallow certain file types that are not relevant for SEO
  • Allow important directories
Back to top