Log In Sign Up

coursefolder.net
robots.txt

Robots Exclusion Standard data for coursefolder.net

Resource Scan

Scan Details

Site Domain coursefolder.net
Base Domain coursefolder.net
Scan Status Ok
Last Scan2025-11-28T17:49:47+00:00
Next Scan 2025-12-05T17:49:47+00:00

Last Scan

Scanned2025-11-28T17:49:47+00:00
URL https://coursefolder.net/robots.txt
Domain IPs 208.91.199.108
Response IP 208.91.199.108
Found Yes
Hash 089d74aa87f47aa49209d87e9309914f511a547b45606a7f9dead1cb1ef71e86
SimHash 25c4d941e402

Groups

*

Rule Path
Allow /
Allow /udemy-coupon-codes.php
Disallow /server.php
Disallow /cron.php
Disallow /generate_*.php
Disallow /sitemap_cron.log
Allow /live-free-udemy-coupon.php
Allow /category/Development
Allow /category/Business
Allow /category/IT-and-Software
Allow /category/Marketing
Allow /category/Design

Other Records

Field Value
crawl-delay 0
crawl-delay 1

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

semrushbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

mj12bot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

baiduspider

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

googlebot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 0

googlebot

Rule Path
Allow /
Disallow /cron.php
Disallow /server.php
Disallow /generate_sitemap*.php
Disallow /.git/

Other Records

Field Value
crawl-delay 1

bingbot

Rule Path
Allow /
Disallow /cron.php
Disallow /server.php
Disallow /generate_sitemap*.php
Disallow /.git/

Other Records

Field Value
crawl-delay 2

yandexbot

Rule Path
Allow /
Disallow /cron.php
Disallow /server.php
Disallow /generate_sitemap*.php
Disallow /.git/

Other Records

Field Value
crawl-delay 3

duckduckbot

Rule Path
Allow /
Disallow /cron.php
Disallow /server.php
Disallow /generate_sitemap*.php
Disallow /.git/

Other Records

Field Value
crawl-delay 3

googlebot-image

Rule Path
Allow /*.jpg$
Allow /*.jpeg$
Allow /*.png$
Allow /*.webp$
Allow /*.gif$

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

*

Rule Path
Allow /*.js$
Allow /*.css$
Allow /*.png$
Allow /*.jpg$
Allow /*.gif$
Allow /*.svg$
Allow /*.webp$
Back to top

Other Records

Field Value
sitemap https://coursefolder.net/sitemap-index.xml
sitemap https://coursefolder.net/sitemap.xml
sitemap https://coursefolder.net/sitemap-courses.xml
sitemap https://coursefolder.net/sitemap-blog.xml
sitemap https://coursefolder.net/sitemap.xml
sitemap https://coursefolder.net/sitemap-index.php
sitemap https://coursefolder.net/sitemap-course.php
sitemap https://coursefolder.net/sitemap-live-course.php
sitemap https://coursefolder.net/sitemap-category.php
sitemap https://coursefolder.net/sitemap-blog.php
Back to top

Comments

  • Important pages - crawled frequently
  • Sitemaps - updated every 10 minutes
  • Crawl delay for heavy crawlers
  • Prioritize the most important pages
  • Sitemaps
  • Special directives for specific bots
  • Speed up Google Image crawler
  • Prevent specific bots from crawling
  • Main SEO directives
Back to top