Log In Sign Up

byteconnect.io
robots.txt

Robots Exclusion Standard data for byteconnect.io

Resource Scan

Scan Details

Site Domain byteconnect.io
Base Domain byteconnect.io
Scan Status Ok
Last Scan2025-08-22T02:49:13+00:00
Next Scan 2025-09-21T02:49:13+00:00

Last Scan

Scanned2025-08-22T02:49:13+00:00
URL https://byteconnect.io/robots.txt
Domain IPs 104.21.95.148, 172.67.145.129, 2606:4700:3034::ac43:9181, 2606:4700:3037::6815:5f94
Response IP 172.67.145.129
Found Yes
Hash a5211b778ada6a8e77c2b59073ee2d23d0252d6f33b719cac33efd7691a0c2fd
SimHash 44178c1246e6

Groups

*

Rule Path
Allow /
Disallow /admin/
Disallow /dashboard/
Disallow /api/
Disallow /dev/
Disallow /*?

*

Rule Path
Disallow /staging/
Disallow /test/
Disallow /dev-*/

googlebot

Rule Path
Allow /

bingbot

Rule Path
Allow /

ultrasearchbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

semrushbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5
Back to top

Other Records

Field Value
sitemap https://byteconnect.io/sitemap.xml
Back to top

Comments

  • ByteConnect robots.txt file
  • Last updated: 2023-06-01
  • Allow all search engines to crawl most of the site
  • Prevent all bots from accessing development environments
  • Allow Googlebot additional access
  • Allow Bingbot additional access
  • Sitemap location
  • Crawl delay for certain bots
  • Slow down bots that don't respect our resources
Back to top