Rill Architecture Documentation

Overview

Rill is a BI-as-code platform that delivers the fastest path from data lake to dashboard.

It provides rapid, self-service dashboards built directly on raw data lakes, with an embedded in-memory database powered by DuckDB or ClickHouse for millisecond query performance.

Core Value Proposition

• For Data Engineers & Analysts: Rapid, self-service dashboards built directly on raw data lakes, eliminating traditional BI complexity.
• For Data Consumers: Reliable, fast-loading dashboards with accurate, real-time metrics.
• For AI Systems: Structured, real-time data access through the metrics layer for quick and accurate AI-driven decision-making.

Design Principles

Rill is built around six core design principles:

1. Lightning Fast - Powered by SvelteKit & DuckDB for conversationally fast performance.
2. Universal Data Support - Works with local and remote datasets (Parquet, CSV, S3, GCS, HTTPS, local files).
3. Automatic Profiling - Build intuition about datasets through automatic profiling.
4. Real-time Response - Responds to each keystroke by re-profiling the resulting dataset.
5. Interactive Dashboards - Thoughtful, opinionated defaults for quick insights.
6. Dashboards as Code - Version control, Git sharing, and easy project rehydration.

Technology Stack

Frontend Stack

• Framework: TypeScript + SvelteKit
• UI Components: Svelte components with modular, reusable design
• Build Tool: Vite
• Styling: Tailwind CSS (configurable)
• State Management: Svelte stores
• Testing: Playwright (E2E), Vitest (unit tests)

Backend Stack

• Language: Go 1.25+
• CLI Framework: Cobra
• Concurrency: Go routines and channels for orchestration
• API Protocols: gRPC + gRPC-Gateway for REST

Data Layer

• Embedded OLAP: DuckDB (primary, for small-to-medium data)
• External OLAP: ClickHouse, Apache Druid (for big data)
• Metadata Storage: PostgreSQL (for cloud/hosted deployments)
• File Storage: Local disk + cloud object stores (S3, GCS)

Monorepo Structure

Rill uses a monorepo to coordinate changes across multiple components in a single PR and provide a unified codebase for contributors.

Key Directories

Core Concepts

BI-as-Code

Rill treats dashboards and metrics as code, enabling version control and reproducibility:

1. SQL Models: Define data transformations via SQL queries.
2. YAML Configuration: Configure metrics views, dashboards, and project settings.
3. Git Integration: Version control for analytics assets.
4. CLI Tools: Deploy and manage from the command line.

Metrics Layer

Purpose: Provide a unified, standardized way to define, compute, and serve business metrics.

Components:

• Dimensions: Grouping attributes (e.g., country, product_category).
• Measures: Aggregated metrics (e.g., total_revenue, order_count).
• Timeseries: Temporal dimension for time-based analysis.

Benefits:

• Reusable metric definitions across dashboards, APIs, and AI agents.
• Consistent business logic.
• Optimized SQL generation.
• Real-time query execution with sub-second latency.

Deployment Configurations

Rill supports two primary deployment modes:

Local Development Mode

• Single-process execution
• All services (CLI, runtime, UI) run together
• DuckDB for OLAP
• No authentication

Cloud Production Mode

• Multi-service architecture
• Services deployed independently (admin, runtime, UI)
• PostgreSQL for metadata
• S3/GCS for file storage
• OAuth authentication

Data Flow Architecture

graph TB
subgraph "User Layer"
CLI[CLI]
WebUI[Web UI<br />SvelteKit]
end

subgraph "Control Plane"
Admin[Admin Service<br />Go + gRPC]
Postgres[(PostgreSQL<br />Metadata)]
end

subgraph "Data Plane"
Runtime[Runtime<br />Go + gRPC]
DuckDB[(DuckDB<br />Embedded OLAP)]
ClickHouse[(ClickHouse<br />External OLAP)]
end

subgraph "Data Sources"
S3[S3/GCS<br />Parquet/CSV]
Local[Local Files]
DB[Databases<br />BigQuery, etc.]
end

CLI -->|gRPC| Runtime
WebUI -->|HTTP/gRPC| Admin
WebUI -->|HTTP/gRPC| Runtime
Admin -->|Provisions| Runtime
Admin -->|Stores Metadata| Postgres
Runtime -->|Queries| DuckDB
Runtime -->|Queries| ClickHouse
Runtime -->|Ingests| S3
Runtime -->|Ingests| Local
Runtime -->|Connects| DB

style Runtime fill:#4285F4,color:#fff
style Admin fill:#34A853,color:#fff
style DuckDB fill:#FBBC04,color:#000
style WebUI fill:#EA4335,color:#fff

Conclusion

Rill provides a powerful platform for transforming raw data into actionable insights through a BI-as-code approach. By leveraging modern technologies like SvelteKit and DuckDB, it offers a fast, reliable, and scalable solution for modern data analytics.