Contributing

Contributing to ncps#

Thank you for your interest in contributing to ncps! This document provides guidelines and instructions for contributing to the project.

Getting Started#

Prerequisites#

The project uses Nix flakes with direnv for reproducible development environments. You'll need:

1. Nix with flakes enabled - Installation guide
2. direnv - Installation guide

Initial Setup#

1. Clone the repository:

   git clone https://github.com/kalbasit/ncps.git
   cd ncps

2. Allow direnv:

   direnv allow

   This will automatically load the development environment with all required tools:

   • Go
   • golangci-lint
   • delve (debugger)
   • watchexec
   • sqlfluff
   • Garage (for S3 testing)
   • PostgreSQL (for database testing)
   • MySQL/MariaDB (for database testing)
   • Redis (for distributed locking testing)

Development Environment#

Available Tools#

Once in the development shell, you have access to:

Tool	Purpose
go	Go compiler and toolchain
golangci-lint	Code linting with 30+ linters
delve	Go debugger
watchexec	File watcher for hot-reloading
sqlfluff	SQL linting and formatting
garage	S3-compatible object storage
postgresql	PostgreSQL database server
mariadb	MySQL/MariaDB database server
redis	Redis server for distributed locks

Development Dependencies#

The project uses process-compose-flake for managing development services. Start dependencies with:

nix run .#deps

This starts:

• Garage - S3-compatible object store (S3 API on port 9000, admin on 3903)
  • Test bucket: test-bucket
  • Credentials: GK1234567890abcdef12345678 / 0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
  • Self-validation ensures proper setup
• PostgreSQL - Database server (port 5432)
  • Test database: test-db
  • Credentials: test-user / test-password
  • Connection URL: postgresql://test-user:test-password@127.0.0.1:5432/test-db?sslmode=disable
• MariaDB - MySQL-compatible database server (port 3306)
  • Test database: test-db
  • Credentials: test-user / test-password
  • Connection URL: mysql://test-user:test-password@127.0.0.1:3306/test-db
• Redis - Distributed locking server (port 6379)
  • No authentication required (test environment)
  • Used for distributed lock testing

Development Workflow#

Running the Development Server#

The development server supports hot-reloading and multiple storage backends:

# Using local filesystem storage (default, no dependencies required)
./dev-scripts/run.sh
# or explicitly
./dev-scripts/run.sh local

# Using S3/Garage storage (requires dependencies to be running)
# In a separate terminal:
nix run .#deps

# Then run the dev server:
./dev-scripts/run.sh s3

The server automatically restarts when you modify code files.

Editing Documentation#

The documentation for this project is managed using Trilium. Follow these steps to contribute to the documentation:

1. Run the documentation editor:

   trilium-edit-docs

   This tool is available in the PATH provided by the Nix flake, see Development Environment in the Developer Guide for more information.

2. Edit the documentation through Trilium: The Trilium interface will open, allowing you to edit the notes. Trilium automatically exports the markdown files back to the repository as you make changes.

3. Wait and close: Wait 5 minutes after you have finished all your edits to ensure all changes are synced, then close Trilium.

4. Format the documentation:

   Run the project formatter to ensure the markdown files follow the project's standards:

   nix fmt

5. Submit your changes: Submit a Pull Request with your changes.

Database Migrations#

The schema is defined in Ent (ent/schema/*.go), per-dialect migrations are generated by Atlas (used as a Go library, never the CLI), and Goose applies them at runtime via the embedded migrations/<dialect>/ directories.

Creating a new migration:

1. Edit ent/schema/<entity>.go (field/edge/index/annotation change).

2. Regenerate the Ent client: go generate ./ent/... (or task ent:generate).

3. Generate per-dialect Atlas migrations:

   task migrations:gen NAME=descriptive_snake_case

   This writes one timestamp-prefixed .sql per dialect under migrations/sqlite/, migrations/postgres/, and migrations/mysql/, plus updates each dialect's atlas.sum integrity file.

Applying migrations:

ncps migrate up --cache-database-url=sqlite:/path/to/db.sqlite
ncps migrate up --cache-database-url=postgresql://user:pass@host:port/db
ncps migrate up --cache-database-url=mysql://user:pass@host:port/db
# Preview without touching the DB:
ncps migrate up --cache-database-url=... --dry-run

The same flag also accepts the env var CACHE_DATABASE_URL. Down migrations are intentionally not supported — see the expand-contract policy in CLAUDE.md for the safe column-change recipe.

Skills under .agent/skills/ codify the day-to-day workflows:

• /migrate-new — edit an Ent schema + generate the per-dialect migrations
• /migrate-up — apply migrations
• /migrate-down — read this to learn why you don't want to roll back

The cmd/ent-lint AST linter (run via task ent:lint or task ent:check) enforces the five Ent codegen invariants documented at the top of cmd/ent-lint/main.go.

Code Quality Standards#

Formatting#

IMPORTANT: Always run formatters first before making manual changes:

# Format all files (Go, Nix, SQL, YAML, Markdown)
nix fmt

The project uses:

• gofumpt - Stricter Go formatting
• goimports - Import organization
• gci - Import grouping (standard → default → alias → localmodule)
• nixfmt - Nix code formatting
• sqlfluff - SQL formatting and linting
• yamlfmt - YAML formatting
• mdformat - Markdown formatting

Linting#

IMPORTANT: Always run golangci-lint run --fix first to automatically fix issues:

# Auto-fix all fixable linting issues
golangci-lint run --fix

# Lint without auto-fix
golangci-lint run

# Lint specific package
golangci-lint run ./pkg/server/...

The project uses 30+ linters including:

• err113 - Explicit error wrapping
• exhaustive - Exhaustive switch statements
• gosec - Security checks
• paralleltest - Parallel test detection
• testpackage - Test package naming

See .golangci.yml for complete linter configuration.

SQL Linting#

# Lint SQL files
sqlfluff lint migrations/sqlite/*.sql
sqlfluff lint migrations/postgres/*.sql
sqlfluff lint migrations/mysql/*.sql

# Format SQL files
sqlfluff format migrations/sqlite/*.sql

Testing#

Running Tests#

# Run all tests with race detector (recommended)
go test -race ./...

# Run tests for specific package
go test -race ./pkg/server/...

# Run a single test
go test -race -run TestName ./pkg/server/...

Integration Tests#

The project includes integration tests for S3, PostgreSQL, MySQL, and Redis. Integration tests are disabled by default and must be explicitly enabled using shell helper functions.

Quick Start:

# In terminal 1: Start development dependencies
nix run .#deps

# In terminal 2: Enable integration tests and run tests
eval "$(enable-integration-tests)"
go test -race ./...

Available Helper Commands:

The development shell provides commands to easily enable/disable integration tests:

Command	Description
eval "$(enable-s3-tests)"	Enable S3/Garage integration tests
eval "$(enable-postgres-tests)"	Enable PostgreSQL integration tests
eval "$(enable-redis-tests)"	Enable Redis integration tests
eval "$(enable-mysql-tests)"	Enable MySQL integration tests
eval "$(enable-integration-tests)"	Enable all integration tests at once
eval "$(disable-integration-tests)"	Disable all integration tests

Running Specific Integration Tests:

# Start dependencies (in a separate terminal)
nix run .#deps

# Enable and run S3 tests only
eval "$(enable-s3-tests)"
go test -race ./pkg/storage/s3

# Enable and run database tests only
eval "$(enable-postgres-tests)"
eval "$(enable-mysql-tests)"
go test -race ./pkg/database

# Enable all tests and run everything
eval "$(enable-integration-tests)"
go test -race ./...

# Disable integration tests when done
eval "$(disable-integration-tests)"

What the Helper Commands Do:

The helper commands output shell export statements that you evaluate in your current shell:

• **enable-s3-tests** exports:
  • NCPS_TEST_S3_BUCKET=test-bucket
  • NCPS_TEST_S3_ENDPOINT=http://127.0.0.1:9000
  • NCPS_TEST_S3_REGION=us-east-1
  • NCPS_TEST_S3_ACCESS_KEY_ID=GK1234567890abcdef12345678
  • NCPS_TEST_S3_SECRET_ACCESS_KEY=0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
• **enable-postgres-tests** exports:
  • NCPS_TEST_POSTGRES_URL=postgresql://test-user:test-password@127.0.0.1:5432/test-db?sslmode=disable
• **enable-mysql-tests** exports:
  • NCPS_TEST_MYSQL_URL=mysql://test-user:test-password@127.0.0.1:3306/test-db
• **enable-redis-tests** exports:
  • NCPS_ENABLE_REDIS_TESTS=1

Tests automatically skip if these environment variables aren't set, so you can run go test -race ./... without enabling integration tests and only unit tests will run.

Test Requirements#

• Use testify for assertions
• Enable race detector (-race flag)
• Use _test package suffix (enforced by testpackage linter)
• Write parallel tests where possible (checked by paralleltest linter)
• Each test should be isolated and not depend on other tests

Nix Build Tests#

# Run all checks including integration tests
nix flake check

# Build package (includes test phase)
nix build

The Nix build automatically:

1. Starts Garage, PostgreSQL, MariaDB, and Redis in preCheck phase
2. Creates test databases and buckets
3. Exports test environment variables
4. Runs all tests (including integration tests)
5. Stops services in postCheck phase

End-to-end testing (unified harness)#

End-to-end testing — including Helm chart validation on a local Kind cluster (Garage, PostgreSQL, MariaDB, Redis) — is driven by a single scenario-based harness, nix run .#e2e (also task test:e2e). The same scenario catalog runs against either a local dev-scripts/run.py deployment or a Kubernetes Kind/Helm deployment, selected with --mode.

Prerequisites: all tools come from the Nix development environment (Docker for Kind; kubectl/helm/kind/skopeo bundled into the e2e package). Local mode also uses the dev shell toolchain (go, dbmate, direnv, watchexec), so run local-mode scenarios via task test:e2e or inside nix develop.

List scenarios:

nix run .#e2e -- --list

Run a scenario locally (run.py + fixed-port nix run .#deps backends, started and torn down automatically):

task test:e2e -- --mode local --scenario cdc-lifecycle
task test:e2e -- --mode local --scenario staging-contention

Run a scenario on Kubernetes (Kind cluster + Helm chart; the cluster is created idempotently and reused across runs):

nix run .#e2e -- --mode kubernetes --scenario single-s3-postgres
nix run .#e2e -- --mode kubernetes --scenario single-local-sqlite -v

A scenario declares which modes it supports; requesting it in an unsupported mode reports SKIP (never PASS).

Scenarios: the catalog covers the single-instance, external-secret, and high-availability Helm permutations (formerly the k8s-tests permutations) plus the cdc-lifecycle (non-CDC → CDC → drain → non-CDC) and staging-contention (multi-replica in-flight-staging) feature drivers.

Adding a scenario: add a permutation entry to nix/e2e-tests/config.nix (the single source of truth for both modes):

{
  permutations = [
    # ... existing permutations
    {
      name = "my-new-scenario";
      description = "My custom test scenario";
      replicas = 1;
      storage = { type = "s3"; };
      database = { type = "postgresql"; };
      redis.enabled = false;
      features = [ ]; # Optional: ["cdc", "ha", "pod-disruption-budget"]
      # Optional harness-only keys:
      # phase = "serve" | "cdc-lifecycle" | "staging-contention";
      # modes = [ "local" "kubernetes" ];
    }
  ];
}

CI note: this harness is manual / opt-in and is intentionally not part of nix flake check (Kind and network-NAR scenarios far exceed the per-PR budget). See nix/e2e-tests/README.md.

For more details: See nix/e2e-tests/README.md

Pull Request Process#

Before Submitting#

1. Format your code:

   nix fmt

2. Fix linting issues:

   golangci-lint run --fix

3. Run tests:

   go test -race ./...

4. Build successfully:

   nix build

Commit Guidelines#

• Use clear, descriptive commit messages
• Follow Conventional Commits when possible:
  • feat: - New features
  • fix: - Bug fixes
  • docs: - Documentation changes
  • refactor: - Code refactoring
  • test: - Test additions/changes
  • chore: - Build/tooling changes

Pull Request Guidelines#

1. Create a feature branch:

   git checkout -b feature/your-feature-name

2. Make your changes following code quality standards

3. Update documentation if needed (README.md, CLAUDE.md, etc.)

4. Add tests for new functionality

5. Submit PR with:

   • Clear description of changes
   • Reference to related issues
   • Screenshots/examples if applicable

CI/CD Notes#

The project uses GitHub Actions for CI/CD:

• Workflows only run on PRs targeting main branch
• This supports Graphite-style stacked PRs efficiently
• When modifying workflows, maintain the branches: [main] restriction

Project Structure#

ncps/
├── cmd/                        # CLI commands
│   ├── serve.go                # Main serve command
│   ├── ent-lint/               # AST linter for Ent codegen invariants
│   ├── generate-migrations/    # Atlas-driven per-dialect generator
│   └── atlas-sum-check/        # CI helper verifying atlas.sum integrity
├── pkg/
│   ├── cache/                  # Core caching logic
│   ├── storage/                # Storage abstraction
│   │   ├── local/              # Local filesystem storage
│   │   └── s3/                 # S3-compatible storage
│   ├── database/               # Thin facade over the generated Ent client
│   │   └── migrate/            # State detection + adoption + apply
│   ├── lock/                   # Lock abstraction
│   │   ├── local/              # Local locks (single-instance)
│   │   └── redis/              # Redis distributed locks (HA)
│   ├── server/                 # HTTP server (Chi router)
│   └── nar/                    # NAR format handling
├── ent/                        # Ent schema definitions
│   ├── schema/                 # Hand-authored entity schemas (*.go)
│   └── ...                     # Generated client (committed)
├── migrations/                 # Per-dialect Atlas migrations
│   ├── sqlite/                 # SQLite migrations + atlas.sum
│   ├── postgres/               # PostgreSQL migrations + atlas.sum
│   └── mysql/                  # MySQL migrations + atlas.sum
├── nix/                        # Nix configuration
│   ├── packages/               # Package definitions
│   ├── devshells/              # Development shells
│   ├── formatter/              # Formatter configuration
│   └── process-compose/        # Development services
└── dev-scripts/                # Development helper scripts
    └── run.py                  # Development server orchestrator

Key Interfaces#

Storage (**pkg/storage/store.go**):

• ConfigStore - Secret key storage
• NarInfoStore - NarInfo metadata storage
• NarStore - NAR file storage

Both local and S3 backends implement these interfaces.

Locks (**pkg/lock/lock.go**):

• Locker - Exclusive locking for download deduplication
• RWLocker - Read-write locking for LRU coordination

Both local (sync.Mutex) and Redis (Redlock) backends implement these interfaces. Redis locks enable high-availability deployments with multiple instances.

Database:

• Supports SQLite, PostgreSQL, and MySQL via Ent (the type-safe ORM)
• Database selection via URL scheme in --cache-database-url
• Schema lives in ent/schema/*.go; generated client lives under ent/
• Per-dialect migrations under migrations/<dialect>/ are produced by Atlas (as a Go library) and applied at runtime by Goose

Common Development Tasks#

Adding a New Database Migration#

1. Edit ent/schema/<entity>.go — add the field/edge/index/annotation. The five codegen invariants are enforced by cmd/ent-lint; see cmd/ent-lint/main.go for the full list (table-level entsql.Check, entsql.OnDelete on edge.To, etc.).

2. Regenerate the Ent client: go generate ./ent/... (or task ent:generate).

3. Generate per-dialect Atlas migrations + update each atlas.sum:

   task migrations:gen NAME=descriptive_snake_case

4. Review the generated .sql files under migrations/sqlite/, migrations/postgres/, migrations/mysql/. Each file is a single timestamp-prefixed file with +goose Up / +goose Down markers. SQLite files that need PRAGMA foreign_keys = OFF must also carry -- +goose NO TRANSACTION.

5. Apply locally to verify:

   ncps migrate up --cache-database-url=sqlite:./test.db --dry-run
   ncps migrate up --cache-database-url=sqlite:./test.db

The expand-contract policy in CLAUDE.md documents the four-step recipe for non-additive column changes (e.g. adding NOT NULL). Down migrations are intentionally not supported.

Regenerating the Ent client#

After editing any ent/schema/*.go:

go generate ./ent/...      # or: task ent:generate
go run ./cmd/ent-lint --root .   # or: task ent:lint

task ent:check runs both and verifies the generated tree is up to date.

Adding a New Storage Backend#

1. Implement the storage interfaces in pkg/storage/
2. Add configuration flags in cmd/serve.go
3. Update documentation in README.md and CLAUDE.md
4. Add integration tests
5. Update Docker and Kubernetes examples if applicable

Debugging#

Use delve for debugging:

# Debug the application
dlv debug . -- serve --cache-hostname=localhost --cache-storage-local=/tmp/ncps

# Debug a specific test
dlv test ./pkg/server -- -test.run TestName

Note: The dev shell disables fortify hardening to allow delve to work.

Building Docker Images#

# Build Docker image
nix build .#docker

# Load into Docker
docker load < result

# Push to registry (requires DOCKER_IMAGE_TAGS environment variable)
DOCKER_IMAGE_TAGS="kalbasit/ncps:latest kalbasit/ncps:v1.0.0" nix run .#push-docker-image

Getting Help#

• Documentation Issues: Check CLAUDE.md for detailed development guidance
• Bug Reports: Open an issue
• Questions: Start a discussion
• Security Issues: Contact maintainers privately

Code of Conduct#

• Be respectful and inclusive
• Provide constructive feedback
• Focus on what's best for the project
• Show empathy towards other contributors

License#

By contributing to ncps, you agree that your contributions will be licensed under the MIT License.

---

Thank you for contributing to ncps! 🎉