Developer Guide
StoreIT is a self-hosted home inventory management system built with:
- Backend: Rust (edition 2024, stable toolchain), Axum web framework
- Frontend: Svelte 5 + Vite + Tailwind CSS, built as a PWA
- Database: SQLite via sqlx with compile-time checked queries
- Auth: OIDC (any provider) + local username/password + mock provider for dev
- AI: Anthropic Claude API for item identification from photos
- Deployment: Single binary with frontend embedded via
rust-embed
Quick Reference
make frontend-install # npm install
make dev-db # create SQLite DB + run migrations
cargo run -p storeit-server # backend on :8080
make frontend-dev # vite dev server on :5173 (proxies to :8080)
make test # all Rust tests
make lint # cargo fmt + clippy
cd frontend && npm test # frontend unit tests (Vitest)
Repository Layout
crates/
storeit-domain/ # Entity types, traits, errors
storeit-db-sqlite/ # SQLite repository implementations
storeit-storage-fs/ # Filesystem image storage + thumbnails
storeit-auth/ # OIDC + local + mock auth
storeit-ai/ # Anthropic API + Claude CLI
storeit-server/ # Axum server, handlers, CLI, interchange
frontend/ # Svelte 5 + Vite + Tailwind PWA
e2e/ # Playwright end-to-end tests
tools/ # storeit-ctl management script
docs/ # mdBook documentation (user + dev)
Architecture Overview
High-Level Design
Svelte 5 PWA (embedded) ──► Axum HTTP Server
│
┌─────────────┼─────────────┐
▼ ▼ ▼
SQLite DB Filesystem Claude API
(+ FTS5) (images) (identification)
Key Principles
Single Binary
The Svelte frontend is compiled to static files, then embedded into the Rust binary via rust-embed. The result is a single executable that serves both the API and the frontend — no separate web server, no file deployment.
Domain-Driven Design
Core types and traits live in storeit-domain. Repository traits define the data access interface. SQLite implementations live in storeit-db-sqlite. This separation means the domain logic has zero knowledge of the database.
Compile-Time SQL
All SQL queries use sqlx::query! macros, which check queries against the database schema at compile time. The .sqlx/ directory contains an offline cache so CI doesn’t need a live database. Run make prepare after changing queries or migrations.
Data Model
Location (has many)──► Container (has many)──► Item
│ │ │
├── Photos ├── Photos ├── Photos
├── NFC Tags ├── NFC Tags ├── NFC Tags
└── Child Locations └── Child Containers └── (leaf)
- Locations, containers, and items belong to a group for multi-tenant isolation
- Containers have polymorphic parents: either a location or another container
- Items can belong to either a container or a location directly
- Photos use content-addressable storage keyed by SHA-256 hash
Multi-Tenant Isolation
All inventory data is scoped to a group. Users belong to one or more groups via memberships. API handlers receive the active group from the session and pass it to repository methods, ensuring users only see their own data.
Backend Crates
The backend is organized as a Cargo workspace with six crates.
storeit-domain
Core domain types, traits, and error definitions. No external dependencies beyond serde/chrono/uuid.
Key types:
Location,Container,Item— entity structsPhoto,NfcTag— supporting entitiesLocationRepo,ContainerRepo,ItemRepo,PhotoRepo,NfcRepo— repository traitsImageStorage— trait for image storage backendsAiIdentification— AI identification result typeDomainError— unified error enum
storeit-db-sqlite
SQLite implementations of all repository traits using sqlx.
- Migrations in
migrations/directory - Uses
sqlx::query!for compile-time SQL checking - FTS5 virtual table for full-text search with
full_reindex()support - Schema versioning via
_metatable (schema_version(),set_schema_version()) EXPECTED_SCHEMA_VERSIONconstant for startup version checks- Offline query cache in
.sqlx/(regenerate withmake prepare)
storeit-storage-fs
Filesystem-based image storage with thumbnail generation.
- Content-addressable storage: files stored as
{sha256[..2]}/{sha256}.{ext} - Automatic WebP thumbnail generation via
webpcrate (lossy VP8, quality 80) - Thumbnails stored alongside originals as
{sha256}_thumb.webp - Max thumbnail dimension: 200px
storeit-auth
Authentication providers:
- OIDC — Full OpenID Connect flow (Authentik, Keycloak, etc.)
- Local — Username/password with bcrypt hashing
- Mock — Accepts any credentials, for development/testing
storeit-ai
AI item identification via:
- Anthropic API — Direct HTTP calls to Claude API with image input
- Claude CLI — Fallback using the
claudeCLI binary
storeit-server
Axum HTTP server that wires everything together:
- Route handlers in
src/handlers/ - Embedded frontend via
rust-embedinsrc/static_files.rs - OpenAPI documentation via
utoipa(Swagger UI at/swagger-ui) - 50MB body limit for photo uploads
- Integration tests in
tests/api/ - CLI subcommands via clap (
src/cli.rs):serve,import <archive>,version - Interchange module (
src/interchange.rs): archive export/import with zstd compression, manifest v2, version transforms - Startup schema version check — refuses to start on mismatch
tools/storeit-ctl
Python 3.8+ management script (stdlib only, no pip dependencies):
status— show running version and healthbackup— download archive from running serverupgrade/downgrade— orchestrate full version migration with rollbackinstall— fresh install from GitHub Releasesrollback/cleanup— manage rollback state
Frontend Structure
The frontend is a Svelte 5 single-page application built with Vite and Tailwind CSS, deployed as a Progressive Web App.
Directory Layout
frontend/src/
api/
client.ts # HTTP client with auth cookie handling
index.ts # All API functions (fetchLocations, createItem, etc.)
types.ts # TypeScript types matching backend DTOs
components/
BottomNav.svelte # Tab navigation bar
Breadcrumbs.svelte # Location path breadcrumbs
CreateDialog.svelte # Modal for creating entities
EntityCard.svelte # Reusable card for lists
MoveFlow.svelte # Multi-step move workflow
ParentPicker.svelte # Tree browser for selecting parents
PhotoGallery.svelte # Photo grid with upload
PhotoLightbox.svelte # Full-screen photo viewer
PhotoThumbnail.svelte # Single thumbnail with loading state
PrintLabel.svelte # QR code print label
QrCode.svelte # QR code SVG generator
NfcTagManager.svelte # NFC tag assignment UI
SearchBar.svelte # Search input with navigation
...
lib/
auth.svelte.ts # Auth store using Svelte 5 runes
offlineQueue.ts # Queue for offline mutations
pages/
HomePage.svelte # Location browser (root view)
LocationPage.svelte # Location detail + children
ContainerPage.svelte # Container detail + children
ItemDetailPage.svelte # Item detail + photos + NFC
SearchPage.svelte # Full-text search results
AddItemPage.svelte # AI-powered item creation
BatchAddItemPage.svelte # Batch item creation
AdminPage.svelte # User/group management + backup
SettingsPage.svelte # Image storage settings
NfcResolvePage.svelte # NFC tag resolution landing
styles/
index.css # Tailwind imports
App.svelte # Root component with router
main.ts # Entry point
Routing
The app uses @mateothegreat/svelte5-router for client-side routing:
| Path | Page |
|---|---|
/ | HomePage |
/locations/:id | LocationPage |
/containers/:id | ContainerPage |
/items/:id | ItemDetailPage |
/search | SearchPage |
/add | AddItemPage |
/add/batch | BatchAddItemPage |
/settings | SettingsPage |
/admin | AdminPage |
/nfc/:tagUri | NfcResolvePage |
Path Alias
The ~ alias maps to src/, configured in vite.config.ts and tsconfig.json. Use it for imports:
import { fetchLocations } from "~/api";
API Client
All API functions are in frontend/src/api/index.ts. They use the HTTP client from client.ts which handles auth cookies and base URL resolution. The Vite dev server proxies /api requests to the Rust backend on port 8080.
Database
StoreIT uses SQLite with the sqlx library for compile-time checked queries.
Migrations
Migrations live in crates/storeit-db-sqlite/migrations/ and follow the naming convention YYYYMMDDHHMMSS_description.sql.
# Run migrations
make migrate
# Create a new migration
sqlx migrate add -r description --source crates/storeit-db-sqlite/migrations
Offline Mode
sqlx checks SQL queries at compile time against a real database. For CI builds (where no database is available), an offline query cache is stored in .sqlx/.
# Regenerate the offline cache after changing queries
make prepare
# Build using offline cache
SQLX_OFFLINE=true cargo build
Full-Text Search
StoreIT uses SQLite FTS5 for search. The FTS virtual table indexes item names, descriptions, and tags. Search queries go through the search repository method which joins FTS results with the main tables.
Schema Overview
Core tables:
locations— physical spaces, self-referencing for sub-locationscontainers— storage containers, polymorphic parent (location or container)items— tracked items, polymorphic parentphotos— image metadata, polymorphic parentnfc_tags— NFC tag registrationsusers/sessions— authenticationsettings— application settingsitems_fts— FTS5 virtual table for search
Development Setup
Prerequisites
- Rust 1.85+ (stable toolchain)
- Node.js 22+ and npm
- SQLite 3
- sqlx-cli —
cargo install sqlx-cli --features sqlite
Initial Setup
# Install frontend dependencies
make frontend-install
# Create development database and run migrations
make dev-db
# Generate sqlx offline cache
make prepare
Running in Development
Run the backend and frontend dev server simultaneously:
# Terminal 1: Backend (port 8080)
cargo run -p storeit-server
# Terminal 2: Frontend dev server (port 5173, proxies API to 8080)
make frontend-dev
Open http://localhost:5173 for hot-reloading frontend development. The Vite dev server proxies all /api requests to the Rust backend.
Without an OIDC provider configured, the mock auth provider is used automatically — any username/password works.
Git Hooks
The project includes a pre-commit hook that automatically runs cargo fmt and cargo clippy --fix before each commit. Enable it with:
git config core.hooksPath .githooks
This ensures formatting and lint issues are fixed locally before they reach CI.
Environment Variables
Copy the defaults or set these in your shell:
export DATABASE_URL="sqlite:./dev.db?mode=rwc"
export STOREIT_BIND="0.0.0.0:8080"
export STOREIT_IMAGE_PATH="./data/images"
See the Configuration page for all options.
Building
Development Build
cargo run -p storeit-server # compiles and runs
Production Build
make build-all # builds frontend, then cargo build --release
This produces a single binary at target/release/storeit-server with the frontend embedded.
What make build-all Does
make frontend-build— runsnpm run buildinfrontend/, producingfrontend/dist/make build— runsmake preparethenSQLX_OFFLINE=true cargo build --workspace --release
The storeit-server crate uses rust-embed to embed frontend/dist/ into the binary at compile time.
Cross-Compilation
The project targets these platforms (via CI):
| Target | Notes |
|---|---|
x86_64-unknown-linux-gnu | Linux x86_64, glibc |
x86_64-unknown-linux-musl | Linux x86_64, static binary |
aarch64-unknown-linux-gnu | Linux ARM64, glibc |
aarch64-unknown-linux-musl | Linux ARM64, static binary |
x86_64-apple-darwin | macOS Intel |
aarch64-apple-darwin | macOS Apple Silicon |
x86_64-pc-windows-msvc | Windows x86_64 |
The musl targets produce fully static binaries with no runtime dependencies.
Note: The webp crate depends on libwebp-sys, which compiles Google’s libwebp from source. This works correctly for all targets including musl.
Docker
docker build -t storeit .
The Docker image uses a multi-stage build: Rust compilation in a builder stage, then a minimal runtime image.
Testing
Four Test Layers
1. Rust Unit Tests
In each crate’s src/ files via #[cfg(test)] modules.
cargo test --workspace
2. Rust Integration Tests
Full HTTP API tests in crates/storeit-server/tests/api/. Each test spins up a real server with an in-memory SQLite database and mock auth.
cargo test -p storeit-server --test integration
3. Frontend Component Tests
Vitest with Svelte testing utilities. Co-located as *.test.ts next to their components.
cd frontend && npm test
4. E2E Tests
Playwright tests in e2e/tests/. Run against a fully built binary with mock auth and a temporary database.
make e2e-test # builds everything first
# or manually:
cd e2e && npx playwright test
E2E tests block service workers (serviceWorkers: "block" in Playwright config) to ensure all requests hit the server directly.
Conventions
- TDD preferred: Write failing tests first, then implement
- Real images: Photo tests use actual PNG data (via
pngjsor hand-crafted minimal PNGs) - WebP validation: Thumbnail tests verify RIFF/WEBP/VP8 headers (must be lossy VP8, not VP8L)
- Mock auth: Integration and E2E tests use the mock auth provider — any credentials work
- Isolated databases: Each test gets its own in-memory SQLite database
Coverage
Enforced at 93% minimum via cargo llvm-cov:
make coverage
Excludes main.rs (CLI entry point) and oidc.rs (requires external provider).
Code Style
Rust
- Formatting:
cargo fmt(enforced in CI, auto-fixed by pre-commit hook) - Linting:
cargo clippy -- -D warnings(all warnings are errors in CI, auto-fixed by pre-commit hook) - Edition: 2024, stable toolchain
- Error handling:
thiserrorfor domain errors,anyhowin application code,?propagation - Async:
async-traitfor trait methods,tokioruntime
Svelte / TypeScript
- Framework: Svelte 5 with runes (
$state,$derived,$props,$effect) - Type checking:
npx tsc --noEmit(enforced in CI) - Component files: PascalCase
.sveltefiles - Test files: Co-located as
ComponentName.test.ts - Path alias:
~maps tosrc/— useimport { foo } from "~/api"not relative paths - Styling: Tailwind CSS utility classes, no separate CSS files per component
General
- Minimal, focused changes — don’t refactor surrounding code
- No unnecessary comments — code should be self-explanatory
- Co-locate tests with the code they test
- Prefer simple solutions over abstractions
API Reference
The full API is documented via OpenAPI 3.0 and published automatically on every push to main.
Interactive Documentation
This is a live Swagger UI loaded from the auto-generated OpenAPI spec. Every endpoint, request body, and response schema is documented and testable.
Local Swagger UI
When running the server locally, the same documentation is available at:
http://localhost:8080/swagger-ui
The local version lets you make real API calls against your running instance.
How It Works
All handlers use #[utoipa::path(...)] annotations that generate the OpenAPI spec at compile time. The CI docs workflow:
- Builds the server
- Starts it briefly with an in-memory database
- Fetches
/api-docs/openapi.json - Publishes it alongside a static Swagger UI page
This means the published API docs always match the code on main — no manual sync needed.
Endpoint Organization
All endpoints are under /api/v1/:
| Prefix | Module | Description |
|---|---|---|
/locations | handlers/locations.rs | Location CRUD + tree |
/containers | handlers/containers.rs | Container CRUD + move |
/items | handlers/items.rs | Item CRUD + batch + move |
/photos | handlers/photos.rs | Upload, serve, thumbnails, rotate |
/search | handlers/search.rs | Full-text search |
/nfc-tags | handlers/nfc.rs | NFC tag management + resolution |
/identify | handlers/identify.rs | AI item identification |
/auth | handlers/auth.rs | OIDC + local login + sessions |
/admin | handlers/admin.rs | Users, groups, settings, backup/restore |
Handler Conventions
- Use Axum extractors:
State,Path,Query,Json,Multipart - All handlers return
Result<impl IntoResponse, AppError> AppErrorwrapsDomainErrorand maps to HTTP status codes- OpenAPI annotations via
#[utoipa::path(...)] - Routes registered with
routes!()macro fromutoipa-axum
Image Storage & Thumbnails
Storage Backend
Images are stored on the filesystem using content-addressable storage. The ImageStorage trait (in storeit-domain) defines the interface:
#![allow(unused)]
fn main() {
#[async_trait]
pub trait ImageStorage: Send + Sync {
async fn store(&self, name: &str, data: &[u8]) -> Result<String>;
async fn retrieve(&self, key: &str) -> Result<(Vec<u8>, String)>;
async fn retrieve_thumbnail(&self, key: &str) -> Result<(Vec<u8>, String)>;
async fn delete(&self, key: &str) -> Result<()>;
}
}
Content-Addressable Keys
Files are stored by SHA-256 hash:
- Path:
{hash[..2]}/{hash}.{extension}(e.g.,a1/a1b2c3...d4.jpg) - First two hex chars as subdirectory to avoid filesystem limits
Thumbnail Generation
Thumbnails are generated at upload time, alongside the original:
- Decode the image using the
imagecrate (supports JPEG, PNG, GIF, WebP) - Resize to fit within 200x200 pixels (preserving aspect ratio)
- Encode as lossy WebP using the
webpcrate (quality 80) - Store as
{hash}_thumb.webp
Why webp crate, not image crate?
The image crate’s built-in WebP encoder (image-webp) only supports lossless VP8L encoding. This produces degenerate output (tiny files that render as gray boxes in browsers). The webp crate wraps Google’s libwebp via libwebp-sys and supports proper lossy VP8 encoding.
Fallback
If thumbnail generation fails (unsupported format, corrupted file), the full original image is served when the thumbnail is requested.
Cache Headers
- Thumbnails:
Cache-Control: public, max-age=31536000, immutable - Original files:
Cache-Control: public, max-age=31536000, immutable
These are safe because storage keys are content-addressed — if the content changes, the key changes.
Authentication
StoreIT supports three authentication providers, configured via environment variables.
Provider Selection
| Configuration | Provider |
|---|---|
STOREIT_AUTH_ISSUER set | OIDC provider |
| No issuer, local users exist | Local login |
| No issuer, no local users | Mock provider (dev only) |
OIDC Flow
- User visits
/api/v1/auth/login - Server generates PKCE challenge and redirects to OIDC provider
- User authenticates with provider
- Provider redirects to
/api/v1/auth/callbackwith authorization code - Server exchanges code for tokens, validates ID token
- Server creates session and sets cookie
- User is redirected to the app
Group-Based Access
OIDC tokens include group claims. StoreIT filters groups by STOREIT_AUTH_GROUP_PREFIX (default: storeit:). Users only see inventory for groups they belong to.
Session Management
Sessions are stored in SQLite with configurable TTL (STOREIT_SESSION_TTL_HOURS, default: 24). Session cookies are signed with STOREIT_SESSION_SECRET.
Mock Provider
When no auth is configured, the mock provider accepts any credentials. This is used for development and E2E tests. Never use in production.
AI Integration
StoreIT uses Claude (Anthropic’s AI) for item identification from photos.
How It Works
The POST /api/v1/identify endpoint accepts a multipart photo upload. The server sends the image to Claude with a prompt asking it to identify the item and return structured metadata.
Response Format
Claude returns JSON with:
name— suggested item namedescription— what the item istags— array of searchable keywords (color, material, category, etc.)
Providers
Anthropic API (Primary)
Direct HTTP calls to https://api.anthropic.com/v1/messages with the image as base64 input. Configured via STOREIT_ANTHROPIC_API_KEY.
Claude CLI (Fallback)
If no API key is set but the claude CLI binary is available, StoreIT shells out to it. The path is configurable via STOREIT_CLAUDE_PATH.
Model Selection
The default model is claude-haiku-4-5-20251001 (fast and inexpensive). Override with STOREIT_AI_MODEL for different quality/cost tradeoffs.
Graceful Degradation
If neither an API key nor CLI is available, the identify endpoint returns an error and the frontend falls back to manual item entry. All other features work normally without AI.
Reactivity & State Management
Svelte 5 Runes
The frontend uses Svelte 5’s rune-based reactivity system:
$state— Reactive mutable state$derived— Computed values that update automatically$effect— Side effects that run when dependencies change$props— Component props
Auth Store
The central auth store (lib/auth.svelte.ts) uses a class with rune-based fields:
class AuthStore {
data = $state<MeResponse | null>(null);
loading = $state(true);
authMode = $state<"oidc" | "local">("oidc");
user = $derived(this.data?.user);
groups = $derived(this.data?.groups ?? []);
activeGroupId = $derived(this.data?.active_group_id);
}
Data Fetching Pattern
Pages fetch data in $effect blocks or use {#await} blocks:
<script lang="ts">
let items = $state<Item[]>([]);
async function loadItems() {
items = await fetchItemsByContainer(containerId);
}
$effect(() => { loadItems(); });
// After mutations, re-fetch to update the UI
async function handleDelete(id: string) {
await deleteItem(id);
await loadItems();
}
</script>
Always re-fetch after mutations to ensure the UI reflects the current server state.
Service Worker Caching
The PWA service worker uses different caching strategies for different data types:
| Data | Strategy | Reason |
|---|---|---|
| Browse/entity API | NetworkFirst | Mutable — always show latest |
| Auth API | NetworkFirst | Session state changes |
| Search API | NetworkFirst | Results depend on current data |
| Photo files | CacheFirst | Immutable (content-addressed) |
| Static assets | Precache | Bundled at build time |
Never use StaleWhileRevalidate for mutable data — it serves stale responses first, which causes confusing UI where changes seem to disappear briefly.
Image Loading
Use loading="eager" (not lazy) for thumbnails in scrollable containers. Lazy loading breaks onLoad events in dynamically-rendered lists, preventing the loading spinner from being replaced.
E2E Test Isolation
Playwright tests run with serviceWorkers: "block" to ensure all requests hit the server directly. This prevents cached responses from masking reactivity bugs.
PWA & Offline Support
StoreIT is a Progressive Web App that can be installed on any device and works offline for browsing cached data.
Service Worker
Configured via vite-plugin-pwa in vite.config.ts.
Caching Strategies
| Pattern | Strategy | TTL | Why |
|---|---|---|---|
/api/v1/locations/**, /api/v1/containers/**, etc. | NetworkFirst | — | Mutable data, show latest |
/api/v1/auth/** | NetworkFirst | 5 min | Session state |
/api/v1/search/** | NetworkFirst | 5 min | Results change with data |
/api/v1/photos/*/file, /api/v1/photos/*/thumbnail | CacheFirst | 30 days | Immutable (content-addressed) |
| Static assets (JS, CSS, HTML) | Precache | — | Bundled at build |
Why Not StaleWhileRevalidate
StaleWhileRevalidate serves a cached response immediately, then updates the cache in the background. For mutable inventory data, this means the UI briefly shows stale data after mutations — items appear to “un-delete” or changes revert momentarily. NetworkFirst avoids this by always trying the server first.
PWA Manifest
{
"name": "StoreIT",
"short_name": "StoreIT",
"display": "standalone",
"theme_color": "#1e293b",
"background_color": "#0f172a"
}
Icons: 192x192 and 512x512 PNG in frontend/public/.
Offline Queue
Mutations (create, update, delete) made while offline are queued in lib/offlineQueue.ts and replayed when connectivity is restored. The queue is stored in memory (not persisted across restarts).
Image Loading
Thumbnails must use loading="eager". The loading="lazy" attribute breaks onLoad events in dynamically-rendered scrollable lists, causing the loading spinner to never be replaced by the actual image.
E2E Testing
Playwright tests block service workers (serviceWorkers: "block" in config) to ensure all requests go directly to the server. This prevents cached responses from masking bugs in data flow.
Components
Key Patterns
All components use Svelte 5 syntax with $props() for inputs and $state/$derived for local state.
PhotoThumbnail
Displays a single photo thumbnail with a loading spinner.
- Uses
photoThumbnailUrl()(not the full-size file URL) - Must use
loading="eager"—loading="lazy"breaksonLoadevents in scrollable containers - Shows a placeholder spinner until the image loads
PhotoGallery
Grid of thumbnails with upload support.
- File input uses
accept="image/*"but nocaptureattribute — addingcaptureblocks access to the photo gallery on mobile, forcing camera-only - After upload, re-fetches the photo list to update the grid
- Supports setting a primary photo
CreateDialog
Modal dialog for creating locations, containers, and items.
- Dialog panels need
max-h-[100dvh] overflow-y-auto+pb-[env(safe-area-inset-bottom)]for mobile - Calls
onCreatedcallback after successful creation for the parent to re-fetch - z-index: dialogs use
z-[60](above bottom nav atz-50)
MoveFlow
Multi-step workflow for moving containers or items to a new parent.
- Step 1: Select destination via ParentPicker tree browser
- Step 2: Confirm the move
- Auto-closes and re-fetches parent data on success
ParentPicker
Tree browser for selecting a location or container as a parent.
- Lazy-loads children on expand
- Highlights the current parent
- Returns a
ParentRef(location or container ID)
PhotoLightbox
Full-screen photo viewer.
- z-index:
z-[70](above dialogs) - Swipe navigation between photos
- Shows rotation controls
PrintLabel
Generates a printable label with QR code.
- Opens in a new window for printing
- Includes entity name, location path, and QR code
- QR code links to
{origin}/nfc/tag?uid={entityType}-{entityId}
QrCode
Renders a QR code as inline SVG using qrcode-generator.
z-index Layering
| Layer | z-index | Component |
|---|---|---|
| Bottom nav | z-50 | BottomNav |
| Dialogs | z-[60] | CreateDialog, MoveFlow |
| Lightbox | z-[70] | PhotoLightbox |
Migration System
StoreIT uses a snapshot-based migration system that allows jumping between any two versions in a single step.
Overview
Instead of incremental SQL migrations between versions, the system:
- Exports all data to a portable
.storeitarchive (zstd-compressed tar) - Imports into a fresh database created by the target version’s binary
- Applies version transforms to the in-memory data if schema versions differ
This means any version can migrate to any other version — no chain of intermediate migrations needed.
Key Components
Schema Versioning (_meta table)
The _meta table stores a schema_version key. Each binary declares SqliteDb::EXPECTED_SCHEMA_VERSION. On startup, the server checks these match and refuses to start on mismatch.
CREATE TABLE _meta (key TEXT PRIMARY KEY, value TEXT NOT NULL);
INSERT INTO _meta VALUES ('schema_version', '1');
Interchange Module (interchange.rs)
Core export/import logic, decoupled from HTTP handlers via the ProgressReporter trait:
export_to_file()— collects all data, writes manifest v2, creates zstd tar archiveimport_from_bytes()— unpacks archive (zstd or gzip), parses data, applies transforms, insertstransform_data()— version transform chain (no-op for v1→v1, add match arms for future versions)
Archive Format (.storeit)
Zstd-compressed tar with this structure:
backup/
manifest.json # format_version, schema_version, app_version, created_at, includes_images
data/
users.json # includes _password_hash field
groups.json
memberships.json
settings.json
locations.json
containers.json
items.json
photos.json # metadata only
nfc_tags.json
images/ # optional
{sha256[..2]}/{sha256}.{ext}
CLI Import Command
storeit-server import archive.storeit --mode replace
- Renames existing DB to
.pre-import - Creates fresh DB, runs sqlx migrations
- Reads archive, applies version transforms
- Inserts all data, copies images
- Rebuilds FTS5 search index
- On failure: restores
.pre-importback
storeit-ctl Orchestrator
Python script that orchestrates the full upgrade flow:
- Backs up via HTTP API
- Downloads target binary from GitHub Releases
- Stops server, swaps binary
- Runs
storeit-server import - Starts new server, health checks
- Auto-rollback on failure
Adding a New Schema Version
When you need to change the database schema:
- Add a new sqlx migration in
crates/storeit-db-sqlite/migrations/ - Increment
SqliteDb::EXPECTED_SCHEMA_VERSION(e.g., 1 → 2) - Update the migration SQL to set
schema_versionin_meta - Add a transform arm in
interchange::transform_data():
#![allow(unused)]
fn main() {
fn transform_data(data: &mut ArchiveData, from: i64, to: i64) -> Result<()> {
if from == to { return Ok(()); }
if from == 1 && to == 2 {
// e.g., add a new field with default value
for item in &mut data.items {
item.new_field = Some(default_value);
}
return Ok(());
}
Err(format!("unsupported: {from} -> {to}").into())
}
}
- Run
make prepareto update the sqlx offline cache
Design Decisions
- No backward-compat gzip export: New exports always use zstd. The HTTP restore endpoint accepts both formats (tries zstd first, falls back to gzip) for importing old archives.
- FTS5 full reindex: After import,
full_reindex()clears and rebuilds the entire search index using the samebuild_*_search_textfunctions used at entity creation time. - Password hashes: Stored in a
_password_hashJSON field (not a DB column) so they survive export/import roundtrips.
Build Pipeline
CI is defined in .github/workflows/.
CI Workflow (ci.yml)
Runs on every push to main or feature/** branches and on pull requests.
Jobs (run in parallel):
| Job | What it does |
|---|---|
| Lint | cargo fmt --check + cargo clippy -D warnings |
| Backend Tests | cargo test --workspace with SQLX_OFFLINE=true |
| Frontend Tests | npx tsc --noEmit + npm test |
| Build Check | Full frontend + backend release build |
Release Workflow (release.yml)
Triggered by pushing a tag matching v* (e.g., v0.1.0).
Build Matrix
| Target | Runner | Asset Name |
|---|---|---|
x86_64-unknown-linux-gnu | ubuntu-latest | storeit-server-linux-x86_64 |
x86_64-unknown-linux-musl | ubuntu-latest | storeit-server-linux-x86_64-musl |
aarch64-unknown-linux-gnu | ubuntu-latest + cross | storeit-server-linux-aarch64 |
aarch64-unknown-linux-musl | ubuntu-latest + cross | storeit-server-linux-aarch64-musl |
x86_64-apple-darwin | macos-14 (cross from ARM) | storeit-server-darwin-x86_64 |
aarch64-apple-darwin | macos-14 | storeit-server-darwin-aarch64 |
x86_64-pc-windows-msvc | windows-latest | storeit-server-windows-x86_64.exe |
ARM Linux targets use the cross tool for Docker-based cross-compilation (compatible with free-tier GitHub runners).
Release Assets
Each release includes:
- Pre-built binaries for all 7 targets
storeit-ctl(Python management tool)SHA256SUMS(checksums for all files)
Local Equivalents
make lint # cargo fmt + clippy
make test # cargo test --workspace
make build-all # frontend build + cargo build --release
make coverage # llvm-cov with 93% minimum
Release Process
Versioning
StoreIT uses semantic versioning (major.minor.patch). The version is defined once in the workspace root:
# Cargo.toml
[workspace.package]
version = "0.1.0"
All crates inherit this version via version.workspace = true.
Schema version is independent — tracked in the _meta table as an integer (currently 1). It only increments when the database schema changes. Defined as SqliteDb::EXPECTED_SCHEMA_VERSION in crates/storeit-db-sqlite/src/lib.rs.
Creating a Release
- Update version in root
Cargo.toml - Commit:
git commit -m "Bump version to 0.2.0" - Tag:
git tag v0.2.0 - Push:
git push origin main v0.2.0 - The release workflow builds all binaries and creates a GitHub Release with auto-generated release notes
Release Artifacts
Each release includes:
| Asset | Description |
|---|---|
storeit-server-linux-x86_64 | Linux x86_64 (glibc) |
storeit-server-linux-x86_64-musl | Linux x86_64 (static, musl) |
storeit-server-linux-aarch64 | Linux ARM64 (glibc) |
storeit-server-linux-aarch64-musl | Linux ARM64 (static, musl) |
storeit-server-darwin-aarch64 | macOS Apple Silicon |
storeit-server-darwin-x86_64 | macOS Intel |
storeit-server-windows-x86_64.exe | Windows |
storeit-ctl | Python management tool |
SHA256SUMS | Checksums for all files |
How Users Upgrade
Users run storeit-ctl upgrade which:
- Backs up data via the running server’s API
- Downloads the new binary from GitHub Releases
- Stops the server, imports data with the new binary, restarts
- Rolls back automatically on failure
See the migration system for the technical details.
Schema Version Changes
When adding a new migration that changes the DB schema:
- Increment
SqliteDb::EXPECTED_SCHEMA_VERSION - Add the sqlx migration file in
crates/storeit-db-sqlite/migrations/ - Add a transform arm in
interchange::transform_data()for the version transition - Run
make prepareto update the sqlx offline cache