Crate — library scan and search v1
docs/plans/crate-library-scan-v1.md
Context & goal
Intended outcome
A producer points Crate at a sample folder and gets a searchable library: every audio file inventoried once, annotated with its detected BPM and musical key, and queryable offline in under 30ms. The index persists across restarts and survives the folder being moved.
Why
Search today re-walks the disk on every keystroke. On the 60k-file library we test against, src/ui/search.svelte blocks the main thread for 4–9
seconds per query — which is why the search box was quietly hidden behind a feature flag.
Non-goals
This plan does not touch the audition player (src/player/), does not add cloud sync or any network calls, and does not attempt genre classification.
Constraint
Crate is offline-first and ships as a single binary. No server, no daemon, no background indexer. Everything here runs in-process.
Current state
Grounded in the files as they stand on main:
- ▪
src/scan/walk.ts— an async generator yielding audio paths under a root. Correct and fast (~2.1s over 60k files). Its only caller is the UI. - ▪
src/library/store.ts— a Map rebuilt from scratch on every mount. Never written to disk. - ▪
src/db/— does not exist. There is no persistence layer yet.
Assumption
walk.ts is sound and stays. This plan treats it as
the inventory source and builds around it rather than rewriting it.
Work breakdown by seam
Ordered by dependency. Each phase lands on its own and leaves the app working.
Phase 1 — seam: the filesystem boundary
Persist the inventory
- What changes
- SQLite table keyed by path with size and mtime. Rescan compares (size, mtime) and skips unchanged files.
- Files
- src/db/index.ts, src/db/migrations/001_samples.sql, src/library/store.ts
- Verification
- cargo test scan::incremental — cold scan under 5s, warm rescan under 300ms.
- Review gate
- Move the fixture root and rescan. Rows must be re-pathed, not duplicated.
Phase 2 — seam: the decoder boundary
Header-only probe for BPM and key
- What changes
- probeHeader(path) alongside decodeFull. BPM from the file’s own tag, falling back to onset estimation. Both fields nullable.
- Files
- src/audio/probe.ts, src/audio/decode.ts, src/scan/annotate.ts
- Verification
- cargo test probe::known_bpm — tag-derived BPM exact; estimated within ±2.
- Review gate
- A truncated file must yield None, not a panic. Annotation must resume after a kill.
Phase 3 — seam: the search API surface
The query API
- What changes
- FTS5 over filename and tags, indexed range predicates on bpm and key. One function is the only read path.
- Files
- src/library/query.ts, src/db/migrations/003_fts.sql
- Verification
- cargo bench query::p95 — under 30ms for text, under 50ms with a BPM range.
- Review gate
- rg "from samples" src/ must return query.ts and nothing else.
Phase 4 — seam: the UI data source
Wire the UI to the index
- What changes
- search.svelte calls query() instead of walk(). Remove the flag. Render BPM/key columns, blank when null.
- Files
- src/ui/search.svelte, src/ui/flags.ts, src/ui/SampleRow.svelte
- Verification
- Playwright: first paint under 100ms, no main-thread block over 50ms.
- Review gate
- Render a scanned-but-unannotated library: blank cells, no errors, search working.
src/library/query.ts
export interface Query {
text?: string; // FTS match over filename + tags
bpm?: { min: number; max: number };
key?: string; // exact, e.g. "Fmin"
limit?: number; // default 200
}
export function query(q: Query): SampleMeta[];| Option | Cold scan (60k) | Complexity | Verdict |
|---|---|---|---|
| Synchronous annotate (Phases 1–4) | ~11 min est. | Low — one pass | Ship this first |
| Worker pool, index-first (Phase 5) | ~90s est. | Medium — resumability | Only if measured pain |
| Skip BPM estimation, tags only | ~40s est. | Lowest | Fallback if estimation is a tarpit |
Adversarial review
The assumption that sinks this if wrong
That BPM lives in the file’s tags for most of a real library. The fixture set is 24 files we curated and 16 are tagged — that is not evidence about a producer’s actual drive. If real libraries are mostly untagged, cold scan goes from ~11 minutes to hours and Phase 5 stops being optional. Get an actual percentage before starting Phase 2.
Riskiest seam
Phase 2. Phases 1, 3 and 4 are plumbing over a schema; Phase 2 is signal processing with no ground truth beyond fixtures we wrote ourselves. “Within ±2 BPM” is a number chosen to be passable, not one derived from what a producer would call correct.
Counter-argument to the whole plan. The problem might be entirely in search.svelte re-walking on every keystroke,
not in the absence of an index. A 20-line debounce plus an in-memory cache could take 4–9s down
to something acceptable and cost nothing. It is worth being explicit that the index is justified
by the annotation requirement, not by the latency number.
Open questions
- 1 Where does $DATA_DIR point on each platform, and is it backed up? Needs a decision before Phase 1 writes the file.
- 2 What is the actual tag-presence rate in a real library? Blocking for Phase 2 — a measurement nobody has taken.
- 3 Is key a free-text tag or an enum? Libraries write Fmin, F minor, Fm and 5A. Exact match is meaningless without normalisation.