[Architecture Question] Optimizing mmap Page Caching and LSM-Tree Compaction for a 3 TB Indexing Run

I am currently modeling the system architecture for a localized, offline text-retrieval engine (Project Dragonfly) to study resource optimization on consumer-grade hardware.

The primary goal of the design is to allow a machine limited to 16GB 32GB of RAM to parse, index, and query roughly 3 Terabytes of raw, unstructured text files completely locally.

The Current Architectural Model:

To balance the 3 TB dataset against the physical memory limit, the design uses a hybrid system architecture:

1. Memory Mapped Files (mmap): Utilizing virtual memory mapping to let the OS kernel manage page caching dynamically, avoiding the need to load large data arrays directly into volatile memory.

2. Log Structured Merge trees (LSM trees): Appending incoming text data sequentially in structured blocks to avoid random, high volume write cycles to disk.

Technical Hurdles I am Trying to Solve:

Question 1 (Memory Isolation): When handling heavy text parsing in systems languages like Rust or Go, what are the standard patterns for implementing explicit heap zeroization? I need to ensure that transient query strings and sensitive index blocks are completely wiped from volatile memory the exact microsecond a search thread closes, preventing data from leaking into swap space.

Question 2 (SSD Longevity & Compaction): On standard consumer hardware, heavy indexing runs face massive write amplification issues during LSM tree compaction. What compaction strategies (Leveled vs. Size Tiered) are best suited to minimize the physical drive write footprint over a 3 TB data set to prevent premature hardware degradation?

I am compiling a design document mapping out these specific backend bottlenecks. I would appreciate any insights, technical documentation, or hardware-level advice from engineers who have built similar data pipelines.

Interesting topic. Not skilled enough to answer unfortunately, will follow-up on comments.

Q1: Explicit Heap Zeroization

Rust: Use zeroize crate with the zeroize(drop) derive macro. It guarantees compiler won't optimize away the wipe. For heap allocations, wrap sensitive buffers in Box<[u8]> and call .zeroize() before drop. If you need to go lower, use std::alloc::dealloc with a manual secure_zero_memory using core::ptr::write_volatile or inline assembly rep stosb to defeat optimizer.

Go: Trickier—GC moves things. Best bet is allocate sensitive data via syscall.Mmap with anonymous mappings, manage lifecycle manually, and use explicit_bzero via CGO or memset_s pattern. Alternatively, pin buffers in C and keep Go references as opaque handles. For query strings specifically, consider using memguard or similar enclave approach since Go's runtime makes no guarantees about not copying your data during GC/compaction.

Swap mitigation: On Linux, mlock your sensitive pages (or mlockall(MCL_FUTURE) for the process) to keep them out of swap entirely. Combine with madvise(MADV_DONTDUMP) to exclude from core dumps.

Q2: Compaction Strategy for SSD Longevity

For 3TB on consumer SSDs, Size-Tiered is usually the pragmatic choice despite Leveled being theoretically better for read amplification. Here's why:

Write amplification tradeoff: Leveled compaction triggers massive rewrites of the same data as levels fill—on 3TB, that's brutal on NAND endurance. Size-Tiered merges similarly-sized SSTables, which is less write-heavy at scale.

Tuning knobs: Set your SSTable size large (256MB–1GB) to reduce total file count and merge frequency. Use a high min_threshold (4–6 files) before triggering compaction to batch work.

Tiered + TTL: If your index has temporal locality, consider Tiered compaction (hybrid) or just aggressive TTL-based dropping to skip compaction of stale data entirely.

Hardware mitigation: Ensure your SSD has PLP (power-loss protection) or at least disable disk cache flushes if you're willing to risk corruption on power loss—this reduces the fsync overhead that LSMs hammer.

One more thought on mmap: For 3TB cold access, you're going to hit the kernel's page cache eviction hard. Consider madvise(MADV_SEQUENTIAL) during indexing and MADV_RANDOM during query serving. Also, vmtouch or similar to warm hot index segments into RAM before query time.

Incredible response, thank you. This is exactly the kind of deep systems breakdown I was hoping to trigger.

The warning about Go's GC copying strings during compaction is a massive point. Honestly, that alone makes a strong case for keeping the core parsing pipeline strictly in Rust. Wrapping the sensitive buffers in Box<[u8]> with the zeroize(drop) macro seems way cleaner than trying to fight Go’s runtime with CGO or manual anonymous memory maps.

Using mlock and madvise(MADV_DONTDUMP) is a solid shout for the swap mitigation layer. That fixes a major blindspot regarding automated core dumps if the process hits an unhandled panic during a heavy parse.

For the storage engine, your point about NAND endurance on 3TB consumer drives completely seals it. Leveled compaction would fry a standard drive way too fast at this scale. I’ll look into tuning the SSTable sizes up to the 512MB 1GB range under a Size-Tiered strategy to batch the merges and keep write amplification down.

Are you currently working with these kinds of LSM compaction trade-offs or memory zeroization pipelines in production, or do you just hack on database internals for fun? If you have any specific Rust repositories or open source database wrappers you like for handling custom allocators like this, I'd love to check them out. Cheers.