fetchr
pomagrenate/fetchr
Overview
A blazing-fast, adaptive multi-connection file downloader and background daemon engine built in Rust. Features dynamic parallel byte-range pipelining, resume recovery, zero-lock positional disk I/O, streaming checksums, and a modern CLI & Desktop UI.
Technologies & Concepts
Project Analysis
🔴 Problem
Single-stream bandwidth underutilization: Standard HTTP download tools and browser engines rely on a single TCP connection per transfer. On high-bandwidth connections (e.g. 500 Mbps to 1 Gbps), high latency, packet loss, or server-side per-connection rate limits often restrict throughput to a fraction of the available capacity (e.g., 2 to 9 MB/s), leaving the connection severely underutilized.
Thread synchronization & disk lock contention: Existing multi-connection download tools often synchronize chunk writes using a shared mutex or a single file pointer. As multiple download workers complete chunks simultaneously, lock contention and continuous flushing on every seek create an I/O bottleneck that starves download workers.
The static partitioning straggler trap: Legacy multi-part downloaders divide a file into fixed, static ranges upfront. If one connection encounters network jitter or packet loss, the entire download stalls waiting for that single "straggler" chunk while other connections sit completely idle.
High memory overhead & post-download verification delays: Buffering large chunks in RAM causes memory usage to balloon into hundreds of megabytes during high-speed multi-gigabyte transfers. Furthermore, verifying file integrity (e.g. SHA-256) usually requires reading the entire file back from disk after the download finishes, adding substantial disk I/O wear and time overhead.
Tool obsolescence on modern HTTPS endpoints: Classic multi-connection tools on Windows (such as Cygwin-based Axel) haven't received architectural updates in years. They fail completely on modern HTTPS/TLS/SNI endpoints with redirect loops or cipher mismatches.
⚪ Baseline
Starting point: Industry-standard utilities tested across identical network workloads: curl (v8.x, single-stream baseline), aria2c (v1.37.0, official multi-connection x86_64 Windows build), and axel (v2.4, Cygwin-based multi-connection downloader).
Baseline characteristics:
- curl: Single TCP stream baseline (1.00x), constrained to ~9.07 MB/s on rate-limited CDN streams and ~2.60 MB/s on live public internet routes
- aria2c: Defaults to a conservative 20 MB minimum split size (
--min-split-size=20M), limiting concurrency on medium-sized files (e.g., 100 MB), with static segment assignments prone to straggler stalls - axel: Suffers from Cygwin POSIX socket emulation overhead on Windows (delivering only ~4.7 MB/s), and fails entirely on modern HTTPS endpoints lacking legacy SSL support
- Integrity checks: Sequential post-download disk reads required for SHA-256 or MD5 verification
Expected performance: Single-stream transfers capped below 10 MB/s, multi-connection tools bottlenecked by static partitioning or socket emulation layers.
🔵 Change
Built Fetchr in 100% safe, async Rust: Architected a modular parallel download engine designed specifically for high throughput, minimal resource utilization, and adaptive network scheduling.
Key architectural changes:
- Dynamic Pipelined Work-Stealing Queue: Rather than statically dividing a file into fixed blocks upfront, Fetchr divides transfers into fine-grained sub-chunks (4x per connection) managed through an asynchronous work queue. Fast connections continuously pull new chunks, eliminating the straggler problem completely.
- Zero-Lock Positional Disk I/O: Leveraged native OS positional write capabilities (
FileExt::seek_writeon Windows andFileExt::write_all_aton Unix). Multiple asynchronous download workers stream incoming bytes directly into non-overlapping file offsets simultaneously with zero mutex locking. - Streaming On-The-Fly Checksums: Computes running SHA-256, SHA-512, or MD5 hashes incrementally as network buffers arrive. The file is completely verified the instant the final byte lands, eliminating post-download sequential disk passes.
- Adaptive AIMD Concurrency Scheduler: Implemented Additive Increase / Multiplicative Decrease logic that monitors real-time chunk throughput and error rates, dynamically adjusting active connection counts to maximize transfer speed without triggering server rate limits.
- Persistent SQLite Task Registry: Embedded SQLite storage tracks chunk completion states. Downloads interrupted by power loss or network disconnection resume seamlessly from exact byte boundaries.
- Dual Interface (CLI + Desktop UI): Built a lightweight background daemon (
fetchr-daemon), a CLI client (fetchr-cli), and a real-time web dashboard (fetchr-desktop) communicating via JSON-RPC IPC.
🟣 Measurement
Test environment: Windows Native workstation, Intel Core i7-8550U, SSD storage, evaluated across both controlled local CDN simulations and live public internet routes.
Benchmark suites:
- Controlled Rate-Limited Mock CDN: Evaluated 10 MB and 100 MB file downloads against an HTTP server configured with a 10 MB/s per-connection rate limit to simulate standard ISP and CDN stream throttling.
- Concurrency Scaling Sweep: Evaluated performance across 1, 2, 4, 8, and 16 concurrent connection workers to observe scaling linearity and thread scheduling overhead.
- Live Public Internet Benchmark: 100 MB HTTPS payload downloaded over public internet from OVH CDN (Roubaix, France) under realistic transatlantic network latency, packet jitter, and routing hops.
- Subsystem Microbenchmarks: Measured range scheduler latency (partitioning 10 GB into 1,280 chunks) and streaming checksum throughput.
Metrics collected: Effective throughput (MB/s), total transfer duration (seconds), speedup factor relative to single-stream curl baseline, scaling curve linearity, and TLS/HTTPS protocol compatibility.
🟢 Result
4.21x peak throughput on 100MB CDN transfers: Fetchr (8 connections) achieved 38.17 MB/s compared to 9.07 MB/s for curl (1 connection), 8.24 MB/s for aria2c (8 connections), and 4.73 MB/s for axel (8 connections).
4.6x faster transfer completion: On a 100 MB payload, Fetchr completed the download in 2.62 seconds, compared to 11.03s for curl, 12.13s for aria2, and 21.16s for axel.
Near-linear concurrency scaling (up to 8.48x): Fetchr scaled from 1.00x at 1 connection to 1.93x (2 conn), 3.66x (4 conn), 5.93x (8 conn), and 8.48x (16 conn). In comparison, aria2 plateaued at ~0.91x due to static splitting constraints, and axel degraded to ~0.5x due to Cygwin socket translation overhead.
4.94x speedup on live public internet (OVH CDN): On a real-world transatlantic 100 MB HTTPS download, Fetchr (8 connections) sustained 12.85 MB/s and finished in 7.78 seconds, versus 38.46 seconds (2.60 MB/s) for curl and 30.80 seconds (3.25 MB/s) for aria2. Axel failed completely due to lack of modern TLS/SNI support.
Microsecond scheduler overhead: The internal range scheduler partitioned a 10 GB file into 1,280 byte ranges in just 133.5 µs (0.13 milliseconds), proving zero CPU scheduling bottleneck.
Performance Visualizations & Empirical Charts
1. Fetchr Empirical Benchmark Overview Dashboard
Comprehensive four-panel overview showing: 100MB CDN throughput (Fetchr achieving 38.17 MB/s vs 9.07 MB/s curl baseline), download duration reduction (2.62s vs 12.13s aria2), live internet CDN performance (12.85 MB/s vs 2.60 MB/s curl), and the architectural comparison matrix across Fetchr, Aria2, and Axel.
2. 100 MB File Transfer Throughput (Controlled Rate-Limited CDN)
Throughput comparison under a 10 MB/s per-stream server cap. Fetchr's zero-lock positional disk writes enable it to saturate aggregate bandwidth, delivering 23.31 MB/s (4 conn), 38.17 MB/s (8 conn, 4.21x peak speedup), and 32.26 MB/s (16 conn).
3. Download Duration Comparison (Lower is Better)
Transfer completion duration for a 100 MB payload. Fetchr completes in 2.62s at 8 connections (4.6x faster than aria2 at 12.13s, and 8.1x faster than axel at 21.16s).
4. Real Public Internet Benchmark: 100MB HTTPS Download (OVH CDN Europe)
Real-world public internet evaluation against OVH CDN in Europe. Fetchr achieves 12.85 MB/s and completes in 7.78s (4.94x speedup vs curl's 38.46s). Axel failed completely due to lack of modern TLS/SNI support.
5. Multi-Connection Scaling Efficiency vs Concurrency (1 to 16 Threads)
Scaling curve illustrating how Fetchr's dynamic pipelined work queue achieves near-linear speedup (up to 8.48x at 16 connections), while aria2 plateaus due to static split limits and axel suffers from POSIX emulation overhead.
🟡 Lesson
Multi-connection downloading is not just about opening more sockets: When I started building Fetchr, I assumed simply spawning multiple async tasks would automatically yield linear speedups. I quickly realized that without careful disk I/O coordination, multiple workers downloading concurrently create heavy file-lock contention and cache thrashing.
Zero-lock positional I/O was the true bottleneck breaker: Moving away from standard sequential file handles to OS-level positional writes (FileExt::seek_write on Windows) was the single biggest architectural improvement. It allowed workers to write non-overlapping byte ranges simultaneously with zero mutex overhead.
Static chunk partitioning is a trap on real networks: In my early prototypes, I statically divided files into equal parts (e.g. 4 parts for 4 connections). On real internet links, one connection invariably hit a high-latency route or packet drop, and the entire download stalled waiting for that single worker. Switching to a dynamic work-stealing queue with fine-grained sub-chunks completely eliminated straggler delays.
Legacy tools carry enormous compatibility debt: Benchmarking against Axel was eye-opening. While Axel is still frequently recommended on forums, testing it against real-world HTTPS CDNs showed that it fails completely on modern TLS/SNI endpoints. Modern tooling requires modern networking stacks (like rustls or native-tls).
More connections have diminishing returns: In my scaling benchmarks, jumping from 8 to 16 connections yielded smaller throughput gains and, in some live public network runs, slightly increased latency due to TCP handshake overhead and server-side socket throttling. 4 to 8 connections proved to be the sweet spot for most web servers.
What I want to explore next: I want to investigate integrating QUIC / HTTP/3 multi-stream support, experiment with BBR-inspired congestion window probing algorithms directly inside the scheduler, and test BitTorrent piece verification protocols alongside HTTP range requests.