S3 to Restic Without the Middleman: Why It’s Hard and What Works Today

The idea makes sense — here’s where it actually gets hard

If you run restic alongside cloud-native storage tools like Longhorn, the workflow feels awkward: your storage system pushes data to an S3 bucket, then something else pulls it back out and feeds it into restic. Two hops, two storage locations, two things to maintain. Cutting straight from source to restic repository is the obvious move.

But there’s no packaged tool that does this cleanly today. The reason is an impedance mismatch that isn’t obvious until you start building.

S3 and restic are not the same abstraction

S3 is an object store: PUT a key with bytes attached, GET it back by key. Restic is a snapshot engine with content-addressed blob packing, an encrypted index, deduplication, and a repository lock protocol. These aren’t two dialects of the same idea.

A valid restic snapshot requires a specific sequence: acquire a repository lock, write content-addressed blobs into pack files, update the index, write the snapshot object, release the lock. An S3-facing bridge has to translate incoming S3 API calls into that sequence — which means it’s not a proxy, it’s a full restic client implementation wearing an S3 server costume.

The snapshot boundary problem

The “wait X seconds with no new data, then complete the snapshot” idea sounds workable, but it breaks down in production. Longhorn and similar tools don’t send one clean sequential stream. They use S3 multipart uploads: a single logical backup splits into many independent PUT requests across multiple connections, in no guaranteed order.

S3’s CompleteMultipartUpload call is the real signal that a logical object is done. A correct bridge would need to track every in-flight multipart upload ID, buffer all parts, reassemble them in part-number order, then feed the assembled byte stream into restic. That’s a non-trivial state machine — and it needs to survive crashes without producing silent partial snapshots.

Timing heuristics are fragile. The multipart completion event is the right trigger.

What exists today and how close it gets

restic –stdin-from-command

If you control the backup source and it can export data as a stream, --stdin-from-command is the cleanest approach and it’s underused. It’s strictly better than the old --stdin pipe because restic watches the subprocess exit code and cancels the snapshot if the command fails.

restic backup --stdin-from-command -- mysqldump --host db mydb

With bare --stdin, a failing dump still produces a partial snapshot. Restic can’t tell the difference between a complete stream and a truncated one. --stdin-from-command eliminates that class of silent failure.

The limitation: Longhorn isn’t a subprocess you can wrap — it pushes over the network on its own schedule. So stdin-based approaches only work if you can interpose something that serializes the push into a stream you control.

rclone serve restic

Rclone’s serve restic command exposes the restic REST protocol over any rclone-supported backend. When you configure an rclone backend in restic, this is what runs internally — restic spawns rclone serve restic --stdio and communicates over HTTP/2 on stdin/stdout, no network socket needed.

This is useful for backing up to storage that restic doesn’t natively support. It runs in the wrong direction for Longhorn’s S3 push.

rclone serve s3 + filesystem watcher

The closest working approximation: run rclone serve s3 pointed at local disk, configure Longhorn’s S3 target to hit that local endpoint, and trigger restic when data arrives.

rclone serve s3 --addr :9000 /mnt/staging

Then watch for completed writes:

inotifywait -m /mnt/staging -e close_write |
while read path _ file; do
  restic backup "$path$file"
done

This still has a local staging layer, but the remote S3 bucket is gone and the staging window is short — restic runs immediately after data lands. The remote round-trip disappears entirely.

One caveat: with real multipart uploads, per-part writes appear as temporary files and the assembled object only lands on disk after CompleteMultipartUpload. rclone’s serve s3 handles S3 multipart correctly and writes the final object atomically, so watching close_write on the assembled file is usually the right trigger — but test with your actual payload sizes and Longhorn version before relying on it in production.

What a proper bridge would actually need

A bridge that takes S3 PUT calls and writes directly into a restic repository — no local staging at all — would need roughly:

• An S3-compatible server layer that handles path-style URLs and full multipart upload semantics
• Per-upload-ID part buffering and correct reassembly on CompleteMultipartUpload
• A restic client implementation to write assembled data into the repository’s pack format
• Repository lock coordination across concurrent uploads
• Snapshot finalization with correct metadata after each completed object

In Go, you could build this using restic’s library internals combined with a minimal S3 server library. The pieces exist. The packaged tool doesn’t.

Concurrent writes and repository locking

Restic’s locking model wasn’t designed for concurrent writers. Two simultaneous Longhorn backup jobs hitting the same repository need careful coordination to avoid index corruption. The REST server protocol supports concurrent clients, but only within the constraints restic’s locking was designed for.

If you have multiple sources pushing at the same time, this is probably harder to solve correctly than the multipart assembly — and it’s the failure mode most likely to cause silent data loss in a naive implementation.

Sources

• restic.readthedocs.io
• rclone.org
• earthly.dev