spill / docs · v0.7.6 · Source · Release notes

v2 and hybrid torrents

spill downloads and seeds BitTorrent v1, v2-only (BEP 52) and hybrid torrents through the same daemon commands and library APIs. A hybrid carries both v1 and v2 metadata for the same files; spill treats it as one torrent with one destination and one set of transfer budgets.

Creating torrents

Use spill create --format v2 or spill create --format hybrid to select the format; v1 is the default. See Creating torrents for ordering, padding and destination rules.

Hashes and IDs

A v1 torrent has a 40-digit SHA-1 hash and a v2-only torrent a 64-digit SHA-256 hash. A hybrid has both. Its stable ID is the SHA-1 hash, but either full hash addresses the same torrent; see IDs. Adding a torrent or magnet with a hash that is already known is a duplicate, even if the other hash or the destination differs. A magnet with so can instead extend an existing entry’s file selection when all its hash topics already identify that entry; see File selection.

Trackers, the DHT and peer handshakes use a 20-byte key: the SHA-1 hash for v1 and the first 20 bytes of the SHA-256 hash for v2. That shortened key only locates peers; spill always checks metadata against the full hash.

Where files are placed

A v2-only torrent whose file tree has exactly one top-level entry, and that entry is a file, is saved as that file with no enclosing directory. Every other v2-only torrent is saved under a directory named after the torrent’s name, keeping all directories from the file tree. This matches libtorrent, except that spill does not drop directories below the top level. When files are saved under name, a name that is not a valid path component makes the torrent invalid; libtorrent substitutes the hash instead. Hybrids are saved like their v1 layout, which spill requires to match the v2 file tree.

Files that share a v2 pieces root must have the same length.

Verification

V2 pieces are checked against SHA-256 Merkle hashes. Each v2 piece belongs to one file, so the gap after a file is zero padding that is not part of any file. Hybrids with authenticated layers check both SHA-1 and SHA-256. If only one passes, the metadata contradicts itself and the torrent fails permanently. If both fail, the data is fetched again, as for v1.

A hybrid missing required piece layers is layers pending. It checks pieces with SHA-1 and announces only in the v1 swarm, but accepts peers using either key. After every piece is verified, spill reads the payload once more, rebuilds the SHA-256 layers and checks every file root, including small files. This background read shares the recheck limit, keeps uploads running, and is canceled on stop. Completion is reported after the payload is synced and a pending-mode checkpoint is saved, without waiting for the rebuild. Contradictory roots fail permanently. Matching layers are saved with the metainfo, then a full-mode checkpoint is saved. The next start uses both swarms for public torrents; private hybrids still announce only v1. The current execution keeps SHA-1 verification and rejects hash requests. An interrupted rebuild starts again on the next run. If selected files complete while other pieces remain unverified, layers stay pending.

spill refuses hybrids whose v1 and v2 file lists disagree on names, order, sizes or padding.

Magnets

Magnets accept incoming TCP and uTP peers during metadata resolution, as well as dialing discovered peers. Listening addresses learned from metadata peers can be reused for payload transfer after metadata validates.

Magnets accept a btih topic, a SHA-256 btmh topic (urn:btmh:1220 followed by 64 hex digits), or both. Other btmh hash types are refused as unsupported.

Resolving a magnet first fetches the info dictionary and checks it against every hash in the magnet. A hybrid can start payload transfer immediately, using its SHA-1 hashes while required layers are pending. V2-only torrents first fetch piece hashes from peers and authenticate them against the file roots. info_validated, hash_bytes, hash_total_bytes and layers_pending in status JSON distinguish these states. Peers that send rejected, invalid or no hashes cannot stop another peer from supplying a v2-only torrent’s layers.

A hybrid can be resolved from either hash. Its stable ID stays the hash the magnet supplied, and the other hash is learned when the info dictionary validates. Both hashes remain usable for aliases and duplicate detection, even while the torrent uses only the v1 swarm. If the learned hash already belongs to another torrent, resolution fails instead of merging the two downloads. A btih or dual-topic hybrid magnet can resolve through v1-only peers. A btmh-only magnet still needs to locate its info dictionary using the supplied v2 key.

Stopping during resolution discards partial metadata and creates no files. V2-only magnets save fetched piece hashes with the metainfo, so they restart and seed without fetching them again. Hybrids save rebuilt layers in the background after full payload verification. Private torrents must be added from a .torrent file.

Swarms and discovery

A v2-only torrent uses its v2 key everywhere. A public hybrid with authenticated layers joins both swarms; a layers-pending hybrid has no upgrade offer or v2 announces. Peers using the v2 key are accepted, and their hash requests are rejected until a restart after the rebuild. Each active swarm has its own tracker announces, intervals, completion delivery and DHT lookups, while payload progress and transfer limits are shared. Both swarms’ trackers receive the same torrent-wide upload and download counters, so adding up their tracker statistics double-counts traffic. Private hybrids announce only their v1 identity and persist completion delivery for that identity, avoiding duplicate tracker credit. They do not discover peers through the separate v2 tracker swarm; pure-v2 private torrents still announce v2.

Peers found through a tracker or the DHT are dialed with the key of the swarm that returned them. Hybrid peers from PEX are dialed with the v1 key, since PEX does not say which keys a peer supports. An upgrade is offered only when layers are authenticated. Local service discovery (BEP 14) announces a hybrid’s v1 hash. For v2-only torrents it uses the first 20 bytes of SHA-256, matching libtorrent’s v2 LSD wire key. Both are encoded as 40 hex digits. Received announcements can match either known hybrid key; the announced key is used when dialing. A btmh-only magnet announces its v2 key before metadata is known. Private torrents disable the DHT, PEX and LSD in every format.

Peer protocol

When spill has authenticated hybrid layers and dials with the v1 key it offers to switch to v2, and accepts a reply on either key. Incoming peers may use either key, or offer the switch themselves.

On a v2 connection spill follows BEP 52 strictly:

  • spill answers every block request with a block or a reject, including after a choke. Peers must do the same: a request spill cancelled still needs an answer within the request timeout, or the peer is disconnected.
  • A block that was never requested disconnects the peer.
  • A duplicate outstanding block request disconnects the peer.
  • Hash responses spill did not ask for disconnect the peer.

spill answers hash requests for verified data:

  • Piece-layer hashes need no disk reads and are answered even while the peer is choked.
  • Hashes below the piece layer require an upload slot, cover at most one piece, and count the bytes read against the upload limit.
  • Leaf hashes are checked against the piece-layer hash before they are sent. If the file changed on disk after it was verified, spill sends a reject instead.

For hybrids, peers in either mode may request the zero padding after a file, because the v1 hashes cover it. Peers of a v2-only torrent cannot request past the end of a file’s data; spill rejects those requests. Malformed requests disconnect the peer. Rejected file-tail requests and unsupported hash requests use up the peer’s ignored-traffic allowance, and repeating them disconnects the peer.

Web seeds

HTTP(S) web seeds work for all three formats. spill fetches only real file bytes and fills padding with zeros locally. A mirror serving bad data is dropped and another source completes the piece.

Compatibility

Hybrid metainfo may omit piece layers, and hybrid magnets need no v2 hash exchange before downloading. spill rebuilds pending layers from its own verified payload rather than requiring v2 peers. The extra background read costs roughly a recheck. V2 peers can use the rebuilt layers after the torrent restarts.

Transmission 4.1.3 reads hybrids as v1 but refuses metainfo with repeated paths. spill and libtorrent name padding .pad/<length>, so a hybrid with two equal-sized gaps between files does not load in Transmission. Single-file hybrids have no padding.

A v2-only .torrent file must contain the piece layers entry of every file larger than one piece. libtorrent can fetch missing layers from peers; spill requires them in the file. V2-only magnets fetch and authenticate layers before creating payload files.

spill accepts single-hash requests because libtorrent 2.1.2.0 sends them, although BEP 52 requires at least two; spill itself always asks for at least two. See Scope for what has been checked against other clients and what has not, including v2 over uTP.