[p2p-hackers] Generalizing BitTorrent..

[cefn.hoile at bt.com]

There could be a big difference between the systems you would use for
the first case (incremental releases to the same file structure), as
opposed to the latter cases (where different distributions sharing files
need to be able to seamlessly interoperate in order to benefit from
whatever optimisations are possible). 

The first case could be solved by a convention to distribute the first
release as a self contained filesystem, and all future releases as diff
patches.

The latter cases you describe are more interesting in their consequences
though. There may be a way in which you could build up the latter case
from conventions too, although encoding these conventions in a tool
would be advantageous. Perhaps a series of file hashes could be chained
together to define a release of an individual file (release 1.0, patch
2.0, patch 3.0, patch 4.0), and distributions (filesystems made up out
of multiple files) in a similar way.

I guess I am suggesting that you might be able to do this without
changing the Bittorrent protocol. Perhaps you could suggest how protocol
changes would improve efficiency over a scheme based on conventions
similar to those described above.

This could also be related to the following project... 
http://www.pdos.lcs.mit.edu/ivy/ 
...which uses publishing of changelogs to maintain a distributed
versioned filesystem.

Cefn
http://cefn.com 


-----Original Message-----
From: p2p-hackers-bounces at zgp.org [mailto:p2p-hackers-bounces at zgp.org]
On Behalf Of Bryan Turner
Sent: 14 January 2005 01:34
To: Peer-to-peer development.
Subject: [p2p-hackers] Generalizing BitTorrent..


Hello p2p-hackers,

    I've been thinking about BitTorrent recently and had some ideas on
improving the protocol.  I'm sure others have had similar thoughts, and
I'm interested to hear your opinions.

    The basic idea is best described with a real-world example.  There
are a number of "Full-MAME" torrents, one for each version of MAME (for
those who don't know, MAME is an arcade emulator and the torrents
contain the ROMs for the arcade games).  As MAME is updated frequently,
there is a string of these torrents on the net.  Each one is very large
(10 GB), and contains 95% of the EXACT SAME data from the previous
torrent.  The other 5% is a small amount of changes, and some new
content.

    If a user is running the v0.7 torrent and has become a Seed, he
serves ONLY the v0.7 peers.  When v0.8 is released, his Seed status is
essentially useless to the peers in the v0.8 crowd, even though he has >
90% of the same data.  And again, when v0.9 is released, the same
problem.  It seems like there should be an extension to the protocol to
allow for this type of 'shared data' among torrents.

    Let me take this concept a bit further.  You could think of the
v0.7, v0.8, and v0.9 torrents as being three separate torrents - or you
could think of them as ONE torrent, with overlapping pieces.  If there
were a tracker that tracked the "meta-torrent" of all three versions, it
would be valuable to ALL peers interested in ANY of the versions.

    Taken to the extreme, if you were to gather up other (non MAME)
torrents in the same manner and glue them all together, the resulting
meta-torrent (and tracker) would be valuable to any client interested in
any data tracked by the meta-torrent tracker.

    Likewise, clients could exchange with other clients for any piece of
data in the meta-torrent shared between the clients.  For example, if
Client A is interested in Fedora 3, a some MP3s, and MAME v0.7, while
Client B is interested in Fedora 2, some MP3s, and MAME v0.8, then their
shared meta-torrent is all the data shared between Fedora 2 & 3, the
mutually-interesting MP3s, and the data shared between MAME v0.7 & v0.8.

    Conceptually, nothing has changed, we're just aggregating the set of
files being exchanged, and allowing torrents to overlap where ever it is
natural.  Many BitTorrent client applications already allow a similar
selection process, they allow the user to choose some subset of data to
transfer from a complete torrent.  The proposed changes are essentially
the same idea, except that the client software automatically selects the
pieces to download/ignore from the meta-torrent based on the sub-torrent
files that the user has selected.

    I have a few ideas on how to implement this as well, but I don't
want to waste bandwidth if the group isn't interested.

Thanks for reading!
--Bryan
bryan.turner at pobox.com

_______________________________________________
p2p-hackers mailing list
p2p-hackers at zgp.org
http://zgp.org/mailman/listinfo/p2p-hackers
_______________________________________________
Here is a web page listing P2P Conferences:
http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences