[p2p-hackers] [i2p] thoughts toward i2p persistent storage (fwd from david@rebirthing.co.nz)

[Eugen Leitl]

----- Forwarded message from David McNab <david at rebirthing.co.nz> -----

From: David McNab <david at rebirthing.co.nz>
Date: Thu, 10 Mar 2005 11:58:57 +1300
To: i2p at i2p.net
Subject: [i2p] thoughts toward i2p persistent storage
User-Agent: Mozilla Thunderbird 1.0 (X11/20041206)

Hi,

Since my return to activity in I2P, I've been pondering the as yet 
unresolved challenge of reliable persistent storage in i2p.

Recent irc discussions have come up with the opinion that a fully 
peer-to-peer architecture might be problematic in i2p, given the volume 
of query traffic as the p2p network grows. Also, that a system of 
syndicated servers, keeping each other up to date, accessed by clients, 
might be the way to go.

So I'm launching this thread with a hope of triggering some detailed 
discussion on how we might architect such a beast.

One option is to go straight nntp, and to build fancy nntp clients which 
can automatically insert/retrieve stuff under program control and 
present it as needed.

But I'm wondering if we need the complexity and baggage of a full 
network of syndicated nntp servers, or whether a cut-down implementation 
of similar ideas would work better.

One of the biggest trade-offs for such an architecture - syndicated 
servers accessed by thin clients - is between performance, convenience 
and scalability.

One extreme is that every server in the syndicate mirrors everything. 
This is great for performance, because each client with each hit 
connects to a random server, and is guaranteed a successful 
insert/retrieve, as long as the server is up. If the server is down, the 
client moves on to another random server till it gets lucky.
This extreme also has the advantage of convenience. Server admins can 
just boot their servers into the network, even run them transient, and 
they will automatically get and stay up to date. However, this situation 
suffers the obvious issue of scalability.

The other option is a syndicate of servers, each of which maintain 
partial mirrors of overall content. However, that takes us at least part 
of the way to the full kademlia query flooding scenario. Here, we have a 
scenario where client 'Alice' knows about servers 'Steve', 'Sally' and 
'Stan' and 'Samantha'. Alice wants to get 'SSK at yadayada/foo/bar'. Assume 
that only Sally has that item. Scenarios include:
(i)   alice tries steve - steve says 'try sally' - alice tries sally,
      sally coughs up the content
(ii)  alice tries steve, steve puts alice on hold, steve then gets the
      content from sally, steve says to alice 'here ya go'

However, a syndicate where each server only partially mirrors opens up a 
can of worms. The big question is - what criteria are used for choice of 
what to mirror or not? Choice will have to be based on automatic or 
manual criteria, or a combination of both.

Fully automated criteria will have to classify content based on some 
metric, such as kademlia. This makes life easy for server admins, but 
requires a good metric to give a good tradeoff between scalability and 
retrievability.

Manual criteria will give server admins more control, but the admins 
will get bugged frequently, needing to choose whether they're willing to 
host SSK at this and SSK at that.

Anyway, I've raved on long enough. What are /your/ thoughts?

-- 
Cheers
David
_______________________________________________
i2p mailing list
i2p at i2p.net
http://i2p.dnsalias.net/mailman/listinfo/i2p

----- End forwarded message -----
-- 
Eugen* Leitl <a href="http://leitl.org">leitl</a>
______________________________________________________________
ICBM: 48.07078, 11.61144            http://www.leitl.org
8B29F6BE: 099D 78BA 2FD3 B014 B08A  7779 75B0 2443 8B29 F6BE
http://moleculardevices.org         http://nanomachines.net
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
Url : http://zgp.org/pipermail/p2p-hackers/attachments/20050310/2cf798b1/attachment.pgp