[p2p-hackers] Re: scalability

[coderman]

On 12/2/05, Bryan Turner <bryan.turner at pobox.com> wrote:
> ...
> 4       Scalability in interest for an object (hot spots)
>...
>         A vision for a network that scales along all axis would be Gnutella
> with some structure to improve domain-specific searches, with BitTorrent as
> the data transfer mechanism.

finding obscure / rare / unpopular resources is the flip side of the
interest coin. in alpine all discovery was done using distinct peer
groups dedicated to a single domain of resource discovery (specific
subjects / applications had distinct groups). peer lists were ordered
within each group according to a relative quality attribute associated
with that group only.

the goal was to make decentralized search efficient for very obscure
resources when a centralized (or partially centralized) index search
was usually required for completeness to make it effective.

the problem with this approach is that it is very hard to model in a
meaningful way due to inherent dependence on relative metrics
associated with human behavior. (or perhaps it will be simple(r) if a
large real world network can be observed and studied)

alpine also used a pluggable module system (dlopen with c++ derived
handlers) to handle arbitrary metadata associated with queries
(different groups may require different search criteria and taxonomy)
and integrate various transport mechanisms (a simple TCP stream
transfer was provided as an example of this ability)

being able to offload such transfers to a system optimized for the
purpose, like bittorrent, was a design goal and definitely makes sense
in any project where cooperative content distribution is useful.