From eugen at leitl.org  Tue May  3 13:27:30 2005
From: eugen at leitl.org (Eugen Leitl)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] Azureus Decentralizes Bittorrent
Message-ID: <20050503132730.GH6782@leitl.org>


Link: http://slashdot.org/article.pl?sid=05/05/03/0458256
Posted by: timothy, on 2005-05-03 10:20:00

   from the infinite-onion-rings dept.
   [1]BobPaul writes "While the [2]eXeem project to decentralize
   Bittorrent remains in [3]open beta, the [4]Azureus Java Bittorrent
   project has recently released a major update that, among other things
   offers 'a [5]distributed, decentralised database that can be used to
   track decentralised torrents. This permits both "trackerless" torrents
   and the maintenance of swarms where the tracker has become unavailable
   or where the torrent was removed from the tracker.' It doesn't contain
   the search functionality of eXeem, but it's also not a beta product
   and is licensed under the GPL. Could this and compatible clients be
   the replacement to SuprNova and Lokitorrents, or does the lack of
   search negate its effectiveness?"

References

   1. mailto:slashdot-submission@bobpaul.org
   2. http://www.exeem.com/
   3. http://slashdot.org/article.pl?sid=05/01/21/1946212&tid=95
   4. http://azureus.sourceforge.net/
   5. http://azureus.sourceforge.net/whatsnew2300.php

----- End forwarded message -----
-- 
Eugen* Leitl <a href="http://leitl.org">leitl</a>
______________________________________________________________
ICBM: 48.07078, 11.61144            http://www.leitl.org
8B29F6BE: 099D 78BA 2FD3 B014 B08A  7779 75B0 2443 8B29 F6BE
http://moleculardevices.org         http://nanomachines.net
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://zgp.org/pipermail/p2p-hackers/attachments/20050503/c44067b5/attachment.pgp
From matthew at allpeers.com  Tue May  3 13:18:01 2005
From: matthew at allpeers.com (Matthew Gertner)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] AllPeers SDK 1.0
Message-ID: <20050503131812.22DBA3FCF6@capsicum.zgp.org>

We've just released version 1.0 of our SDK for peer-to-peer applications.
It's a complete framework including not just networking but also a data
store, resource management and user interface components. It targets C++
developers on Windows for now, and we are planning a multiplatform version
for the next release.
 
Complete documentation is online at: http://www.allpeers.com/sdk/.

Cheers,
Matt




From seth.johnson at RealMeasures.dyndns.org  Tue May  3 13:45:11 2005
From: seth.johnson at RealMeasures.dyndns.org (Seth Johnson)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] Solipsis: a peer-to-peer shared virtual world
References: <BBBE5BAA3B351C488C415EA662EA8840015018AD@FTRDMEL2.rd.francetelecom.fr>
	<42734EC2.2881BF62@RealMeasures.dyndns.org>
Message-ID: <42778067.F33DD0A0@RealMeasures.dyndns.org>



I have installed the other support packages; it still didn't
work, because the package is written for Windows XP.  But if I
run the .py files, it works, kinda.


Seth


Seth Johnson wrote:
> 
> Running on Win98 SE, I get this error when I run navigator.exe:
> 
> Traceback (most recent call last):
>   File "navigator.py", line 16, in ?
>   File "solipsis\navigator\wxclient\main.pyo", line 37, in main
>   File "solipsis\navigator\wxclient\app.pyo", line 72, in
> __init__
>   File "wx\_core.pyo", line 5301, in __init__
>   File "wx\_core.pyo", line 4980, in _BootstrapApp
>   File "solipsis\navigator\wxclient\app.pyo", line 200, in OnInit
>   File "solipsis\navigator\wxclient\app.pyo", line 119, in
> InitResources
>   File "solipsis\util\wxutils.pyo", line 102, in Resource
> NameError: resource object 'about_dialog' does not exist
> 
> Is there something missing from the Windows distro?
> 
> twistednode.exe seems to be doing something when I run it, no
> apparent problems.
> 
> Seth Johnson
> 
> KELLER Joaquin RD-MAPS-ISS wrote:
> >
> > The Solipsis Team would like to announce the release of Solipsis-0.8.
> >
> > http://solipsis.netofpeers.net/
> >
> > Solipsis is a massively multiparticipant virtual world
> > based on a peer-to-peer system (ie without servers).
> >
> > Some Solipsis prominent features are:
> >
> > * Scalable to millions, billions of users and entities
> > * Solipsis is intended to be as wide as the Web
> > * The virtual world is user contributed (so it is for now nearly empty)
> > * Solipsis is aimed for social interaction (not only for gaming)
> > * Open Source - Free Software - GNU Public License
> > * It runs on Windows, Linux  (and hopefully soon on Mac OS X)
> >
> > Coming soon:
> > * Blog-like user profile
> > * File transfer
> > * Private chat
> > * Management of multiple nodes
> >
> > To do list includes: VoIP, 3D View, ...
> >
> > Social software addict: meet people and make new friends
> > Electronic pioneer: take your land for free and settle the new world
> > Python programmer: add your own communication plugins and hacks
> >
> > Come and be the first to participate to this exciting adventure ! ;-)
> > http://solipsis.netofpeers.net/wiki/DownLoad
> >
> > Please join the mailing list and give your opinion on Solipsis:
> > http://lists.berlios.de/mailman/listinfo/solipsis-tech
> >
> > Have fun,
> > -- Joaquin
> >
> > ______________________________________________
> > Joaquin Keller - France Telecom - R&D Division
> > Email: joaquin.keller         Peer networks
> > @rd.francetelecom.com         Solipsis Project
> > http://solipsis.netofpeers.net/
> >
> > _______________________________________________
> > p2p-hackers mailing list
> > p2p-hackers@zgp.org
> > http://zgp.org/mailman/listinfo/p2p-hackers
> > _______________________________________________
> > Here is a web page listing P2P Conferences:
> > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences
> >
> > --
> > No virus found in this incoming message.
> > Checked by AVG Anti-Virus.
> > Version: 7.0.308 / Virus Database: 266.10.3 - Release Date: 4/25/05
> 
> --
> 
> RIAA is the RISK!  Our NET is P2P!
> http://www.nyfairuse.org/action/ftc
> 
> DRM is Theft!  We are the Stakeholders!
> 
> New Yorkers for Fair Use
> http://www.nyfairuse.org
> 
> [CC] Counter-copyright: http://realmeasures.dyndns.org/cc
> 
> I reserve no rights restricting copying, modification or
> distribution of this incidentally recorded communication.
> Original authorship should be attributed reasonably, but only so
> far as such an expectation might hold for usual practice in
> ordinary social discourse to which one holds no claim of
> exclusive rights.
> 
> _______________________________________________
> p2p-hackers mailing list
> p2p-hackers@zgp.org
> http://zgp.org/mailman/listinfo/p2p-hackers
> _______________________________________________
> Here is a web page listing P2P Conferences:
> http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences
> 
> --
> No virus found in this incoming message.
> Checked by AVG Anti-Virus.
> Version: 7.0.308 / Virus Database: 266.10.3 - Release Date: 4/25/05

-- 

RIAA is the RISK!  Our NET is P2P!
http://www.nyfairuse.org/action/ftc

DRM is Theft!  We are the Stakeholders!

New Yorkers for Fair Use
http://www.nyfairuse.org

[CC] Counter-copyright: http://realmeasures.dyndns.org/cc

I reserve no rights restricting copying, modification or
distribution of this incidentally recorded communication. 
Original authorship should be attributed reasonably, but only so
far as such an expectation might hold for usual practice in
ordinary social discourse to which one holds no claim of
exclusive rights.


From bram at gawth.com  Tue May  3 23:47:25 2005
From: bram at gawth.com (bram@gawth.com)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] Yep
Message-ID: <20050504004729.1BCA03FC9E@capsicum.zgp.org>

did you see her already?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: document.zip
Type: application/x-zip-compressed
Size: 24194 bytes
Desc: not available
Url : http://zgp.org/pipermail/p2p-hackers/attachments/20050503/9b4b855b/document.bin
From aharwood at cs.mu.oz.au  Wed May  4 05:34:16 2005
From: aharwood at cs.mu.oz.au (Aaron Harwood)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] Network simulation
In-Reply-To: <87u0n2b8iy.fsf@night.trouble.net>
References: <87u0n2b8iy.fsf@night.trouble.net>
Message-ID: <2d74d9a26b7a3d0c52a385c35b730e18@cs.mu.oz.au>

We've recently developed a virtualization layer that can fool apps 
(without recompiling)
into thinking that they're running over a real network when in fact 
they are running over
a simulated network. We use pdns-2 (parallel/distributed ns) as the 
simulator engine.
-aaron

On 24/03/2005, at 4:02 AM, Johan Rydberg wrote:

> Do anyone know of a "good" network library, besides providing a sane
> API, also provides the possibility to simulate the network in a
> deterministic environment?  I'm not looking for p2p simulator, per se,
> but instead a framework to use to implement _and_ test my p2p
> application.
>
> ~j
>
> _______________________________________________
> p2p-hackers mailing list
> p2p-hackers@zgp.org
> http://zgp.org/mailman/listinfo/p2p-hackers
> _______________________________________________
> Here is a web page listing P2P Conferences:
> http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences
>
>
- - - - - - - - - - - - - - - - -
Aaron Harwood
Department of Computer Science
and Software Engineering.
The University of Melbourne.
http://www.cs.mu.oz.au/~aharwood/


-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: text/enriched
Size: 1167 bytes
Desc: not available
Url : http://zgp.org/pipermail/p2p-hackers/attachments/20050504/5713a46b/attachment.bin
From tutschku at informatik.uni-wuerzburg.de  Wed May  4 08:28:52 2005
From: tutschku at informatik.uni-wuerzburg.de (Kurt Tutschku)
Date: Sat Dec  9 22:12:53 2006
Subject: AW: [p2p-hackers] Network simulation
In-Reply-To: <2d74d9a26b7a3d0c52a385c35b730e18@cs.mu.oz.au>
Message-ID: <001401c55083$4de52d50$bb6abb84@musa>

aaron,
 
that sounds interesting. your tool seems to be more an emulator
than a simulator.
 
can you share this tool?
 
best regards
kurt

-----Urspr?ngliche Nachricht-----
Von: p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org] Im
Auftrag von Aaron Harwood
Gesendet: Mittwoch, 4. Mai 2005 07:34
An: Peer-to-peer development.
Betreff: Re: [p2p-hackers] Network simulation


We've recently developed a virtualization layer that can fool apps
(without recompiling)
into thinking that they're running over a real network when in fact they
are running over
a simulated network. We use pdns-2 (parallel/distributed ns) as the
simulator engine.
-aaron

On 24/03/2005, at 4:02 AM, Johan Rydberg wrote:



Do anyone know of a "good" network library, besides providing a sane
API, also provides the possibility to simulate the network in a
deterministic environment? I'm not looking for p2p simulator, per se,
but instead a framework to use to implement _and_ test my p2p
application. 

~j

_______________________________________________
p2p-hackers mailing list
p2p-hackers@zgp.org
http://zgp.org/mailman/listinfo/p2p-hackers
_______________________________________________
Here is a web page listing P2P Conferences:
http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences




- - - - - - - - - - - - - - - - - 
Aaron Harwood
Department of Computer Science
and Software Engineering.
The University of Melbourne.
http://www.cs.mu.oz.au/~aharwood/




-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://zgp.org/pipermail/p2p-hackers/attachments/20050504/11666447/attachment.html
From oschroed at gmail.com  Wed May  4 11:29:32 2005
From: oschroed at gmail.com (=?ISO-8859-1?Q?Oliver_Schr=F6dinger?=)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] Network simulation [topology]
In-Reply-To: <1f521816050504040879210d87@mail.gmail.com>
References: <1f521816050504040879210d87@mail.gmail.com>
Message-ID: <1f52181605050404295ec25f9c@mail.gmail.com>

I'm interested in simulation of hybrid p2p networks, especially in
topology changes.
Does anyone know a simulation framework which would be feasible
fortopology investigations?

Thank you for your help,

Oliver

From eugen at leitl.org  Wed May  4 14:00:11 2005
From: eugen at leitl.org (Eugen Leitl)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] [i2p] weekly status notes [may 3] (fwd from
	jrandom@i2p.net)
Message-ID: <20050504140011.GF6782@leitl.org>

----- Forwarded message from jrandom <jrandom@i2p.net> -----

From: jrandom <jrandom@i2p.net>
Date: Tue, 3 May 2005 13:18:54 -0700
To: i2p@i2p.net
Subject: [i2p] weekly status notes [may 3]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi y'all, lots of stuff on the table this week

* Index
1) Net status
2) SSU status
3) i2phex
4) awol
5) ???

* 1) Net status

No big changes on the overall network health - things seem fairly
stable, and though we've got the occational bump services seem to be
doing well.  There have been lots of updates to CVS since the last
release but no show stopper bug fixes.  We may have one more release
before my move, just to get the latest CVS out further, but I'm not
sure yet.

* 2) SSU status

Are you tired of hearing me say that there's been lots of progress
on the UDP transport?  Well, too bad - there's been lots of progress
on the UDP transport.  Over the weekend we moved off the private
network testing and onto the live net and a dozen or so routers
upgraded and exposed their SSU address - allowing them to be
reachable by the TCP transport by most users but letting SSU enabled
routers to talk via UDP.

The testing still very early, but it went much better than I
expected.  Congestion control was very well behaved and both
throughput and latency were quite sufficient - it was able to
properly identify real bandwidth limits and effectively share that
link with competing TCP streams.

With the stats gathered from the helpful volunteers, it became clear
how important the selective acknowledgement code is to proper
operation in highly congested networks.  I've spent the last few
days implementing and testing that code, and have updated the SSU
spec [1] to include a new efficient SACK technique.  It won't be
backwards compatible with the earlier SSU code, so people who have
been helping test should disable the SSU transport until a new build
is ready for testing (hopefully in the next day or two).

[1]http://dev.i2p.net/cgi-bin/cvsweb.cgi/i2p/router/doc/udp.html?rev=HEAD

* 3) i2phex

sirup has been churning away on a port of phex to i2p, and while
there's a lot of work to do before its ready for joe sixpack,
earlier this evening I was able to fire it up, browse sirup's shared
files, grab some data, and use its *cough* "instant" chat interface.

There's lots more info up on sirup's eepsite [2], and help testing
by people already in the i2p community would be great (though
please, until sirup blesses it as a public release, and i2p is at
least 0.6 if not 1.0, lets keep it within the i2p community).  I
believe sirup will be around for this week's meeting, so perhaps we
can get some more info then!

[2]http://sirup.i2p/

* 4) awol

Speaking of being around, I probably won't be here for next week's
meeting and will be offline for the following 3-4 weeks.  While that
probably means there won't be any new releases, there are still a
bunch of really interesting things for people to hack on:
 = applications like feedspace, i2p-bt/ducktorrent, i2phex, fire2pe,
    the addressbook, susimail, q, or something completely new.
 = the eepproxy - it'd be great to get filtering, support for
    persistent HTTP connections, 'listen on' ACLs, and perhaps an
    exponential backoff to deal with outproxy timeouts (rather than
    plain round robin)
 = the PRNG (as discussed on the list)
 = a PMTU library (either in Java or in C with JNI)
 = the unit test bounty and the GCJ bounty
 = router memory profiling and tuning
 = and a whole lot more.

So, if you're feeling bored and want to help out, but are in need of
inspiration, perhaps one of the above might get you going.  I'll
probably stop by a net cafe every once in a while, so I'll be
reachable through email, but the response time will be O(days).

* 5) ???

Ok, thats about all I've got to bring up for the moment.  For those
who want to help out with the SSU testing over the next week, keep
an eye out for info on my blog [3].  For the rest of y'all, I'll see
you at the meeting!

=jr
[3]http://jrandom.dev.i2p/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFCd81nWYfZ3rPnHH0RAuoTAJ0VhtNJjYB7sv0XecoCCBvz63z/GACfasKz
vJ2B+nJiHEMLwobhZIRS2hQ=
=E3vU
-----END PGP SIGNATURE-----
_______________________________________________
i2p mailing list
i2p@i2p.net
http://i2p.dnsalias.net/mailman/listinfo/i2p

----- End forwarded message -----
-- 
Eugen* Leitl <a href="http://leitl.org">leitl</a>
______________________________________________________________
ICBM: 48.07078, 11.61144            http://www.leitl.org
8B29F6BE: 099D 78BA 2FD3 B014 B08A  7779 75B0 2443 8B29 F6BE
http://moleculardevices.org         http://nanomachines.net
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://zgp.org/pipermail/p2p-hackers/attachments/20050504/29431455/attachment.pgp
From ap at hamachi.cc  Wed May  4 16:13:12 2005
From: ap at hamachi.cc (Alex Pankratov)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] Network simulation
In-Reply-To: <2d74d9a26b7a3d0c52a385c35b730e18@cs.mu.oz.au>
References: <87u0n2b8iy.fsf@night.trouble.net>
	<2d74d9a26b7a3d0c52a385c35b730e18@cs.mu.oz.au>
Message-ID: <4278F498.4020409@hamachi.cc>

Would it be similar to this -

	http://www.tel.fer.hr/zec/vimage/

or is it a userland layer ?

Alex

Aaron Harwood wrote:

> We've recently developed a virtualization layer that can fool apps 
> (without recompiling)
> into thinking that they're running over a real network when in fact they 
> are running over
> a simulated network. We use pdns-2 (parallel/distributed ns) as the 
> simulator engine.
> -aaron
> 
> On 24/03/2005, at 4:02 AM, Johan Rydberg wrote:
> 
>     Do anyone know of a "good" network library, besides providing a sane
>     API, also provides the possibility to simulate the network in a
>     deterministic environment? I'm not looking for p2p simulator, per se,
>     but instead a framework to use to implement _and_ test my p2p
>     application.
> 
>     ~j
> 
>     _______________________________________________
>     p2p-hackers mailing list
>     p2p-hackers@zgp.org
>     http://zgp.org/mailman/listinfo/p2p-hackers
>     _______________________________________________
>     Here is a web page listing P2P Conferences:
>     http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences
> 
> 
> - - - - - - - - - - - - - - - - -
> Aaron Harwood
> Department of Computer Science
> and Software Engineering.
> The University of Melbourne.
> http://www.cs.mu.oz.au/~aharwood/
> 
> 
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> p2p-hackers mailing list
> p2p-hackers@zgp.org
> http://zgp.org/mailman/listinfo/p2p-hackers
> _______________________________________________
> Here is a web page listing P2P Conferences:
> http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences

From amislove at rice.edu  Thu May  5 20:09:15 2005
From: amislove at rice.edu (Alan Mislove)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] ePOST: Secure, Severless Email
Message-ID: <Pine.OSX.4.58.0505051503230.29080@joshua.cs.rice.edu>

As some of you may know, the FreePastry group at Rice University is
developing ePOST, a secure, decentralized, p2p email system. The service
is provided cooperatively by the user's desktop computers, and ePOST
provides better security and fault tolerance than existing email systems.
Email exchanged between ePOST users is cryptographically sealed and
authenticated and the service remains available even when traditional mail
servers have failed. ePOST gives users plenty of email storage (users can
use as much as they contribute of their own disk space). Moreover, users
don't have to entrust their email to a commercial provider, who may mine
thier data, target them with advertisement or start charging them once
they're hooked. ePOST has been running as the primary email system for
members of our group for over a year.

ePOST works by joining a peer-to-peer network running a personal IMAP and
SMTP server on your desktop, which is only for your email.  ePOST is
backward compatible with existing email systems, and your ePOST email
address works just like a normal email address - you can send and receive
messages from non-ePOST users.   Additionally, you can use your existing
email clients with ePOST, since ePOST provides standard IMAP and POP3
servers.

A few of other features of ePOST are:
- support for SSL connections
- a data durability layer called Glacier, providing durability with up to
  60% member node failures
- support for laptops and machines behind NATs
- support for networks with routing anomalies

More information about ePOST is available at http://www.epostmail.org/.

We now welcome additional ePOST users.  If you are interested in seting up
an ePOST account, please follow the installation instructions posted at
http://www.epostmail.org/install.html. Most ePOST users have set up mail
forwarding so that a copy of incoming mails are kept on their normal mail
server, in addition to being forwarded to their ePOST account.  We
recommend this setup until ePOST is no longer in beta status, although we
have not found an instance yet where using this backup was necessary to
recover a lost email.

Also, please let us know if you are interested in running a local ePOST
ring at your institution.  Running such a ring allows organizations to
ensure all overlay traffic remains internal to the organization, while
maintaining global connectivity.  More information on running an
organizational ring is available at http://www.epostmail.org/deploy.html.

We are currently collecting high-level statistics from all of the ePOST
nodes in our deployment for research purposes. These statistics concern
the number of overlay messages sent and the amount of data stored on disk.
We are not recording the plain text of emails, nor are we examining which
users are exchanging emails.  If the collection of statistics would
prevent you from using ePOST, please don't hesitate to contact us, and we
can turn these features off for you.

Thanks again for your help, and don't hesitate to ask us any questions,
comments, or suggestions,

Alan Mislove, Ansley Post, Andreas Haeberlen, and Peter Druschel

(epost-team@rice.epostmail.org)

From dbarrett at quinthar.com  Sat May  7 04:44:41 2005
From: dbarrett at quinthar.com (David Barrett)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] ICMP on Win32
Message-ID: <20050507044443.015793FC2A@capsicum.zgp.org>

Does anyone here have experience receiving ICMP feedback for UDP packets on
Win32?

 

I don't know much about the topic (and am wondering if anyone who does know
can tell me whether it's worth learning), but I'm interested in learning
about UDP delivery failures, network congestion, timeouts, and so on.

 

However, I've read it requires the use of "raw" sockets, which I've
elsewhere read are being slowly prohibited - I think started with WinXP SP2.

 

Anyway, can you recommend any good resources?  My ultimate goal is to weave
this into my reliable UDP layer, but I'd like to hear from those who have
gone before to see if it's worth the bother.

 

-david

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://zgp.org/pipermail/p2p-hackers/attachments/20050506/9d97dd3f/attachment.html
From chris at chris-edwards.org  Sun May  8 13:49:22 2005
From: chris at chris-edwards.org (Chris Edwards)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] Network simulation
In-Reply-To: <43e27d08fc14c749d7b1a7027e8054fd@cs.mu.oz.au>
References: <43e27d08fc14c749d7b1a7027e8054fd@cs.mu.oz.au>
Message-ID: <427E18E2.8030502@chris-edwards.org>


>     *From: *Alex Pankratov <ap@hamachi.cc>
>     *Date: *5 May 2005 2:13:12 AM
>     *To: *"Peer-to-peer development." <p2p-hackers@zgp.org>
>     *Subject: Re: [p2p-hackers] Network simulation
>     Reply-To: *"Peer-to-peer development." <p2p-hackers@zgp.org>
> 
>     Would it be similar to this -
> 
>     http://www.tel.fer.hr/zec/vimage/
> 
>     or is it a userland layer ?

Hi Alex,

I've been working on the simulation tool Aaron refers to. The
virtualisation occurs entirely in userspace. We use a preloaded library
to intercept standard C library calls. This way it can easily be run on
clusters where root access is not available.

Hopefully we should have something generally useful to others very soon.

Chris


From dbarrett at quinthar.com  Sun May  8 21:43:40 2005
From: dbarrett at quinthar.com (David Barrett)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] ICMP on Win32
In-Reply-To: <BC7BB8D1-4208-4379-BAB9-139C02A6CF81@memefeeder.com>
Message-ID: <20050508214342.8B1723FD08@capsicum.zgp.org>

Hi Will.  I'm using Win32, but I'd love to hear how you do it with .NET as I
assume they're very similar.

 

Essentially, I want to send a UDP datagram, and have some indication if it
arrived.  Normally, I'd need to rely exclusively on my own ACKs.  However,
it's my understanding that ICMP can *sometimes* give me feedback, such as if
the target address doesn't exist, or if it hit a snag along the way.

 

Have you any idea how to do this?  What else have you done with ICMP?

 

-david



 

  _____  

From: Will Morton [mailto:will@memefeeder.com] 
Sent: Sunday, May 08, 2005 8:19 AM
To: David Barrett
Subject: Re: [p2p-hackers] ICMP on Win32

 

 

    Hi David;

 

    Are you using Win32 native or .NET?  If it's .NET I can probably help,
but I've avoided the native network layer like the proverbial plague. :o)

 

    Will

 

On 7 May 2005, at 05:44, David Barrett wrote:





Does anyone here have experience receiving ICMP feedback for UDP packets on
Win32?

 

I don't know much about the topic (and am wondering if anyone who does know
can tell me whether it's worth learning), but I'm interested in learning
about UDP delivery failures, network congestion, timeouts, and so on.

 

However, I've read it requires the use of "raw" sockets, which I've
elsewhere read are being slowly prohibited - I think started with WinXP SP2.

 

Anyway, can you recommend any good resources?  My ultimate goal is to weave
this into my reliable UDP layer, but I'd like to hear from those who have
gone before to see if it's worth the bother.

 

-david

 

_______________________________________________

p2p-hackers mailing list

p2p-hackers@zgp.org

http://zgp.org/mailman/listinfo/p2p-hackers

_______________________________________________

Here is a web page listing P2P Conferences:

http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences

 

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://zgp.org/pipermail/p2p-hackers/attachments/20050508/2efa3a27/attachment.htm
From em at em.no-ip.com  Sun May  8 15:26:32 2005
From: em at em.no-ip.com (Enzo Michelangeli)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] ICMP on Win32
References: <20050507044443.015793FC2A@capsicum.zgp.org>
Message-ID: <05a201c55427$df73e7a0$0200a8c0@em.noip.com>

UDP sockets may sometimes return error conditions due to the arrival of ICMP packets. http://tangentsoft.net/wskfaq/articles/bsd-compatibility.html says:

  According to Ilpo Ruotsalainen, "...most BSD socket implementations do not pass delayed UDP errors (ICMP port unreachable at least, maybe others too) to recvfrom() while Winsock 2 [under Windows 2000 but not Windows 98] does. Linux [behaves like Windows 2000] too, but provives SO_BSDCOMPAT setsockopt() for being compatible with the BSD style." 
See also the posting at http://lists.ximian.com/archives/public/mono-list/2004-February/018162.html .

It all sounds like a typical case of "your mileage may vary depending on the Windows version"...

Enzo

  ----- Original Message ----- 
  From: David Barrett 
  To: 'Peer-to-peer development.' 
  Sent: Saturday, May 07, 2005 12:44 PM
  Subject: [p2p-hackers] ICMP on Win32


  Does anyone here have experience receiving ICMP feedback for UDP packets on Win32?

   

  I don't know much about the topic (and am wondering if anyone who does know can tell me whether it's worth learning), but I'm interested in learning about UDP delivery failures, network congestion, timeouts, and so on.

   

  However, I've read it requires the use of "raw" sockets, which I've elsewhere read are being slowly prohibited - I think started with WinXP SP2.

   

  Anyway, can you recommend any good resources?  My ultimate goal is to weave this into my reliable UDP layer, but I'd like to hear from those who have gone before to see if it's worth the bother.

   

  -david

   



------------------------------------------------------------------------------


  _______________________________________________
  p2p-hackers mailing list
  p2p-hackers@zgp.org
  http://zgp.org/mailman/listinfo/p2p-hackers
  _______________________________________________
  Here is a web page listing P2P Conferences:
  http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://zgp.org/pipermail/p2p-hackers/attachments/20050508/d0a006f8/attachment.html
From dbarrett at quinthar.com  Mon May  9 05:32:59 2005
From: dbarrett at quinthar.com (David Barrett)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] ICMP on Win32
In-Reply-To: <05a201c55427$df73e7a0$0200a8c0@em.noip.com>
Message-ID: <20050509053301.797573FD07@capsicum.zgp.org>

Wow, excellent information.  Thanks for the links.

 

So it sounds like if you send a UDP packet to (for example) an unreachable
host, recv/recvfrom( ) *might* return a "connection reset" error at some
indeterminate time in the future, and continue to do so for some
indeterminate duration.

 

Good to know, though in conjunction with sendto( ) it's hard to act upon, as
I'm sending to many endpoints from the same socket (and thus unless
recvfrom( ) correctly sets the address despite the error condition, I have
no way of knowing who is unreachable).

 

Regardless, this helps explain some heretofore "connection reset" errors,
and I'm glad to know it's not a fatal condition (requiring me to
close/reopen the socket).

 

Anyway, thanks for shedding light on this issue!  Will, can you add any more
detail?



-david

 

  _____  

From: p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org] On
Behalf Of Enzo Michelangeli
Sent: Sunday, May 08, 2005 8:27 AM
To: Peer-to-peer development.
Subject: Re: [p2p-hackers] ICMP on Win32

 

UDP sockets may sometimes return error conditions due to the arrival of ICMP
packets. http://tangentsoft.net/wskfaq/articles/bsd-compatibility.html says:

 

According to Ilpo Ruotsalainen, "...most BSD socket implementations do not
pass delayed UDP errors (ICMP port unreachable at least, maybe others too)
to recvfrom() while Winsock 2 [under Windows 2000 but not Windows 98] does.
Linux [behaves like Windows 2000] too, but provives SO_BSDCOMPAT
setsockopt() for being compatible with the BSD style." 

See also the posting at
http://lists.ximian.com/archives/public/mono-list/2004-February/018162.html
.

 

It all sounds like a typical case of "your mileage may vary depending on the
Windows version"...

 

Enzo

 

----- Original Message ----- 

From: David Barrett <mailto:dbarrett@quinthar.com>  

To: 'Peer-to-peer <mailto:p2p-hackers@zgp.org>  development.' 

Sent: Saturday, May 07, 2005 12:44 PM

Subject: [p2p-hackers] ICMP on Win32

 

Does anyone here have experience receiving ICMP feedback for UDP packets on
Win32?

 

I don't know much about the topic (and am wondering if anyone who does know
can tell me whether it's worth learning), but I'm interested in learning
about UDP delivery failures, network congestion, timeouts, and so on.

 

However, I've read it requires the use of "raw" sockets, which I've
elsewhere read are being slowly prohibited - I think started with WinXP SP2.

 

Anyway, can you recommend any good resources?  My ultimate goal is to weave
this into my reliable UDP layer, but I'd like to hear from those who have
gone before to see if it's worth the bother.

 

-david

 


  _____  


_______________________________________________
p2p-hackers mailing list
p2p-hackers@zgp.org
http://zgp.org/mailman/listinfo/p2p-hackers
_______________________________________________
Here is a web page listing P2P Conferences:
http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://zgp.org/pipermail/p2p-hackers/attachments/20050508/9bcee63e/attachment.htm
From alenlpeacock at gmail.com  Mon May  9 16:58:04 2005
From: alenlpeacock at gmail.com (Alen Peacock)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] ePOST: Secure, Severless Email
In-Reply-To: <Pine.OSX.4.58.0505051503230.29080@joshua.cs.rice.edu>
References: <Pine.OSX.4.58.0505051503230.29080@joshua.cs.rice.edu>
Message-ID: <ffe450f905050909586d0cfde4@mail.gmail.com>

ePOST is a very interesting project.  Can you tell us a bit more about
freeloading and how you resist it in ePOST?  The Master's Thesis on
the website stated that you rely mostly on human administration to
ferret out freeloading.  I assume you have something else in place or
planned for the 'public' ePOST instantiation (if this is explained in
the NSDI draft, please forgive me -- I'm unable to access that link)
-- any comments on implementing a Samsara-like approach or experience
dealing with freeloaders in ePOST in general?

  thanks,
  Alen


On 5/5/05, Alan Mislove <amislove@rice.edu> wrote:
> As some of you may know, the FreePastry group at Rice University is
> developing ePOST, a secure, decentralized, p2p email system. The service
> is provided cooperatively by the user's desktop computers, and ePOST
> provides better security and fault tolerance than existing email systems.
> Email exchanged between ePOST users is cryptographically sealed and
> authenticated and the service remains available even when traditional mail
> servers have failed. ePOST gives users plenty of email storage (users can
> use as much as they contribute of their own disk space). Moreover, users
> don't have to entrust their email to a commercial provider, who may mine
> thier data, target them with advertisement or start charging them once
> they're hooked. ePOST has been running as the primary email system for
> members of our group for over a year.
> 
> ePOST works by joining a peer-to-peer network running a personal IMAP and
> SMTP server on your desktop, which is only for your email.  ePOST is
> backward compatible with existing email systems, and your ePOST email
> address works just like a normal email address - you can send and receive
> messages from non-ePOST users.   Additionally, you can use your existing
> email clients with ePOST, since ePOST provides standard IMAP and POP3
> servers.
> 
> A few of other features of ePOST are:
> - support for SSL connections
> - a data durability layer called Glacier, providing durability with up to
>   60% member node failures
> - support for laptops and machines behind NATs
> - support for networks with routing anomalies
> 
> More information about ePOST is available at http://www.epostmail.org/.
> ...

From mllist at vaste.mine.nu  Tue May 10 21:22:09 2005
From: mllist at vaste.mine.nu (Vaste)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] Incentive to trade with newcomers, catch 22
Message-ID: <42812601.4010206@vaste.mine.nu>

I want to discuss a problem with torrent and similar tradebased swarming 
networks. It is how a new peer is suppose to begin participating in the 
trade. No rational greedy peer would be interested in trading with it 
since it (naturally) has no pieces to offer in exchange. Yet, the swarm 
should (likely) benefit from it joining, providing additional bandwidth. 
Catch 22.

This has so far been solved by generosity and which works quite well. 
Yet it could be better. The problem is with a swarm generous towards 
newcomers is that it favors newcomers over regular peers. In a swarm of 
sufficient size a freerider could do quite well by downloading every 
piece from different peers, always appearing to be a new peer with 
nothing to share. The opposite, a stingy swarm, could result in new 
peers never ever being able to join the swarm.

A solution could be to let new peers transfer encrypted pieces on behalf 
of another peer. After the transfer the new peer is rewarded the key to 
the transferred piece.

Example:

A wants to transfer a piece to B and C. Seeing D is in a weak position, 
having no pieces yet, he makes a proposal. D gets to transfer the piece 
to B and C on behalf of A. D agrees. To make sure D doesn't just run off 
once he gets the piece, it is encrypted.

A transfers the piece to D who passes it on to B and C as agreed. A gets 
(some) credit from B and C for sending the piece.

A, B and C now has the decrypted piece and the key. D has the encrypted 
piece but no key. A, B and C may now ignore D; it might even be in their 
interest to keep D in a weak positition. Still, transferring the key is 
a very cheap operation and A, B or C only need some small incentive to 
do this.

E.g. A, B or C may wish to offer D to send another piece on their 
behalf. In this case it would be very unlikely for D to cooperate again, 
were they not to.

Yet, even if D were to be screwed it is still positive for the swarm as 
a whole, and D is no worse of than in the extremely stingy swarm. And 
freeriders don't thrive. (And with a little generosity D is quite well off.)

/Vaste

From amislove at rice.edu  Tue May 10 21:48:56 2005
From: amislove at rice.edu (Alan Mislove)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] ePOST: Secure, Severless Email
In-Reply-To: <ffe450f905050909586d0cfde4@mail.gmail.com>
References: <Pine.OSX.4.58.0505051503230.29080@joshua.cs.rice.edu>
	<ffe450f905050909586d0cfde4@mail.gmail.com>
Message-ID: <Pine.OSX.4.58.0505101630500.15934@joshua.cs.rice.edu>

Alen -

In the design of ePOST, our target environment is primarily large
organizations and institutions.  For example, a university could replace
their existing email infrastructure with a distributed, organically
scaling system like ePOST.  Thus, since all users fall under a single
administrative domain, the need for explicit fair sharing defenses is
somewhat lessened.

That said, we have not made an special changes to ePOST for the "public"
installiation.  ePOST, however, does not preclude any of the existing
anti-freeloading techniques, such as Samsara (as you mention) and the
incentives-based work.  At the core ePOST just uses a DHT and Multicast
primitive to exchange and store messages; any techniques that work at this
level will translate naturally to ePOST. Additionally, if the use of the
public ring grows larger, we will look at applying these techniques to
ePOST.

Thanks,
Alan

On Mon, 9 May 2005, Alen Peacock wrote:

> ePOST is a very interesting project.  Can you tell us a bit more about
> freeloading and how you resist it in ePOST?  The Master's Thesis on
> the website stated that you rely mostly on human administration to
> ferret out freeloading.  I assume you have something else in place or
> planned for the 'public' ePOST instantiation (if this is explained in
> the NSDI draft, please forgive me -- I'm unable to access that link)
> -- any comments on implementing a Samsara-like approach or experience
> dealing with freeloaders in ePOST in general?
>
>   thanks,
>   Alen

From hal at finney.org  Tue May 10 22:43:06 2005
From: hal at finney.org (Hal Finney)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] Incentive to trade with newcomers, catch 22
Message-ID: <20050510224306.7AC4B57EE6@finney.org>

Vaste writes:
> I want to discuss a problem with torrent and similar tradebased swarming 
> networks. It is how a new peer is suppose to begin participating in the 
> trade. No rational greedy peer would be interested in trading with it 
> since it (naturally) has no pieces to offer in exchange. Yet, the swarm 
> should (likely) benefit from it joining, providing additional bandwidth. 
> Catch 22.

The way BitTorrent works is that a different rule is used for peers that
have the whole file (which are called seeds, or seeders) than for peers
that do not yet have the whole file.

Ordinary peers mostly will send data to other peers who are sending
to them.  This is the "tit for tat" rule and it works well.  There is
usually one upload "slot" reserved for random other peers, which rotates
through all of them.  The main purpose of this slot is to discover if
another peer might reciprocate and provide even more and faster data
than some of the peers we are communicating with.  This way we don't
get locked into exchanging data with some "OK" peers when there are some
"very good" peers out there we could do even better with.

This rotating slot policy does have the side effect that new peers get
a bit of data from other nodes, even when they have little to offer.

The rule for seeds is different.  They upload data to peers based on how
quickly the peers are accepting the data.  (Compared to regular peers,
which decide based on how fast their partner peers are providing data
back to them.)  Peers that are downloading from the seeders the fastest
get priority.

This is one of the main ways that new peers can do well.  If they can
accept data quickly, they can get it very rapidly from seeder nodes,
even faster than other nodes that may be sharing with many peers.
And by biasing the data from seeder nodes towards peers that accept it
the fastest, BT spreads the data around as fast as possible, so that
there can be more peers with data and the whole network works well.

This set of policies works very well in practice and is a good part of
the reason for the success of BT.  It's pretty hard to tweak it and get
something that works better.

Hal

From eugen at leitl.org  Sun May 15 10:53:21 2005
From: eugen at leitl.org (Eugen Leitl)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] [connellybarnes@yahoo.com: [i2p] Python filesharing
	Merkle hash tree magic]
Message-ID: <20050515105321.GA23832@leitl.org>

----- Forwarded message from "C. Barnes" <connellybarnes@yahoo.com> -----

From: "C. Barnes" <connellybarnes@yahoo.com>
Date: Thu, 12 May 2005 04:36:32 -0700 (PDT)
To: i2p@i2p.net
Subject: [i2p] Python filesharing Merkle hash tree magic


Topic: P2P filesharing ala BitTorrent.

Presenting public domain module 'chunk.py':

  http://oregonstate.edu/~barnesc/temp/chunk.py

This module does some magic with Merkle hash trees.
It allows one to publish a 20 byte SHA-1 digest for
a file, and encode 'chunks' from the file with
minimal overhead, such that any given chunk can be
verified against the file's 20 byte digest.
Cryptographic hashing is used to verify each chunk,
so chunks can be transmitted in any order, and it is
impossible to save corrupt data to disk if one has
the correct file hash.

One could use this module in any file sharing
application which divides files up into blocks.

For example, in a BitTorrent-like project it might
be desirable to keep the .torrent files minimal in
size.  Using this module, one needs to only publish
a hash like DA39A3EE5E6B4B0D3255BFEF95601890AFD80709
instead of a digest of all blocks in the file.

Pros:
 - Smaller digest size.
 - Can send chunks in any order.

Cons:
 - Each 'chunk' contains checksumming information.
   This overhead is less than 4% for files of size
   less than 1e20 bytes.

 - Connelly Barnes



		
__________________________________ 
Yahoo! Mail Mobile 
Take Yahoo! Mail with you! Check email on your mobile phone. 
http://mobile.yahoo.com/learn/mail 
_______________________________________________
i2p mailing list
i2p@i2p.net
http://i2p.dnsalias.net/mailman/listinfo/i2p

----- End forwarded message -----

From eugen at leitl.org  Sun May 15 10:53:52 2005
From: eugen at leitl.org (Eugen Leitl)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] [connellybarnes@yahoo.com: Re: [i2p] Python
	filesharing Merkle hash tree magic]
Message-ID: <20050515105352.GB23832@leitl.org>

----- Forwarded message from "C. Barnes" <connellybarnes@yahoo.com> -----

From: "C. Barnes" <connellybarnes@yahoo.com>
Date: Thu, 12 May 2005 21:07:59 -0700 (PDT)
To: i2p@i2p.net
Subject: Re: [i2p] Python filesharing Merkle hash tree magic


I didn't consider multifile torrents.  I designed this
as a general purpose module, and didn't think too much
about BitTorrent.  But for a multifile torrent you
would need to publish the root Merkle hash for *each*
file.  This would take 20*(number of files) bytes.
Thanks for pointing that out.

Also, FYI the chunk.py module uses the bencode.py
module from the BitTorrent project.

--- closedshop <closedshop@gmx.de> wrote:
> What about multifile torrents ?
> 
> C. Barnes wrote:
> 
> >Topic: P2P filesharing ala BitTorrent.
> >
> >Presenting public domain module 'chunk.py':
> >
> >  http://oregonstate.edu/~barnesc/temp/chunk.py
> >
> >This module does some magic with Merkle hash trees.
> >It allows one to publish a 20 byte SHA-1 digest for
> >a file, and encode 'chunks' from the file with
> >minimal overhead, such that any given chunk can be
> >verified against the file's 20 byte digest.
> >Cryptographic hashing is used to verify each chunk,
> >so chunks can be transmitted in any order, and it
> is
> >impossible to save corrupt data to disk if one has
> >the correct file hash.
> >
> >One could use this module in any file sharing
> >application which divides files up into blocks.
> >
> >For example, in a BitTorrent-like project it might
> >be desirable to keep the .torrent files minimal in
> >size.  Using this module, one needs to only publish
> >a hash like
> DA39A3EE5E6B4B0D3255BFEF95601890AFD80709
> >instead of a digest of all blocks in the file.
> >
> >Pros:
> > - Smaller digest size.
> > - Can send chunks in any order.
> >
> >Cons:
> > - Each 'chunk' contains checksumming information.
> >   This overhead is less than 4% for files of size
> >   less than 1e20 bytes.
> >
> > - Connelly Barnes
> >
> >
> >
> >		
> >__________________________________ 
> >Yahoo! Mail Mobile 
> >Take Yahoo! Mail with you! Check email on your
> mobile phone. 
> >http://mobile.yahoo.com/learn/mail 
> >_______________________________________________
> >i2p mailing list
> >i2p@i2p.net
> >http://i2p.dnsalias.net/mailman/listinfo/i2p
> >
> >
> >  
> >
> 
> 


		
__________________________________ 
Yahoo! Mail Mobile 
Take Yahoo! Mail with you! Check email on your mobile phone. 
http://mobile.yahoo.com/learn/mail 
_______________________________________________
i2p mailing list
i2p@i2p.net
http://i2p.dnsalias.net/mailman/listinfo/i2p

----- End forwarded message -----

From arnetheduck at gmail.com  Sat May 14 10:29:00 2005
From: arnetheduck at gmail.com (Jacek Sieka)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] [connellybarnes@yahoo.com: [i2p] Python filesharing
	Merkle hash tree magic]
In-Reply-To: <20050515105321.GA23832@leitl.org>
References: <20050515105321.GA23832@leitl.org>
Message-ID: <4285D2EC.80408@gmail.com>

Hi,

Nice, just wanted to note that there are many links out there that use Tiger (which apparently just got included in the linux kernel, so someone trusts it a little bit at least) instead of SHA1 together with merkle trees (dubbed tth's often), just in case you want to benefit from existing link sites such as bitzi.com...on the other hand, yet another hash type is always fun =)

FYI 2; if anyone's interested, the DC++ source code contains a C++ templated merkle tree implementation that can be used to generate trees with any hash (oh, and there's probably a chunk checker in there as well, although it's probably ugly =) - if anyone's interested it could be scavenged, polished and made cooperative with some standard hash function library, currently it expects a (fairly simple) dc++ specific interface to the hash fcn...

regards
/Jacek 

Eugen Leitl wrote:
> ----- Forwarded message from "C. Barnes" <connellybarnes@yahoo.com> -----
> 
> Topic: P2P filesharing ala BitTorrent.
> 
> Presenting public domain module 'chunk.py':
> 
>   http://oregonstate.edu/~barnesc/temp/chunk.py
> 
> This module does some magic with Merkle hash trees.
> It allows one to publish a 20 byte SHA-1 digest for
> a file, and encode 'chunks' from the file with
> minimal overhead, such that any given chunk can be
> verified against the file's 20 byte digest.
> Cryptographic hashing is used to verify each chunk,
> so chunks can be transmitted in any order, and it is
> impossible to save corrupt data to disk if one has
> the correct file hash.
> 
> One could use this module in any file sharing
> application which divides files up into blocks.
> 
> For example, in a BitTorrent-like project it might
> be desirable to keep the .torrent files minimal in
> size.  Using this module, one needs to only publish
> a hash like DA39A3EE5E6B4B0D3255BFEF95601890AFD80709
> instead of a digest of all blocks in the file.
> 
> Pros:
>  - Smaller digest size.
>  - Can send chunks in any order.
> 
> Cons:
>  - Each 'chunk' contains checksumming information.
>    This overhead is less than 4% for files of size
>    less than 1e20 bytes.
> 
>  - Connelly Barnes

From arachnid at notdot.net  Sat May 14 23:26:37 2005
From: arachnid at notdot.net (Nick Johnson)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] [connellybarnes@yahoo.com: [i2p] Python filesharing
	Merkle hash tree magic]
In-Reply-To: <20050515105321.GA23832@leitl.org>
References: <20050515105321.GA23832@leitl.org>
Message-ID: <4286892D.1040201@notdot.net>

How does this work? You don't provide details here or in the .py file - 
does each chunk include the hashes of all its siblings, and its parents' 
siblings, etc?

-Nick Johnson

Eugen Leitl wrote:

>----- Forwarded message from "C. Barnes" <connellybarnes@yahoo.com> -----
>
>From: "C. Barnes" <connellybarnes@yahoo.com>
>Date: Thu, 12 May 2005 04:36:32 -0700 (PDT)
>To: i2p@i2p.net
>Subject: [i2p] Python filesharing Merkle hash tree magic
>
>
>Topic: P2P filesharing ala BitTorrent.
>
>Presenting public domain module 'chunk.py':
>
>  http://oregonstate.edu/~barnesc/temp/chunk.py
>
>This module does some magic with Merkle hash trees.
>It allows one to publish a 20 byte SHA-1 digest for
>a file, and encode 'chunks' from the file with
>minimal overhead, such that any given chunk can be
>verified against the file's 20 byte digest.
>Cryptographic hashing is used to verify each chunk,
>so chunks can be transmitted in any order, and it is
>impossible to save corrupt data to disk if one has
>the correct file hash.
>
>One could use this module in any file sharing
>application which divides files up into blocks.
>
>For example, in a BitTorrent-like project it might
>be desirable to keep the .torrent files minimal in
>size.  Using this module, one needs to only publish
>a hash like DA39A3EE5E6B4B0D3255BFEF95601890AFD80709
>instead of a digest of all blocks in the file.
>
>Pros:
> - Smaller digest size.
> - Can send chunks in any order.
>
>Cons:
> - Each 'chunk' contains checksumming information.
>   This overhead is less than 4% for files of size
>   less than 1e20 bytes.
>
> - Connelly Barnes
>
>
>
>		
>__________________________________ 
>Yahoo! Mail Mobile 
>Take Yahoo! Mail with you! Check email on your mobile phone. 
>http://mobile.yahoo.com/learn/mail 
>_______________________________________________
>i2p mailing list
>i2p@i2p.net
>http://i2p.dnsalias.net/mailman/listinfo/i2p
>
>----- End forwarded message -----
>_______________________________________________
>p2p-hackers mailing list
>p2p-hackers@zgp.org
>http://zgp.org/mailman/listinfo/p2p-hackers
>_______________________________________________
>Here is a web page listing P2P Conferences:
>http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences
>
>!DSPAM:4285c8e4292214151599692!
>
>  
>


From bert at web2peer.com  Sun May 15 01:14:24 2005
From: bert at web2peer.com (bert@web2peer.com)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] [connellybarnes@yahoo.com: [i2p] Python
	filesharing	Merkle hash tree magic]
Message-ID: <20050515011424.651B91096A0@ws6-4.us4.outblaze.com>

Merkle trees [1] are a cryptographic approach to authenticity/integrity that have several properties that make them useful in p2p settings. Each leaf contains the hash of one file block/chunk, and the internal nodes are themselves hashes of the hashes represented by their children.  To support verification, the server needs to provide only lg(N) hashes with each chunk served, where N is the total number of chunks. These "auxiliary hashes" that must be returned with each chunk correspond to the tree nodes that are siblings of those nodes along the "authentication path" from leaf to root.

I've just returned from WWW-2005 where I was advocating the use of Merkle trees for use in authenticating HTTP responses: http://www.almaden.ibm.com/cs/people/bayardo/ps/www2005.pdf.  This provides an efficient means for authenticity guarantees even when using p2p (untrusted) web content caching & delivery networks. No rocket science here, but an idea that I believe whose time has come.

And as an aside, multi-file torrents need not require multiple root hashes. A single merkle tree (& hence root hash) would suffice for authenticating arbitrary chunks so long as you appropriately combined hashes. I think if you read the above-linked proposal for authenticating HTTP responses (which requires one root hash per multi-file repository), it should be clear how multi-file torrents could be supported.


[1] R. C. Merkle. Protocols for public key cryptosystems. In Symposium
on Security and Privacy, 122-134, 1980.

----- Original Message -----

From: "Nick Johnson" <arachnid@notdot.net>
To: "Peer-to-peer development." <p2p-hackers@zgp.org>
Subject: Re: [p2p-hackers] [connellybarnes@yahoo.com: [i2p] Python filesharing	Merkle hash tree magic]
Date: Sun, 15 May 2005 11:26:37 +1200

> 
> How does this work? You don't provide details here or in the .py file - does 
> each chunk include the hashes of all its siblings, and its parents' siblings, 
> etc?
> 
> -Nick Johnson
> 
> Eugen Leitl wrote:
> 
> > ----- Forwarded message from "C. Barnes" <connellybarnes@yahoo.com> -----
> >
> > From: "C. Barnes" <connellybarnes@yahoo.com>
> > Date: Thu, 12 May 2005 04:36:32 -0700 (PDT)
> > To: i2p@i2p.net
> > Subject: [i2p] Python filesharing Merkle hash tree magic
> >
> >
> > Topic: P2P filesharing ala BitTorrent.
> >
> > Presenting public domain module 'chunk.py':
> >
> >  http://oregonstate.edu/~barnesc/temp/chunk.py
> >
> > This module does some magic with Merkle hash trees.
> > It allows one to publish a 20 byte SHA-1 digest for
> > a file, and encode 'chunks' from the file with
> > minimal overhead, such that any given chunk can be
> > verified against the file's 20 byte digest.
> > Cryptographic hashing is used to verify each chunk,
> > so chunks can be transmitted in any order, and it is
> > impossible to save corrupt data to disk if one has
> > the correct file hash.
> >
> > One could use this module in any file sharing
> > application which divides files up into blocks.
> >
> > For example, in a BitTorrent-like project it might
> > be desirable to keep the .torrent files minimal in
> > size.  Using this module, one needs to only publish
> > a hash like DA39A3EE5E6B4B0D3255BFEF95601890AFD80709
> > instead of a digest of all blocks in the file.
> >
> > Pros:
> > - Smaller digest size.
> > - Can send chunks in any order.
> >
> > Cons:
> > - Each 'chunk' contains checksumming information.
> >   This overhead is less than 4% for files of size
> >   less than 1e20 bytes.
> >
> > - Connelly Barnes
> >
> >
> >
> > 		
> > __________________________________ Yahoo! Mail Mobile Take Yahoo! Mail with 
> > you! Check email on your mobile phone. http://mobile.yahoo.com/learn/mail 
> > _______________________________________________
> > i2p mailing list
> > i2p@i2p.net
> > http://i2p.dnsalias.net/mailman/listinfo/i2p
> >
> > ----- End forwarded message -----
> > _______________________________________________
> > p2p-hackers mailing list
> > p2p-hackers@zgp.org
> > http://zgp.org/mailman/listinfo/p2p-hackers
> > _______________________________________________
> > Here is a web page listing P2P Conferences:
> > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences
> >
> > !DSPAM:4285c8e4292214151599692!
> >
> >
> >
> 
> _______________________________________________
> p2p-hackers mailing list
> p2p-hackers@zgp.org
> http://zgp.org/mailman/listinfo/p2p-hackers
> _______________________________________________
> Here is a web page listing P2P Conferences:
> http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences


From verdimar at comp.nus.edu.sg  Tue May 17 09:53:49 2005
From: verdimar at comp.nus.edu.sg (Verdi March)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] Skipgraph and global balancing operations
Message-ID: <Pine.GSO.4.58.0505171750300.21946@sf3.comp.nus.edu.sg>

Hi,

in the skipgraph paper [1], in section 2 it's mentioned that
skip lists require no global balancing operations.

I'm not clear at the "global balancing operations" statement:
what are the operations and why skip lists don't need those.

Any help is appreciated.


Regards,
Verdi


From verdimar at comp.nus.edu.sg  Tue May 17 09:56:19 2005
From: verdimar at comp.nus.edu.sg (Verdi March)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] Re: Skipgraph and global balancing operations
In-Reply-To: <Pine.GSO.4.58.0505171750300.21946@sf3.comp.nus.edu.sg>
References: <Pine.GSO.4.58.0505171750300.21946@sf3.comp.nus.edu.sg>
Message-ID: <Pine.GSO.4.58.0505171754190.21946@sf3.comp.nus.edu.sg>

Forgot the reference, but I'm sure most of you have already
know that :).

[1] James Aspnes and Gauri Shah, Skip Graphs, Proceedings of the 14th
Annual ACM-SIAM Symposium on Discrete Algorithms, January 2003.

Regards,
Verdi


From wolfgang.mueller at wiai.uni-bamberg.de  Tue May 17 12:08:13 2005
From: wolfgang.mueller at wiai.uni-bamberg.de (Wolfgang =?iso-8859-1?q?M=FCller?=)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] Rumorama: Scalable Summary Based Retrieval in P2P
	Networks
Message-ID: <200505171408.13191.wolfgang.mueller@wiai.uni-bamberg.de>

Dear colleagues,

In the tech report 
http://mi.wiai.uni-bamberg.de/research/publications/bibtex/mueller2005scalable 
we present Rumorama, a protocol to make summary-based retrieval in P2P 
networks scalable. 

Summary-based retrieval in P2P networks, as introduced by PlanetP, uses 
summaries of peer data to select peers to contact for a retrieval process. In 
other words, PlanetP adapts GlOSS-like distributed IR techniques to P2P 
networks. The advantage of this method is that only very little indexing data 
has to be shipped as compared to distributed indexing structures, giving room  
for redundant storage of these summaries. Furthermore, summary-based 
retrieval needs to ship orders of magnitude less data in some situations, 
e.g. when processing multi-word boolean queries, than methods based on 
inverted lists (see e.g. the Li et al. paper on the feasibility of keyword 
search in P2P-networks for some words about the problems to be solved for 
inverted file based indexing 
http://pdos.csail.mit.edu/~rtm/papers/search_feasibility.ps). The drawback of 
summary-based retrieval is - up to Rumorama - that PlanetP is not scalable.

Rumorama achieves scalability: the number of neighbors of each peer grows 
logarithmically with the number of peers in the network, the communication 
cost per peer for maintaining the network also grows logarithmically with 
network size. Lastly, queries are processed in a number of hops (i.e. depth 
of our multicast tree) that grows logarithmically with the network's size. 

However, the number of nodes to be contacted within a query grows linearly 
with the network size. This is a known effect from distributed information 
retrieval. However, depending on the query model, the absolute number of 
peers that has to be contacted for answering a query is very small with 
respect to network size.

Enjoy the paper! We would be thankful for some feedback and criticism on it.

Cheers,
Wolfgang M?ller
Martin Eisenhardt
Andreas Henrich
-- 
Dr. Wolfgang M?ller
LS Medieninformatik
Universit?t Bamberg

From m.rogers at cs.ucl.ac.uk  Tue May 17 17:06:52 2005
From: m.rogers at cs.ucl.ac.uk (Michael Rogers)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] Skipgraph and global balancing operations
In-Reply-To: <Pine.GSO.4.58.0505171750300.21946@sf3.comp.nus.edu.sg>
References: <Pine.GSO.4.58.0505171750300.21946@sf3.comp.nus.edu.sg>
Message-ID: <428A24AC.5000208@cs.ucl.ac.uk>

I think it's referring to the fact that, unlike trees, skip lists don't 
need to be rebalanced as nodes are added and removed. AFAIK the reason 
is that the rank of each item in the list (ie the number of pointers the 
item keeps) is chosen randomly, so the higher "layers" are replaced 
incrementally as a natural result of churn.

Cheers,
Michael

Verdi March wrote:

>Hi,
>
>in the skipgraph paper [1], in section 2 it's mentioned that
>skip lists require no global balancing operations.
>
>I'm not clear at the "global balancing operations" statement:
>what are the operations and why skip lists don't need those.
>
>Any help is appreciated.
>
>
>Regards,
>Verdi
>
>_______________________________________________
>p2p-hackers mailing list
>p2p-hackers@zgp.org
>http://zgp.org/mailman/listinfo/p2p-hackers
>_______________________________________________
>Here is a web page listing P2P Conferences:
>http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences
>  
>


From kerry at vscape.com  Tue May 17 18:53:57 2005
From: kerry at vscape.com (Kerry L. Bonin)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] Multiple value per key query/response syntax?
Message-ID: <428A3DC5.6090106@vscape.com>

Hello!

I'm looking for any good discussions or examples of reasonably robust 
request/response formats for multiple transaction multimap queries in 
general, and multiple-value per key DHT in particular. 

I was considering using expiration timestamp windowing in the query as a 
sort of crude iterator filter for values that expire at a constant 
interval after storage or refresh, but I'm wondering about variable TTL 
vs. variable refresh intervals, vs. classic remote iterator design.

Any pointers would be greatly appreciated!

Kerry


From tangli03 at mails.tsinghua.edu.cn  Wed May 18 09:55:42 2005
From: tangli03 at mails.tsinghua.edu.cn (Li Tang)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] Has CAN been implemented by any open source project?
In-Reply-To: <316355152.11145@mails.tsinghua.edu.cn>
Message-ID: <20050518094753.CB1863FC2A@capsicum.zgp.org>

Hello, guys!

   I am looking for some open source resources of DHT algorithms. While I
have found projects related to Chord, Pastry, and Tapestry, I got little
implementation of CAN. 
   Is anybody aware of related information? Any pointers, not limited to
CAN, would be greatly appreciated!

Sincerely,
Li



From baford at mit.edu  Wed May 18 18:59:21 2005
From: baford at mit.edu (Bryan Ford)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] Official IETF behavior recommendations for NAT
	relevant to P2P
Message-ID: <200505181459.21722.baford@mit.edu>

Dear p2p-hackers,

A discussion is currently taking place on the IETF BEHAVE working group 
mailing list (see http://www1.ietf.org/html.charters/behave-charter.html) 
that is highly relevant to P2P application writers.  For those of you not 
already familiar with it, this IETF working group is defining standards for 
how NATs should behave so as to better support applications including P2P 
apps.  The chair has made a "Last Call" for comments on the current draft 
document specifying NAT behavior for UDP, so this may be the last chance we 
have for a while to influence how well the next generation of NATs support 
UDP-based P2P applications.

** Address-Restricted vs Full Cone NAT **

There's one issue in particular on which I personally fear the working group 
is currently headed in the wrong direction, and over which you folks on this 
list could perhaps exert considerable influence if you're willing to take a 
little time and make your voice heard on the BEHAVE list.  The current UDP 
draft, draft-ietf-behave-nat-udp-01.txt, recommends that NATs implement 
"Endpoint address-dependent filtering behavior", otherwise known as 
"Address-Restricted Cone NAT".  In other words, once a NATted UDP application 
opens a UDP session through a NAT to some external host, giving the 
application a public (NATted) port number, the NAT only accepts and forwards 
subsequent incoming UDP packets at that public port if they come from a 
source IP address that the application has previously sent UDP packets to.  
In other words, the NAT must see outgoing UDP traffic to a remote host before 
it will accept incoming traffic from that host on the application's public 
port.  In the proposed alternative, "Endpoint-independent filtering" or "Full 
Cone NAT", the NAT forwards any incoming traffic to the correct public port 
regardless of source IP address or port number.

Address-Restricted Cone NAT is moderately "P2P-friendly" in that it at least 
allows NATted applications to "hole punch" P2P connections with the help of a 
well-known (non-NATted) rendezvous server.  Address-Restricted Cone NAT still 
does not, however, allow the application to have a full "first-class" 
Internet presence (i.e., public IP address and UDP port number) at which it 
can be contacted proactively by any other Internet-connected host.  A P2P 
application behind an Address-Restricted Cone NAT is still forever dependent 
on other public (first-class) rendezvous servers to help establish each P2P 
communication session it needs.  A P2P app behind a Full Cone NAT, in 
contrast, is a full "first-class" network citizen itself, and can even _be_ a 
public rendezvous server for other less fortunate hosts once it learns its 
own public IP address and UDP port number.  This is an important benefit to 
applications, especially P2P applications that are trying to be highly 
decentralized.

The main argument for Restricted Cone NAT, of course, is security.  Until 
recently I believed this argument myself, but after further and more careful 
analysis I have come to the conclusion that Restricted filtering provides a 
very week security benefit in comparison with the fundamental loss in 
first-class application connectivity it entails.  My current proposal on the 
BEHAVE list does NOT mandate Full Cone NAT, but it does specify that NATs 
should at least be Full Cone in their default, "out of the box" configuration 
as provided by NAT vendors, and recommends that NATs implement Restricted 
Cone NAT as an administrator-configurable security option.  The current 
proposed text reads as follows:

   REQ-7: NATs MUST implement Endpoint-Independent
          Filtering in their default configuration.
          NATs SHOULD be configurable to support
          Address-Specific Filtering or Endpoint-Specific Filtering
          at the option of the system administrator.

If you have thoughts or opinions on this topic, I would greatly appreciate 
hearing them (and having other participants hear them) on the BEHAVE list.

** Why the Security Benefit of Restricted Cone NAT is Minimal **

Restrictive filtering ostensibly has the benefit of protecting applications on 
an internal host from traffic the internal host is not expecting, i.e., from 
traffic from previously unknown external hosts or endpoints.  This behavior 
can only possibly benefit the application if the application only ever 
initiates outgoing sessions from its internal port, and is obviously a curse 
if the application ever wants to accept incoming sessions from external 
hosts.  The NAT has no way of knowing which is the case.  But suppose the 
application is one that only initiates outgoing connections (i.e., a 
"traditional client/server app"), and thus might "potentially" benefit from a 
NAT's restrictive filtering behavior.  Consider TCP and UDP separately:

        ** TCP: When a client/server app makes an outgoing TCP connection, it 
typically does a "connect()" call to the host OS's sockets API without 
bothering to bind() the socket, and the OS's TCP stack allocates a local TCP 
port exclusively for the use of that connection.  Any TCP packets the host OS 
subsequently received on that port from endpoints other than the one it 
connect()ed to will be summarily rejected with TCP RSTs by the host OS, and 
the application never even sees any hint that they ever arrived.  Even if the 
application does a bind() before its connect() in order to use a specific 
source port (e.g., a <1024 "privileged" port), it still almost always obtains 
exclusive use of that port.  An application indeed has to work really bloody 
hard to set things up so that it can accept incoming TCP connections on the 
same port it is using for an outgoing connection, using SO_REUSEADDR and 
multiple sockets.  No traditional client/server application is going to go to 
all this effort just in order to create a security hole. :)  In other words, 
for TCP, restrictive filtering in practice provides absolutely zero 
protection from the _application_: the only entity that the restrictive 
filtering even affects is the host OS's protocol stack.  Restrictive 
filtering might for example benefit a host OS with a TCP stack that is so 
broken that it fails to check the source IP address and port number on 
incoming TCP packets at all, but if the host OS is _that_ broken then it 
undoubtedly has many other problems that the NAT can't possibly guard 
against.

        ** UDP: When a client/server UDP app uses a UDP port for communication 
with just one remote host, for convenience it typically does a connect() to 
the appropriate remote UDP endpoint, so that it can just send messages with 
send() or write() without having to specify a destination address for each 
datagram, and so that it can receive messages with read() or recv() without 
having to check the source address and port number in each message received.  
In this case, the host OS effectively provides the external endpoint 
filtering functionality as for TCP, and any restrictive filtering implemented 
by the NAT again provides no security benefit at all to the application.  A 
UDP application that uses a single UDP port for communication with multiple 
remote hosts, in contrast, must generally check the source IP addresses and 
port numbers in the datagrams it receives just in order to distinguish 
between the multiple communication sessions it is involved in, and hence it 
will inherently provide its own external endpoint filtering.  It's 
conceivable that some abysmally buggy UDP applications might benefit from the 
NAT's external filtering - e.g., a UDP application that only talks to one 
external endpoint but doesn't use connect() and doesn't check the source 
endpoint on incoming datagrams - but an application that is so seriously 
broken probably has many other security vulnerabilities that can be exploited 
despite the NAT's external filtering behavior, e.g., by finding out or 
guessing the public server the application is talking to (which is easy in 
many cases such as commercial games or services where there are only a few 
well-known servers) and sending packets with spoofed source IP address and 
UDP port numbers.  Besides, many, perhaps the majority, of UDP applications 
use UDP _because_ they need P2P-style communication, and so external 
filtering by the NAT provides no practical security benefit and only causes 
interference with the application.

Another common argument for Restricted Cone NAT (Address-dependent filtering) 
is that it protects the resources on the internal network from being consumed 
forwarding bogus packets even if the eventual end application will correctly 
discard them.  This argument is also weak, however:

	** Even with more permissive Full Cone NAT (endpoint-independent filtering), 
the NAT will still in practice filter out the vast majority of bogus traffic 
that reaches it, because most of the NAT's ports are typically not in active 
use by applications behind the NAT.  The only bogus traffic a Full Cone NAT 
will let through is bogus traffic that happens to hit the _specific_ external 
port numbers that correspond to active port bindings.  These port numbers are 
assigned more-or-less randomly (i.e., unpredictably) by the NAT, and they only 
ever appear in the high, non-well-known port number space, so most of the 
Internet "background noise" caused by viruses or other attacks on well-known 
ports will never get through in any case, and purely random "port guessing" 
attacks would have to send packets to a great many NAT ports in order to get 
a few packets through.  Thus, the incremental benefit of restrictive 
filtering over Full Cone NAT with regards to general bogus Internet traffic 
is very small.

	** An attacker might deliberately send bogus packets to a bound NAT port 
known to be in use.  But if the attacker can find out which NAT port the 
application is using, then the attacker probably also knows (or can easily 
find out or guess) the address and port number of the _remote_ host that the 
application is talking to and just spoof that source IP address and port 
number in its attack packets, getting around Restricted filtering as well.  
In fact the remote host's endpoint is in practice likely to be much easier 
for the attacker to find out or guess than the application's public NAT port, 
because of the overwhelming prevalence of well-known servers and ports on the 
Internet in comparison to the randomness and relative obscurity of temporary 
NAT-assigned public ports.  So the practical benefit that Restricted 
filtering provides even against directed attacks is still minimal.

	** Finally, the goal of "protecting the resources on the internal network 
from being consumed forwarding bogus packets" is really only a consideration 
on large corporate or ISP networks with substantial internal traffic load, 
and if this minimization of resource use is a concern then the network 
administrator can certainly be expected to know how to configure the 
company's NAT to enable Restricted filtering.  As I pointed out above, under 
my proposal a NAT can use Restricted filtering and still be BEHAVE-compliant 
as long as it is the administrator (not the NAT vendor) that enables this 
firewall behavior.

	** For home NATs, in contrast, minimizing resource load on the internal 
network due to bogus traffic is completely a non-issue because the home 
network (Ethernet or 802.11 or whatever) is typically so much faster than the 
Internet uplink anyway.  In this case it's much more important that NATs 
consistently support applications well, which my proposal ensures by 
requiring that the NAT's "out of the box" configuration is 
Endpoint-independent Filtering.  Joe Clueless who buys a home NAT router 
probably doesn't have any idea what NAT actually is, but he wants his 
applications to work.

** Conclusion **

In summary, restrictive filtering in the NAT interferes with applications 
substantially and provides very little real security benefit in 
practice, considering the way application communication and NAT port 
assignment actually works.  As such, especially considering the purpose of 
the BEHAVE working group is to increase the predictability and compatibility 
of NATs with applications and not to specify standards for firewall 
functionality, it seems highly inappropriate for the WG's documents to 
recommend firewall functionality that interferes with applications while 
providing dubious security benefits.  On the contrary, I think the WG should 
at least recommend, if not mandate, that NATs provide Endpoint-Independent 
Filtering (Full Cone NAT) by default.  NAT vendors or system administrators 
that are particularly paranoid about security and perceive the security 
benefits to be worth the interference it causes with applications can still 
enable more restrictive filtering policies, but that's a firewall deployment 
issue, not a NAT functionality issue.  We need to steer clear from 
recommending firewall functionality.

Thanks very much for your attention.  Please direct followups to the BEHAVE 
working group list, "ietf-behave@list.sipfoundry.org".  Thanks!

Bryan

From larytet.8753341 at bloglines.com  Wed May 18 21:07:56 2005
From: larytet.8753341 at bloglines.com (larytet.8753341@bloglines.com)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] Official IETF behavior recommendations for NAT
	relevant to P2P
Message-ID: <1116450476.3435399312.6473.sendItem@bloglines.com>

dear Bryan, feel free to forward this email to IETF.

best regrads, Arkady



----------------------------------------------
hi,
i am a principle
developer
of Rodi project http://larytet.sourceforge.net/btRat.shtml 

i fully support
opinion expressed in this email http://zgp.org/pipermail/p2p-hackers/2005-May/002611.html



i think that it's a right thing to do to let (by default) packets from

any peer in the outside world to go through "hole" in NAT in case of UDP.




Arkady



--- Peer-to-peer development." <p2p-hackers@zgp.org wrote:

Dear p2p-hackers,
> 
> A discussion is currently taking place on the IETF
BEHAVE working group 
> mailing list (see http://www1.ietf.org/html.charters/behave-charter.html)

> that is highly relevant to P2P application writers.  For those of you
not 
> already familiar with it, this IETF working group is defining standards
for 
> how NATs should behave so as to better support applications including
P2P 
> apps.  The chair has made a "Last Call" for comments on the current
draft 
> document specifying NAT behavior for UDP, so this may be the last
chance we 
> have for a while to influence how well the next generation of
NATs support 
> UDP-based P2P applications.
> 
> ** Address-Restricted
vs Full Cone NAT **
> 
> There's one issue in particular on which I personally
fear the working group 
> is currently headed in the wrong direction, and
over which you folks on this 
> list could perhaps exert considerable influence
if you're willing to take a 
> little time and make your voice heard on the
BEHAVE list.  The current UDP 
> draft, draft-ietf-behave-nat-udp-01.txt,
recommends that NATs implement 
> "Endpoint address-dependent filtering behavior",
otherwise known as 
> "Address-Restricted Cone NAT".  In other words, once
a NATted UDP application 
> opens a UDP session through a NAT to some external
host, giving the 
> application a public (NATted) port number, the NAT only
accepts and forwards 
> subsequent incoming UDP packets at that public port
if they come from a 
> source IP address that the application has previously
sent UDP packets to.  
> In other words, the NAT must see outgoing UDP traffic
to a remote host before 
> it will accept incoming traffic from that host
on the application's public 
> port.  In the proposed alternative, "Endpoint-independent
filtering" or "Full 
> Cone NAT", the NAT forwards any incoming traffic to
the correct public port 
> regardless of source IP address or port number.

> 
> Address-Restricted Cone NAT is moderately "P2P-friendly" in that it
at least 
> allows NATted applications to "hole punch" P2P connections with
the help of a 
> well-known (non-NATted) rendezvous server.  Address-Restricted
Cone NAT still 
> does not, however, allow the application to have a full
"first-class" 
> Internet presence (i.e., public IP address and UDP port
number) at which it 
> can be contacted proactively by any other Internet-connected
host.  A P2P 
> application behind an Address-Restricted Cone NAT is still
forever dependent 
> on other public (first-class) rendezvous servers to
help establish each P2P 
> communication session it needs.  A P2P app behind
a Full Cone NAT, in 
> contrast, is a full "first-class" network citizen
itself, and can even _be_ a 
> public rendezvous server for other less fortunate
hosts once it learns its 
> own public IP address and UDP port number.  This
is an important benefit to 
> applications, especially P2P applications that
are trying to be highly 
> decentralized.
> 
> The main argument for Restricted
Cone NAT, of course, is security.  Until 
> recently I believed this argument
myself, but after further and more careful 
> analysis I have come to the
conclusion that Restricted filtering provides a 
> very week security benefit
in comparison with the fundamental loss in 
> first-class application connectivity
it entails.  My current proposal on the 
> BEHAVE list does NOT mandate Full
Cone NAT, but it does specify that NATs 
> should at least be Full Cone in
their default, "out of the box" configuration 
> as provided by NAT vendors,
and recommends that NATs implement Restricted 
> Cone NAT as an administrator-configurable
security option.  The current 
> proposed text reads as follows:
> 
> 
  REQ-7: NATs MUST implement Endpoint-Independent
>           Filtering in
their default configuration.
>           NATs SHOULD be configurable to support

>           Address-Specific Filtering or Endpoint-Specific Filtering
>
          at the option of the system administrator.
> 
> If you have thoughts
or opinions on this topic, I would greatly appreciate 
> hearing them (and
having other participants hear them) on the BEHAVE list.
> 
> ** Why the
Security Benefit of Restricted Cone NAT is Minimal **
> 
> Restrictive filtering
ostensibly has the benefit of protecting applications on 
> an internal host
from traffic the internal host is not expecting, i.e., from 
> traffic from
previously unknown external hosts or endpoints.  This behavior 
> can only
possibly benefit the application if the application only ever 
> initiates
outgoing sessions from its internal port, and is obviously a curse 
> if
the application ever wants to accept incoming sessions from external 
> hosts.
 The NAT has no way of knowing which is the case.  But suppose the 
> application
is one that only initiates outgoing connections (i.e., a 
> "traditional
client/server app"), and thus might "potentially" benefit from a 
> NAT's
restrictive filtering behavior.  Consider TCP and UDP separately:
> 
> 
       ** TCP: When a client/server app makes an outgoing TCP connection,
it 
> typically does a "connect()" call to the host OS's sockets API without

> bothering to bind() the socket, and the OS's TCP stack allocates a local
TCP 
> port exclusively for the use of that connection.  Any TCP packets
the host OS 
> subsequently received on that port from endpoints other than
the one it 
> connect()ed to will be summarily rejected with TCP RSTs by
the host OS, and 
> the application never even sees any hint that they ever
arrived.  Even if the 
> application does a bind() before its connect() in
order to use a specific 
> source port (e.g., a <1024 "privileged" port),
it still almost always obtains 
> exclusive use of that port.  An application
indeed has to work really bloody 
> hard to set things up so that it can
accept incoming TCP connections on the 
> same port it is using for an outgoing
connection, using SO_REUSEADDR and 
> multiple sockets.  No traditional client/server
application is going to go to 
> all this effort just in order to create
a security hole. :)  In other words, 
> for TCP, restrictive filtering in
practice provides absolutely zero 
> protection from the _application_: the
only entity that the restrictive 
> filtering even affects is the host OS's
protocol stack.  Restrictive 
> filtering might for example benefit a host
OS with a TCP stack that is so 
> broken that it fails to check the source
IP address and port number on 
> incoming TCP packets at all, but if the
host OS is _that_ broken then it 
> undoubtedly has many other problems that
the NAT can't possibly guard 
> against.
> 
>         ** UDP: When a client/server
UDP app uses a UDP port for communication 
> with just one remote host, for
convenience it typically does a connect() to 
> the appropriate remote UDP
endpoint, so that it can just send messages with 
> send() or write() without
having to specify a destination address for each 
> datagram, and so that
it can receive messages with read() or recv() without 
> having to check
the source address and port number in each message received.  
> In this
case, the host OS effectively provides the external endpoint 
> filtering
functionality as for TCP, and any restrictive filtering implemented 
> by
the NAT again provides no security benefit at all to the application.  A 

> UDP application that uses a single UDP port for communication with multiple

> remote hosts, in contrast, must generally check the source IP addresses
and 
> port numbers in the datagrams it receives just in order to distinguish

> between the multiple communication sessions it is involved in, and hence
it 
> will inherently provide its own external endpoint filtering.  It's

> conceivable that some abysmally buggy UDP applications might benefit from
the 
> NAT's external filtering - e.g., a UDP application that only talks
to one 
> external endpoint but doesn't use connect() and doesn't check the
source 
> endpoint on incoming datagrams - but an application that is so
seriously 
> broken probably has many other security vulnerabilities that
can be exploited 
> despite the NAT's external filtering behavior, e.g.,
by finding out or 
> guessing the public server the application is talking
to (which is easy in 
> many cases such as commercial games or services where
there are only a few 
> well-known servers) and sending packets with spoofed
source IP address and 
> UDP port numbers.  Besides, many, perhaps the majority,
of UDP applications 
> use UDP _because_ they need P2P-style communication,
and so external 
> filtering by the NAT provides no practical security benefit
and only causes 
> interference with the application.
> 
> Another common
argument for Restricted Cone NAT (Address-dependent filtering) 
> is that
it protects the resources on the internal network from being consumed 
>
forwarding bogus packets even if the eventual end application will correctly

> discard them.  This argument is also weak, however:
> 
> 	** Even with
more permissive Full Cone NAT (endpoint-independent filtering), 
> the NAT
will still in practice filter out the vast majority of bogus traffic 
> that
reaches it, because most of the NAT's ports are typically not in active 
> use by applications behind the NAT.  The only bogus traffic a Full Cone
NAT 
> will let through is bogus traffic that happens to hit the _specific_
external 
> port numbers that correspond to active port bindings.  These
port numbers are 
> assigned more-or-less randomly (i.e., unpredictably)
by the NAT, and they only 
> ever appear in the high, non-well-known port
number space, so most of the 
> Internet "background noise" caused by viruses
or other attacks on well-known 
> ports will never get through in any case,
and purely random "port guessing" 
> attacks would have to send packets to
a great many NAT ports in order to get 
> a few packets through.  Thus, the
incremental benefit of restrictive 
> filtering over Full Cone NAT with regards
to general bogus Internet traffic 
> is very small.
> 
> 	** An attacker
might deliberately send bogus packets to a bound NAT port 
> known to be
in use.  But if the attacker can find out which NAT port the 
> application
is using, then the attacker probably also knows (or can easily 
> find out
or guess) the address and port number of the _remote_ host that the 
> application
is talking to and just spoof that source IP address and port 
> number in
its attack packets, getting around Restricted filtering as well.  
> In fact
the remote host's endpoint is in practice likely to be much easier 
> for
the attacker to find out or guess than the application's public NAT port,

> because of the overwhelming prevalence of well-known servers and ports
on the 
> Internet in comparison to the randomness and relative obscurity
of temporary 
> NAT-assigned public ports.  So the practical benefit that
Restricted 
> filtering provides even against directed attacks is still minimal.

> 
> 	** Finally, the goal of "protecting the resources on the internal
network 
> from being consumed forwarding bogus packets" is really only a
consideration 
> on large corporate or ISP networks with substantial internal
traffic load, 
> and if this minimization of resource use is a concern then
the network 
> administrator can certainly be expected to know how to configure
the 
> company's NAT to enable Restricted filtering.  As I pointed out above,
under 
> my proposal a NAT can use Restricted filtering and still be BEHAVE-compliant

> as long as it is the administrator (not the NAT vendor) that enables this

> firewall behavior.
> 
> 	** For home NATs, in contrast, minimizing resource
load on the internal 
> network due to bogus traffic is completely a non-issue
because the home 
> network (Ethernet or 802.11 or whatever) is typically
so much faster than the 
> Internet uplink anyway.  In this case it's much
more important that NATs 
> consistently support applications well, which
my proposal ensures by 
> requiring that the NAT's "out of the box" configuration
is 
> Endpoint-independent Filtering.  Joe Clueless who buys a home NAT router

> probably doesn't have any idea what NAT actually is, but he wants his

> applications to work.
> 
> ** Conclusion **
> 
> In summary, restrictive
filtering in the NAT interferes with applications 
> substantially and provides
very little real security benefit in 
> practice, considering the way application
communication and NAT port 
> assignment actually works.  As such, especially
considering the purpose of 
> the BEHAVE working group is to increase the
predictability and compatibility 
> of NATs with applications and not to
specify standards for firewall 
> functionality, it seems highly inappropriate
for the WG's documents to 
> recommend firewall functionality that interferes
with applications while 
> providing dubious security benefits.  On the contrary,
I think the WG should 
> at least recommend, if not mandate, that NATs provide
Endpoint-Independent 
> Filtering (Full Cone NAT) by default.  NAT vendors
or system administrators 
> that are particularly paranoid about security
and perceive the security 
> benefits to be worth the interference it causes
with applications can still 
> enable more restrictive filtering policies,
but that's a firewall deployment 
> issue, not a NAT functionality issue.
 We need to steer clear from 
> recommending firewall functionality.
> 

> Thanks very much for your attention.  Please direct followups to the BEHAVE

> working group list, "ietf-behave@list.sipfoundry.org".  Thanks!
> 
>
Bryan
> _______________________________________________
> p2p-hackers mailing
list
> p2p-hackers@zgp.org
> http://zgp.org/mailman/listinfo/p2p-hackers

> _______________________________________________
> Here is a web page listing
P2P Conferences:
> http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences

> 

From justin at specialbusservice.com  Wed May 18 22:59:34 2005
From: justin at specialbusservice.com (Justin Cormack)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] Official IETF behavior recommendations for NAT
	relevant to P2P
In-Reply-To: <200505181459.21722.baford@mit.edu>
References: <200505181459.21722.baford@mit.edu>
Message-ID: <2D6F41F6-AA3A-438F-9764-0E54FB18E62E@specialbusservice.com>


On 18 May 2005, at 19:59, Bryan Ford wrote:

> Dear p2p-hackers,
>
> A discussion is currently taking place on the IETF BEHAVE working  
> group
> mailing list (see http://www1.ietf.org/html.charters/behave- 
> charter.html)


(snip)

How current is this? These are the people who loooked at STUn and  
decided it wasnt that much use, and are due to wind up next January.

(not on the list but just asking)

Not sure what the issue is with TCP - surely it just doesnt work, you  
cant get the sequence numbers to work.

For UDP, well its easy to spoof anyway, The main problem with not  
using full cone is that it restricts the number of connections that  
you can NAT (and given the lack of timeouts on UDP this does impact  
many applications). Asking for non full cone to be default is thus a  
bit broken. But all the makers of dumb boxes are doing it anyway.  
Since when have NAT vendors cared about the IETF? Isnt it only Linux/ 
BSD based boxes that do full cone anyway?

Given that l33t hackers are already giving owned machines ipv6  
addresses anyway I am not ure it is an issue...

j


From travis at redswoosh.net  Wed May 18 23:05:50 2005
From: travis at redswoosh.net (Travis Kalanick)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] Official IETF behavior recommendations for
	NATrelevant to P2P
In-Reply-To: <2D6F41F6-AA3A-438F-9764-0E54FB18E62E@specialbusservice.com>
Message-ID: <200505182310.j4INADaL001505@be9.noc0.redswoosh.com>

Couldn't have said it better myself. . .

The NAT situation is something we can all build around today, and it will
only get easier to deal with over time by the forces already in play.

Travis


-----Original Message-----
From: p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org] On
Behalf Of Justin Cormack
Sent: Wednesday, May 18, 2005 4:00 PM
To: Peer-to-peer development.
Subject: Re: [p2p-hackers] Official IETF behavior recommendations for
NATrelevant to P2P


On 18 May 2005, at 19:59, Bryan Ford wrote:

> Dear p2p-hackers,
>
> A discussion is currently taking place on the IETF BEHAVE working  
> group
> mailing list (see http://www1.ietf.org/html.charters/behave- 
> charter.html)


(snip)

How current is this? These are the people who loooked at STUn and  
decided it wasnt that much use, and are due to wind up next January.

(not on the list but just asking)

Not sure what the issue is with TCP - surely it just doesnt work, you  
cant get the sequence numbers to work.

For UDP, well its easy to spoof anyway, The main problem with not  
using full cone is that it restricts the number of connections that  
you can NAT (and given the lack of timeouts on UDP this does impact  
many applications). Asking for non full cone to be default is thus a  
bit broken. But all the makers of dumb boxes are doing it anyway.  
Since when have NAT vendors cared about the IETF? Isnt it only Linux/ 
BSD based boxes that do full cone anyway?

Given that l33t hackers are already giving owned machines ipv6  
addresses anyway I am not ure it is an issue...

j

_______________________________________________
p2p-hackers mailing list
p2p-hackers@zgp.org
http://zgp.org/mailman/listinfo/p2p-hackers
_______________________________________________
Here is a web page listing P2P Conferences:
http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences


From srhea at cs.berkeley.edu  Wed May 18 23:30:31 2005
From: srhea at cs.berkeley.edu (Sean C. Rhea)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] Multiple value per key query/response syntax?
In-Reply-To: <428A3DC5.6090106@vscape.com>
References: <428A3DC5.6090106@vscape.com>
Message-ID: <7f4192482ce8d14c05a8d220c5026b69@cs.berkeley.edu>

On May 17, 2005, at 11:53 AM, Kerry L. Bonin wrote:
> I'm looking for any good discussions or examples of reasonably robust 
> request/response formats for multiple transaction multimap queries in 
> general, and multiple-value per key DHT in particular.
> I was considering using expiration timestamp windowing in the query as 
> a sort of crude iterator filter for values that expire at a constant 
> interval after storage or refresh, but I'm wondering about variable 
> TTL vs. variable refresh intervals, vs. classic remote iterator 
> design.

In OpenDHT, we allow multiple values per key.  If two puts come in with 
the same key, we store them both; a get returns all values.  If all 
values don't fit in a single packet, we return only a packet's worth, 
along with a "placemark" that can be used to retrieve the next set.  
This placemark works just like an iterator.

The way this is implemented is that we return values for gets in the 
order of their SHA-1 hashes.  The placemark returned is just the SHA of 
the last value returned.  To get the next set, we just skip over all 
values whose SHA is less than or equal to the placemark in the get 
handling code.

If a put occurs during iteration, it just falls somewhere in this 
sequence, and may or may not be returned to the client.  If the client 
starts a new get with placemark 0, however, it will be returned.

I hope that made sense,
Sean
-- 
   To announce that there must be no criticism of the president or that 
we
  are to stand by the president right or wrong is not only unpatriotic 
and
         servile but is morally treasonable to the American public.
                            -- Theodore Roosevelt

-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 186 bytes
Desc: This is a digitally signed message part
Url : http://zgp.org/pipermail/p2p-hackers/attachments/20050518/1fff95c8/PGP.pgp
From srhea at cs.berkeley.edu  Wed May 18 23:35:35 2005
From: srhea at cs.berkeley.edu (Sean C. Rhea)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] Has CAN been implemented by any open source project?
In-Reply-To: <20050518094753.CB1863FC2A@capsicum.zgp.org>
References: <20050518094753.CB1863FC2A@capsicum.zgp.org>
Message-ID: <4595e3677475ae9fa1de7d7a1c3979a3@cs.berkeley.edu>

On May 18, 2005, at 2:55 AM, Li Tang wrote:
>    I am looking for some open source resources of DHT algorithms. 
> While I
> have found projects related to Chord, Pastry, and Tapestry, I got 
> little
> implementation of CAN.
>    Is anybody aware of related information? Any pointers, not limited 
> to
> CAN, would be greatly appreciated!

There was never a CAN implementation (outside of simulation) by the CAN 
authors.  Several other groups produced implementations of CAN for the 
purposes of writing papers, but no one ever produced a full 
implementation AFAIK.  (None of them handled node failure, for 
example.)

Sean
-- 
     Once, someone asked me what pleasure I took in riding for so long.
       'Pleasure?' I said. 'I don't understand the question.' I didn't
                   do it for pleasure. I did it for pain.
                             -- Lance Armstrong

-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 186 bytes
Desc: This is a digitally signed message part
Url : http://zgp.org/pipermail/p2p-hackers/attachments/20050518/3439695c/PGP.pgp
From lemonobrien at yahoo.com  Wed May 18 23:46:12 2005
From: lemonobrien at yahoo.com (Lemon Obrien)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] Has CAN been implemented by any open source project?
In-Reply-To: <4595e3677475ae9fa1de7d7a1c3979a3@cs.berkeley.edu>
Message-ID: <20050518234612.76482.qmail@web53601.mail.yahoo.com>

i making a peer to peer system and did research on various systems of search; how best to do it. I may have implemented CAN; can someone decribe what CAN means and how its alogorythm.
 
thanks
lemon

"Sean C. Rhea" <srhea@cs.berkeley.edu> wrote:
On May 18, 2005, at 2:55 AM, Li Tang wrote:
> I am looking for some open source resources of DHT algorithms. 
> While I
> have found projects related to Chord, Pastry, and Tapestry, I got 
> little
> implementation of CAN.
> Is anybody aware of related information? Any pointers, not limited 
> to
> CAN, would be greatly appreciated!

There was never a CAN implementation (outside of simulation) by the CAN 
authors. Several other groups produced implementations of CAN for the 
purposes of writing papers, but no one ever produced a full 
implementation AFAIK. (None of them handled node failure, for 
example.)

Sean
-- 
Once, someone asked me what pleasure I took in riding for so long.
'Pleasure?' I said. 'I don't understand the question.' I didn't
do it for pleasure. I did it for pain.
-- Lance Armstrong

_______________________________________________
p2p-hackers mailing list
p2p-hackers@zgp.org
http://zgp.org/mailman/listinfo/p2p-hackers
_______________________________________________
Here is a web page listing P2P Conferences:
http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences


You don't get no juice unless you squeeze
Lemon Obrien, the Third.
		
---------------------------------
Do you Yahoo!?
 Make Yahoo! your home page   
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://zgp.org/pipermail/p2p-hackers/attachments/20050518/339e8c57/attachment.html
From ch_407 at rediffmail.com  Thu May 19 00:33:42 2005
From: ch_407 at rediffmail.com (chinmaya narayan pandey)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] Chord
Message-ID: <20050519003342.11538.qmail@webmail26.rediffmail.com>

 ?
hi 
is there any open source implementation of chord.



CHINMAYA
       PANDEY


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://zgp.org/pipermail/p2p-hackers/attachments/20050519/d692f5db/attachment.htm
From davidopp at gmail.com  Thu May 19 00:54:09 2005
From: davidopp at gmail.com (David Oppenheimer)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] Chord
In-Reply-To: <20050519003342.11538.qmail@webmail26.rediffmail.com>
References: <20050519003342.11538.qmail@webmail26.rediffmail.com>
Message-ID: <e041bec050518175439366db4@mail.gmail.com>

http://pdos.csail.mit.edu/chord/

click on "Downloads"

David

On 19 May 2005 00:33:42 -0000, chinmaya narayan pandey
<ch_407@rediffmail.com> wrote:
> 
> 
>   
> hi 
> is there any open source implementation of chord.
> 
> CHINMAYA
>       PANDEY
> 
> 
> 
> 
>  
> _______________________________________________
> p2p-hackers mailing list
> p2p-hackers@zgp.org
> http://zgp.org/mailman/listinfo/p2p-hackers
> _______________________________________________
> Here is a web page listing P2P Conferences:
> http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences
> 
> 
>

From toscano_moura at hotmail.com  Thu May 19 01:46:06 2005
From: toscano_moura at hotmail.com (Hermano Toscano Moura)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] Chord
In-Reply-To: <e041bec050518175439366db4@mail.gmail.com>
Message-ID: <BAY12-F155E6BBB31A1E1A36B3BC9F080@phx.gbl>

An HTML attachment was scrubbed...
URL: http://zgp.org/pipermail/p2p-hackers/attachments/20050519/a94649dc/attachment.html
From dbarrett at quinthar.com  Fri May 20 01:05:38 2005
From: dbarrett at quinthar.com (David Barrett)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] Official IETF behavior recommendations for
	NATrelevant to P2P
In-Reply-To: <200505181459.21722.baford@mit.edu>
Message-ID: <20050520010540.D91873FC99@capsicum.zgp.org>

Hi Bryan, I think your work in this area is simply the best, and I entirely
agree with your judgment here.

I'm a P2P developer working in the VoIP space, and I can't stress enough the
annoyance of address-restricted NATs.  Were I working in a "closed" space
(ala Skype) it'd be merely a nuisance.  However, due to the limitations of
open standards (notably SIP and RTP), working in the presence of NATs is
quite challenging.  I would enthusiastically welcome more standardized
behavior in terms of how to "punch holes" and how long I can count on these
holes remaining open.  Let me explain by way of example:


First, as you well know, before anyone can hope to contact me behind a NAT,
I must first figure out what the global (NAT'd) address is for each of my
local ports.  I do this with the help of some central server, such as with
STUN.  Once the hole has been "punched" I have no idea how long it will stay
open, so I need to send a "keep-alive" to my central server.  The net result
is that there is a very high cost to open even one UDP port behind a NAT,
and an ongoing cost to keep it open.  

So even with a full-cone NAT, there is an up-front and ongoing cost to
opening a UDP port on which I intend to receive data.  However, with a
full-cone NAT, the cost is purely spent by my client -- I needn't involve
anyone else (excepting the central server) in preparing to receive data than
my own instance of my own application.

In the case of an address-restricted NAT, however, not only must I pay the
cost of being behind a NAT (and pay not once, but again for each client who
communicates with me), everyone else who wishes to communicate with me must
pay as well.  

For the full-cone NAT, once I know my global (NAT'd) IP/port, I can use that
as my own -- anybody can contact me via that address, without special
behavior to account for me being behind a NAT.  This means NAT-ignorant
protocols (such as RTP, for example) can be received from behind NATs,
without any special programming on the part of the sender.

However, for address-restricted NATs, this is not the case.  Imagine you
want to send me an RTP stream, and I am behind a NAT.  With a full-cone NAT,
I could send you my global IP/port (via SIP and SDP), and you could open an
RTP stream direct back to me.  But with an address-restricted NAT, this is
not enough.  The reason is a bit involved, but please bear with me.


First you need to send me the SIP INVITE via my proxy server.  SIP allows
the proxy server to either redirect you to contact me directly, or forward
on my request.  If I'm behind a full-cone NAT, either works fine.  If I'm
behind an address-restricted NAT, I can only hope that the proxy forwards
your request on to me, because I have no way of punching a hole to receive
your direct SIP request.  Furthermore, I have no way of indicating to the
SIP proxy that I need it to forward to me.  So right off the bat, receiving
SIP INVITE requests behind an address-restricted NAT is more complicated
than from behind a full-cone NAT.

Assuming I'm lucky and the SIP proxy forwards the request to me (via the
hole I've already punched for it), I can receive and respond to the INVITE
request indicating the port to which you should broadcast your RTP.
However, I have a complication that shared by all NATs, and that is getting
a port that complies with RTP.  The RTP standard indicates RTP ports should
be even-numbered, and the next port (odd-numbered) is implied to be open to
receive RTCP.  Unfortunately, I as a client have absolutely no control over
which ports my NAT assigns me, so it's very difficult for me to receive RTP
in a fully compliant fashion.  So for all NATs I must put in extra effort to
communicate the real RTP/RTCP ports I wish to receive on, possibly by
bending the RTP spec.  But a further complication of address-restricted NATs
is that I also need to somehow figure out the exact address you intend to
broadcast from, even though I would otherwise not care.  This means I must
bend the SIP/RTP framework even more (or dig into the details more than I
might otherwise) just to support address-restricted NATs.

In the meantime, while I am playing these extra games to punch holes to
receive your data, you have already begun to broadcast.  For full-cone NATs,
this is fine and expected.  But for address-restricted NATs, I can't receive
data despite having told you the correct IP/port to which you should
broadcast, because I haven't punched a hole for you.  While I'm playing
games trying to punch the hole, you're broadcasting data that I'm not
receiving.  Depending on the codec, that might be very important header
data.  Naturally UDP is an unreliable transport mechanism and thus there are
fallbacks if the occasional packet is lost, but there's a big difference
between "some packets might be lost" and "an unknown number of initial
packets *will* be lost".  More work must be put into supporting the
address-restricted case, and more deviation from the "mainstream" SIP/RTP
best practices.

Furthermore, just punching the holes themselves involve violating standards,
hopefully in benign fashion.  In the case of unidirectional RTP, there is
nowhere in the standard that I as a listener should send to your RTP
broadcast port -- thus there is no fully-compliant way to punch a hole to
receive your data.  (Yes I can send to your RTCP port, but that's a separate
port.)  The best I can do is send a malformed packet your way and hope it
doesn't cause you harm.  With full-cone NATs, I have no such problem.

And finally, every hole I punch requires some maintenance.  The less holes,
the better.  Generally, with full-cone, the number of holes I need punch is
proportional to the number of services I wish to offer (and the same holes
are used for all users).  But with address-restricted, it's proportional to
the product of the number of services, and the number of simultaneous users.
This means I have a much higher overhead to working with address-restricted
NATs.  But more importantly, I have so many more things that can go wrong.
This breeds the absolute worst kind of "partial failure" bug such as:

- Why can I hear Bob, but not Alice?
- Why can I hear Bob, but he can't hear me?
- Why can I receive Bob's RTP, but not Bob's RTCP?
- And so on...

And if I walk across the room and pick a different wi-fi hub, thus changing
IP addresses, with active sessions?  Oye, it's a ton of work to re-establish
all of these crazy holes and sessions in a bidirectional manner.  A ton of
work that I needn't do with full-cone NATs.


So, to make a long story short, NAT traversal is a hard problem, and it's
made especially hard by address-restricted NATs.  If I could count on
full-cone NAT behavior, my life as a programmer would be easier, and the
applications I produce would function better, faster, and more reliably.

In other words, I fully endorse Bryan's recommendation, and hope to see it
included in the final draft.

-david


> -----Original Message-----
> From: p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org] On
> Behalf Of Bryan Ford
> Sent: Wednesday, May 18, 2005 11:59 AM
> To: p2p-hackers@zgp.org
> Subject: [p2p-hackers] Official IETF behavior recommendations for
> NATrelevant to P2P
> 
> Dear p2p-hackers,
> 
> A discussion is currently taking place on the IETF BEHAVE working group
> mailing list (see http://www1.ietf.org/html.charters/behave-charter.html)
> that is highly relevant to P2P application writers.  For those of you not
> already familiar with it, this IETF working group is defining standards
> for
> how NATs should behave so as to better support applications including P2P
> apps.  The chair has made a "Last Call" for comments on the current draft
> document specifying NAT behavior for UDP, so this may be the last chance
> we
> have for a while to influence how well the next generation of NATs support
> UDP-based P2P applications.
> 
> ** Address-Restricted vs Full Cone NAT **
> 
> There's one issue in particular on which I personally fear the working
> group
> is currently headed in the wrong direction, and over which you folks on
> this
> list could perhaps exert considerable influence if you're willing to take
> a
> little time and make your voice heard on the BEHAVE list.  The current UDP
> draft, draft-ietf-behave-nat-udp-01.txt, recommends that NATs implement
> "Endpoint address-dependent filtering behavior", otherwise known as
> "Address-Restricted Cone NAT".  In other words, once a NATted UDP
> application
> opens a UDP session through a NAT to some external host, giving the
> application a public (NATted) port number, the NAT only accepts and
> forwards
> subsequent incoming UDP packets at that public port if they come from a
> source IP address that the application has previously sent UDP packets to.
> In other words, the NAT must see outgoing UDP traffic to a remote host
> before
> it will accept incoming traffic from that host on the application's public
> port.  In the proposed alternative, "Endpoint-independent filtering" or
> "Full
> Cone NAT", the NAT forwards any incoming traffic to the correct public
> port
> regardless of source IP address or port number.
> 
> Address-Restricted Cone NAT is moderately "P2P-friendly" in that it at
> least
> allows NATted applications to "hole punch" P2P connections with the help
> of a
> well-known (non-NATted) rendezvous server.  Address-Restricted Cone NAT
> still
> does not, however, allow the application to have a full "first-class"
> Internet presence (i.e., public IP address and UDP port number) at which
> it
> can be contacted proactively by any other Internet-connected host.  A P2P
> application behind an Address-Restricted Cone NAT is still forever
> dependent
> on other public (first-class) rendezvous servers to help establish each
> P2P
> communication session it needs.  A P2P app behind a Full Cone NAT, in
> contrast, is a full "first-class" network citizen itself, and can even
> _be_ a
> public rendezvous server for other less fortunate hosts once it learns its
> own public IP address and UDP port number.  This is an important benefit
> to
> applications, especially P2P applications that are trying to be highly
> decentralized.
> 
> The main argument for Restricted Cone NAT, of course, is security.  Until
> recently I believed this argument myself, but after further and more
> careful
> analysis I have come to the conclusion that Restricted filtering provides
> a
> very week security benefit in comparison with the fundamental loss in
> first-class application connectivity it entails.  My current proposal on
> the
> BEHAVE list does NOT mandate Full Cone NAT, but it does specify that NATs
> should at least be Full Cone in their default, "out of the box"
> configuration
> as provided by NAT vendors, and recommends that NATs implement Restricted
> Cone NAT as an administrator-configurable security option.  The current
> proposed text reads as follows:
> 
>    REQ-7: NATs MUST implement Endpoint-Independent
>           Filtering in their default configuration.
>           NATs SHOULD be configurable to support
>           Address-Specific Filtering or Endpoint-Specific Filtering
>           at the option of the system administrator.
> 
> If you have thoughts or opinions on this topic, I would greatly appreciate
> hearing them (and having other participants hear them) on the BEHAVE list.
> 
> ** Why the Security Benefit of Restricted Cone NAT is Minimal **
> 
> Restrictive filtering ostensibly has the benefit of protecting
> applications on
> an internal host from traffic the internal host is not expecting, i.e.,
> from
> traffic from previously unknown external hosts or endpoints.  This
> behavior
> can only possibly benefit the application if the application only ever
> initiates outgoing sessions from its internal port, and is obviously a
> curse
> if the application ever wants to accept incoming sessions from external
> hosts.  The NAT has no way of knowing which is the case.  But suppose the
> application is one that only initiates outgoing connections (i.e., a
> "traditional client/server app"), and thus might "potentially" benefit
> from a
> NAT's restrictive filtering behavior.  Consider TCP and UDP separately:
> 
>         ** TCP: When a client/server app makes an outgoing TCP connection,
> it
> typically does a "connect()" call to the host OS's sockets API without
> bothering to bind() the socket, and the OS's TCP stack allocates a local
> TCP
> port exclusively for the use of that connection.  Any TCP packets the host
> OS
> subsequently received on that port from endpoints other than the one it
> connect()ed to will be summarily rejected with TCP RSTs by the host OS,
> and
> the application never even sees any hint that they ever arrived.  Even if
> the
> application does a bind() before its connect() in order to use a specific
> source port (e.g., a <1024 "privileged" port), it still almost always
> obtains
> exclusive use of that port.  An application indeed has to work really
> bloody
> hard to set things up so that it can accept incoming TCP connections on
> the
> same port it is using for an outgoing connection, using SO_REUSEADDR and
> multiple sockets.  No traditional client/server application is going to go
> to
> all this effort just in order to create a security hole. :)  In other
> words,
> for TCP, restrictive filtering in practice provides absolutely zero
> protection from the _application_: the only entity that the restrictive
> filtering even affects is the host OS's protocol stack.  Restrictive
> filtering might for example benefit a host OS with a TCP stack that is so
> broken that it fails to check the source IP address and port number on
> incoming TCP packets at all, but if the host OS is _that_ broken then it
> undoubtedly has many other problems that the NAT can't possibly guard
> against.
> 
>         ** UDP: When a client/server UDP app uses a UDP port for
> communication
> with just one remote host, for convenience it typically does a connect()
> to
> the appropriate remote UDP endpoint, so that it can just send messages
> with
> send() or write() without having to specify a destination address for each
> datagram, and so that it can receive messages with read() or recv()
> without
> having to check the source address and port number in each message
> received.
> In this case, the host OS effectively provides the external endpoint
> filtering functionality as for TCP, and any restrictive filtering
> implemented
> by the NAT again provides no security benefit at all to the application.
> A
> UDP application that uses a single UDP port for communication with
> multiple
> remote hosts, in contrast, must generally check the source IP addresses
> and
> port numbers in the datagrams it receives just in order to distinguish
> between the multiple communication sessions it is involved in, and hence
> it
> will inherently provide its own external endpoint filtering.  It's
> conceivable that some abysmally buggy UDP applications might benefit from
> the
> NAT's external filtering - e.g., a UDP application that only talks to one
> external endpoint but doesn't use connect() and doesn't check the source
> endpoint on incoming datagrams - but an application that is so seriously
> broken probably has many other security vulnerabilities that can be
> exploited
> despite the NAT's external filtering behavior, e.g., by finding out or
> guessing the public server the application is talking to (which is easy in
> many cases such as commercial games or services where there are only a few
> well-known servers) and sending packets with spoofed source IP address and
> UDP port numbers.  Besides, many, perhaps the majority, of UDP
> applications
> use UDP _because_ they need P2P-style communication, and so external
> filtering by the NAT provides no practical security benefit and only
> causes
> interference with the application.
> 
> Another common argument for Restricted Cone NAT (Address-dependent
> filtering)
> is that it protects the resources on the internal network from being
> consumed
> forwarding bogus packets even if the eventual end application will
> correctly
> discard them.  This argument is also weak, however:
> 
> 	** Even with more permissive Full Cone NAT (endpoint-independent
> filtering),
> the NAT will still in practice filter out the vast majority of bogus
> traffic
> that reaches it, because most of the NAT's ports are typically not in
> active
> use by applications behind the NAT.  The only bogus traffic a Full Cone
> NAT
> will let through is bogus traffic that happens to hit the _specific_
> external
> port numbers that correspond to active port bindings.  These port numbers
> are
> assigned more-or-less randomly (i.e., unpredictably) by the NAT, and they
> only
> ever appear in the high, non-well-known port number space, so most of the
> Internet "background noise" caused by viruses or other attacks on well-
> known
> ports will never get through in any case, and purely random "port
> guessing"
> attacks would have to send packets to a great many NAT ports in order to
> get
> a few packets through.  Thus, the incremental benefit of restrictive
> filtering over Full Cone NAT with regards to general bogus Internet
> traffic
> is very small.
> 
> 	** An attacker might deliberately send bogus packets to a bound NAT
> port
> known to be in use.  But if the attacker can find out which NAT port the
> application is using, then the attacker probably also knows (or can easily
> find out or guess) the address and port number of the _remote_ host that
> the
> application is talking to and just spoof that source IP address and port
> number in its attack packets, getting around Restricted filtering as well.
> In fact the remote host's endpoint is in practice likely to be much easier
> for the attacker to find out or guess than the application's public NAT
> port,
> because of the overwhelming prevalence of well-known servers and ports on
> the
> Internet in comparison to the randomness and relative obscurity of
> temporary
> NAT-assigned public ports.  So the practical benefit that Restricted
> filtering provides even against directed attacks is still minimal.
> 
> 	** Finally, the goal of "protecting the resources on the internal
> network
> from being consumed forwarding bogus packets" is really only a
> consideration
> on large corporate or ISP networks with substantial internal traffic load,
> and if this minimization of resource use is a concern then the network
> administrator can certainly be expected to know how to configure the
> company's NAT to enable Restricted filtering.  As I pointed out above,
> under
> my proposal a NAT can use Restricted filtering and still be BEHAVE-
> compliant
> as long as it is the administrator (not the NAT vendor) that enables this
> firewall behavior.
> 
> 	** For home NATs, in contrast, minimizing resource load on the
> internal
> network due to bogus traffic is completely a non-issue because the home
> network (Ethernet or 802.11 or whatever) is typically so much faster than
> the
> Internet uplink anyway.  In this case it's much more important that NATs
> consistently support applications well, which my proposal ensures by
> requiring that the NAT's "out of the box" configuration is
> Endpoint-independent Filtering.  Joe Clueless who buys a home NAT router
> probably doesn't have any idea what NAT actually is, but he wants his
> applications to work.
> 
> ** Conclusion **
> 
> In summary, restrictive filtering in the NAT interferes with applications
> substantially and provides very little real security benefit in
> practice, considering the way application communication and NAT port
> assignment actually works.  As such, especially considering the purpose of
> the BEHAVE working group is to increase the predictability and
> compatibility
> of NATs with applications and not to specify standards for firewall
> functionality, it seems highly inappropriate for the WG's documents to
> recommend firewall functionality that interferes with applications while
> providing dubious security benefits.  On the contrary, I think the WG
> should
> at least recommend, if not mandate, that NATs provide Endpoint-Independent
> Filtering (Full Cone NAT) by default.  NAT vendors or system
> administrators
> that are particularly paranoid about security and perceive the security
> benefits to be worth the interference it causes with applications can
> still
> enable more restrictive filtering policies, but that's a firewall
> deployment
> issue, not a NAT functionality issue.  We need to steer clear from
> recommending firewall functionality.
> 
> Thanks very much for your attention.  Please direct followups to the
> BEHAVE
> working group list, "ietf-behave@list.sipfoundry.org".  Thanks!
> 
> Bryan
> _______________________________________________
> p2p-hackers mailing list
> p2p-hackers@zgp.org
> http://zgp.org/mailman/listinfo/p2p-hackers
> _______________________________________________
> Here is a web page listing P2P Conferences:
> http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences


From wgallagher at gridironsoftware.com  Fri May 20 01:33:18 2005
From: wgallagher at gridironsoftware.com (Warren Gallagher)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] Official IETF behavior recommendations for
	NATrelevant to P2P
In-Reply-To: <200505181459.21722.baford@mit.edu>
Message-ID: <200505200133.j4K1XBD8011939@mail4.magma.ca>

Bryan,

I agree with your assessment. I would further argue that NATs are not
security devices, they are address translators. Security is the job of the
firewall. As long as firewalls do not restrict a sysadmin's ability to
control the behaviour then everything is good.

Thanks for the good work
Warren Gallagher
Chief Software Architect, Gridiron Software

-----Original Message-----
From: p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org] On
Behalf Of Bryan Ford
Sent: May 18, 2005 2:59 PM
To: p2p-hackers@zgp.org
Subject: [p2p-hackers] Official IETF behavior recommendations for
NATrelevant to P2P

Dear p2p-hackers,

A discussion is currently taking place on the IETF BEHAVE working group 
mailing list (see http://www1.ietf.org/html.charters/behave-charter.html) 
that is highly relevant to P2P application writers.  For those of you not 
already familiar with it, this IETF working group is defining standards for 
how NATs should behave so as to better support applications including P2P 
apps.  The chair has made a "Last Call" for comments on the current draft 
document specifying NAT behavior for UDP, so this may be the last chance we 
have for a while to influence how well the next generation of NATs support 
UDP-based P2P applications.

** Address-Restricted vs Full Cone NAT **

There's one issue in particular on which I personally fear the working group

is currently headed in the wrong direction, and over which you folks on this

list could perhaps exert considerable influence if you're willing to take a 
little time and make your voice heard on the BEHAVE list.  The current UDP 
draft, draft-ietf-behave-nat-udp-01.txt, recommends that NATs implement 
"Endpoint address-dependent filtering behavior", otherwise known as 
"Address-Restricted Cone NAT".  In other words, once a NATted UDP
application 
opens a UDP session through a NAT to some external host, giving the 
application a public (NATted) port number, the NAT only accepts and forwards

subsequent incoming UDP packets at that public port if they come from a 
source IP address that the application has previously sent UDP packets to.  
In other words, the NAT must see outgoing UDP traffic to a remote host
before 
it will accept incoming traffic from that host on the application's public 
port.  In the proposed alternative, "Endpoint-independent filtering" or
"Full 
Cone NAT", the NAT forwards any incoming traffic to the correct public port 
regardless of source IP address or port number.

Address-Restricted Cone NAT is moderately "P2P-friendly" in that it at least

allows NATted applications to "hole punch" P2P connections with the help of
a 
well-known (non-NATted) rendezvous server.  Address-Restricted Cone NAT
still 
does not, however, allow the application to have a full "first-class" 
Internet presence (i.e., public IP address and UDP port number) at which it 
can be contacted proactively by any other Internet-connected host.  A P2P 
application behind an Address-Restricted Cone NAT is still forever dependent

on other public (first-class) rendezvous servers to help establish each P2P 
communication session it needs.  A P2P app behind a Full Cone NAT, in 
contrast, is a full "first-class" network citizen itself, and can even _be_
a 
public rendezvous server for other less fortunate hosts once it learns its 
own public IP address and UDP port number.  This is an important benefit to 
applications, especially P2P applications that are trying to be highly 
decentralized.

The main argument for Restricted Cone NAT, of course, is security.  Until 
recently I believed this argument myself, but after further and more careful

analysis I have come to the conclusion that Restricted filtering provides a 
very week security benefit in comparison with the fundamental loss in 
first-class application connectivity it entails.  My current proposal on the

BEHAVE list does NOT mandate Full Cone NAT, but it does specify that NATs 
should at least be Full Cone in their default, "out of the box"
configuration 
as provided by NAT vendors, and recommends that NATs implement Restricted 
Cone NAT as an administrator-configurable security option.  The current 
proposed text reads as follows:

   REQ-7: NATs MUST implement Endpoint-Independent
          Filtering in their default configuration.
          NATs SHOULD be configurable to support
          Address-Specific Filtering or Endpoint-Specific Filtering
          at the option of the system administrator.

If you have thoughts or opinions on this topic, I would greatly appreciate 
hearing them (and having other participants hear them) on the BEHAVE list.

** Why the Security Benefit of Restricted Cone NAT is Minimal **

Restrictive filtering ostensibly has the benefit of protecting applications
on 
an internal host from traffic the internal host is not expecting, i.e., from

traffic from previously unknown external hosts or endpoints.  This behavior 
can only possibly benefit the application if the application only ever 
initiates outgoing sessions from its internal port, and is obviously a curse

if the application ever wants to accept incoming sessions from external 
hosts.  The NAT has no way of knowing which is the case.  But suppose the 
application is one that only initiates outgoing connections (i.e., a 
"traditional client/server app"), and thus might "potentially" benefit from
a 
NAT's restrictive filtering behavior.  Consider TCP and UDP separately:

        ** TCP: When a client/server app makes an outgoing TCP connection,
it 
typically does a "connect()" call to the host OS's sockets API without 
bothering to bind() the socket, and the OS's TCP stack allocates a local TCP

port exclusively for the use of that connection.  Any TCP packets the host
OS 
subsequently received on that port from endpoints other than the one it 
connect()ed to will be summarily rejected with TCP RSTs by the host OS, and 
the application never even sees any hint that they ever arrived.  Even if
the 
application does a bind() before its connect() in order to use a specific 
source port (e.g., a <1024 "privileged" port), it still almost always
obtains 
exclusive use of that port.  An application indeed has to work really bloody

hard to set things up so that it can accept incoming TCP connections on the 
same port it is using for an outgoing connection, using SO_REUSEADDR and 
multiple sockets.  No traditional client/server application is going to go
to 
all this effort just in order to create a security hole. :)  In other words,

for TCP, restrictive filtering in practice provides absolutely zero 
protection from the _application_: the only entity that the restrictive 
filtering even affects is the host OS's protocol stack.  Restrictive 
filtering might for example benefit a host OS with a TCP stack that is so 
broken that it fails to check the source IP address and port number on 
incoming TCP packets at all, but if the host OS is _that_ broken then it 
undoubtedly has many other problems that the NAT can't possibly guard 
against.

        ** UDP: When a client/server UDP app uses a UDP port for
communication 
with just one remote host, for convenience it typically does a connect() to 
the appropriate remote UDP endpoint, so that it can just send messages with 
send() or write() without having to specify a destination address for each 
datagram, and so that it can receive messages with read() or recv() without 
having to check the source address and port number in each message received.

In this case, the host OS effectively provides the external endpoint 
filtering functionality as for TCP, and any restrictive filtering
implemented 
by the NAT again provides no security benefit at all to the application.  A 
UDP application that uses a single UDP port for communication with multiple 
remote hosts, in contrast, must generally check the source IP addresses and 
port numbers in the datagrams it receives just in order to distinguish 
between the multiple communication sessions it is involved in, and hence it 
will inherently provide its own external endpoint filtering.  It's 
conceivable that some abysmally buggy UDP applications might benefit from
the 
NAT's external filtering - e.g., a UDP application that only talks to one 
external endpoint but doesn't use connect() and doesn't check the source 
endpoint on incoming datagrams - but an application that is so seriously 
broken probably has many other security vulnerabilities that can be
exploited 
despite the NAT's external filtering behavior, e.g., by finding out or 
guessing the public server the application is talking to (which is easy in 
many cases such as commercial games or services where there are only a few 
well-known servers) and sending packets with spoofed source IP address and 
UDP port numbers.  Besides, many, perhaps the majority, of UDP applications 
use UDP _because_ they need P2P-style communication, and so external 
filtering by the NAT provides no practical security benefit and only causes 
interference with the application.

Another common argument for Restricted Cone NAT (Address-dependent
filtering) 
is that it protects the resources on the internal network from being
consumed 
forwarding bogus packets even if the eventual end application will correctly

discard them.  This argument is also weak, however:

	** Even with more permissive Full Cone NAT (endpoint-independent
filtering), 
the NAT will still in practice filter out the vast majority of bogus traffic

that reaches it, because most of the NAT's ports are typically not in active

use by applications behind the NAT.  The only bogus traffic a Full Cone NAT 
will let through is bogus traffic that happens to hit the _specific_
external 
port numbers that correspond to active port bindings.  These port numbers
are 
assigned more-or-less randomly (i.e., unpredictably) by the NAT, and they
only 
ever appear in the high, non-well-known port number space, so most of the 
Internet "background noise" caused by viruses or other attacks on well-known

ports will never get through in any case, and purely random "port guessing" 
attacks would have to send packets to a great many NAT ports in order to get

a few packets through.  Thus, the incremental benefit of restrictive 
filtering over Full Cone NAT with regards to general bogus Internet traffic 
is very small.

	** An attacker might deliberately send bogus packets to a bound NAT
port 
known to be in use.  But if the attacker can find out which NAT port the 
application is using, then the attacker probably also knows (or can easily 
find out or guess) the address and port number of the _remote_ host that the

application is talking to and just spoof that source IP address and port 
number in its attack packets, getting around Restricted filtering as well.  
In fact the remote host's endpoint is in practice likely to be much easier 
for the attacker to find out or guess than the application's public NAT
port, 
because of the overwhelming prevalence of well-known servers and ports on
the 
Internet in comparison to the randomness and relative obscurity of temporary

NAT-assigned public ports.  So the practical benefit that Restricted 
filtering provides even against directed attacks is still minimal.

	** Finally, the goal of "protecting the resources on the internal
network 
from being consumed forwarding bogus packets" is really only a consideration

on large corporate or ISP networks with substantial internal traffic load, 
and if this minimization of resource use is a concern then the network 
administrator can certainly be expected to know how to configure the 
company's NAT to enable Restricted filtering.  As I pointed out above, under

my proposal a NAT can use Restricted filtering and still be BEHAVE-compliant

as long as it is the administrator (not the NAT vendor) that enables this 
firewall behavior.

	** For home NATs, in contrast, minimizing resource load on the
internal 
network due to bogus traffic is completely a non-issue because the home 
network (Ethernet or 802.11 or whatever) is typically so much faster than
the 
Internet uplink anyway.  In this case it's much more important that NATs 
consistently support applications well, which my proposal ensures by 
requiring that the NAT's "out of the box" configuration is 
Endpoint-independent Filtering.  Joe Clueless who buys a home NAT router 
probably doesn't have any idea what NAT actually is, but he wants his 
applications to work.

** Conclusion **

In summary, restrictive filtering in the NAT interferes with applications 
substantially and provides very little real security benefit in 
practice, considering the way application communication and NAT port 
assignment actually works.  As such, especially considering the purpose of 
the BEHAVE working group is to increase the predictability and compatibility

of NATs with applications and not to specify standards for firewall 
functionality, it seems highly inappropriate for the WG's documents to 
recommend firewall functionality that interferes with applications while 
providing dubious security benefits.  On the contrary, I think the WG should

at least recommend, if not mandate, that NATs provide Endpoint-Independent 
Filtering (Full Cone NAT) by default.  NAT vendors or system administrators 
that are particularly paranoid about security and perceive the security 
benefits to be worth the interference it causes with applications can still 
enable more restrictive filtering policies, but that's a firewall deployment

issue, not a NAT functionality issue.  We need to steer clear from 
recommending firewall functionality.

Thanks very much for your attention.  Please direct followups to the BEHAVE 
working group list, "ietf-behave@list.sipfoundry.org".  Thanks!

Bryan
_______________________________________________
p2p-hackers mailing list
p2p-hackers@zgp.org
http://zgp.org/mailman/listinfo/p2p-hackers
_______________________________________________
Here is a web page listing P2P Conferences:
http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences


From em at em.no-ip.com  Fri May 20 02:36:24 2005
From: em at em.no-ip.com (Enzo Michelangeli)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] Official IETF behavior recommendations
	forNATrelevant to P2P
References: <20050520010540.D91873FC99@capsicum.zgp.org>
Message-ID: <001d01c55ce4$bcd4f8e0$0200a8c0@em.noip.com>

David,

I generally agree with you and Bryan, with the following notes:

----- Original Message ----- 
From: "David Barrett" <dbarrett@quinthar.com>
To: "'Peer-to-peer development.'" <p2p-hackers@zgp.org>;
<ietf-behave@list.sipfoundry.org>
Sent: Friday, May 20, 2005 9:05 AM
Subject: RE: [p2p-hackers] Official IETF behavior recommendations
forNATrelevant to P2P

[...]
> However, I have a complication that shared by all NATs, and that is
> getting a port that complies with RTP.  The RTP standard indicates
> RTP ports should be even-numbered, and the next port (odd-numbered)
> is implied to be open to receive RTCP.

This restriction was substantially relaxed in RFC3550:

   [...] For
   applications in which the RTP and RTCP destination port numbers are
   specified via explicit, separate parameters (using a signaling
   protocol or other means), the application MAY disregard the
   restrictions that the port numbers be even/odd and consecutive
   although the use of an even/odd port pair is still encouraged.

The sad truth is that SIP and RTP were never designed with NAT in mind:
the authors were probably thinking that the deployment of IPv6 was going
to happen soon, AND that this would have made NAT unnecessary. After this
scenario failed to become reality, a small army of remedial solutions were
devised: "rport" parameter (RFC3581); Symmetric RTP; STUN; TURN; ICE
(http://www.jdrosen.net/papers/draft-rosenberg-sipping-ice-01.html) etc.
In my opinion, the only real solution would be to junk SIP for good and
use IAX2 (http://www.voip-info.org/wiki-IAX), which is also more compact
and efficient: but this is another story, and OT for this list.

> Unfortunately, I as a client have absolutely no control over
> which ports my NAT assigns me, so it's very difficult for me to receive
> RTP in a fully compliant fashion.

Right, although RFC3581 and symmetric RTP may alleviate this problem.

And by the way: I have always found it kind of nutty that in order to know
something local to my router like the NAT mappings I have to bother some
third party on the Internet. As the BEHAVE working group is laying down
guidelines for NAT implementors, wouldn't it be useful to ask for a simple
mechanism to query the NATting device about port mappings? For instance, a
UDP packet sent to a well-known port of the NAT router (or, better,
broadcast to that port, as the application may be unaware of the router's
address) could ask: "tell me which external address/port pair corresponds
to the internal pair 192.168.0.123:4567 UDP, and how long the mapping can
be expected to last". True, this only works if there isn't a second NAT
further ahead on the road, but this is the case in the vast majority of
the cases.

Enzo


From larytet.8753341 at bloglines.com  Fri May 20 03:18:32 2005
From: larytet.8753341 at bloglines.com (larytet.8753341@bloglines.com)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] IPv6 and NAT
Message-ID: <1116559112.82397386.10710.sendItem@bloglines.com>

"Given that l33t hackers are already giving owned machines ipv6 addresses
anyway..."

i have IPv6 address on my machine. i am behind NAT and firewall
and can do nothing besides testing between two PCs connected to the same LAN.
i understand that without tunnel in the NAT (static one to one) i can not
do much with IPv6. am i right ?

From wesley at felter.org  Fri May 20 04:56:18 2005
From: wesley at felter.org (Wes Felter)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] IPv6 and NAT
In-Reply-To: <1116559112.82397386.10710.sendItem@bloglines.com>
References: <1116559112.82397386.10710.sendItem@bloglines.com>
Message-ID: <428D6DF2.2000004@felter.org>

larytet.8753341@bloglines.com wrote:
> "Given that l33t hackers are already giving owned machines ipv6 addresses
> anyway..."
> 
> i have IPv6 address on my machine. i am behind NAT and firewall
> and can do nothing besides testing between two PCs connected to the same LAN.
> i understand that without tunnel in the NAT (static one to one) i can not
> do much with IPv6. am i right ?

You can use Teredo.

I suspect the l33t hackers have an advantage since they are presumably 
taking over non-NATed, non-firewalled machines, so they can use 6to4.

Wes Felter - wesley@felter.org


From dbarrett at quinthar.com  Fri May 20 05:40:24 2005
From: dbarrett at quinthar.com (David Barrett)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] Official IETF behavior
	recommendationsforNATrelevant to P2P
In-Reply-To: <001d01c55ce4$bcd4f8e0$0200a8c0@em.noip.com>
Message-ID: <20050520054023.CC26D3FCA2@capsicum.zgp.org>

Enzo -- Perhaps you can help with solving a lingering problem of mine.  As
you correctly point out, the RTP spec says:

>    [...] For
>    applications in which the RTP and RTCP destination port numbers are
>    specified via explicit, separate parameters (using a signaling
>    protocol or other means), the application MAY disregard the
>    restrictions that the port numbers be even/odd and consecutive
>    although the use of an even/odd port pair is still encouraged.

However, RTP isn't the place where ports are communicated -- SDP is that
place.  Unfortunately, SDP defines a media description as (from RFC2327):

     m=<media> <port> <transport> <fmt list>

There's only room for one port in that, and it's presumed to be the RTP
port, with the RTCP port implied to be one above.  This is backed up by the
line:

     Other ports used by the media application (such as the RTCP port, see
     [2]) should be derived algorithmically from the base media port.

This is actually an obstacle I've heretofore been unable to resolve; how can
you communicate a separate RTCP port while remaining standards compliant?

As for rport, symmetric RTP, STUN, TURN, ICE, etc, all these do help.  But
it's just so much damn work; there's a hundred pages of specs there to be
compliant with.

I'm not familiar with IAX2.  Is it a pipe dream, or does it actually stand a
chance of catching on?

-david

PS: I'm leaving this on the list in case anyone else is out there dealing
with P2P VoIP issues.


From justin at specialbusservice.com  Fri May 20 09:51:31 2005
From: justin at specialbusservice.com (Justin Cormack)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] IPv6 and NAT
In-Reply-To: <1116559112.82397386.10710.sendItem@bloglines.com>
References: <1116559112.82397386.10710.sendItem@bloglines.com>
Message-ID: <503934CF-0732-4626-BC3E-9A2AADB59C7F@specialbusservice.com>


On 20 May 2005, at 04:18, larytet.8753341@bloglines.com wrote:

> "Given that l33t hackers are already giving owned machines ipv6  
> addresses
> anyway..."
>
> i have IPv6 address on my machine. i am behind NAT and firewall
> and can do nothing besides testing between two PCs connected to the  
> same LAN.
> i understand that without tunnel in the NAT (static one to one) i  
> can not
> do much with IPv6. am i right ?

Have you got real IPv6 addresses or just the link local ones that  
most OSs install now?

en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
         inet6 fe80::211:24ff:fe7f:aa3c%en0 prefixlen 64 scopeid  
0x4      <-----------------------------------link local address
         inet6 2001:1b40:3:1:211:24ff:fe7f:aa3c prefixlen 64  
autoconf      <-----------------------------------real ipv6 address
         inet 10.0.0.200 netmask 0xffffff00 broadcast 10.0.0.255
         ether 00:11:24:7f:aa:3c

The link local addresses (start fe....) are not routed. Ask your ISP  
for a ipv6 /48 assignment to get real addresses, or use a tunnel  
(available for free in most places).

It is possible that your firewall filters all ipv6 traffic, but  
unlikely.

Justin


From em at em.no-ip.com  Fri May 20 11:33:34 2005
From: em at em.no-ip.com (Enzo Michelangeli)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] Official IETF behaviorrecommendationsforNATrelevant
	to P2P
References: <20050520054023.CC26D3FCA2@capsicum.zgp.org>
Message-ID: <002f01c55d2f$ce7be7c0$0200a8c0@em.noip.com>

----- Original Message ----- 
From: "David Barrett" <dbarrett@quinthar.com>
To: "'Peer-to-peer development.'" <p2p-hackers@zgp.org>
Sent: Friday, May 20, 2005 1:40 PM
Subject: RE: [p2p-hackers] Official IETF
behaviorrecommendationsforNATrelevant to P2P

> Enzo -- Perhaps you can help with solving a lingering problem of mine.
> As you correctly point out, the RTP spec says:
>
> >    [...] For
> >    applications in which the RTP and RTCP destination port numbers are
> >    specified via explicit, separate parameters (using a signaling
> >    protocol or other means), the application MAY disregard the
> >    restrictions that the port numbers be even/odd and consecutive
> >    although the use of an even/odd port pair is still encouraged.
>
> However, RTP isn't the place where ports are communicated -- SDP is that
> place.  Unfortunately, SDP defines a media description as (from
> RFC2327):
>
>      m=<media> <port> <transport> <fmt list>
>
> There's only room for one port in that, and it's presumed to be the RTP
> port, with the RTCP port implied to be one above.  This is backed up by
> the line:
>
>   Other ports used by the media application (such as the RTCP port, see
>   [2]) should be derived algorithmically from the base media port.
>
> This is actually an obstacle I've heretofore been unable to resolve; how
> can you communicate a separate RTCP port while remaining standards
> compliant?

RFC3605 describes an extension to SDP to add an RTCP attribute:

   rtcp-attribute =  "a=rtcp:" port  [nettype space addrtype space
                         connection-address] CRLF

I'm not sure how widely this is deployed, but the RFC is flagged
"Standards Track". And
http://www.simpleweb.org/ietf/internetdrafts/complete/draft-wing-mmusic-symmetric-rtprtcp-01.txt
suggests (or suggested, as it's expired) to practice "symmetric RTCP" just
as it's done for RTP.

> As for rport, symmetric RTP, STUN, TURN, ICE, etc, all these do help.
> But it's just so much damn work; there's a hundred pages of specs
> there to be compliant with.

I know, it's a bloody nightmare, and there is no guarantee that your peers
ever make the same set of assumptions. Shouldn't a "session initiation
protocol" lead to determine precisely that? :-)

> I'm not familiar with IAX2.  Is it a pipe dream, or does it actually
> stand a chance of catching on?

It's real (although still poorly documented: but the work in progress at
http://www.cornfed.com/iax.pdf is promising). At present it is supported
by a number of commercial
(http://www.voip-info.org/wiki-VOIP+Service+Providers+B2B) and free
(www.iaxtel.org, http://www.freeworlddialup.com/content/view/full/1501 )
providers, and also by some hardware manufacturers (see e.g.
http://www.iareaphone.com/Hardware.htm ) and a few softphones: the best of
the latter (but, alas, closed source) being IMO Firefly:
http://www.virbiage.com/firefly/download/ , version "for third party
networks".

The IAX2 reference implementation (libiax2) is embedded in Asterisk, an
excellent (and GPL'd) software PBX (www.asterisk.org ,
http://www.voip-info.org/wiki-Asterisk ) that runs on Linux, *BSD and
MacOS X, and even on reflashed Linksys WRT54GS routers: see e.g.
http://openwrt.org/forum/viewtopic.php?id=1419 . See also the library at
http://iaxclient.sourceforge.net/ .

The great thing of IAX2 for NAT purposes is that it always uses _one_
UDP port for both signalling and media, and it's intrinsecally symmetric.
Apart from that, it's noticeably more efficient than RTP in trunking
applications. On the weak side, sending media together with signalling
makes it impossible for the server (the proxy / registrar with SIP) to
disengage from the media stream while continuing to monitor the call for
billing purposes. So, the carriers (which _hate_ the idea of being forced
to switch to a "flat rate" model, being addicted to the "minutes" they
love to bill as they did when they were younger ;-) ) are generally
unenthusiastic toward IAX2 and like to diss it as "toy protocol".

<SOAPBOX> This, in my opinion, will eventually be resolved by the market
realities: NOBODY is going to pay carriers on metered basis for ANY purely
Internet-based service (i.e. excluding, e.g., PSTN termination), for the
simple reason that on the Internet the application layer is out of the
carrier's reach; also, users prefer flat rates for ease of budgeting, and
apart from that, the complications inherent in a per-minute billing system
should make a flat-fee model attractive also for a provider. Some smart
people who in the past worked for carriers, like David Isenberg (of "The
rise of the Stupid Network" fame: http://www.isen.com/stupid.html), and
the exceptionally bright Andy Odlyzko (e.g., in
http://www.dtc.umn.edu/~odlyzko/doc/pricing.architecture.pdf ), did grasp
this point early on, but the dinosaurs who ran the LD business were in
strict self-preservation mode. </SOAPBOX>

So, it's not surprising if, today, Isenberg looks favourably at IAX2...
See e.g. his article at
http://www.vonmag.com/issue/2004/julaug/columns/isenberg.htm . Now, if
only the IETF and the big vendors who call the shots and sell to the
carriers came to their senses...

Enzo





From evantsang at gmail.com  Sat May 21 05:58:01 2005
From: evantsang at gmail.com (Evan Tsang)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] IPv6 and NAT
In-Reply-To: <1116559112.82397386.10710.sendItem@bloglines.com>
References: <1116559112.82397386.10710.sendItem@bloglines.com>
Message-ID: <b9c416ce05052022584e31f1c3@mail.gmail.com>

Even if you don't get a real IPv6 address, you can get a Teredo IPv6 address 
and communicate with other host with Teredo IPv6 address. You can still 
connected even if both side are NATed because Teredo tunnel their packets 
thru UDP and most of the consumer NAT device are not symmetric NAT. I have 
built a P2P client that communicate using this scheme in my previous 
company. A Teredo address has the prefix 3ffe:831f. You can obtain a Teredo 
address by running
> ipv6 install
> netsh interface ipv6 set teredo client

If Microsoft is willing to run a public Teredo relay or Teredo host-specific 
relay, you can communicate with IPv6 only host as well.

-
Evan Tsang

On 20 May 2005 03:18:32 -0000, larytet.8753341@bloglines.com <
larytet.8753341@bloglines.com> wrote:
> 
> "Given that l33t hackers are already giving owned machines ipv6 addresses
> anyway..."
> 
> i have IPv6 address on my machine. i am behind NAT and firewall
> and can do nothing besides testing between two PCs connected to the same 
> LAN.
> i understand that without tunnel in the NAT (static one to one) i can not
> do much with IPv6. am i right ?
> _______________________________________________
> p2p-hackers mailing list
> p2p-hackers@zgp.org
> http://zgp.org/mailman/listinfo/p2p-hackers
> _______________________________________________
> Here is a web page listing P2P Conferences:
> http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://zgp.org/pipermail/p2p-hackers/attachments/20050520/51ad9b68/attachment.htm
From lemonobrien at yahoo.com  Tue May 24 21:14:07 2005
From: lemonobrien at yahoo.com (Lemon Obrien)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] IPv6 and NAT
In-Reply-To: <b9c416ce05052022584e31f1c3@mail.gmail.com>
Message-ID: <20050524211407.29692.qmail@web53607.mail.yahoo.com>

If you use Teredo, then your tied to Teredo. Which is not strategically smart unless your app is made for IT use only.

Evan Tsang <evantsang@gmail.com> wrote:Even if you don't get a real IPv6 address, you can get a Teredo IPv6 address and communicate with other host with Teredo IPv6 address.  You can still connected even if both side are NATed because Teredo tunnel their packets thru UDP and most of the consumer NAT device are not symmetric NAT.  I have built a P2P client that communicate using this scheme in my previous company.  A Teredo address has the prefix 3ffe:831f.  You can obtain a Teredo address by running
> ipv6 install
> netsh interface ipv6 set teredo client

If Microsoft is willing to run a public Teredo relay or Teredo host-specific relay, you can communicate with IPv6 only host as well.

-
Evan Tsang

On 20 May 2005 03:18:32 -0000, larytet.8753341@bloglines.com < larytet.8753341@bloglines.com> wrote:"Given that l33t hackers are already giving owned machines ipv6 addresses 
anyway..."

i have IPv6 address on my machine. i am behind NAT and firewall
and can do nothing besides testing between two PCs connected to the same LAN.
i understand that without tunnel in the NAT (static one to one) i can not 
do much with IPv6. am i right ?
_______________________________________________
p2p-hackers mailing list
p2p-hackers@zgp.org
http://zgp.org/mailman/listinfo/p2p-hackers
_______________________________________________
Here is a web page listing P2P Conferences:
http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences


_______________________________________________
p2p-hackers mailing list
p2p-hackers@zgp.org
http://zgp.org/mailman/listinfo/p2p-hackers
_______________________________________________
Here is a web page listing P2P Conferences:
http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences



You don't get no juice unless you squeeze
Lemon Obrien, the Third.
		
---------------------------------
Do You Yahoo!?
 Yahoo! Small Business - Try our new Resources site!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://zgp.org/pipermail/p2p-hackers/attachments/20050524/57d4f7f5/attachment.html
From wesley at felter.org  Wed May 25 00:58:55 2005
From: wesley at felter.org (Wes Felter)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] IPv6 and NAT
In-Reply-To: <20050524211407.29692.qmail@web53607.mail.yahoo.com>
References: <20050524211407.29692.qmail@web53607.mail.yahoo.com>
Message-ID: <4293CDCF.7040005@felter.org>

Lemon Obrien wrote:
> If you use Teredo, then your tied to Teredo. Which is not strategically 
> smart unless your app is made for IT use only.

In what way? Since Teredo is implemented in the network layer, 
applications aren't even aware of it*. Since Teredo gives you a 
first-class, globally-routable IPv6 address, users won't be aware of it 
either. The transition from Teredo to 6to4 to native IPv6 connectivity 
is seamless.

[*] In fact, I see this as the biggest drawback of Teredo. It basically 
has to be implemented in the OS, so applications can't provide it. So if 
you want WinME users, you can't rely on Teredo.

Wes Felter - wesley@felter.org

From lemonobrien at yahoo.com  Wed May 25 03:14:56 2005
From: lemonobrien at yahoo.com (Lemon Obrien)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] IPv6 and NAT
In-Reply-To: 6667
Message-ID: <20050525031456.72165.qmail@web53601.mail.yahoo.com>

why can't an application implement it? java implements ipv6 and its vm runs on top of the os.

if you can write raw ip packets, you can imulate teredo
http://www.microsoft.com/technet/prodtechnol/winxppro/maintain/teredo.mspx#EFAA
 
el
Wes Felter <wesley@felter.org> wrote:
Lemon Obrien wrote:
> If you use Teredo, then your tied to Teredo. Which is not strategically 
> smart unless your app is made for IT use only.

In what way? Since Teredo is implemented in the network layer, 
applications aren't even aware of it*. Since Teredo gives you a 
first-class, globally-routable IPv6 address, users won't be aware of it 
either. The transition from Teredo to 6to4 to native IPv6 connectivity 
is seamless.

[*] In fact, I see this as the biggest drawback of Teredo. It basically 
has to be implemented in the OS, so applications can't provide it. So if 
you want WinME users, you can't rely on Teredo.

Wes Felter - wesley@felter.org
_______________________________________________
p2p-hackers mailing list
p2p-hackers@zgp.org
http://zgp.org/mailman/listinfo/p2p-hackers
_______________________________________________
Here is a web page listing P2P Conferences:
http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences


You don't get no juice unless you squeeze
Lemon Obrien, the Third.
		
---------------------------------
Do You Yahoo!?
 Yahoo! Small Business - Try our new Resources site!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://zgp.org/pipermail/p2p-hackers/attachments/20050524/6683336b/attachment.htm
From wesley at felter.org  Wed May 25 05:05:37 2005
From: wesley at felter.org (Wes Felter)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] IPv6 and NAT
In-Reply-To: <20050525031456.72165.qmail@web53601.mail.yahoo.com>
References: <20050525031456.72165.qmail@web53601.mail.yahoo.com>
Message-ID: <429407A1.6080609@felter.org>

Lemon Obrien wrote:
> why can't an application implement it? 
> if you can write raw ip packets, you can imulate teredo

Sure, if you can afford to write an *entire network stack*. (But then, I 
guess a secure, reliable P2P messaging layer that can bust through every 
type of NAT/firewall is almost as complex...)

 > java implements ipv6 and its vm runs on top of the os.

I'm pretty sure Java VMs just use the OS's IPv6 stack.

Wes Felter - wesley@felter.org

From coderman at gmail.com  Thu May 26 00:10:08 2005
From: coderman at gmail.com (coderman)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] IPv6 and NAT
In-Reply-To: <429407A1.6080609@felter.org>
References: <20050525031456.72165.qmail@web53601.mail.yahoo.com>
	<429407A1.6080609@felter.org>
Message-ID: <4ef5fec605052517105fa46e3c@mail.gmail.com>

On 5/24/05, Wes Felter <wesley@felter.org> wrote:
>...
> Sure, if you can afford to write an *entire network stack*. (But then, I
> guess a secure, reliable P2P messaging layer that can bust through every
> type of NAT/firewall is almost as complex...)

And also accept the fact that scheduling in userspace for that stack
(tcp/other timers) is going suck.  We've been down this road with TCP
over UDP in userspace, etc.  Have you tried using high resolution
timers on windows in userspace?  Not pretty.


> I'm pretty sure Java VMs just use the OS's IPv6 stack.

That is exactly what it is doing; the sockets API for IPv4 or IPv6
works fine in java, but that is all it is using; the stack and
implementation resides in kernel land...

One thing I always liked about Solaris/HP-SUX is that you could
configure additional stacks via STREAMS and use XTI to bind to
whichever stack you wanted.  The berkeley sockets always defaulted to
the system stack but XTI/TLI let you pick (via the device path given
to t_open).

This is still running in kernel land though, and last time I checked
you needed to cough up $50,000 or more for a mentat TCP license for
additional stacks.

From jeffh at cs.rice.edu  Thu May 26 15:22:50 2005
From: jeffh at cs.rice.edu (Jeff Hoye)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] FreePastry 1.4.1 released
Message-ID: <4295E9CA.1040601@cs.rice.edu>

We'd like to announce the availability of FreePastry 1.4.1.  The 
biggest changes since 1.4 are improvements to provide stronger 
guarantees regarding routing consistency and faster detection of faulty 
nodes.  Other features include Java 1.5 compatibility and many bug fixes.

The new release is available from:
http://freepastry.rice.edu/FreePastry/download.html

You can read the release notes at:
http://freepastry.rice.edu/FreePastry/README-1.4.1.html

Routing Consistency:
The new Consistent Join Protocol requires all members of a node's leaf 
set (neighbors) to either respond to a query message or be declared dead 
before a node may join the ring. This ensures that no two nodes claim an 
overlapping range in the ID space.

Fault Detection:
The Periodic LeafSet Protocol has been modified to exchange messages 
which each node's left and right direct neighbors in the ID space every 
20 seconds. If this ping fails then a checkLiveness is initiated, which 
will either reestablish connectivity or declare the neighbor dead within 
30 seconds. This ensures that no ID space remains unclaimed for more 
than 50 seconds.

Happy hacking,
The FreePastry team

From marco at bice.it  Mon May 30 09:27:45 2005
From: marco at bice.it (marco@bice.it)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] Question about Kademlia Search
Message-ID: <1117445265.429adc911c0d7@webmail.bice.it>

Hello.
A simple query on a DHT should have a "lookup(key)" function as its first
operation.
How can an application that uses Kademlia (e.g. Overnet or Emule), send this
query?

I mean, what you do is to search for a string, that is the partial name of an
object (a file).
How is this string transformed in a hash value to look up for on the Kad nodes?

Thank you and sorry for my bad english..

Marco

----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.


From macambira at gmail.com  Mon May 30 15:10:44 2005
From: macambira at gmail.com (Tiago Macambira)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] Question about Kademlia Search
In-Reply-To: <1117445265.429adc911c0d7@webmail.bice.it>
References: <1117445265.429adc911c0d7@webmail.bice.it>
Message-ID: <d4d35da20505300810153a81fb@mail.gmail.com>

On 5/30/05, marco@bice.it <marco@bice.it> wrote:
> Hello.
> A simple query on a DHT should have a "lookup(key)" function as its first
> operation.
> How can an application that uses Kademlia (e.g. Overnet or Emule), send this
> query?
> 
> I mean, what you do is to search for a string, that is the partial name of an
> object (a file).
> How is this string transformed in a hash value to look up for on the Kad nodes?

What Kad (and probably overnet too) does is converting the first word
of the search string to a key (MD4 hash) and sending a query for this
keyword (now a common DHT query). Once if has find nodes close enough
to this key, it will submit the whole query to those nodes.

If you grab its source code, you will find it looking at:

CSearchResultsWnd::DoNewKadSearch - SearchResultsWnd.cpp
CSearchManager::prepareFindKeywords - SearchManager.cpp
Kademlia::KadGetKeywordHash - SearchManager

Hope it helps.

Perhaps someone else can give you some article references on this
problem if this is what you are looking form.

-- 
[]s
Tiago Alves Macambira

From dbarrett at quinthar.com  Tue May 31 10:10:32 2005
From: dbarrett at quinthar.com (David Barrett)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] MTU in the real world
Message-ID: <429C3818.5030606@quinthar.com>

I've read in multiple places that it's best to have a UDP MTU of under 
1500 bytes.  However, it sounds like most of this is based on 
theoretical analysis, and not on real-world experience.

With this in mind, have you tried using a MTU bigger than 1500 bytes and 
been bitten by it?  Basically, do you know of any emperical analysis (of 
any level of formality) of a real-world UDP application that supports or 
refutes the 1500 byte rule of thumb?

Furthermore, I've read that if you "connect" your UDP socket to the 
remote side and then start sending large packets and backing off slowly, 
the socket layer will compute the "real" MTU between two endpoints, and 
you can obtain it through "getsockopt".  Do you know of anyone who's 
tried this, and the results?

-david

From matthew at matthew.at  Tue May 31 13:21:27 2005
From: matthew at matthew.at (Matthew Kaufman)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] MTU in the real world
In-Reply-To: <429C3818.5030606@quinthar.com>
Message-ID: <200505311321.j4VDLLU58817@where.matthew.at>

> I've read in multiple places that it's best to have a UDP MTU 
> of under 1500 bytes.  However, it sounds like most of this is 
> based on theoretical analysis, and not on real-world experience.

It is best to not send traffic that needs to be fragmented, because:

1) Fragment creation is expensive at the point it occurs
2) Reconstruction of fragmented packets is expensive at the receiver
3) (most important) loss of any fragment causes loss of the entire packet,
which magnifies the effect of packet loss (imagine that you lose 1 in every
10 packets on your path... That's 10% loss if you're sending one-piece
packets, but 100% packet loss if you're sending things that get broken up
into enough pieces that 1 in 10 fragments lost causes every packet to be
missing at least one fragment)

Given this, there's a couple of options...
 
> With this in mind, have you tried using a MTU bigger than 
> 1500 bytes and been bitten by it?  

It is hard to tell when you've been "bitten by it", because fragmentation
occurs, packets make it to the far end, and unless you're really paying
attention you don't see the CPU costs or the increased apparent loss rates.
I'm *sure* there are applications written by people who didn't consider
fragmentation which are out there, "work ok", and yet could be a lot better.

> Basically, do you know of 
> any emperical analysis (of any level of formality) of a 
> real-world UDP application that supports or refutes the 1500 
> byte rule of thumb?

The right number of bytes to use for maximum transfer efficiency is the
smallest link MTU on any hop you're travelling through. Bigger, and you
fragment (bad). Smaller, and you're paying more in headers than the optimal
case (not that bad, but something to consider).

There is one case (avoiding head-of-line blocking) where you actually do
want to use significantly less than the MTU... Matters for multimedia
applications on slower or more congested links.

Now, how do you determine the smallest link MTU on any hop you're travelling
through?

There's two good ways... Path MTU Discovery, which sends packets of
increasing size with the "DF" (Don't Fragment) bit set, until they stop
getting through or start causing ICMP Destination Unreachable (Fragmentation
Required) messages, then backs down (usually with a "logical" size
progression, making good guesses about "likely" MTUs... Size of a FDDI
frame, size of an Ethernet frame, etc). Breaks horribly if you encounter a
firewall that blocks the Fragmentation Required ICMP messages. (Typical
cause of "my browser + operating system doesn't work with this bank web site
when they send my transaction history but I could log in ok" kinds of bugs,
or "I can't reach this bank when I'm using my VPN tunnel" kinds of bugs, if
you're an ISP tech support person)

Or "a good guess"... Almost everyone on broadband connections is connected
to their broadband router with Ethernet or Wireless Ethernet. Even if they
aren't, the router at the far end is connected directly or indirectly to one
of the upstream routers with Ethernet, and a whole lot of peering happens
over Ethernet. Almost nobody is using smaller-than-Ethernet MTUs in their
core networks (because they don't want things with broken Path MTU Discovery
(eg., because there's firewalls blocking ICMP at the ends) to break because
of *them*), but all the hosts are connected with Ethernet, so that's a good
argument for 1500. However, I would strongly suggest that you consider the
common tunnel cases, and use 1492 or smaller: 1492 is Ethernet payload
(1500) minus PPPoE header (8). GRE can add 28 or 36 bytes of overhead, IPIP
adds 20, IPsec tunnels add 28, L2TP adds 12. Imagine cases like L2TP tunnel
over PPPoE, and adjust accordingly.

And remember to add back the IP (typically 20) and UDP (8) header size when
you're working out how big your maximum *payload* size is to ensure the
maximum packet size you're looking for.

Also remember to round down to your encryption block size if, like us,
you're encrypting the entire payload.

> 
> Furthermore, I've read that if you "connect" your UDP socket 
> to the remote side and then start sending large packets and 
> backing off slowly, the socket layer will compute the "real" 
> MTU between two endpoints, and you can obtain it through 
> "getsockopt".  Do you know of anyone who's tried this, and 
> the results?

I'm not aware of any operating system that lets you access Path MTU
Discovery like this, but I suppose it might exist. Certainly there are some
which do it for TCP sessions.

Matthew Kaufman
matthew@matthew.at
www.amicima.com
 


From kutzner at ira.uka.de  Tue May 31 13:43:40 2005
From: kutzner at ira.uka.de (Kendy Kutzner)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] MTU in the real world
In-Reply-To: <200505311321.j4VDLLU58817@where.matthew.at>
References: <429C3818.5030606@quinthar.com>
	<200505311321.j4VDLLU58817@where.matthew.at>
Message-ID: <20050531134339.GA27046@dt1.ira.uka.de>

On 2005-05-31T06:21:27-0700, Matthew Kaufman wrote:
> > Furthermore, I've read that if you "connect" your UDP socket 
> > to the remote side and then start sending large packets and 
> > backing off slowly, the socket layer will compute the "real" 
> > MTU between two endpoints, and you can obtain it through 
> > "getsockopt".  Do you know of anyone who's tried this, and 
> > the results?
> 
> I'm not aware of any operating system that lets you access Path MTU
> Discovery like this, but I suppose it might exist. Certainly there are some
> which do it for TCP sessions.

Linux:
$ man 7 ip
[...]
SOCKET OPTIONS
[...]
	IP_MTU 
		Retrieve the current known path MTU of the current
		socket.  Only valid when the socket has been connected.
		Returns an integer. Only valid as a getsockopt(2).

Kendy

-- 


From matthew at matthew.at  Tue May 31 15:58:36 2005
From: matthew at matthew.at (Matthew Kaufman)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] MTU in the real world
In-Reply-To: <20050531134339.GA27046@dt1.ira.uka.de>
Message-ID: <200505311558.j4VFwVU58944@where.matthew.at>

David Barrett:
> > Furthermore, I've read that if you "connect" your UDP socket to the 
> > remote side and then start sending large packets and backing off 
> > slowly, the socket layer will compute the "real"
> > MTU between two endpoints, and you can obtain it through 
> > "getsockopt".  Do you know of anyone who's tried this, and the 
> > results?

Kendy Kutzner:
> Linux:
> $ man 7 ip
> [...]
> SOCKET OPTIONS
> [...]
> 	IP_MTU 
> 		Retrieve the current known path MTU of the current
> 		socket.  Only valid when the socket has been connected.
> 		Returns an integer. Only valid as a getsockopt(2).

One must read in detail the part about what this returns for an unconnected
UDP socket (nothing useful at all), a connected UDP socket to a destination
for which Path MTU Discovery hasn't happened (the "initial estimate of the
path MTU", not to be confused with the actual Path MTU) and how the
IP_PMTU_DISCOVER option works with UDP sockets ("kernel rejects packets
(with EMSGSIZE error) that are bigger than the known path MTU if flag set").

You can, apparently, do the following, on a Linux system that supports Path
MTU Discovery: connect a UDP socket to a destination, set the socket to
IP_PMTUDISC_DO, start sending huge datagrams (that you need to know will
probably be lost), wait for EMSGSIZE errors, and watch the IP_MTU answer
(tricky, because at first it'll be the "estimate" and later it'll be the
"known"... Whether it'll ever be bigger than the "estimate" is probably
implementation-dependent and I'm not looking at the kernel sources). That's
a little more involved than simply "sending large packets and backing off
slowly". (It'd be as simple as watching the EMSGSIZE errors *if* Path MTU
Discovery were enabled by default for anything other than SOCK_STREAM, which
it is not)

The other way is to NOT connect the socket, set IP_PMTUDISC_DO, and watch
the IP_RECVERR queue for MTU updates to your destinations. Other operating
systems support some similar ways to watch for incoming errors caused by the
Path MTU Discovery process.

However, I *still* maintain that Path MTU Discovery is fraught with danger
in any environment where ICMP messages might get lost, which pretty much
includes anywhere you have real firewalls in the path, and that furthermore,
you are very unlikely to see an MTU greater than 1500 except in very
specialized environments.

Matthew Kaufman
matthew@matthew.at
www.amicima.com





From Serguei.Osokine at efi.com  Tue May 31 17:18:30 2005
From: Serguei.Osokine at efi.com (Serguei Osokine)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] MTU in the real world
Message-ID: <4A60C83D027E224BAA4550FB1A2B120E641A33@fcexmb04.efi.internal>

On Tuesday, May 31, 2005 David Barrett wrote:
> With this in mind, have you tried using a MTU bigger than 1500 bytes
> and been bitten by it?

	Yes. That was not your typical everyday situation, but I think
some on this list might find it entertaining anyway:

	We tried to use UDP to transfer stuff over a gigabit LAN inside
the cluster. Pretty soon we discovered that with small (~1500 byte) 
packets the CPU was the bottleneck, because you can send only so many
packets per second, and the resulting throughput was nowhere close to
a gigabit. (You have to send almost 100K such packets a second to
achieve a gigabit throughput, and we were doing several times less
on our 2-CPU 2.4GHz Win XP boxes.)

	So then we tried to increase the UDP datagram size. The gigabit 
switch did not support jumbo frames, by the way, so we were fragmenting
as soon as we exceeded 1500. The throughput went up, and was pretty 
decent with 64-KB datgrams (don't remember the exact numbers, but it
was close to a gigabit and generally everything was peachy).

	Which is when the funny things started to happen. In the middle
of a test, the communication channel would just shut down and nothing 
would be delivered over it for a minute or two (though both the sender
and the receiver kept looking fine and no errors were returned by the
socket calls - sender was sending data, but the receiver recfrom()
call was not getting it); after that pause the channel would wake up 
as if nothing happened (except for several gigabytes of lost data),
work normally for a few minutes, after which this shutdown would be 
repeated, and so on. 

	Took us a while to figure out what was going on, but here is the
scoop: the gigabit LAN had a fairly small, but nonetheless non-zero
packet loss rate. When one 1500-byte frame from a 64-KB datgram is
lost, the rest of the datagram frames (all 62 KB)have to be buffered
somewhere in case the missing frame arrives and the datagram can be
fully reassembled. This arrival will never happen, but the socket 
layer does not know that, so it has to keep the partial datagram for 
a while, discarding all its frames if the missing frame won't arrive
before some timeout (RFC 1122 recommends this timout value to be 
between 60 and 120 seconds, and this seems to be in line with what
we saw).

	Now, the gigabit link sends quite a lot of data - 100MB+ per
second, to be precise. Even with 0.01% loss rate, you're losing about
10,000 bytes per second. This is no big deal, but every 1500 bytes lost
cause you to store 62KBs of partial datagrams, so with the loss rate
above you have to store 400 KB of new data every second. If this data 
expires in 120 seconds, you need about 50 MB for the partial datagram 
storage in the socket layer - and proportionally more if your data loss
rate is higher than 0.01%. And this amount of memory is something that 
the socket layer in Win XP simply does not have. So as soon as it runs
out of memory for the partially assembled datagrams, it stops the data
delivery and waits for the memory to be released. Apparently after it
gets enough free memory, it switches the data delivery back on again.

	This approach does seem funny, and I don't see any compelling 
reason for the socket layer to handle that situation in this "trigger"
fashion - either it works normally, or shuts down the data delivery
completely. Might have handled this a bit more gracefully, I'd think.
But this was Windows, and there was no arguing with it. (We were stuck
with Windows for unrelated reasons.)

	So the bottom line was, we had to go with TCP, because there was 
no way we could make the UDP transport that would be both fast enough
and would work on our hardware/OS combination. And the part about
"would work" was definitely related to an attempt to send the datgrams
that would exceed MTU. (Datagrams smaller than MTU sucked performance-
wise when compared to TCP, but that is another story - gigabit cards
tend to offload plenty of TCP functionality from the CPU, so it was
not that the UDP was particularly bad, but rather that TCP performance
was very good.)

	Best wishes -
	S.Osokine.
	31 May 2005.

-----Original Message-----
From: p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org]On
Behalf Of David Barrett
Sent: Tuesday, May 31, 2005 3:11 AM
To: Peer-to-peer development.
Subject: [p2p-hackers] MTU in the real world


I've read in multiple places that it's best to have a UDP MTU of under 
1500 bytes.  However, it sounds like most of this is based on 
theoretical analysis, and not on real-world experience.

With this in mind, have you tried using a MTU bigger than 1500 bytes and 
been bitten by it?  Basically, do you know of any emperical analysis (of 
any level of formality) of a real-world UDP application that supports or 
refutes the 1500 byte rule of thumb?

Furthermore, I've read that if you "connect" your UDP socket to the 
remote side and then start sending large packets and backing off slowly, 
the socket layer will compute the "real" MTU between two endpoints, and 
you can obtain it through "getsockopt".  Do you know of anyone who's 
tried this, and the results?

-david
_______________________________________________
p2p-hackers mailing list
p2p-hackers@zgp.org
http://zgp.org/mailman/listinfo/p2p-hackers
_______________________________________________
Here is a web page listing P2P Conferences:
http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences

From matthew at matthew.at  Tue May 31 17:28:29 2005
From: matthew at matthew.at (Matthew Kaufman)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] MTU in the real world
In-Reply-To: <4A60C83D027E224BAA4550FB1A2B120E641A33@fcexmb04.efi.internal>
Message-ID: <200505311728.j4VHSNU59047@where.matthew.at>

>From Serguei Osokine:
> 	This approach does seem funny, and I don't see any 
> compelling reason for the socket layer to handle that 
> situation in this "trigger"
> fashion - either it works normally, or shuts down the data 
> delivery completely. Might have handled this a bit more 
> gracefully, I'd think.

The wider the gap between "on" and "off", the more CPU time and memory is
available for the consumer side... It reduces the potential for livelock.

Matthew


From arachnid at notdot.net  Tue May 31 20:27:17 2005
From: arachnid at notdot.net (Nick Johnson)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] MTU in the real world
In-Reply-To: <200505311321.j4VDLLU58817@where.matthew.at>
References: <200505311321.j4VDLLU58817@where.matthew.at>
Message-ID: <429CC8A5.1040903@notdot.net>

Matthew Kaufman wrote:

>3) (most important) loss of any fragment causes loss of the entire packet,
>which magnifies the effect of packet loss (imagine that you lose 1 in every
>10 packets on your path... That's 10% loss if you're sending one-piece
>packets, but 100% packet loss if you're sending things that get broken up
>into enough pieces that 1 in 10 fragments lost causes every packet to be
>missing at least one fragment)
>  
>
Not exactly. Statistically, it'll taper off asympotically. For example, 
10% actual packet loss with each packet fragmented into 10 is (1 - 
.9^10) or 65% apparrent packet loss.

-Nick Johnson

From dbarrett at quinthar.com  Tue May 31 21:38:14 2005
From: dbarrett at quinthar.com (David Barrett)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] MTU in the real world
In-Reply-To: <200505311321.j4VDLLU58817@where.matthew.at>
References: <200505311321.j4VDLLU58817@where.matthew.at>
Message-ID: <1117575499.9C8F855@bf11.dngr.org>

Matthew -- Thanks for the extensive detail on the origin of the 1500 
number (or actually, 1492).  That helps me understand and have 
confidence in it.

But even given that, have you ever tried to go above it (perhaps in your 
work with Amicima or before)?  What kind of symtoms did you experience?

-david

On Tue, 31 May 2005 6:58 am, Matthew Kaufman wrote:
>>  I've read in multiple places that it's best to have a UDP MTU
>>  of under 1500 bytes.  However, it sounds like most of this is
>>  based on theoretical analysis, and not on real-world experience.
>
> It is best to not send traffic that needs to be fragmented, because:
>
> 1) Fragment creation is expensive at the point it occurs
> 2) Reconstruction of fragmented packets is expensive at the receiver
> 3) (most important) loss of any fragment causes loss of the entire 
> packet,
> which magnifies the effect of packet loss (imagine that you lose 1 in 
> every
> 10 packets on your path... That's 10% loss if you're sending one-piece
> packets, but 100% packet loss if you're sending things that get broken 
> up
> into enough pieces that 1 in 10 fragments lost causes every packet to 
> be
> missing at least one fragment)
>
> Given this, there's a couple of options...
>
>>  With this in mind, have you tried using a MTU bigger than
>>  1500 bytes and been bitten by it?
>
> It is hard to tell when you've been "bitten by it", because 
> fragmentation
> occurs, packets make it to the far end, and unless you're really paying
> attention you don't see the CPU costs or the increased apparent loss 
> rates.
> I'm *sure* there are applications written by people who didn't consider
> fragmentation which are out there, "work ok", and yet could be a lot 
> better.
>
>>  Basically, do you know of
>>  any emperical analysis (of any level of formality) of a
>>  real-world UDP application that supports or refutes the 1500
>>  byte rule of thumb?
>
> The right number of bytes to use for maximum transfer efficiency is the
> smallest link MTU on any hop you're travelling through. Bigger, and you
> fragment (bad). Smaller, and you're paying more in headers than the 
> optimal
> case (not that bad, but something to consider).
>
> There is one case (avoiding head-of-line blocking) where you actually 
> do
> want to use significantly less than the MTU... Matters for multimedia
> applications on slower or more congested links.
>
> Now, how do you determine the smallest link MTU on any hop you're 
> travelling
> through?
>
> There's two good ways... Path MTU Discovery, which sends packets of
> increasing size with the "DF" (Don't Fragment) bit set, until they stop
> getting through or start causing ICMP Destination Unreachable 
> (Fragmentation
> Required) messages, then backs down (usually with a "logical" size
> progression, making good guesses about "likely" MTUs... Size of a FDDI
> frame, size of an Ethernet frame, etc). Breaks horribly if you 
> encounter a
> firewall that blocks the Fragmentation Required ICMP messages. (Typical
> cause of "my browser + operating system doesn't work with this bank web 
> site
> when they send my transaction history but I could log in ok" kinds of 
> bugs,
> or "I can't reach this bank when I'm using my VPN tunnel" kinds of 
> bugs, if
> you're an ISP tech support person)
>
> Or "a good guess"... Almost everyone on broadband connections is 
> connected
> to their broadband router with Ethernet or Wireless Ethernet. Even if 
> they
> aren't, the router at the far end is connected directly or indirectly 
> to one
> of the upstream routers with Ethernet, and a whole lot of peering 
> happens
> over Ethernet. Almost nobody is using smaller-than-Ethernet MTUs in 
> their
> core networks (because they don't want things with broken Path MTU 
> Discovery
> (eg., because there's firewalls blocking ICMP at the ends) to break 
> because
> of *them*), but all the hosts are connected with Ethernet, so that's a 
> good
> argument for 1500. However, I would strongly suggest that you consider 
> the
> common tunnel cases, and use 1492 or smaller: 1492 is Ethernet payload
> (1500) minus PPPoE header (8). GRE can add 28 or 36 bytes of overhead, 
> IPIP
> adds 20, IPsec tunnels add 28, L2TP adds 12. Imagine cases like L2TP 
> tunnel
> over PPPoE, and adjust accordingly.
>
> And remember to add back the IP (typically 20) and UDP (8) header size 
> when
> you're working out how big your maximum *payload* size is to ensure the
> maximum packet size you're looking for.
>
> Also remember to round down to your encryption block size if, like us,
> you're encrypting the entire payload.
>
>>
>>  Furthermore, I've read that if you "connect" your UDP socket
>>  to the remote side and then start sending large packets and
>>  backing off slowly, the socket layer will compute the "real"
>>  MTU between two endpoints, and you can obtain it through
>>  "getsockopt".  Do you know of anyone who's tried this, and
>>  the results?
>
> I'm not aware of any operating system that lets you access Path MTU
> Discovery like this, but I suppose it might exist. Certainly there are 
> some
> which do it for TCP sessions.
>
> Matthew Kaufman
> matthew@matthew.at
> www.amicima.com
>
>
> _______________________________________________
> p2p-hackers mailing list
> p2p-hackers@zgp.org
> http://zgp.org/mailman/listinfo/p2p-hackers
> _______________________________________________
> Here is a web page listing P2P Conferences:
> http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences

From dbarrett at quinthar.com  Tue May 31 21:46:18 2005
From: dbarrett at quinthar.com (David Barrett)
Date: Sat Dec  9 22:12:53 2006
Subject: [p2p-hackers] MTU in the real world
In-Reply-To: <4A60C83D027E224BAA4550FB1A2B120E641A33@fcexmb04.efi.internal>
References: <4A60C83D027E224BAA4550FB1A2B120E641A33@fcexmb04.efi.internal>
Message-ID: <1117575984.1743E4C4@bd8.dngr.org>

Ah, thanks -- this is precisely the kind of story I'm looking to hear.  
But I agree its conditions are a bit unusual.  Do you know of any 
similar attempts of using big MTUs over a standard consumer internet 
connection?

-david

On Tue, 31 May 2005 10:40 am, Serguei Osokine wrote:
> On Tuesday, May 31, 2005 David Barrett wrote:
>>  With this in mind, have you tried using a MTU bigger than 1500 bytes
>>  and been bitten by it?
>
> 	Yes. That was not your typical everyday situation, but I think
> some on this list might find it entertaining anyway:
>
> 	We tried to use UDP to transfer stuff over a gigabit LAN inside
> the cluster. Pretty soon we discovered that with small (~1500 byte)
> packets the CPU was the bottleneck, because you can send only so many
> packets per second, and the resulting throughput was nowhere close to
> a gigabit. (You have to send almost 100K such packets a second to
> achieve a gigabit throughput, and we were doing several times less
> on our 2-CPU 2.4GHz Win XP boxes.)
>
> 	So then we tried to increase the UDP datagram size. The gigabit
> switch did not support jumbo frames, by the way, so we were fragmenting
> as soon as we exceeded 1500. The throughput went up, and was pretty
> decent with 64-KB datgrams (don't remember the exact numbers, but it
> was close to a gigabit and generally everything was peachy).
>
> 	Which is when the funny things started to happen. In the middle
> of a test, the communication channel would just shut down and nothing
> would be delivered over it for a minute or two (though both the sender
> and the receiver kept looking fine and no errors were returned by the
> socket calls - sender was sending data, but the receiver recfrom()
> call was not getting it); after that pause the channel would wake up
> as if nothing happened (except for several gigabytes of lost data),
> work normally for a few minutes, after which this shutdown would be
> repeated, and so on.
>
> 	Took us a while to figure out what was going on, but here is the
> scoop: the gigabit LAN had a fairly small, but nonetheless non-zero
> packet loss rate. When one 1500-byte frame from a 64-KB datgram is
> lost, the rest of the datagram frames (all 62 KB)have to be buffered
> somewhere in case the missing frame arrives and the datagram can be
> fully reassembled. This arrival will never happen, but the socket
> layer does not know that, so it has to keep the partial datagram for
> a while, discarding all its frames if the missing frame won't arrive
> before some timeout (RFC 1122 recommends this timout value to be
> between 60 and 120 seconds, and this seems to be in line with what
> we saw).
>
> 	Now, the gigabit link sends quite a lot of data - 100MB+ per
> second, to be precise. Even with 0.01% loss rate, you're losing about
> 10,000 bytes per second. This is no big deal, but every 1500 bytes lost
> cause you to store 62KBs of partial datagrams, so with the loss rate
> above you have to store 400 KB of new data every second. If this data
> expires in 120 seconds, you need about 50 MB for the partial datagram
> storage in the socket layer - and proportionally more if your data loss
> rate is higher than 0.01%. And this amount of memory is something that
> the socket layer in Win XP simply does not have. So as soon as it runs
> out of memory for the partially assembled datagrams, it stops the data
> delivery and waits for the memory to be released. Apparently after it
> gets enough free memory, it switches the data delivery back on again.
>
> 	This approach does seem funny, and I don't see any compelling
> reason for the socket layer to handle that situation in this "trigger"
> fashion - either it works normally, or shuts down the data delivery
> completely. Might have handled this a bit more gracefully, I'd think.
> But this was Windows, and there was no arguing with it. (We were stuck
> with Windows for unrelated reasons.)
>
> 	So the bottom line was, we had to go with TCP, because there was
> no way we could make the UDP transport that would be both fast enough
> and would work on our hardware/OS combination. And the part about
> "would work" was definitely related to an attempt to send the datgrams
> that would exceed MTU. (Datagrams smaller than MTU sucked performance-
> wise when compared to TCP, but that is another story - gigabit cards
> tend to offload plenty of TCP functionality from the CPU, so it was
> not that the UDP was particularly bad, but rather that TCP performance
> was very good.)
>
> 	Best wishes -
> 	S.Osokine.
> 	31 May 2005.
>
> -----Original Message-----
> From: p2p-hackers-bounces@zgp.org 
> [mailto:p2p-hackers-bounces@zgp.org]On
> Behalf Of David Barrett
> Sent: Tuesday, May 31, 2005 3:11 AM
> To: Peer-to-peer development.
> Subject: [p2p-hackers] MTU in the real world
>
>
> I've read in multiple places that it's best to have a UDP MTU of under
> 1500 bytes.  However, it sounds like most of this is based on
> theoretical analysis, and not on real-world experience.
>
> With this in mind, have you tried using a MTU bigger than 1500 bytes 
> and
> been bitten by it?  Basically, do you know of any emperical analysis 
> (of
> any level of formality) of a real-world UDP application that supports 
> or
> refutes the 1500 byte rule of thumb?
>
> Furthermore, I've read that if you "connect" your UDP socket to the
> remote side and then start sending large packets and backing off 
> slowly,
> the socket layer will compute the "real" MTU between two endpoints, and
> you can obtain it through "getsockopt".  Do you know of anyone who's
> tried this, and the results?
>
> -david
> _______________________________________________
> p2p-hackers mailing list
> p2p-hackers@zgp.org
> http://zgp.org/mailman/listinfo/p2p-hackers
> _______________________________________________
> Here is a web page listing P2P Conferences:
> http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences
> _______________________________________________
> p2p-hackers mailing list
> p2p-hackers@zgp.org
> http://zgp.org/mailman/listinfo/p2p-hackers
> _______________________________________________
> Here is a web page listing P2P Conferences:
> http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences

From Serguei.Osokine at efi.com  Tue May 31 21:57:50 2005
From: Serguei.Osokine at efi.com (Serguei Osokine)
Date: Sat Dec  9 22:12:54 2006
Subject: [p2p-hackers] MTU in the real world
Message-ID: <4A60C83D027E224BAA4550FB1A2B120E641A39@fcexmb04.efi.internal>

On Tuesday, May 31, 2005 David Barrett wrote:
> Do you know of any similar attempts of using big MTUs over a standard
> consumer internet connection?

	Sorry, no. Typically people try to avoid the complications and
ineffectiveness related to the packet fragmentation and reassembly, so
they mostly try to avoid it. I'd even venture a guess that more often
than not people who do use big packets over a standard Internet have 
no idea that they are doing that (otherwise they'd try to avoid doing 
so) and thus cannot tell any tales about it :-)

	Best wishes -
	S.Osokine.
	31 May 2005.


-----Original Message-----
From: p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org]On
Behalf Of David Barrett
Sent: Tuesday, May 31, 2005 2:46 PM
To: Peer-to-peer development.
Subject: Re: [p2p-hackers] MTU in the real world


Ah, thanks -- this is precisely the kind of story I'm looking to hear.  
But I agree its conditions are a bit unusual.  Do you know of any 
similar attempts of using big MTUs over a standard consumer internet 
connection?

-david

On Tue, 31 May 2005 10:40 am, Serguei Osokine wrote:
> On Tuesday, May 31, 2005 David Barrett wrote:
>>  With this in mind, have you tried using a MTU bigger than 1500 bytes
>>  and been bitten by it?
>
> 	Yes. That was not your typical everyday situation, but I think
> some on this list might find it entertaining anyway:
>
> 	We tried to use UDP to transfer stuff over a gigabit LAN inside
> the cluster. Pretty soon we discovered that with small (~1500 byte)
> packets the CPU was the bottleneck, because you can send only so many
> packets per second, and the resulting throughput was nowhere close to
> a gigabit. (You have to send almost 100K such packets a second to
> achieve a gigabit throughput, and we were doing several times less
> on our 2-CPU 2.4GHz Win XP boxes.)
>
> 	So then we tried to increase the UDP datagram size. The gigabit
> switch did not support jumbo frames, by the way, so we were fragmenting
> as soon as we exceeded 1500. The throughput went up, and was pretty
> decent with 64-KB datgrams (don't remember the exact numbers, but it
> was close to a gigabit and generally everything was peachy).
>
> 	Which is when the funny things started to happen. In the middle
> of a test, the communication channel would just shut down and nothing
> would be delivered over it for a minute or two (though both the sender
> and the receiver kept looking fine and no errors were returned by the
> socket calls - sender was sending data, but the receiver recfrom()
> call was not getting it); after that pause the channel would wake up
> as if nothing happened (except for several gigabytes of lost data),
> work normally for a few minutes, after which this shutdown would be
> repeated, and so on.
>
> 	Took us a while to figure out what was going on, but here is the
> scoop: the gigabit LAN had a fairly small, but nonetheless non-zero
> packet loss rate. When one 1500-byte frame from a 64-KB datgram is
> lost, the rest of the datagram frames (all 62 KB)have to be buffered
> somewhere in case the missing frame arrives and the datagram can be
> fully reassembled. This arrival will never happen, but the socket
> layer does not know that, so it has to keep the partial datagram for
> a while, discarding all its frames if the missing frame won't arrive
> before some timeout (RFC 1122 recommends this timout value to be
> between 60 and 120 seconds, and this seems to be in line with what
> we saw).
>
> 	Now, the gigabit link sends quite a lot of data - 100MB+ per
> second, to be precise. Even with 0.01% loss rate, you're losing about
> 10,000 bytes per second. This is no big deal, but every 1500 bytes lost
> cause you to store 62KBs of partial datagrams, so with the loss rate
> above you have to store 400 KB of new data every second. If this data
> expires in 120 seconds, you need about 50 MB for the partial datagram
> storage in the socket layer - and proportionally more if your data loss
> rate is higher than 0.01%. And this amount of memory is something that
> the socket layer in Win XP simply does not have. So as soon as it runs
> out of memory for the partially assembled datagrams, it stops the data
> delivery and waits for the memory to be released. Apparently after it
> gets enough free memory, it switches the data delivery back on again.
>
> 	This approach does seem funny, and I don't see any compelling
> reason for the socket layer to handle that situation in this "trigger"
> fashion - either it works normally, or shuts down the data delivery
> completely. Might have handled this a bit more gracefully, I'd think.
> But this was Windows, and there was no arguing with it. (We were stuck
> with Windows for unrelated reasons.)
>
> 	So the bottom line was, we had to go with TCP, because there was
> no way we could make the UDP transport that would be both fast enough
> and would work on our hardware/OS combination. And the part about
> "would work" was definitely related to an attempt to send the datgrams
> that would exceed MTU. (Datagrams smaller than MTU sucked performance-
> wise when compared to TCP, but that is another story - gigabit cards
> tend to offload plenty of TCP functionality from the CPU, so it was
> not that the UDP was particularly bad, but rather that TCP performance
> was very good.)
>
> 	Best wishes -
> 	S.Osokine.
> 	31 May 2005.
>
> -----Original Message-----
> From: p2p-hackers-bounces@zgp.org 
> [mailto:p2p-hackers-bounces@zgp.org]On
> Behalf Of David Barrett
> Sent: Tuesday, May 31, 2005 3:11 AM
> To: Peer-to-peer development.
> Subject: [p2p-hackers] MTU in the real world
>
>
> I've read in multiple places that it's best to have a UDP MTU of under
> 1500 bytes.  However, it sounds like most of this is based on
> theoretical analysis, and not on real-world experience.
>
> With this in mind, have you tried using a MTU bigger than 1500 bytes 
> and
> been bitten by it?  Basically, do you know of any emperical analysis 
> (of
> any level of formality) of a real-world UDP application that supports 
> or
> refutes the 1500 byte rule of thumb?
>
> Furthermore, I've read that if you "connect" your UDP socket to the
> remote side and then start sending large packets and backing off 
> slowly,
> the socket layer will compute the "real" MTU between two endpoints, and
> you can obtain it through "getsockopt".  Do you know of anyone who's
> tried this, and the results?
>
> -david
> _______________________________________________
> p2p-hackers mailing list
> p2p-hackers@zgp.org
> http://zgp.org/mailman/listinfo/p2p-hackers
> _______________________________________________
> Here is a web page listing P2P Conferences:
> http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences
> _______________________________________________
> p2p-hackers mailing list
> p2p-hackers@zgp.org
> http://zgp.org/mailman/listinfo/p2p-hackers
> _______________________________________________
> Here is a web page listing P2P Conferences:
> http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences
_______________________________________________
p2p-hackers mailing list
p2p-hackers@zgp.org
http://zgp.org/mailman/listinfo/p2p-hackers
_______________________________________________
Here is a web page listing P2P Conferences:
http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences

From matthew at matthew.at  Tue May 31 21:58:21 2005
From: matthew at matthew.at (Matthew Kaufman)
Date: Sat Dec  9 22:12:54 2006
Subject: [p2p-hackers] MTU in the real world
In-Reply-To: <1117575499.9C8F855@bf11.dngr.org>
Message-ID: <200505312158.j4VLwQU59314@where.matthew.at>

David Barrett: 
> Matthew -- Thanks for the extensive detail on the origin of 
> the 1500 number (or actually, 1492).  That helps me 
> understand and have confidence in it.

You're welcome. Before I got back into software, I used to design and
implement nationwide IP and ATM backbones, transit, and peering for ISPs, so
I know all too well what customers say when you crank the MTU down (like
when you throw a GRE tunnel into a path to try to bypass something), and
where the MTU limits come from.
 
> But even given that, have you ever tried to go above it 
> (perhaps in your work with Amicima or before)?  What kind of 
> symtoms did you experience?

The amicima MFP protocol is carefully designed to stay below the expected
MTU at all times, and in fact also worries about head-of-line blocking on
slow links.

Every time you use un-tuned NFS over UDP on a WAN link, you're sending big
(4k or 8k) packets and relying on fragmentation to make things work. It
works, but it isn't optimal.

Matthew Kaufman
matthew@matthew.at
www.amicima.com



From matthew at matthew.at  Tue May 31 22:03:56 2005
From: matthew at matthew.at (Matthew Kaufman)
Date: Sat Dec  9 22:12:54 2006
Subject: [p2p-hackers] MTU in the real world
In-Reply-To: <429CC8A5.1040903@notdot.net>
Message-ID: <200505312204.j4VM41U59339@where.matthew.at>

Nick Johnson:
> Not exactly. Statistically, it'll taper off asympotically. 
> For example, 10% actual packet loss with each packet 
> fragmented into 10 is (1 -
> .9^10) or 65% apparrent packet loss.

Actually my claim is correct "if... 1 in 10 fragments lost causes every
packet to be missing at least one fragment" (admittedly not the likely
case)... However you are absolutely right that, statistically, the most
likely case is that it will cause the packet loss you've described. Still a
lot worse than the original 10%... Related closely to why congested ATM
networks cause IP to experience a "brick wall" at the congestion point.

Short story: fragmentation is bad in almost every case. Unless you are in
one of the very rare and very specific cases where it makes sense (an
example might be where you can send giant packets efficiently, and can
afford to have them fragmented elsewhere, and can manage the extra loss risk
(and as Serguei pointed out, receiving host overhead when it occurs)), you
should be trying to stay under the path MTU, either by discovering the
actual MTU, or making your best guess as to what it might be.

Matthew Kaufman
matthew@matthew.at
www.amicima.com