From dcarboni at gmail.com Thu Feb 2 17:57:09 2006 From: dcarboni at gmail.com (Davide "dada" Carboni) Date: Sat Dec 9 22:13:09 2006 Subject: [p2p-hackers] overlay netowork Message-ID: <71b79fa90602020957n28557e11j7f0b4cbbd1a22b35@mail.gmail.com> Hi, I'm searching a definition for 'overlay network' and how this concept is related to P2P computing. Thanx. -- La gente vuole le manopole (Anonimo inventore) -- http://people.crs4.it/dcarboni From trep at cs.ucr.edu Fri Feb 3 00:08:08 2006 From: trep at cs.ucr.edu (Thomas Repantis) Date: Sat Dec 9 22:13:09 2006 Subject: [p2p-hackers] overlay netowork In-Reply-To: <71b79fa90602020957n28557e11j7f0b4cbbd1a22b35@mail.gmail.com> References: <71b79fa90602020957n28557e11j7f0b4cbbd1a22b35@mail.gmail.com> Message-ID: <20060203000808.GA2888@angeldust.chaos> http://en.wikipedia.org/wiki/Overlay_network Thomas On Thu, Feb 02, 2006 at 06:57:09PM +0100, Davide dada Carboni wrote: > Hi, > I'm searching a definition for 'overlay network' and how this concept > is related to P2P computing. > Thanx. -- http://www.cs.ucr.edu/~trep From dcarboni at gmail.com Fri Feb 3 11:59:15 2006 From: dcarboni at gmail.com (Davide "dada" Carboni) Date: Sat Dec 9 22:13:09 2006 Subject: [p2p-hackers] overlay netowork In-Reply-To: <20060203000808.GA2888@angeldust.chaos> References: <71b79fa90602020957n28557e11j7f0b4cbbd1a22b35@mail.gmail.com> <20060203000808.GA2888@angeldust.chaos> Message-ID: <71b79fa90602030359q22d5d55bp10d26052c6291908@mail.gmail.com> On 2/3/06, Thomas Repantis wrote: > http://en.wikipedia.org/wiki/Overlay_network > > Thomas Oops. So easy?! Thanx From matthew at allpeers.com Wed Feb 8 20:03:50 2006 From: matthew at allpeers.com (Matthew Gertner) Date: Sat Dec 9 22:13:09 2006 Subject: [p2p-hackers] Keep Alive and network performance In-Reply-To: <71b79fa90602030359q22d5d55bp10d26052c6291908@mail.gmail.com> References: <71b79fa90602020957n28557e11j7f0b4cbbd1a22b35@mail.gmail.com> <20060203000808.GA2888@angeldust.chaos> <71b79fa90602030359q22d5d55bp10d26052c6291908@mail.gmail.com> Message-ID: <43EA4EA6.8040803@allpeers.com> Hi, We have a straightforward P2P topology with each peer in the network attached to hub. We want to detect efficiently that a peer has disconnected from the hub. The obvious way to do this would be to use our current TCP connection with a relatively short Keep Alive value (a few seconds so we could see the peer's presence updated quickly if they don't disconnect cleanly). Does anyone have a feeling for the performance implications of this approach if many peers are attached to a single hub? Is there another approach that is recommend (such as using a custom UDP-based keep alive protocol)? Regards, Matt From alenlpeacock at gmail.com Thu Feb 9 04:46:33 2006 From: alenlpeacock at gmail.com (Alen Peacock) Date: Sat Dec 9 22:13:09 2006 Subject: [p2p-hackers] Petnames Go Mainstream Message-ID: First Schneier linked to Mark Stiegler's paper in his blog today ( http://www.schneier.com/blog/archives/2006/02/petnames.html), then boingboing picked up the story ( http://www.boingboing.net/2006/02/08/fascinating_solution.html). I imagine you are getting a nice little surge in traffic at your site today, Zooko? Nice to see some of this stuff being explained to the masses (even if those masses are mostly made up of technical folk anyway) -- it makes proselytizing this type of technology more feasible when a general audience has some grasp of the concepts. Alen -------------- next part -------------- An HTML attachment was scrubbed... URL: http://zgp.org/pipermail/p2p-hackers/attachments/20060208/d9f82be9/attachment.htm From lgonze at panix.com Thu Feb 9 05:14:52 2006 From: lgonze at panix.com (Lucas Gonze) Date: Sat Dec 9 22:13:09 2006 Subject: [p2p-hackers] Petnames Go Mainstream In-Reply-To: References: Message-ID: Popularization is always nice, I guess, but I can't believe that neither Schneier nor Cory knew about this stuff, which is as basic as air in the p2p subculture. On Wed, 8 Feb 2006, Alen Peacock wrote: > First Schneier linked to Mark Stiegler's paper in his blog today ( > http://www.schneier.com/blog/archives/2006/02/petnames.html), then > boingboing picked up the story ( > http://www.boingboing.net/2006/02/08/fascinating_solution.html). > > I imagine you are getting a nice little surge in traffic at your site today, > Zooko? > > Nice to see some of this stuff being explained to the masses (even if those > masses are mostly made up of technical folk anyway) -- it makes > proselytizing this type of technology more feasible when a general audience > has some grasp of the concepts. > > Alen > From turbogeek at cluck.com Thu Feb 9 07:34:53 2006 From: turbogeek at cluck.com (Daniel B.) Date: Sat Dec 9 22:13:09 2006 Subject: [p2p-hackers] Keep Alive and network performance In-Reply-To: <43EA4EA6.8040803@allpeers.com> References: <71b79fa90602020957n28557e11j7f0b4cbbd1a22b35@mail.gmail.com> <20060203000808.GA2888@angeldust.chaos> <71b79fa90602030359q22d5d55bp10d26052c6291908@mail.gmail.com> <43EA4EA6.8040803@allpeers.com> Message-ID: <5037146A-FF9F-4786-BD8A-200BABF490DD@cluck.com> Keep alive is both a missing component in practical P2P and a reason why P2P is also hard to make practical. The trap is that the keep-alive is between two machines. Imagine hundreds of thousands of machines!!!! Even with small signatures matching each machine, we are talking about a flooded network. A secondary trap is managing reliability. Reliability in P2P is not moving bytes back and forth, but also taking into account the machine may be a laptop or other personal computer that has a pathetic uptime. The better choice is online/offline signaling and optimistic communications with good error control. On Feb 8, 2006, at 12:03 PM, Matthew Gertner wrote: > Hi, > > We have a straightforward P2P topology with each peer in the > network attached to hub. We want to detect efficiently that a peer > has disconnected from the hub. The obvious way to do this would be > to use our current TCP connection with a relatively short Keep > Alive value (a few seconds so we could see the peer's presence > updated quickly if they don't disconnect cleanly). Does anyone have > a feeling for the performance implications of this approach if many > peers are attached to a single hub? Is there another approach that > is recommend (such as using a custom UDP-based keep alive protocol)? > > Regards, > Matt > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > From matthew at allpeers.com Thu Feb 9 09:34:19 2006 From: matthew at allpeers.com (Matthew Gertner) Date: Sat Dec 9 22:13:09 2006 Subject: [p2p-hackers] Keep Alive and network performance In-Reply-To: <5037146A-FF9F-4786-BD8A-200BABF490DD@cluck.com> References: <71b79fa90602020957n28557e11j7f0b4cbbd1a22b35@mail.gmail.com> <20060203000808.GA2888@angeldust.chaos> <71b79fa90602030359q22d5d55bp10d26052c6291908@mail.gmail.com> <43EA4EA6.8040803@allpeers.com> <5037146A-FF9F-4786-BD8A-200BABF490DD@cluck.com> Message-ID: <43EB0C9B.7090506@allpeers.com> Daniel B. wrote: > Keep alive is both a missing component in practical P2P and a reason > why P2P is also hard to make practical. > > The trap is that the keep-alive is between two machines. Imagine > hundreds of thousands of machines!!!! Even with small signatures > matching each machine, we are talking about a flooded network. A > secondary trap is managing reliability. Reliability in P2P is not > moving bytes back and forth, but also taking into account the machine > may be a laptop or other personal computer that has a pathetic uptime. > > The better choice is online/offline signaling and optimistic > communications with good error control. I assume that by "optimistic communications" you mean that we should assume a connection exists and confirm this using periodic pings? Isn't this basically the equivalent of KA? It seems to me that the fundamental tradeoff is speed of notification vs. extra network traffic (i.e. you know more quickly when someone's network connection is cut if you ping more often), and this is the case both for KA and for a custom optimistic approach that uses pings. Am I missing something? Cheers, Matt From turbogeek at cluck.com Thu Feb 9 15:30:44 2006 From: turbogeek at cluck.com (Daniel B.) Date: Sat Dec 9 22:13:09 2006 Subject: [p2p-hackers] Keep Alive and network performance In-Reply-To: <43EB0C9B.7090506@allpeers.com> References: <71b79fa90602020957n28557e11j7f0b4cbbd1a22b35@mail.gmail.com> <20060203000808.GA2888@angeldust.chaos> <71b79fa90602030359q22d5d55bp10d26052c6291908@mail.gmail.com> <43EA4EA6.8040803@allpeers.com> <5037146A-FF9F-4786-BD8A-200BABF490DD@cluck.com> <43EB0C9B.7090506@allpeers.com> Message-ID: In an optimistic communications should not use pings per say. My preference is just to write the data. To deal with optimism, in some cases, you do need feedback. A couple of choices are to use checksums or last-heard-from in your data. There are many complex issues here. Many of them depend on the P2P network and your application. Speed of notification is only sometimes important. If you are sending data via relayed machines, it would be better to stop a send soon, when the destination disappears. If the relays store for a later forward, the source does not care that the destination is up or down because the destination will eventually get the message. If we assume infinite bandwidth and computers that can never be flooded by bandwidth, its all cool. If we assume spotty wifi and 5 year old computers, the solutions need to be carefully implemented to avoid failures. On Feb 9, 2006, at 1:34 AM, Matthew Gertner wrote: > Daniel B. wrote: > >> Keep alive is both a missing component in practical P2P and a >> reason why P2P is also hard to make practical. >> >> The trap is that the keep-alive is between two machines. Imagine >> hundreds of thousands of machines!!!! Even with small signatures >> matching each machine, we are talking about a flooded network. A >> secondary trap is managing reliability. Reliability in P2P is not >> moving bytes back and forth, but also taking into account the >> machine may be a laptop or other personal computer that has a >> pathetic uptime. >> >> The better choice is online/offline signaling and optimistic >> communications with good error control. > > I assume that by "optimistic communications" you mean that we > should assume a connection exists and confirm this using periodic > pings? Isn't this basically the equivalent of KA? It seems to me > that the fundamental tradeoff is speed of notification vs. extra > network traffic (i.e. you know more quickly when someone's network > connection is cut if you ping more often), and this is the case > both for KA and for a custom optimistic approach that uses pings. > Am I missing something? > > Cheers, > Matt > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > From dbarrett at quinthar.com Thu Feb 9 17:53:14 2006 From: dbarrett at quinthar.com (David Barrett) Date: Sat Dec 9 22:13:09 2006 Subject: [p2p-hackers] Keep Alive and network performance In-Reply-To: Message-ID: <20060209175320.9A6923FCA7@capsicum.zgp.org> Well, in practice keep-alives seem to do well despite the theoretical worst case. Really it's a balance between the importance of fast dirty-disconnect detection and bandwidth. If the feature is critical to your application, the bandwidth is worth it. If the feature isn't, then the bandwidth is wasted. If you're writing an IM-like tool, the general standard seems to be to detect coming online instantly (eg, you contact everyone on your buddy list the moment you come online), detect clean disconnects instantly (eg, notify everyone before you go offline), and then notice dirty disconnects after several seconds or more (eg, using keepalives). Thus IM-like presence in a world where most clients disconnect in a clean fashion (which, incidentally, is the world we live in) doesn't take much keep-alive traffic at all because the ping frequencies are so low (10+ seconds?). But another entirely separate (and more valuable?) use of keepalives is to keep NAT mappings alive in a hole-punching scenario. With iGlance (which is being presented at CodeCon on Saturday) I use an adaptive approach where I look for the slowest keepalive frequency that works. Adaptiveness is critical, as I've seen some NATs that start with 5-second windows, but grow up to 256-second windows over time. -david > -----Original Message----- > From: p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org] On > Behalf Of Daniel B. > Sent: Thursday, February 09, 2006 7:31 AM > To: Peer-to-peer development. > Subject: Re: [p2p-hackers] Keep Alive and network performance > > In an optimistic communications should not use pings per say. My > preference is just to write the data. To deal with optimism, in some > cases, you do need feedback. A couple of choices are to use checksums > or last-heard-from in your data. > > There are many complex issues here. Many of them depend on the P2P > network and your application. Speed of notification is only sometimes > important. If you are sending data via relayed machines, it would be > better to stop a send soon, when the destination disappears. If the > relays store for a later forward, the source does not care that the > destination is up or down because the destination will eventually get > the message. > > If we assume infinite bandwidth and computers that can never be > flooded by bandwidth, its all cool. If we assume spotty wifi and 5 > year old computers, the solutions need to be carefully implemented to > avoid failures. > > On Feb 9, 2006, at 1:34 AM, Matthew Gertner wrote: > > > Daniel B. wrote: > > > >> Keep alive is both a missing component in practical P2P and a > >> reason why P2P is also hard to make practical. > >> > >> The trap is that the keep-alive is between two machines. Imagine > >> hundreds of thousands of machines!!!! Even with small signatures > >> matching each machine, we are talking about a flooded network. A > >> secondary trap is managing reliability. Reliability in P2P is not > >> moving bytes back and forth, but also taking into account the > >> machine may be a laptop or other personal computer that has a > >> pathetic uptime. > >> > >> The better choice is online/offline signaling and optimistic > >> communications with good error control. > > > > I assume that by "optimistic communications" you mean that we > > should assume a connection exists and confirm this using periodic > > pings? Isn't this basically the equivalent of KA? It seems to me > > that the fundamental tradeoff is speed of notification vs. extra > > network traffic (i.e. you know more quickly when someone's network > > connection is cut if you ping more often), and this is the case > > both for KA and for a custom optimistic approach that uses pings. > > Am I missing something? > > > > Cheers, > > Matt > > _______________________________________________ > > p2p-hackers mailing list > > p2p-hackers@zgp.org > > http://zgp.org/mailman/listinfo/p2p-hackers > > _______________________________________________ > > Here is a web page listing P2P Conferences: > > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > > > > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences From brturn at cisco.com Thu Feb 9 18:02:44 2006 From: brturn at cisco.com (Bryan Turner (brturn)) Date: Sat Dec 9 22:13:09 2006 Subject: [p2p-hackers] Keep Alive and network performance In-Reply-To: <43EA4EA6.8040803@allpeers.com> Message-ID: <000301c62da3$09905190$b5576640@amer.cisco.com> Matt, TCP already uses keep-alives (typically set to 3-minutes). Adding additional keep-alives on top of this should be carefully deployed. For instance, if you send a keep-alive 3 minutes after the last packet received, you will be coinciding with the "natural" TCP keep-alive and not gaining any new information. Overhead: TCP messages over the wire typically cost Ethernet + IP + TCP header sizes, plus the size of your keep alive (the TCP keep alive on Windows was 64 bytes last I checked). A good overview of packet overhead is [1]. So a typical keep-alive is: ~16 + 20 + 20 + 64 bytes = 120 bytes Plus the ACK response - also 120 bytes (although the ACK may be acking other packets too, so this is a conservative estimate). Thus the TCP keep-alives consume: 240 bytes/3 min = 80 bytes/min = 4 bytes/sec = 32 bits/sec If your link capacity is a conservative 256 Kbit/sec broadband, and you wanted to use 1% for keep-alives, that gives you a budget of: ~2600 bits/sec = 325 bytes/sec. That's about one keep-alive per second for 1% overhead. If you have control over the hardware abstraction layer in your app, I would suggest piggybacking the socket write of the keep-alive with an outgoing data payload so the OS will send them out as one packet on the wire, saving the protocol overhead. --Bryan bryan.turner@pobox.com [1] http://sd.wareonearth.com/~phil/net/overhead/ -----Original Message----- From: p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org] On Behalf Of Matthew Gertner Sent: Wednesday, February 08, 2006 3:04 PM To: Peer-to-peer development. Subject: [p2p-hackers] Keep Alive and network performance Hi, We have a straightforward P2P topology with each peer in the network attached to hub. We want to detect efficiently that a peer has disconnected from the hub. The obvious way to do this would be to use our current TCP connection with a relatively short Keep Alive value (a few seconds so we could see the peer's presence updated quickly if they don't disconnect cleanly). Does anyone have a feeling for the performance implications of this approach if many peers are attached to a single hub? Is there another approach that is recommend (such as using a custom UDP-based keep alive protocol)? Regards, Matt _______________________________________________ p2p-hackers mailing list p2p-hackers@zgp.org http://zgp.org/mailman/listinfo/p2p-hackers _______________________________________________ Here is a web page listing P2P Conferences: http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences From gbildson at limepeer.com Thu Feb 9 18:04:23 2006 From: gbildson at limepeer.com (Greg Bildson) Date: Sat Dec 9 22:13:09 2006 Subject: [p2p-hackers] Keep Alive and network performance In-Reply-To: <43EA4EA6.8040803@allpeers.com> Message-ID: Rather than using a dedicated message for keepalives, LimeWire first looks for message traffic of any kind on a connection. If there is traffic on the connection during a time window, then the connection is assumed to still be alive. If there has been no natural traffic, we will send a keepalive ping to it from the hub/ultrapeer. If there is still no network traffic coming from the host after a recheck time, we assume the connection is dead and kill it on the ultrapeer. We were concerned both about unclean disconnects and connections where the bandwidth was getting so choked that it was effectively dead. Thanks -greg > -----Original Message----- > From: p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org]On > Behalf Of Matthew Gertner > Sent: Wednesday, February 08, 2006 3:04 PM > To: Peer-to-peer development. > Subject: [p2p-hackers] Keep Alive and network performance > > > Hi, > > We have a straightforward P2P topology with each peer in the network > attached to hub. We want to detect efficiently that a peer has > disconnected from the hub. The obvious way to do this would be to use > our current TCP connection with a relatively short Keep Alive value (a > few seconds so we could see the peer's presence updated quickly if they > don't disconnect cleanly). Does anyone have a feeling for the > performance implications of this approach if many peers are attached to > a single hub? Is there another approach that is recommend (such as using > a custom UDP-based keep alive protocol)? > > Regards, > Matt > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences From gwendal.simon at francetelecom.com Thu Feb 9 19:15:55 2006 From: gwendal.simon at francetelecom.com (SIMON Gwendal RD-MAPS-ISS) Date: Sat Dec 9 22:13:09 2006 Subject: [p2p-hackers] Keep Alive and network performance Message-ID: In Solipsis, we also use a unique message for both "heartbeat" (confirm aliveness) and "keepalive" (maintain NAT hole-punching). A french saying is "One stone, two hits !" :-) We also intended to adjust heartbeat periodicity to the "link value". Some links are essential to the network connectivity, so they deserve a small periodicity. Some links are valuable because of some criteria (in our case, the peer is graphically displayed). A "NAT hole-punching" periodicity is enough. Finally, some links are optional. We just keep them alive with a long periodicity because (i) why not, (ii) handshaking may be costly. This may be applied in other p2p networks : adjust keepalive periodicity to the number of shared files, to the redudancy (in DHT), to the estimated half-life of peer... -- Gwendal http://solipsis.netofpeers.net > -----Message d'origine----- > De : p2p-hackers-bounces@zgp.org > [mailto:p2p-hackers-bounces@zgp.org] De la part de Greg Bildson > Envoy? : jeudi 9 f?vrier 2006 19:04 > ? : Peer-to-peer development. > Objet : RE: [p2p-hackers] Keep Alive and network performance > > Rather than using a dedicated message for keepalives, > LimeWire first looks > for message traffic of any kind on a connection. If there is > traffic on the > connection during a time window, then the connection is > assumed to still be > alive. If there has been no natural traffic, we will send > a keepalive > ping to it from the hub/ultrapeer. If there is still no > network traffic > coming from the host after a recheck time, we assume the > connection is dead > and kill it on the ultrapeer. > > We were concerned both about unclean disconnects and > connections where the > bandwidth was getting so choked that it was effectively dead. > > Thanks > -greg > > > -----Original Message----- > > From: p2p-hackers-bounces@zgp.org > [mailto:p2p-hackers-bounces@zgp.org]On > > Behalf Of Matthew Gertner > > Sent: Wednesday, February 08, 2006 3:04 PM > > To: Peer-to-peer development. > > Subject: [p2p-hackers] Keep Alive and network performance > > > > > > Hi, > > > > We have a straightforward P2P topology with each peer in the network > > attached to hub. We want to detect efficiently that a peer has > > disconnected from the hub. The obvious way to do this would > be to use > > our current TCP connection with a relatively short Keep > Alive value (a > > few seconds so we could see the peer's presence updated > quickly if they > > don't disconnect cleanly). Does anyone have a feeling for the > > performance implications of this approach if many peers are > attached to > > a single hub? Is there another approach that is recommend > (such as using > > a custom UDP-based keep alive protocol)? > > > > Regards, > > Matt > > _______________________________________________ > > p2p-hackers mailing list > > p2p-hackers@zgp.org > > http://zgp.org/mailman/listinfo/p2p-hackers > > _______________________________________________ > > Here is a web page listing P2P Conferences: > > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > > > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > From alain.damiral at student.uclouvain.be Mon Feb 13 10:36:16 2006 From: alain.damiral at student.uclouvain.be (Alain Damiral) Date: Sat Dec 9 22:13:09 2006 Subject: [p2p-hackers] Secure communication in p2p networks Message-ID: <43F06120.50902@student.uclouvain.be> Hello list, This is my first post so please let me know if I'm already doing it wrong. I am a student and am currently working on establishing SSL/TLS enabled communication channels in a peer-to-peer network (for one-to-one secure communication). In my report, I would also like to mention some current research that might be in progress on secure protocols specifically designed for one to many communication. According to what I last heard, hardly anything has been done in this regard (creating shared secrets between more than 2 entities). I was hoping that maybe some of the members of this fine community could direct me to some documentation on current research on this topic. Well that's it for now, thank you all for your time and if you never received this message, please disregard it. -- Alain Damiral, I hope this message makes me look like a very intelligent person Universit? Catholique de Louvain - student alain.damiral'at'student.info.ucl.ac.be From fis at wiwi.hu-berlin.de Mon Feb 13 13:54:03 2006 From: fis at wiwi.hu-berlin.de (Matthias Fischmann) Date: Sat Dec 9 22:13:09 2006 Subject: [p2p-hackers] Secure communication in p2p networks In-Reply-To: <43F06120.50902@student.uclouvain.be> References: <43F06120.50902@student.uclouvain.be> Message-ID: <20060213135403.GD6557@localhost.localdomain> hi alain, the reason there is not much research might be that solutions better than the trivial ones are hard to find. with trivial i mean that one of the communicating group broadcasts a symmetric key over ssl-secured channels to all other group members and then everybody uses symmetric crypto. slightly more efficiently, i think DH key agreement [1] can be adapted for more than two parties. one property of these approaches is that the receiver of a secured package only knows that *some* member of the group sent it, not which one. is that helpful? cheers, matthias [1] http://www.rsasecurity.com/rsalabs/node.asp?id=2248 On Mon, Feb 13, 2006 at 11:36:16AM +0100, Alain Damiral wrote: > To: p2p-hackers@zgp.org > From: Alain Damiral > Date: Mon, 13 Feb 2006 11:36:16 +0100 > Subject: [p2p-hackers] Secure communication in p2p networks > > Hello list, > > This is my first post so please let me know if I'm already doing it wrong. > > I am a student and am currently working on establishing SSL/TLS enabled > communication channels in a peer-to-peer network (for one-to-one secure > communication). In my report, I would also like to mention some current > research that might be in progress on secure protocols specifically > designed for one to many communication. According to what I last heard, > hardly anything has been done in this regard (creating shared secrets > between more than 2 entities). > > I was hoping that maybe some of the members of this fine community could > direct me to some documentation on current research on this topic. > > Well that's it for now, thank you all for your time and if you never > received this message, please disregard it. > > -- > Alain Damiral, > > I hope this message makes me look like a very intelligent person > > Universit? Catholique de Louvain - student > alain.damiral'at'student.info.ucl.ac.be > > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences -- Institute of Information Systems, Humboldt-Universitaet zu Berlin web: http://www.wiwi.hu-berlin.de/~fis/ e-mail: fis@wiwi.hu-berlin.de tel: +49 30 2093-5742 fax: +49 30 2093-5741 office: Spandauer Strasse 1, R.324, 10178 Berlin, Germany -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://zgp.org/pipermail/p2p-hackers/attachments/20060213/51378d4f/attachment.pgp From networksimulator at gmail.com Mon Feb 13 16:46:57 2006 From: networksimulator at gmail.com (Ranus) Date: Sat Dec 9 22:13:09 2006 Subject: [p2p-hackers] Secure communication in p2p networks In-Reply-To: <43F06120.50902@student.uclouvain.be> Message-ID: <000c01c630bd$1cce4470$ccc96fa6@thinkingfish> End-to-End encryption is tested by (mu)torrent (the greek letter \mu), they use DH public-key scheme to exchange the secret and symmetric key encryption for data transfer. read this interview http://www.slyck.com/news.php?story=1083 I don't know if it helps, are you trying to find a way to achieve something like group signature or merely secure communication for each connection? Hope you can tell more about your project, sounds interesting. -- Ranus Yue Tsinghua University ( -----Original Mail----- Sender: p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org] Alain Damiral time: 2006/02/13 18:36 receiver: p2p-hackers@zgp.org Subject: [p2p-hackers] Secure communication in p2p networks Hello list, This is my first post so please let me know if I'm already doing it wrong. I am a student and am currently working on establishing SSL/TLS enabled communication channels in a peer-to-peer network (for one-to-one secure communication). In my report, I would also like to mention some current research that might be in progress on secure protocols specifically designed for one to many communication. According to what I last heard, hardly anything has been done in this regard (creating shared secrets between more than 2 entities). I was hoping that maybe some of the members of this fine community could direct me to some documentation on current research on this topic. Well that's it for now, thank you all for your time and if you never received this message, please disregard it. -- Alain Damiral, I hope this message makes me look like a very intelligent person Universit? Catholique de Louvain - student alain.damiral'at'student.info.ucl.ac.be _______________________________________________ p2p-hackers mailing list p2p-hackers@zgp.org http://zgp.org/mailman/listinfo/p2p-hackers _______________________________________________ Here is a web page listing P2P Conferences: http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences From alain.damiral at student.uclouvain.be Mon Feb 13 17:51:13 2006 From: alain.damiral at student.uclouvain.be (Alain Damiral) Date: Sat Dec 9 22:13:09 2006 Subject: [p2p-hackers] Secure communication in p2p networks In-Reply-To: <000c01c630bd$1cce4470$ccc96fa6@thinkingfish> References: <000c01c630bd$1cce4470$ccc96fa6@thinkingfish> Message-ID: <43F0C711.505@student.uclouvain.be> Thanks, (and to Matthias also) So here are some extra details about my project... What I am trying to achieve is rather simple. I'm using OpenSSL but instead of letting it send data directly over a TCP stream, I feed the data back to the peer-to-peer middleware. I'm basically moving SSL/TLS's transport layer one level up. This can be done as long as the actual transport of messages accross the network is reliable. This allows for secure communication between two peers much in the same way SSL usually does, without having to trust any intermediate peer in the network. The most exotic aspect of my work is that the p2p middleware I am working with is written in Oz (hello Oz coders !) while the OpenSSL API is defined in C. Simple end-to-end TLS enabled channels can already give the user some modest solutions for group communication, if a symmetric key algorithm is available. This can be done by having a service responsible for providing peers (if they can be authentified) with a shared secret key, that he can update regularly. However I believe that it is possible to design a protocol that would achieve something more efficient and maybe stronger. I do not intend to support such a protocol in my work, but if something exists or is being researched today, I would like to mention it in my written report :) So it is absolutely not critical for my project. Once again, thanks for the help ! -Alain Ranus wrote: >End-to-End encryption is tested by (mu)torrent (the greek letter \mu), they >use DH public-key scheme to exchange the secret and symmetric key encryption >for data transfer. read this interview >http://www.slyck.com/news.php?story=1083 > >I don't know if it helps, are you trying to find a way to achieve something >like group signature or merely secure communication for each connection? >Hope you can tell more about your project, sounds interesting. > > >-- >Ranus Yue >Tsinghua University > >( >-----Original Mail----- >Sender: p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org] >Alain Damiral >time: 2006/02/13 18:36 >receiver: p2p-hackers@zgp.org >Subject: [p2p-hackers] Secure communication in p2p networks > >Hello list, > >This is my first post so please let me know if I'm already doing it wrong. > >I am a student and am currently working on establishing SSL/TLS enabled >communication channels in a peer-to-peer network (for one-to-one secure >communication). In my report, I would also like to mention some current >research that might be in progress on secure protocols specifically designed >for one to many communication. According to what I last heard, hardly >anything has been done in this regard (creating shared secrets between more >than 2 entities). > >I was hoping that maybe some of the members of this fine community could >direct me to some documentation on current research on this topic. > >Well that's it for now, thank you all for your time and if you never >received this message, please disregard it. > >-- >Alain Damiral, > >I hope this message makes me look like a very intelligent person > >Universit? Catholique de Louvain - student >alain.damiral'at'student.info.ucl.ac.be > >_______________________________________________ >p2p-hackers mailing list >p2p-hackers@zgp.org >http://zgp.org/mailman/listinfo/p2p-hackers >_______________________________________________ >Here is a web page listing P2P Conferences: >http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > >_______________________________________________ >p2p-hackers mailing list >p2p-hackers@zgp.org >http://zgp.org/mailman/listinfo/p2p-hackers >_______________________________________________ >Here is a web page listing P2P Conferences: >http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > > From coderman at gmail.com Mon Feb 13 20:48:17 2006 From: coderman at gmail.com (coderman) Date: Sat Dec 9 22:13:09 2006 Subject: [p2p-hackers] Secure communication in p2p networks In-Reply-To: <43F06120.50902@student.uclouvain.be> References: <43F06120.50902@student.uclouvain.be> Message-ID: <4ef5fec60602131248m6f348876xdb44758b0a4bdfc6@mail.gmail.com> On 2/13/06, Alain Damiral wrote: > ... > I am a student and am currently working on establishing SSL/TLS enabled > communication channels in a peer-to-peer network (for one-to-one secure > communication). you might find openvpn and the latest 4.3 openssh with tun/tap relevant/interesting then. there are some other tun/tap p2p projects in progress if you search around a bit as well. > In my report, I would also like to mention some current > research that might be in progress on secure protocols specifically > designed for one to many communication. According to what I last heard, > hardly anything has been done in this regard (creating shared secrets > between more than 2 entities). are you thinking of group key distribution and management? this is tricky, but there is also a lot of work being done in this regard, from efficient pre distribution methods to n-party public key mechanisms. From greg at capepaterson.com Mon Feb 13 21:20:43 2006 From: greg at capepaterson.com (Greg Rolan) Date: Sat Dec 9 22:13:09 2006 Subject: [p2p-hackers] gig announcement In-Reply-To: <4ef5fec60602131248m6f348876xdb44758b0a4bdfc6@mail.gmail.com> References: <43F06120.50902@student.uclouvain.be> <4ef5fec60602131248m6f348876xdb44758b0a4bdfc6@mail.gmail.com> Message-ID: <43F0F82B.2080101@capepaterson.com> Hi people, I'm not sure if this sort of post is allowed on this list so apologies if this contravenes the etiquette conventions here. We're a small startup developing a broadly distributed application which requires an dht overlay network for information lookup. I've been playing around with various open-source dht implementations, lurking in this list and trying to find a good solution for the NAT traversal issues. Many members have posted good suggestions for approaches to a number of problems and it appears that there is the makings of a fairly good (and I acknowledge that good is not perfect or even great) solution. What we would like to do is engage someone to do some integration work based on an existing code-base or even buy a commercial ready-to-go implementation of dht node code which addresses NAT/Firewall traversal issues using a mix of Firewall port-forwarding registration, hole punching and proxy nodes (super-node) strategies. We would possibly be interested in working with another organisations to achieve common goals in this area. I appreciate that the search for a solution will be an ongoing activity. For questions, offers, suggestions or more information (a specification document) please reply off-list (unless you think it is relevant to the list). Thanks - and apologies again if this post was inappropriate. Greg -- No virus found in this outgoing message. Checked by AVG Free Edition. Version: 7.1.375 / Virus Database: 267.15.6/257 - Release Date: 2/10/2006 From agthorr at cs.uoregon.edu Mon Feb 13 23:51:10 2006 From: agthorr at cs.uoregon.edu (Daniel Stutzbach) Date: Sat Dec 9 22:13:09 2006 Subject: [p2p-hackers] Secure communication in p2p networks In-Reply-To: <4ef5fec60602131248m6f348876xdb44758b0a4bdfc6@mail.gmail.com> References: <43F06120.50902@student.uclouvain.be> <4ef5fec60602131248m6f348876xdb44758b0a4bdfc6@mail.gmail.com> Message-ID: <20060213235110.GB5752@cs.uoregon.edu> On 2/13/06, Alain Damiral wrote: > In my report, I would also like to mention some current > research that might be in progress on secure protocols specifically > designed for one to many communication. According to what I last heard, > hardly anything has been done in this regard (creating shared secrets > between more than 2 entities). Another group at my University is working on something like that: http://www.acsa-admin.org/2005/abstracts/166.html -- Daniel Stutzbach Computer Science Ph.D Student http://www.barsoom.org/~agthorr University of Oregon From supercanadian at gmail.com Tue Feb 14 08:35:32 2006 From: supercanadian at gmail.com (Charles Iliya Krempeaux) Date: Sat Dec 9 22:13:09 2006 Subject: [p2p-hackers] Tighter HTTP and P2P integration?? Message-ID: <84ce626f0602140035t608263dar1d63f0eecfa92ccb@mail.gmail.com> Hello, (First, I should say that I'm writing this around midnight, so let me first apologise for all the spelling and grammar mistakes I know I'll probably make but won't notice until I read this tomorrow. But anyways....) One thing that's been on my mind for a while is making "tighter integration" between HTTP and P2P technologies. (I've finally been motivated to write this e-mail since I need to put this to practice now. Although I really wanted to write a longer and more detailed blog post on this. And do more research first.... Maybe later.) The current methods that bring some integration between HTTP and P2P technologies include the magnet-uri and BitTorrent (via BitTorrent's "application/x-bittorrent" MIME type). However, one could get even tighter integration. (Here's a few different ways that may work... I'd like to get people's opinions on these proposals.) One possible avenue for getting even tighter integration is the creation of something like a "conditional get" for P2P technologies. (For those unfamiliar with the HTTP "conditional get", look for information of the "If-None-Match" and "If-Modified-Since" HTTP headers.) A "P2P conditional get" (sent by the client) would tell the server NOT to send the requested document if there is an alternative way of getting the document via a P2P technology supported by the client. (The client would tell the server which P2P technologies it supports.) (This "P2P conditional get" would work similar to the "If-None-Match" and "If-Modified-Since" "conditional get" headers.). For example, it might be something like: X-If-No-Alternate: bittorrent, something-else, magnet First, this says that the client supports the protocols: "bittorrent", "something-else", and "magnet". It also says it prefers "bittorrent" first, then "something-else", and then "magnet". (Really, there's nothing that mandates that these protocols be for P2P technologies. But that is the use case that is motivating this.) This could also include "q" factors. Perhaps something like: X-If-No-Alternate: protocol1; q=1.0, protocol2; q=0.7, protocol3; q=0.5 And there could be ways of including more information too. (Which could be used to add protocol specific information.) For example: X-If-No-Alternate: protocol1; q=1.0; a=b;c="d", magnet; q=0.5;urn-allowed="sha1 md5" Note, I'm NOT trying to propose the exact format here or the names used. Just trying to illustrate what I'm talking about through examples. (I've tried to follow similar formats as other HTTP headers though.) The server might then respond with an "HTTP 302" (Found) and then redirect the user to a P2P version using a URI for the P2P technology. This "redirect" would be done via the HTTP "Location" header. For example: HTTP/1.1 302 Found Location: magnet:?xt=urn:sha1:WRCIRZV5ZO56CWMNHFV4FRGNPWPPDVKT or: HTTP/1.1 302 Found Location: x-some-new-p2p-protocol:etc/etc/etc However, I'm not sure if "302" is the correct HTTP response code to use. (Anyone have any suggestions?) Another possible solution would be to use the "Accept" header (sent by the client). Now, this would only be useful for P2P technologies that are shunted into HTTP via a MIME type (like BitTorrent and the "application/x-bittorrent" MIME type.). But for technologies which have this, the client could list the P2P shunt MIME types first, in the HTTP "Accept" header. For example: Accept: application/x-bittorrent, application/ogg However, with BitTorrent's case, this brings the problem of confusing the torrent and files in the torrent. So perhaps a "accept extension" is needed: Accept: application/x-bittorrent; enclosure-content-type="application/ogg", application/ogg Although I still think this still conveys the wrong semantics. So perhaps this really isn't a good solution. Another alternative is to use the "Upgrade" header (from the client) and the "HTTP 101" (Switching Protocols) (from the server). This would let you switch to a P2P protocol. However, this seems to somewhat imply that whatever protocol is "upgraded to" will have communication going from between the same client and server. (At least that's my feeling.) Any Comments? Anyone have any suggestions or advice? Also, are there already any proposals or specs along these lines? (No sense in reinventing the wheel.) TIA See ya -- Charles Iliya Krempeaux, B.Sc. charles @ reptile.ca supercanadian @ gmail.com developer weblog: http://ChangeLog.ca/ ___________________________________________________________________________ Make Television http://maketelevision.com/ From kmagdsick at limewire.com Tue Feb 14 16:07:34 2006 From: kmagdsick at limewire.com (Karl A. Magdsick) Date: Sat Dec 9 22:13:09 2006 Subject: [p2p-hackers] Tighter HTTP and P2P integration?? In-Reply-To: <84ce626f0602140035t608263dar1d63f0eecfa92ccb@mail.gmail.com> References: <84ce626f0602140035t608263dar1d63f0eecfa92ccb@mail.gmail.com> Message-ID: <43F20046.4040709@limewire.com> Hello Charles, A couple of weeks ago, there was some work done to integrate both BitTorrent and Gnutella technologies with the Plone CMS/Zope webserver. http://plone.org/events/sprints/snow-sprint3 I've done some preliminary work (along with Matt Hamilton from NetSight) using an Apache plugin to add the X-Alt and X-Node HTTP headers that allow Gnutella clients downloading the same file to find each other and take load off the server. I haven't gone past proof-of-concept, but I have confirmed that two LimeWire clients are able to find each other through an Apache server using a plugin I helped write. The two LimeWire clients then share portions of the file with each other and thereby reduce load on the server. Note that Apache 2.2 on OS X 10.4 (Tiger) deals very poorly with a single connection utilizing multiple 128 kb GET requests in random order. We got very high CPU useage on the PowerBook being used as the webserver, and even got a kernel panic. We were forced to use the latest Apache 2.0, which seemed to work fine. (This was the weekend of Feb 4th, 2006.) Good Luck, Karl Magdsick Software Engineer, Lime Wire LLC Charles Iliya Krempeaux wrote: >Hello, > >(First, I should say that I'm writing this around midnight, so let me >first apologise for all the spelling and grammar mistakes I know I'll >probably make but won't notice until I read this tomorrow. But >anyways....) > >One thing that's been on my mind for a while is making "tighter >integration" between HTTP and P2P technologies. (I've finally been >motivated to write this e-mail since I need to put this to practice >now. Although I really wanted to write a longer and more detailed >blog post on this. And do more research first.... Maybe later.) > >The current methods that bring some integration between HTTP and P2P >technologies include the magnet-uri and BitTorrent (via BitTorrent's >"application/x-bittorrent" MIME type). > >However, one could get even tighter integration. (Here's a few >different ways that may work... I'd like to get people's opinions on >these proposals.) > >One possible avenue for getting even tighter integration is the >creation of something like a "conditional get" for P2P technologies. >(For those unfamiliar with the HTTP "conditional get", look for >information of the "If-None-Match" and "If-Modified-Since" HTTP >headers.) > >A "P2P conditional get" (sent by the client) would tell the server NOT >to send the requested document if there is an alternative way of >getting the document via a P2P technology supported by the client. >(The client would tell the server which P2P technologies it supports.) > (This "P2P conditional get" would work similar to the "If-None-Match" >and "If-Modified-Since" "conditional get" headers.). > >For example, it might be something like: > > X-If-No-Alternate: bittorrent, something-else, magnet > >First, this says that the client supports the protocols: "bittorrent", >"something-else", and "magnet". It also says it prefers "bittorrent" >first, then "something-else", and then "magnet". > >(Really, there's nothing that mandates that these protocols be for P2P >technologies. But that is the use case that is motivating this.) > >This could also include "q" factors. Perhaps something like: > > X-If-No-Alternate: protocol1; q=1.0, protocol2; q=0.7, protocol3; q=0.5 > >And there could be ways of including more information too. (Which >could be used to add protocol specific information.) For example: > > X-If-No-Alternate: protocol1; q=1.0; a=b;c="d", magnet; >q=0.5;urn-allowed="sha1 md5" > >Note, I'm NOT trying to propose the exact format here or the names >used. Just trying to illustrate what I'm talking about through >examples. (I've tried to follow similar formats as other HTTP headers >though.) > >The server might then respond with an "HTTP 302" (Found) and then >redirect the user to a P2P version using a URI for the P2P technology. > This "redirect" would be done via the HTTP "Location" header. For >example: > > HTTP/1.1 302 Found > Location: magnet:?xt=urn:sha1:WRCIRZV5ZO56CWMNHFV4FRGNPWPPDVKT > >or: > > HTTP/1.1 302 Found > Location: x-some-new-p2p-protocol:etc/etc/etc > >However, I'm not sure if "302" is the correct HTTP response code to >use. (Anyone have any suggestions?) > > >Another possible solution would be to use the "Accept" header (sent by >the client). Now, this would only be useful for P2P technologies that >are shunted into HTTP via a MIME type (like BitTorrent and the >"application/x-bittorrent" MIME type.). But for technologies which >have this, the client could list the P2P shunt MIME types first, in >the HTTP "Accept" header. For example: > > Accept: application/x-bittorrent, application/ogg > >However, with BitTorrent's case, this brings the problem of confusing >the torrent and files in the torrent. So perhaps a "accept extension" >is needed: > > Accept: application/x-bittorrent; >enclosure-content-type="application/ogg", application/ogg > >Although I still think this still conveys the wrong semantics. So >perhaps this really isn't a good solution. > > >Another alternative is to use the "Upgrade" header (from the client) >and the "HTTP 101" (Switching Protocols) (from the server). This would >let you switch to a P2P protocol. However, this seems to somewhat >imply that whatever protocol is "upgraded to" will have communication >going from between the same client and server. (At least that's my >feeling.) > > >Any Comments? Anyone have any suggestions or advice? > >Also, are there already any proposals or specs along these lines? (No >sense in reinventing the wheel.) > > >TIA > > >See ya > >-- > Charles Iliya Krempeaux, B.Sc. > > charles @ reptile.ca > supercanadian @ gmail.com > > developer weblog: http://ChangeLog.ca/ >___________________________________________________________________________ > Make Television http://maketelevision.com/ >_______________________________________________ >p2p-hackers mailing list >p2p-hackers@zgp.org >http://zgp.org/mailman/listinfo/p2p-hackers >_______________________________________________ >Here is a web page listing P2P Conferences: >http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > > From travis at redswoosh.net Tue Feb 14 18:35:53 2006 From: travis at redswoosh.net (Travis Kalanick) Date: Sat Dec 9 22:13:09 2006 Subject: [p2p-hackers] Tighter HTTP and P2P integration?? In-Reply-To: <43F20046.4040709@limewire.com> Message-ID: <200602141841.k1EIf9qx019157@be9.noc0.redswoosh.com> Web integration has been key to Red Swoosh's architecture. We have been doing a more advanced version of your "conditional get" where clients simply look at http as another source. In addition we have several parameters that are embedded in the links that allow for different streaming/download methods (peers only, peers and server, peers but fallback on server, etc.). You can get a Basic version of the API (a limited set of the parameters available in the advanced API) emailed to you here: http://redswoosh.net/tryus_step_by_step.php We've had live customers (i.e. websites) up on this web-integration architecture since 2001 (Deviantart.com launched with us in October 2001) and have found it to be quite a powerful way to employ P2P tech. Thanks, Travis -----Original Message----- From: p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org] On Behalf Of Karl A. Magdsick Sent: Tuesday, February 14, 2006 8:08 AM To: charles@reptile.ca; Peer-to-peer development. Subject: Re: [p2p-hackers] Tighter HTTP and P2P integration?? Hello Charles, A couple of weeks ago, there was some work done to integrate both BitTorrent and Gnutella technologies with the Plone CMS/Zope webserver. http://plone.org/events/sprints/snow-sprint3 I've done some preliminary work (along with Matt Hamilton from NetSight) using an Apache plugin to add the X-Alt and X-Node HTTP headers that allow Gnutella clients downloading the same file to find each other and take load off the server. I haven't gone past proof-of-concept, but I have confirmed that two LimeWire clients are able to find each other through an Apache server using a plugin I helped write. The two LimeWire clients then share portions of the file with each other and thereby reduce load on the server. Note that Apache 2.2 on OS X 10.4 (Tiger) deals very poorly with a single connection utilizing multiple 128 kb GET requests in random order. We got very high CPU useage on the PowerBook being used as the webserver, and even got a kernel panic. We were forced to use the latest Apache 2.0, which seemed to work fine. (This was the weekend of Feb 4th, 2006.) Good Luck, Karl Magdsick Software Engineer, Lime Wire LLC Charles Iliya Krempeaux wrote: >Hello, > >(First, I should say that I'm writing this around midnight, so let me >first apologise for all the spelling and grammar mistakes I know I'll >probably make but won't notice until I read this tomorrow. But >anyways....) > >One thing that's been on my mind for a while is making "tighter >integration" between HTTP and P2P technologies. (I've finally been >motivated to write this e-mail since I need to put this to practice >now. Although I really wanted to write a longer and more detailed >blog post on this. And do more research first.... Maybe later.) > >The current methods that bring some integration between HTTP and P2P >technologies include the magnet-uri and BitTorrent (via BitTorrent's >"application/x-bittorrent" MIME type). > >However, one could get even tighter integration. (Here's a few >different ways that may work... I'd like to get people's opinions on >these proposals.) > >One possible avenue for getting even tighter integration is the >creation of something like a "conditional get" for P2P technologies. >(For those unfamiliar with the HTTP "conditional get", look for >information of the "If-None-Match" and "If-Modified-Since" HTTP >headers.) > >A "P2P conditional get" (sent by the client) would tell the server NOT >to send the requested document if there is an alternative way of >getting the document via a P2P technology supported by the client. >(The client would tell the server which P2P technologies it supports.) > (This "P2P conditional get" would work similar to the "If-None-Match" >and "If-Modified-Since" "conditional get" headers.). > >For example, it might be something like: > > X-If-No-Alternate: bittorrent, something-else, magnet > >First, this says that the client supports the protocols: "bittorrent", >"something-else", and "magnet". It also says it prefers "bittorrent" >first, then "something-else", and then "magnet". > >(Really, there's nothing that mandates that these protocols be for P2P >technologies. But that is the use case that is motivating this.) > >This could also include "q" factors. Perhaps something like: > > X-If-No-Alternate: protocol1; q=1.0, protocol2; q=0.7, protocol3; q=0.5 > >And there could be ways of including more information too. (Which >could be used to add protocol specific information.) For example: > > X-If-No-Alternate: protocol1; q=1.0; a=b;c="d", magnet; >q=0.5;urn-allowed="sha1 md5" > >Note, I'm NOT trying to propose the exact format here or the names >used. Just trying to illustrate what I'm talking about through >examples. (I've tried to follow similar formats as other HTTP headers >though.) > >The server might then respond with an "HTTP 302" (Found) and then >redirect the user to a P2P version using a URI for the P2P technology. > This "redirect" would be done via the HTTP "Location" header. For >example: > > HTTP/1.1 302 Found > Location: magnet:?xt=urn:sha1:WRCIRZV5ZO56CWMNHFV4FRGNPWPPDVKT > >or: > > HTTP/1.1 302 Found > Location: x-some-new-p2p-protocol:etc/etc/etc > >However, I'm not sure if "302" is the correct HTTP response code to >use. (Anyone have any suggestions?) > > >Another possible solution would be to use the "Accept" header (sent by >the client). Now, this would only be useful for P2P technologies that >are shunted into HTTP via a MIME type (like BitTorrent and the >"application/x-bittorrent" MIME type.). But for technologies which >have this, the client could list the P2P shunt MIME types first, in >the HTTP "Accept" header. For example: > > Accept: application/x-bittorrent, application/ogg > >However, with BitTorrent's case, this brings the problem of confusing >the torrent and files in the torrent. So perhaps a "accept extension" >is needed: > > Accept: application/x-bittorrent; >enclosure-content-type="application/ogg", application/ogg > >Although I still think this still conveys the wrong semantics. So >perhaps this really isn't a good solution. > > >Another alternative is to use the "Upgrade" header (from the client) >and the "HTTP 101" (Switching Protocols) (from the server). This would >let you switch to a P2P protocol. However, this seems to somewhat >imply that whatever protocol is "upgraded to" will have communication >going from between the same client and server. (At least that's my >feeling.) > > >Any Comments? Anyone have any suggestions or advice? > >Also, are there already any proposals or specs along these lines? (No >sense in reinventing the wheel.) > > >TIA > > >See ya > >-- > Charles Iliya Krempeaux, B.Sc. > > charles @ reptile.ca > supercanadian @ gmail.com > > developer weblog: http://ChangeLog.ca/ >___________________________________________________________________________ > Make Television http://maketelevision.com/ >_______________________________________________ >p2p-hackers mailing list >p2p-hackers@zgp.org >http://zgp.org/mailman/listinfo/p2p-hackers >_______________________________________________ >Here is a web page listing P2P Conferences: >http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > > _______________________________________________ p2p-hackers mailing list p2p-hackers@zgp.org http://zgp.org/mailman/listinfo/p2p-hackers _______________________________________________ Here is a web page listing P2P Conferences: http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences From ian at locut.us Tue Feb 14 18:43:58 2006 From: ian at locut.us (Ian Clarke) Date: Sat Dec 9 22:13:09 2006 Subject: [p2p-hackers] Tighter HTTP and P2P integration?? In-Reply-To: <84ce626f0602140035t608263dar1d63f0eecfa92ccb@mail.gmail.com> References: <84ce626f0602140035t608263dar1d63f0eecfa92ccb@mail.gmail.com> Message-ID: <41879D03-2A20-4D3B-80EF-724C4A1B6629@locut.us> You should take a look at Dijjer - http://dijjer.org/. Dijjer is a Java client that runs on your computer, and emulates a local HTTP server. You can request (almost) any file through Dijjer that is available on the WWW by requesting http://127.0.0.1:9115/http://the.url.of/the/ file.mov . Unlike BitTorrent, files are downloaded from Dijjer starting at the beginning, and will be streamed to the browser over the local HTTP connection as they are downloaded. This allows people to do neat things like embed Quicktime movies in web browsers and have the video play as it is downloaded from the P2P network. This will appear to your web browser as if you are downloading the file from an ordinary HTTP server, but in fact, the file will be downloaded through a distributed P2P cache. The first time the file is requested it will be pulled into the cache, subsequent requests will come from other nodes in the P2P network. Other Dijjer features: * NAT hole-punching (see http://en.wikipedia.org/wiki/ UDP_hole_punching) * RESTful design (see above) * Decentralized algorithm (based on small-world principle) * Two complimentary forms of integrity checking (http://dijjer.org/ wiki/IntegrityChecking) * TCP friendly data transfer (http://dijjer.org/wiki/DataTransmission) * Neat internal network abstraction layer that has formed the basis for the next major release of Freenet (http://dijjer.org/wiki/MessagingLayer) * Free as in speech (GPL license) If this sounds like advocacy, that's because it is :-) Dijjer needs developers, the code is relatively clean, and relatively well documented, but right now I am basically the only active developer, and I haven't had much free time lately with the next version of Freenet, not to mention my day job. In particular, it needs work on deployment, there is no OSX installer right now, and Windows version relies on Excelsior Jet - an easy to install but pure Java version would be nice. If you are a Java programmer and looking for something interesting to do, check out the Development section of the site and join our mailing list. Ian. On 14 Feb 2006, at 00:35, Charles Iliya Krempeaux wrote: > Hello, > > (First, I should say that I'm writing this around midnight, so let me > first apologise for all the spelling and grammar mistakes I know I'll > probably make but won't notice until I read this tomorrow. But > anyways....) > > One thing that's been on my mind for a while is making "tighter > integration" between HTTP and P2P technologies. (I've finally been > motivated to write this e-mail since I need to put this to practice > now. Although I really wanted to write a longer and more detailed > blog post on this. And do more research first.... Maybe later.) > > The current methods that bring some integration between HTTP and P2P > technologies include the magnet-uri and BitTorrent (via BitTorrent's > "application/x-bittorrent" MIME type). > > However, one could get even tighter integration. (Here's a few > different ways that may work... I'd like to get people's opinions on > these proposals.) > > One possible avenue for getting even tighter integration is the > creation of something like a "conditional get" for P2P technologies. > (For those unfamiliar with the HTTP "conditional get", look for > information of the "If-None-Match" and "If-Modified-Since" HTTP > headers.) > > A "P2P conditional get" (sent by the client) would tell the server NOT > to send the requested document if there is an alternative way of > getting the document via a P2P technology supported by the client. > (The client would tell the server which P2P technologies it supports.) > (This "P2P conditional get" would work similar to the "If-None-Match" > and "If-Modified-Since" "conditional get" headers.). > > For example, it might be something like: > > X-If-No-Alternate: bittorrent, something-else, magnet > > First, this says that the client supports the protocols: "bittorrent", > "something-else", and "magnet". It also says it prefers "bittorrent" > first, then "something-else", and then "magnet". > > (Really, there's nothing that mandates that these protocols be for P2P > technologies. But that is the use case that is motivating this.) > > This could also include "q" factors. Perhaps something like: > > X-If-No-Alternate: protocol1; q=1.0, protocol2; q=0.7, > protocol3; q=0.5 > > And there could be ways of including more information too. (Which > could be used to add protocol specific information.) For example: > > X-If-No-Alternate: protocol1; q=1.0; a=b;c="d", magnet; > q=0.5;urn-allowed="sha1 md5" > > Note, I'm NOT trying to propose the exact format here or the names > used. Just trying to illustrate what I'm talking about through > examples. (I've tried to follow similar formats as other HTTP headers > though.) > > The server might then respond with an "HTTP 302" (Found) and then > redirect the user to a P2P version using a URI for the P2P technology. > This "redirect" would be done via the HTTP "Location" header. For > example: > > HTTP/1.1 302 Found > Location: magnet:?xt=urn:sha1:WRCIRZV5ZO56CWMNHFV4FRGNPWPPDVKT > > or: > > HTTP/1.1 302 Found > Location: x-some-new-p2p-protocol:etc/etc/etc > > However, I'm not sure if "302" is the correct HTTP response code to > use. (Anyone have any suggestions?) > > > Another possible solution would be to use the "Accept" header (sent by > the client). Now, this would only be useful for P2P technologies that > are shunted into HTTP via a MIME type (like BitTorrent and the > "application/x-bittorrent" MIME type.). But for technologies which > have this, the client could list the P2P shunt MIME types first, in > the HTTP "Accept" header. For example: > > Accept: application/x-bittorrent, application/ogg > > However, with BitTorrent's case, this brings the problem of confusing > the torrent and files in the torrent. So perhaps a "accept extension" > is needed: > > Accept: application/x-bittorrent; > enclosure-content-type="application/ogg", application/ogg > > Although I still think this still conveys the wrong semantics. So > perhaps this really isn't a good solution. > > > Another alternative is to use the "Upgrade" header (from the client) > and the "HTTP 101" (Switching Protocols) (from the server). This would > let you switch to a P2P protocol. However, this seems to somewhat > imply that whatever protocol is "upgraded to" will have communication > going from between the same client and server. (At least that's my > feeling.) > > > Any Comments? Anyone have any suggestions or advice? > > Also, are there already any proposals or specs along these lines? (No > sense in reinventing the wheel.) > > > TIA > > > See ya > > -- > Charles Iliya Krempeaux, B.Sc. > > charles @ reptile.ca > supercanadian @ gmail.com > > developer weblog: http://ChangeLog.ca/ > ______________________________________________________________________ > _____ > Make Television http:// > maketelevision.com/ > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > From supercanadian at gmail.com Tue Feb 14 23:49:02 2006 From: supercanadian at gmail.com (Charles Iliya Krempeaux) Date: Sat Dec 9 22:13:09 2006 Subject: [p2p-hackers] Tighter HTTP and P2P integration?? In-Reply-To: <43F20046.4040709@limewire.com> References: <84ce626f0602140035t608263dar1d63f0eecfa92ccb@mail.gmail.com> <43F20046.4040709@limewire.com> Message-ID: <84ce626f0602141549l2d994826g84b01eafb61da5bd@mail.gmail.com> Hello Karl, On 2/14/06, Karl A. Magdsick wrote: > Hello Charles, > > A couple of weeks ago, there was some work done to integrate > both BitTorrent and Gnutella technologies with the Plone > CMS/Zope webserver. http://plone.org/events/sprints/snow-sprint3 Would this be a more direct link?: http://plone.org/products/plonetramline > I've done some preliminary work (along with Matt Hamilton from NetSight) > using an Apache plugin to add the X-Alt and X-Node HTTP headers > that allow Gnutella clients downloading the same file to find each other > and take load off the server. (I know I could probably read the code to get this info, but I thought it might be easier to just ask, so....) Could you explain the semantics and usage of X-Alt and X-Node more. As well as elaborate more on how all this works, please. (My guess is that the client sends an X-Alt... and then the server responds with a X-Node, probably giving a list of nodes. But please elaborate.) > I haven't gone past proof-of-concept, but I have confirmed that two > LimeWire clients are able to find each other through an Apache > server using a plugin I helped write. The two LimeWire clients > then share portions of the file with each other and thereby reduce > load on the server. If I remember the Gnutella protocol correctly... Do the 2 clients still share this data via HTTP? And it is a normally HTTP download?... or is BitTorrent being employed for this? > Note that Apache 2.2 on OS X 10.4 (Tiger) deals very poorly with a single > connection utilizing multiple 128 kb GET requests in random order. We > got very high CPU useage on the PowerBook being used as the webserver, > and even got a kernel panic. We were forced to use the latest Apache 2.0, > which seemed to work fine. (This was the weekend of Feb 4th, 2006.) Is this for the clients. I.e., is this for the client to client transfer of file pieces? [...] Thanks. See ya -- Charles Iliya Krempeaux, B.Sc. charles @ reptile.ca supercanadian @ gmail.com developer weblog: http://ChangeLog.ca/ ___________________________________________________________________________ Make Television http://maketelevision.com/ From supercanadian at gmail.com Wed Feb 15 00:38:25 2006 From: supercanadian at gmail.com (Charles Iliya Krempeaux) Date: Sat Dec 9 22:13:09 2006 Subject: [p2p-hackers] Tighter HTTP and P2P integration?? In-Reply-To: <41879D03-2A20-4D3B-80EF-724C4A1B6629@locut.us> References: <84ce626f0602140035t608263dar1d63f0eecfa92ccb@mail.gmail.com> <41879D03-2A20-4D3B-80EF-724C4A1B6629@locut.us> Message-ID: <84ce626f0602141638r1550bf0aq8fdd0e0b8114ba7a@mail.gmail.com> Hello, I've been actually considering using a local HTTP server to implement things. (Which is something you are doing too.) Although I thought some extra "sugar" was warranted, to help with (user) usabiliy and developer usability. I skimmed through the stuff on dijjer.org, and I have a questions. Does digger pay attention to HTTP headers? And act basically as a web cache? For example, here's some (but not all) "rules" that I was planning on following.... If HTTP headers "say" that something should NOT be cached, then it should NOT be on the P2P network. Stuff on the P2P network should be removed after the date for the "Expires" header passes? Things from non-HTTP GET methods should NOT be put onto the P2P network? Etc. Does dijjer follow these web caching rules? See ya On 2/14/06, Ian Clarke wrote: > You should take a look at Dijjer - http://dijjer.org/. > > Dijjer is a Java client that runs on your computer, and emulates a > local HTTP server. > > You can request (almost) any file through Dijjer that is available on > the WWW by requesting http://127.0.0.1:9115/http://the.url.of/the/ > file.mov . Unlike BitTorrent, files are downloaded from Dijjer > starting at the beginning, and will be streamed to the browser over > the local HTTP connection as they are downloaded. This allows people > to do neat things like embed Quicktime movies in web browsers and > have the video play as it is downloaded from the P2P network. > > This will appear to your web browser as if you are downloading the > file from an ordinary HTTP server, but in fact, the file will be > downloaded through a distributed P2P cache. The first time the file > is requested it will be pulled into the cache, subsequent requests > will come from other nodes in the P2P network. > > Other Dijjer features: > > * NAT hole-punching (see http://en.wikipedia.org/wiki/ > UDP_hole_punching) > * RESTful design (see above) > * Decentralized algorithm (based on small-world principle) > * Two complimentary forms of integrity checking (http://dijjer.org/ > wiki/IntegrityChecking) > * TCP friendly data transfer (http://dijjer.org/wiki/DataTransmission) > * Neat internal network abstraction layer that has formed the basis > for the next > major release of Freenet (http://dijjer.org/wiki/MessagingLayer) > * Free as in speech (GPL license) > > If this sounds like advocacy, that's because it is :-) Dijjer needs > developers, the code is relatively clean, and relatively well > documented, but right now I am basically the only active developer, > and I haven't had much free time lately with the next version of > Freenet, not to mention my day job. > > In particular, it needs work on deployment, there is no OSX installer > right now, and Windows version relies on Excelsior Jet - an easy to > install but pure Java version would be nice. > > If you are a Java programmer and looking for something interesting to > do, check out the Development section of the site and join our > mailing list. > > Ian. > > On 14 Feb 2006, at 00:35, Charles Iliya Krempeaux wrote: > > > Hello, > > > > (First, I should say that I'm writing this around midnight, so let me > > first apologise for all the spelling and grammar mistakes I know I'll > > probably make but won't notice until I read this tomorrow. But > > anyways....) > > > > One thing that's been on my mind for a while is making "tighter > > integration" between HTTP and P2P technologies. (I've finally been > > motivated to write this e-mail since I need to put this to practice > > now. Although I really wanted to write a longer and more detailed > > blog post on this. And do more research first.... Maybe later.) > > > > The current methods that bring some integration between HTTP and P2P > > technologies include the magnet-uri and BitTorrent (via BitTorrent's > > "application/x-bittorrent" MIME type). > > > > However, one could get even tighter integration. (Here's a few > > different ways that may work... I'd like to get people's opinions on > > these proposals.) > > > > One possible avenue for getting even tighter integration is the > > creation of something like a "conditional get" for P2P technologies. > > (For those unfamiliar with the HTTP "conditional get", look for > > information of the "If-None-Match" and "If-Modified-Since" HTTP > > headers.) > > > > A "P2P conditional get" (sent by the client) would tell the server NOT > > to send the requested document if there is an alternative way of > > getting the document via a P2P technology supported by the client. > > (The client would tell the server which P2P technologies it supports.) > > (This "P2P conditional get" would work similar to the "If-None-Match" > > and "If-Modified-Since" "conditional get" headers.). > > > > For example, it might be something like: > > > > X-If-No-Alternate: bittorrent, something-else, magnet > > > > First, this says that the client supports the protocols: "bittorrent", > > "something-else", and "magnet". It also says it prefers "bittorrent" > > first, then "something-else", and then "magnet". > > > > (Really, there's nothing that mandates that these protocols be for P2P > > technologies. But that is the use case that is motivating this.) > > > > This could also include "q" factors. Perhaps something like: > > > > X-If-No-Alternate: protocol1; q=1.0, protocol2; q=0.7, > > protocol3; q=0.5 > > > > And there could be ways of including more information too. (Which > > could be used to add protocol specific information.) For example: > > > > X-If-No-Alternate: protocol1; q=1.0; a=b;c="d", magnet; > > q=0.5;urn-allowed="sha1 md5" > > > > Note, I'm NOT trying to propose the exact format here or the names > > used. Just trying to illustrate what I'm talking about through > > examples. (I've tried to follow similar formats as other HTTP headers > > though.) > > > > The server might then respond with an "HTTP 302" (Found) and then > > redirect the user to a P2P version using a URI for the P2P technology. > > This "redirect" would be done via the HTTP "Location" header. For > > example: > > > > HTTP/1.1 302 Found > > Location: magnet:?xt=urn:sha1:WRCIRZV5ZO56CWMNHFV4FRGNPWPPDVKT > > > > or: > > > > HTTP/1.1 302 Found > > Location: x-some-new-p2p-protocol:etc/etc/etc > > > > However, I'm not sure if "302" is the correct HTTP response code to > > use. (Anyone have any suggestions?) > > > > > > Another possible solution would be to use the "Accept" header (sent by > > the client). Now, this would only be useful for P2P technologies that > > are shunted into HTTP via a MIME type (like BitTorrent and the > > "application/x-bittorrent" MIME type.). But for technologies which > > have this, the client could list the P2P shunt MIME types first, in > > the HTTP "Accept" header. For example: > > > > Accept: application/x-bittorrent, application/ogg > > > > However, with BitTorrent's case, this brings the problem of confusing > > the torrent and files in the torrent. So perhaps a "accept extension" > > is needed: > > > > Accept: application/x-bittorrent; > > enclosure-content-type="application/ogg", application/ogg > > > > Although I still think this still conveys the wrong semantics. So > > perhaps this really isn't a good solution. > > > > > > Another alternative is to use the "Upgrade" header (from the client) > > and the "HTTP 101" (Switching Protocols) (from the server). This would > > let you switch to a P2P protocol. However, this seems to somewhat > > imply that whatever protocol is "upgraded to" will have communication > > going from between the same client and server. (At least that's my > > feeling.) > > > > > > Any Comments? Anyone have any suggestions or advice? > > > > Also, are there already any proposals or specs along these lines? (No > > sense in reinventing the wheel.) -- Charles Iliya Krempeaux, B.Sc. charles @ reptile.ca supercanadian @ gmail.com developer weblog: http://ChangeLog.ca/ ___________________________________________________________________________ Make Television http://maketelevision.com/ From supercanadian at gmail.com Wed Feb 15 00:59:36 2006 From: supercanadian at gmail.com (Charles Iliya Krempeaux) Date: Sat Dec 9 22:13:09 2006 Subject: [p2p-hackers] Tighter HTTP and P2P integration?? In-Reply-To: <200602141841.k1EIf9qx019157@be9.noc0.redswoosh.com> References: <43F20046.4040709@limewire.com> <200602141841.k1EIf9qx019157@be9.noc0.redswoosh.com> Message-ID: <84ce626f0602141659r75488e35tf90a27dbb7b11a64@mail.gmail.com> Hello, I took a look at the site, but I had trouble finding any technical information. Is there any available? (Or is everything proprietary?) See ya On 2/14/06, Travis Kalanick wrote: > Web integration has been key to Red Swoosh's architecture. > > We have been doing a more advanced version of your "conditional get" where > clients simply look at http as another source. In addition we have several > parameters that are embedded in the links that allow for different > streaming/download methods (peers only, peers and server, peers but fallback > on server, etc.). > > You can get a Basic version of the API (a limited set of the parameters > available in the advanced API) emailed to you here: > > http://redswoosh.net/tryus_step_by_step.php > > > We've had live customers (i.e. websites) up on this web-integration > architecture since 2001 (Deviantart.com launched with us in October 2001) > and have found it to be quite a powerful way to employ P2P tech. > > > Thanks, > > Travis > > -----Original Message----- > From: p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org] On > Behalf Of Karl A. Magdsick > Sent: Tuesday, February 14, 2006 8:08 AM > To: charles@reptile.ca; Peer-to-peer development. > Subject: Re: [p2p-hackers] Tighter HTTP and P2P integration?? > > Hello Charles, > > A couple of weeks ago, there was some work done to integrate > both BitTorrent and Gnutella technologies with the Plone > CMS/Zope webserver. http://plone.org/events/sprints/snow-sprint3 > > I've done some preliminary work (along with Matt Hamilton from NetSight) > using an Apache plugin to add the X-Alt and X-Node HTTP headers > that allow Gnutella clients downloading the same file to find each other > and take load off the server. > > I haven't gone past proof-of-concept, but I have confirmed that two > LimeWire clients are able to find each other through an Apache > server using a plugin I helped write. The two LimeWire clients > then share portions of the file with each other and thereby reduce > load on the server. > > Note that Apache 2.2 on OS X 10.4 (Tiger) deals very poorly with a single > connection utilizing multiple 128 kb GET requests in random order. We > got very high CPU useage on the PowerBook being used as the webserver, > and even got a kernel panic. We were forced to use the latest Apache 2.0, > which seemed to work fine. (This was the weekend of Feb 4th, 2006.) > > > Good Luck, > > Karl Magdsick > Software Engineer, Lime Wire LLC > > Charles Iliya Krempeaux wrote: > > >Hello, > > > >(First, I should say that I'm writing this around midnight, so let me > >first apologise for all the spelling and grammar mistakes I know I'll > >probably make but won't notice until I read this tomorrow. But > >anyways....) > > > >One thing that's been on my mind for a while is making "tighter > >integration" between HTTP and P2P technologies. (I've finally been > >motivated to write this e-mail since I need to put this to practice > >now. Although I really wanted to write a longer and more detailed > >blog post on this. And do more research first.... Maybe later.) > > > >The current methods that bring some integration between HTTP and P2P > >technologies include the magnet-uri and BitTorrent (via BitTorrent's > >"application/x-bittorrent" MIME type). > > > >However, one could get even tighter integration. (Here's a few > >different ways that may work... I'd like to get people's opinions on > >these proposals.) > > > >One possible avenue for getting even tighter integration is the > >creation of something like a "conditional get" for P2P technologies. > >(For those unfamiliar with the HTTP "conditional get", look for > >information of the "If-None-Match" and "If-Modified-Since" HTTP > >headers.) > > > >A "P2P conditional get" (sent by the client) would tell the server NOT > >to send the requested document if there is an alternative way of > >getting the document via a P2P technology supported by the client. > >(The client would tell the server which P2P technologies it supports.) > > (This "P2P conditional get" would work similar to the "If-None-Match" > >and "If-Modified-Since" "conditional get" headers.). > > > >For example, it might be something like: > > > > X-If-No-Alternate: bittorrent, something-else, magnet > > > >First, this says that the client supports the protocols: "bittorrent", > >"something-else", and "magnet". It also says it prefers "bittorrent" > >first, then "something-else", and then "magnet". > > > >(Really, there's nothing that mandates that these protocols be for P2P > >technologies. But that is the use case that is motivating this.) > > > >This could also include "q" factors. Perhaps something like: > > > > X-If-No-Alternate: protocol1; q=1.0, protocol2; q=0.7, protocol3; q=0.5 > > > >And there could be ways of including more information too. (Which > >could be used to add protocol specific information.) For example: > > > > X-If-No-Alternate: protocol1; q=1.0; a=b;c="d", magnet; > >q=0.5;urn-allowed="sha1 md5" > > > >Note, I'm NOT trying to propose the exact format here or the names > >used. Just trying to illustrate what I'm talking about through > >examples. (I've tried to follow similar formats as other HTTP headers > >though.) > > > >The server might then respond with an "HTTP 302" (Found) and then > >redirect the user to a P2P version using a URI for the P2P technology. > > This "redirect" would be done via the HTTP "Location" header. For > >example: > > > > HTTP/1.1 302 Found > > Location: magnet:?xt=urn:sha1:WRCIRZV5ZO56CWMNHFV4FRGNPWPPDVKT > > > >or: > > > > HTTP/1.1 302 Found > > Location: x-some-new-p2p-protocol:etc/etc/etc > > > >However, I'm not sure if "302" is the correct HTTP response code to > >use. (Anyone have any suggestions?) > > > > > >Another possible solution would be to use the "Accept" header (sent by > >the client). Now, this would only be useful for P2P technologies that > >are shunted into HTTP via a MIME type (like BitTorrent and the > >"application/x-bittorrent" MIME type.). But for technologies which > >have this, the client could list the P2P shunt MIME types first, in > >the HTTP "Accept" header. For example: > > > > Accept: application/x-bittorrent, application/ogg > > > >However, with BitTorrent's case, this brings the problem of confusing > >the torrent and files in the torrent. So perhaps a "accept extension" > >is needed: > > > > Accept: application/x-bittorrent; > >enclosure-content-type="application/ogg", application/ogg > > > >Although I still think this still conveys the wrong semantics. So > >perhaps this really isn't a good solution. > > > > > >Another alternative is to use the "Upgrade" header (from the client) > >and the "HTTP 101" (Switching Protocols) (from the server). This would > >let you switch to a P2P protocol. However, this seems to somewhat > >imply that whatever protocol is "upgraded to" will have communication > >going from between the same client and server. (At least that's my > >feeling.) > > > > > >Any Comments? Anyone have any suggestions or advice? > > > >Also, are there already any proposals or specs along these lines? (No > >sense in reinventing the wheel.) -- Charles Iliya Krempeaux, B.Sc. charles @ reptile.ca supercanadian @ gmail.com developer weblog: http://ChangeLog.ca/ ___________________________________________________________________________ Make Television http://maketelevision.com/ From kmagdsick at limewire.com Wed Feb 15 02:00:34 2006 From: kmagdsick at limewire.com (Karl A. Magdsick) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Tighter HTTP and P2P integration?? In-Reply-To: <84ce626f0602141549l2d994826g84b01eafb61da5bd@mail.gmail.com> References: <84ce626f0602140035t608263dar1d63f0eecfa92ccb@mail.gmail.com> <43F20046.4040709@limewire.com> <84ce626f0602141549l2d994826g84b01eafb61da5bd@mail.gmail.com> Message-ID: <43F28B42.8070009@limewire.com> Charles Iliya Krempeaux wrote: [snip] >Would this be a more direct link?: http://plone.org/products/plonetramline > > No. Tramline is just a somewhat clever hack to prevent the Zope webserver from storing large files as objects in its object database. If you're not familiar with Zope, looking at Tramilne will likely just confuse you. Tramline was involved with my work only in a round-about way. >>I've done some preliminary work (along with Matt Hamilton from NetSight) >>using an Apache plugin to add the X-Alt and X-Node HTTP headers >>that allow Gnutella clients downloading the same file to find each other >>and take load off the server. >> >> > >(I know I could probably read the code to get this info, but I thought >it might be easier to just ask, so....) Could you explain the >semantics and usage of X-Alt and X-Node more. As well as elaborate >more on how all this works, please. > > First, a short bit about the relationship between Gnutella and HTTP: Gnutella uses HTTP to transfer all file data. This is a huge advantage for integration with webservers, as Gnutella clients can treat the webserver as just another Gnutella client. Many P2P networks invent their own file-transfer protocols, but the overhead of HTTP isn't very large, and the ability to pretend that regular webservers are peers is a huge win. Optional HTTP headers are used to exchange information so that Gnutella clients that are downloading the same file can form a "download mesh" -- a set of clients that share and share alike chunks of a file they are all trying to download. Once you've installed a webserver plugin to allow the webserver to understand and send a very small number of optional headers, the webserver can be thought of as a special case of Gnutella client: one that can't search or download, but one that is capable of uploading files, helping coordinate download meshes, and participating in download meshes. It's all just HTTP with a handful of optional headers. The Gnutella protocol itself is only necessary for searching, and for getting files from hosts that are unable to punch holes in NAT. It's a fair assumption that your webserver isn't behind NAT, or you've punched a hole in NAT for your webserver. In this case, the webserver only needs to speak the Gnutella protocol if you want your webserver to respond to Gnutella searches or you want some kind of web interface for searching Gnutella. Facilitating a Gnutella mesh through a webserver is very simple, much simpler than implementing a BitTorrent tracker. There is nearly no extra intelligence required in the webserver, and the clients all treat the webserver as just another member of the download mesh. Next, a bit about the headers: X-Node: a Gnutella client that's able to punch through NAT/firewalls and get an externally contactable IP address will send its external IP and Gnutella port number in this header. X-Alt: a Gnutella client will send IP:port pairs of Gnutella clients in the download mesh using the X-Alt header. If a client is able to punch through NAT, it'll include itself in the X-Alt list once it is sharing part of the file. X-NAlt: this header contains a list of "bad" alternate locations. In essence, this header says "you gave me some bad entries in an X-Alt header. stop sending out the following IP:port pairs". X-FAlt: this is the X-Alt header, but for mesh members that are unable to punch through NAT/firewalls. The entries contain information about which proxies to use to contact these firewalled clients. X-NFAlt: this is the firewalled version of X-NAlt. How this all works: For each file to be downloaded, the webserver remembers information that it gets in X-Alt headers, and spits the same information back out in X-Alt headers. It purges X-Alt entries that it sees in X-NAlt headers. It does the same thing for X-FAlt and X-NFAlt headers. The X-Node information may be useful in deciding which entries in its internal X-Alt pool should be sent out. >(My guess is that the client sends an X-Alt... and then the server >responds with a X-Node, probably giving a list of nodes. But please >elaborate.) > > Unlike semi-centralized systems like Napster or BitTorrent, there is no distinction between client and server in Gnutella. X-Node is for "this is me", and X-Alt is for "you may want to talk to these". Both the "client" and "server" side send the X-Alt headers. Presumably, the "client" side knows how it contacted the "server" and therefore knows how the "server" is externally contactable, so it's useless for the "server" to send X-Node. >>I haven't gone past proof-of-concept, but I have confirmed that two >>LimeWire clients are able to find each other through an Apache >>server using a plugin I helped write. The two LimeWire clients >>then share portions of the file with each other and thereby reduce >>load on the server. >> >> > >If I remember the Gnutella protocol correctly... Do the 2 clients >still share this data via HTTP? And it is a normally HTTP >download?... or is BitTorrent being employed for this? > > It's pure HTTP. There's no need to thrown in some other protocol. I hope this helps, Karl Magdsick Software Engineer, Lime Wire LLC From ian at locut.us Wed Feb 15 02:13:31 2006 From: ian at locut.us (Ian Clarke) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Tighter HTTP and P2P integration?? In-Reply-To: <84ce626f0602141638r1550bf0aq8fdd0e0b8114ba7a@mail.gmail.com> References: <84ce626f0602140035t608263dar1d63f0eecfa92ccb@mail.gmail.com> <41879D03-2A20-4D3B-80EF-724C4A1B6629@locut.us> <84ce626f0602141638r1550bf0aq8fdd0e0b8114ba7a@mail.gmail.com> Message-ID: <097AC0C1-CFE5-46A4-A49C-42691CA3F69D@locut.us> On 14 Feb 2006, at 16:38, Charles Iliya Krempeaux wrote: > I've been actually considering using a local HTTP server to implement > things. (Which is something you are doing too.) Although I thought > some extra "sugar" was warranted, to help with (user) usabiliy and > developer usability. We have added some sugar too. For example, we have javascript available on our site that can detect whether Dijjer is present on the user's machine and redirect the user to a download page if it isn't. > I skimmed through the stuff on dijjer.org, and I have a questions. > Does digger pay attention to HTTP headers? And act basically as a web > cache? Yes, we have attempted to stick as closely to the HTTP protocol as possible. > For example, here's some (but not all) "rules" that I was planning on > following.... If HTTP headers "say" that something should NOT be > cached, then it should NOT be on the P2P network. Yes, we obey the various HTTP no-cache headers. We consider it important for Dijjer to act as much like a HTTP cache as possible so that we are afforded the protection the DMCA gives to caches for copyright infringement. > Stuff on the P2P > network should be removed after the date for the "Expires" header > passes? We don't obey "Expires" explicitly, but since Dijjer pings the original web server each-time something is downloaded to ensure that the file is still there and hasn't changed, if it is removed from the web server, it will effectively be removed from Dijjer at the same instant. > Things from non-HTTP GET methods should NOT be put onto the > P2P network? Etc. We attempt to be smart about our interaction with HTTP, if there are shortcomings in this regard they will be easy to fix. Ian. From travis at redswoosh.net Wed Feb 15 08:40:33 2006 From: travis at redswoosh.net (Travis Kalanick) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Tighter HTTP and P2P integration?? In-Reply-To: <84ce626f0602141659r75488e35tf90a27dbb7b11a64@mail.gmail.com> Message-ID: <200602150845.k1F8jnqx028119@be9.noc0.redswoosh.com> An email with the Basic API is available at the link I provided. http://redswoosh.net/tryus_step_by_step.php Check it out, let me know what you think. Also, keep in mind that our installation system will be re-vamped within 45 days. We'll have a much cleaner, tighter .exe based installation. T -----Original Message----- From: Charles Iliya Krempeaux [mailto:supercanadian@gmail.com] Sent: Tuesday, February 14, 2006 5:00 PM To: Peer-to-peer development. Cc: Travis Kalanick Subject: Re: [p2p-hackers] Tighter HTTP and P2P integration?? Hello, I took a look at the site, but I had trouble finding any technical information. Is there any available? (Or is everything proprietary?) See ya On 2/14/06, Travis Kalanick wrote: > Web integration has been key to Red Swoosh's architecture. > > We have been doing a more advanced version of your "conditional get" where > clients simply look at http as another source. In addition we have several > parameters that are embedded in the links that allow for different > streaming/download methods (peers only, peers and server, peers but fallback > on server, etc.). > > You can get a Basic version of the API (a limited set of the parameters > available in the advanced API) emailed to you here: > > http://redswoosh.net/tryus_step_by_step.php > > > We've had live customers (i.e. websites) up on this web-integration > architecture since 2001 (Deviantart.com launched with us in October 2001) > and have found it to be quite a powerful way to employ P2P tech. > > > Thanks, > > Travis > > -----Original Message----- > From: p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org] On > Behalf Of Karl A. Magdsick > Sent: Tuesday, February 14, 2006 8:08 AM > To: charles@reptile.ca; Peer-to-peer development. > Subject: Re: [p2p-hackers] Tighter HTTP and P2P integration?? > > Hello Charles, > > A couple of weeks ago, there was some work done to integrate > both BitTorrent and Gnutella technologies with the Plone > CMS/Zope webserver. http://plone.org/events/sprints/snow-sprint3 > > I've done some preliminary work (along with Matt Hamilton from NetSight) > using an Apache plugin to add the X-Alt and X-Node HTTP headers > that allow Gnutella clients downloading the same file to find each other > and take load off the server. > > I haven't gone past proof-of-concept, but I have confirmed that two > LimeWire clients are able to find each other through an Apache > server using a plugin I helped write. The two LimeWire clients > then share portions of the file with each other and thereby reduce > load on the server. > > Note that Apache 2.2 on OS X 10.4 (Tiger) deals very poorly with a single > connection utilizing multiple 128 kb GET requests in random order. We > got very high CPU useage on the PowerBook being used as the webserver, > and even got a kernel panic. We were forced to use the latest Apache 2.0, > which seemed to work fine. (This was the weekend of Feb 4th, 2006.) > > > Good Luck, > > Karl Magdsick > Software Engineer, Lime Wire LLC > > Charles Iliya Krempeaux wrote: > > >Hello, > > > >(First, I should say that I'm writing this around midnight, so let me > >first apologise for all the spelling and grammar mistakes I know I'll > >probably make but won't notice until I read this tomorrow. But > >anyways....) > > > >One thing that's been on my mind for a while is making "tighter > >integration" between HTTP and P2P technologies. (I've finally been > >motivated to write this e-mail since I need to put this to practice > >now. Although I really wanted to write a longer and more detailed > >blog post on this. And do more research first.... Maybe later.) > > > >The current methods that bring some integration between HTTP and P2P > >technologies include the magnet-uri and BitTorrent (via BitTorrent's > >"application/x-bittorrent" MIME type). > > > >However, one could get even tighter integration. (Here's a few > >different ways that may work... I'd like to get people's opinions on > >these proposals.) > > > >One possible avenue for getting even tighter integration is the > >creation of something like a "conditional get" for P2P technologies. > >(For those unfamiliar with the HTTP "conditional get", look for > >information of the "If-None-Match" and "If-Modified-Since" HTTP > >headers.) > > > >A "P2P conditional get" (sent by the client) would tell the server NOT > >to send the requested document if there is an alternative way of > >getting the document via a P2P technology supported by the client. > >(The client would tell the server which P2P technologies it supports.) > > (This "P2P conditional get" would work similar to the "If-None-Match" > >and "If-Modified-Since" "conditional get" headers.). > > > >For example, it might be something like: > > > > X-If-No-Alternate: bittorrent, something-else, magnet > > > >First, this says that the client supports the protocols: "bittorrent", > >"something-else", and "magnet". It also says it prefers "bittorrent" > >first, then "something-else", and then "magnet". > > > >(Really, there's nothing that mandates that these protocols be for P2P > >technologies. But that is the use case that is motivating this.) > > > >This could also include "q" factors. Perhaps something like: > > > > X-If-No-Alternate: protocol1; q=1.0, protocol2; q=0.7, protocol3; q=0.5 > > > >And there could be ways of including more information too. (Which > >could be used to add protocol specific information.) For example: > > > > X-If-No-Alternate: protocol1; q=1.0; a=b;c="d", magnet; > >q=0.5;urn-allowed="sha1 md5" > > > >Note, I'm NOT trying to propose the exact format here or the names > >used. Just trying to illustrate what I'm talking about through > >examples. (I've tried to follow similar formats as other HTTP headers > >though.) > > > >The server might then respond with an "HTTP 302" (Found) and then > >redirect the user to a P2P version using a URI for the P2P technology. > > This "redirect" would be done via the HTTP "Location" header. For > >example: > > > > HTTP/1.1 302 Found > > Location: magnet:?xt=urn:sha1:WRCIRZV5ZO56CWMNHFV4FRGNPWPPDVKT > > > >or: > > > > HTTP/1.1 302 Found > > Location: x-some-new-p2p-protocol:etc/etc/etc > > > >However, I'm not sure if "302" is the correct HTTP response code to > >use. (Anyone have any suggestions?) > > > > > >Another possible solution would be to use the "Accept" header (sent by > >the client). Now, this would only be useful for P2P technologies that > >are shunted into HTTP via a MIME type (like BitTorrent and the > >"application/x-bittorrent" MIME type.). But for technologies which > >have this, the client could list the P2P shunt MIME types first, in > >the HTTP "Accept" header. For example: > > > > Accept: application/x-bittorrent, application/ogg > > > >However, with BitTorrent's case, this brings the problem of confusing > >the torrent and files in the torrent. So perhaps a "accept extension" > >is needed: > > > > Accept: application/x-bittorrent; > >enclosure-content-type="application/ogg", application/ogg > > > >Although I still think this still conveys the wrong semantics. So > >perhaps this really isn't a good solution. > > > > > >Another alternative is to use the "Upgrade" header (from the client) > >and the "HTTP 101" (Switching Protocols) (from the server). This would > >let you switch to a P2P protocol. However, this seems to somewhat > >imply that whatever protocol is "upgraded to" will have communication > >going from between the same client and server. (At least that's my > >feeling.) > > > > > >Any Comments? Anyone have any suggestions or advice? > > > >Also, are there already any proposals or specs along these lines? (No > >sense in reinventing the wheel.) -- Charles Iliya Krempeaux, B.Sc. charles @ reptile.ca supercanadian @ gmail.com developer weblog: http://ChangeLog.ca/ ___________________________________________________________________________ Make Television http://maketelevision.com/ From supercanadian at gmail.com Wed Feb 15 22:39:29 2006 From: supercanadian at gmail.com (Charles Iliya Krempeaux) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Tighter HTTP and P2P integration?? In-Reply-To: <43F28B42.8070009@limewire.com> References: <84ce626f0602140035t608263dar1d63f0eecfa92ccb@mail.gmail.com> <43F20046.4040709@limewire.com> <84ce626f0602141549l2d994826g84b01eafb61da5bd@mail.gmail.com> <43F28B42.8070009@limewire.com> Message-ID: <84ce626f0602151439w2031da3cl966c0d0a08aefa67@mail.gmail.com> Hello Karl, On 2/14/06, Karl A. Magdsick wrote: > Charles Iliya Krempeaux wrote: [...] > >>I've done some preliminary work (along with Matt Hamilton from NetSight) > >>using an Apache plugin to add the X-Alt and X-Node HTTP headers > >>that allow Gnutella clients downloading the same file to find each other > >>and take load off the server. > >> > >> > > > >(I know I could probably read the code to get this info, but I thought > >it might be easier to just ask, so....) Could you explain the > >semantics and usage of X-Alt and X-Node more. As well as elaborate > >more on how all this works, please. > > > > > > First, a short bit about the relationship between Gnutella and HTTP: > > Gnutella uses HTTP to transfer all file data. This is a huge advantage > for integration with webservers, as Gnutella clients can treat the webserver > as just another Gnutella client. Many P2P networks invent their own > file-transfer protocols, but the overhead of HTTP isn't very large, and > the ability to pretend that regular webservers are peers is a huge win. > > Optional HTTP headers are used to exchange information so that Gnutella > clients that are downloading the same file can form a "download mesh" -- a > set of clients that share and share alike chunks of a file they are all > trying to download. > > Once you've installed a webserver plugin to allow the webserver to > understand > and send a very small number of optional headers, the webserver can be > thought of as a special case of Gnutella client: one that can't search > or download, > but one that is capable of uploading files, helping coordinate download > meshes, > and participating in download meshes. It's all just HTTP with a handful of > optional headers. > > The Gnutella protocol itself is only necessary for searching, and for > getting files from hosts that are unable to punch holes in NAT. It's a fair > assumption that your webserver isn't behind NAT, or you've punched a hole > in NAT for your webserver. In this case, the webserver only needs to speak > the Gnutella protocol if you want your webserver to respond to Gnutella > searches or you want some kind of web interface for searching Gnutella. > > Facilitating a Gnutella mesh through a webserver is very simple, much > simpler than implementing a BitTorrent tracker. There is nearly no > extra intelligence > required in the webserver, and the clients all treat the webserver as > just another > member of the download mesh. > > > Next, a bit about the headers: > > X-Node: a Gnutella client that's able to punch through NAT/firewalls > and get > an externally contactable IP address will send its external IP and > Gnutella port > number in this header. > > X-Alt: a Gnutella client will send IP:port pairs of Gnutella clients in > the download > mesh using the X-Alt header. If a client is able to punch through NAT, > it'll > include itself in the X-Alt list once it is sharing part of the file. > > X-NAlt: this header contains a list of "bad" alternate locations. In > essence, this > header says "you gave me some bad entries in an X-Alt header. stop sending > out the following IP:port pairs". > > X-FAlt: this is the X-Alt header, but for mesh members that are unable > to punch > through NAT/firewalls. The entries contain information about which proxies > to use to contact these firewalled clients. > > X-NFAlt: this is the firewalled version of X-NAlt. > > > How this all works: > > For each file to be downloaded, the webserver remembers information that > it gets > in X-Alt headers, and spits the same information back out in X-Alt > headers. It purges > X-Alt entries that it sees in X-NAlt headers. It does the same thing > for X-FAlt and > X-NFAlt headers. The X-Node information may be useful in deciding which > entries in its internal X-Alt pool should be sent out. That illuminates things a bit. I guess my knowledge of the Gnutella protocol is a bit lacking :-) (It's been a while since I looked at the protocol :-) ) I did a bit of searching (after you wrote this), and came to these 2 documents that were helpful too: http://www.the-gdf.org/wiki/index.php?title=Known_HTTP_Download_Headers http://www.the-gdf.org/wiki/index.php?title=The_Download_Mesh The "X-Alt" headers seem to be pretty close to what I was thinking. Although (in my original post) I had this type of information communicated with the "Location" header. And I was also thinking of somehow "encoding" all the different alternate locations into one URL. But, the "X-Alt" header seems to be more flexible, in that you can list all the alternate sources easily. One problem I see with it (at least for my purposes) is that it only let's you list a IP address with a TCP port number. I guess because Gnutella has a special format for requesting files; as documented here: http://www.the-gdf.org/wiki/index.php?title=File_Transfer I think the "X-Alt" would have been able to be used for more things if you were able to use full URLs or full URIs. But anyways, something like "X-Alt" seems better than using the "Location" header. In the HTTP 1.1 spec, there is mention of a "Link" header. (See section 19.6.2.4 of RFC 2068.) This could probably be used. The "Link" header basically has the same semantics as the HTML element. So, something like the following could be done. Link: ; rel="alternate" Link: ; rel="alternate" Link: ; rel="alternate" Link: ; rel="alternate" (Note, in the example above, the server is sending multiple "Link" headers in one response.) With the link you can add other "attributes" too. (Think "attributes" in terms of HTML "attribute"... on the element.) You can have "title", "class", "media", "type", etc. (Just like the HTML element.) As a side note, Gnutella could even use the "Link" header instead of the "X-Alt" header. For example, the following: X-Alt: 192.0.2.17:6347, 192.0.2.44 could be translated into: Link: ; rel="alternate" Link: ; rel="alternate" (Where "FILE_INDEX" and "FILE_NAME" would be replaced with whatever they actually are.) So, having said all that, I'm still thinking that a "P2P conditional get" is a good idea. (For reason I'll explain in the next paragraph. But please argue if you disagree.) And that, the shunting of the client (or servent or whatever) onto the P2P network could be done via those "Link" headers. Now, if some of those "alternate" locations are Gnutella servents, then you'd use the Gnutella extension to HTTP. And if they are using a different protocol, the you do whatever is necessary for it. The reason I'm still inclined to go with a "P2P conditional get" is because it gives some level of control to the publisher. That way they are in some sense sanctioning it. (Which under some situations matters.) Now, like with web caching, one could always go ahead and use a P2P method of getting the file unilaterally -- without asking or needing the publisher to get involved. So right now my thinking is something like the following example. (Please argue if you disagree.) Client: GET /some/file HTTP/1.1 Host: example.com X-If-No-Alternate: ## PUT STUFF HERE ABOUT WHAT I CAN ACCEPT ## Server: HTTP/1.1 204 No Content Link: ; rel="alternate" Link: ; rel="alternate"; Link: ; rel="alternate", class="gnutella" (Note sure if the "204" HTTP result code is the correct one to use though. But anyways....) Note, I'm using a "URI reference" on the BitTorrent URI to access the file "within" the torrent. I'm not sure if anyone has used that kind of notation before. But it seems like a good way to "give a link to" what's inside of the torrent. Also note, that I've maked the Gnutella link with a "class" parameters... so that the client (or servernt or whatever) can identify it as being Gnutella. Thoughts? Comments? [...] See ya -- Charles Iliya Krempeaux, B.Sc. charles @ reptile.ca supercanadian @ gmail.com developer weblog: http://ChangeLog.ca/ ___________________________________________________________________________ Make Television http://maketelevision.com/ From coderman at gmail.com Thu Feb 16 00:34:16 2006 From: coderman at gmail.com (coderman) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Tighter HTTP and P2P integration?? In-Reply-To: <84ce626f0602140035t608263dar1d63f0eecfa92ccb@mail.gmail.com> References: <84ce626f0602140035t608263dar1d63f0eecfa92ccb@mail.gmail.com> Message-ID: <4ef5fec60602151634w2f37e25dhb338c9af2e7b50c1@mail.gmail.com> On 2/14/06, Charles Iliya Krempeaux wrote: > ... > One thing that's been on my mind for a while is making "tighter > integration" between HTTP and P2P technologies. > ... > A "P2P conditional get" (sent by the client) would tell the server NOT > to send the requested document if there is an alternative way of > getting the document via a P2P technology supported by the client. > ... > Also, are there already any proposals or specs along these lines? another route to http / p2p integration is through the file system with each peer running a local web server instance for UI on top of a shared fs tree / archive. [decentralized and/or distributed file systems are another topic, so i'll assume the presence of a robust shared file system view among peers in this explanation] our context is a bit different, as this mainly applies to private group networks with known client configurations (everyone running off a live CD / hdd install of the same shared OS) the desired handlers for various MIME types / filters / views can be accomplished per user with custom browser configurations and plugins. for interactive / collaborative / real time functions the private group network endpoints can be used to host services. consider: a pre-distributed archive / fs tree associated with the group: .../mygroup/public/music/ video/ docs/ all peers can reference these resources via local methods like file:///... or aliases in a drupal instance on localhost - http://127.0.0.1/video/favorites. aliases/vhosts for 127.0.0.1 (localhost, home, mygroup, etc) can also be used to provide distinct shared group apps/instances with distinct document roots as desired. (again, simply apache and drupal with a common httpd.conf bound to 127.0.0.1:80 using vservers for names in /etc/hosts) services and other resources are bound to peer identities and IPv6, for example a private IPv6 network using feed:7ba3:c779:e5f5:cc1a:8bc7:5fe1:ed08 where feed: is the common subnet and the remaining identifier space is based off a hash of public node identity. feed:7ba3:c779:e5f5:cc1a:8bc7:5fe1:ed08 can then be aliased like 127.0.0.1 and will remain valid as long as communication (direct or via proxy) with that peer can be accomplished. using private networks also means that UDP|TCP/IPv4/v6 aware applications can be hosted on these endpoints and used as private p2p group services. so while this approach requires a heavy endpoint (custom live CD / hdd OS linux image with group key material and base|cached fs distributed securely out of band) the benefits from a security|privacy and flexibility viewpoint are significant. this moves a lot of the complexity out of application space (where it usually resides for p2p) and into the networking / file system layers, where it is more appropriate and can be handled much more robustly. (think access restrictions, network and distributed filesystems, netfilter hooks, security modules / caps, etc rather than trying to reproduce a subset of this poorly in user space applications) this is one of the many reasons i am so fond of the tun/tap based networking software (openvpn, openssh 4.3), IPsec, and virtual machines (Xen, VMware player, etc). From supercanadian at gmail.com Thu Feb 16 06:39:31 2006 From: supercanadian at gmail.com (Charles Iliya Krempeaux) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Tighter HTTP and P2P integration?? In-Reply-To: <84ce626f0602151439w2031da3cl966c0d0a08aefa67@mail.gmail.com> References: <84ce626f0602140035t608263dar1d63f0eecfa92ccb@mail.gmail.com> <43F20046.4040709@limewire.com> <84ce626f0602141549l2d994826g84b01eafb61da5bd@mail.gmail.com> <43F28B42.8070009@limewire.com> <84ce626f0602151439w2031da3cl966c0d0a08aefa67@mail.gmail.com> Message-ID: <84ce626f0602152239t451de6b6of0b7d603a827b510@mail.gmail.com> Hello, On 2/15/06, Charles Iliya Krempeaux wrote: [...] > So right now my thinking is something like the following example. > (Please argue if you disagree.) > > Client: > > GET /some/file HTTP/1.1 > Host: example.com > X-If-No-Alternate: ## PUT STUFF HERE ABOUT WHAT I CAN ACCEPT ## > > > Server: > > HTTP/1.1 204 No Content > Link: ; rel="alternate" > Link: ; rel="alternate"; > Link: ; rel="alternate", class="gnutella" I suppose if the "Link" header is being used to return information, this could be generalized to make this "conditional get" be a way to probe for "Link" data. So, instead of using "X-If-No-Alternate" it could be "X-If-No-Link" (and then specify what relation or other meta data you want). For example, I could get a blogs RSS feed with: X-If-No-Link: rel="alternate"; type="application/rss+xml" Although the exact notation may need to be different since using the "=" sign may not be enough. (For example, "rel" is a space separated list. And "type" is a content type, not a MIME type... and we're really trying to match a MIME type.) Maybe using CSS selector style notation. And in that case, the above example would be: X-If-No-Link: [rel~="alternate"][type="application/rss+xml"] (The "~=" operator is like "element of".) But anways, back to the original example, it might then be: Client: GET /some/file HTTP/1.1 Host: example.com X-If-No-Link: [rel~="alternate"][href^="http:"][class~="gnutella"] X-If-No-Link: [rel~="alternate"][type="application/x-bittorrent"] Server: HTTP/1.1 204 No Content Link: ; rel="alternate"; class="gnutella" Link: ; rel="alternate"; class="gnutella" Link: ; rel="alternate"; Which says, don't give me any content if I can get this over gnutella or bittorrent. (And give me links to those.) (The syntax for "X-If-No-Link" probably needs to be improved though, to make it more inline with the syntax of other HTTP headers.) Comments? Suggestions? [...] See ya -- Charles Iliya Krempeaux, B.Sc. charles @ reptile.ca supercanadian @ gmail.com developer weblog: http://ChangeLog.ca/ ___________________________________________________________________________ Make Television http://maketelevision.com/ From kmagdsick at limewire.com Thu Feb 16 15:41:24 2006 From: kmagdsick at limewire.com (Karl A. Magdsick) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Tighter HTTP and P2P integration?? In-Reply-To: <84ce626f0602152239t451de6b6of0b7d603a827b510@mail.gmail.com> References: <84ce626f0602140035t608263dar1d63f0eecfa92ccb@mail.gmail.com> <43F20046.4040709@limewire.com> <84ce626f0602141549l2d994826g84b01eafb61da5bd@mail.gmail.com> <43F28B42.8070009@limewire.com> <84ce626f0602151439w2031da3cl966c0d0a08aefa67@mail.gmail.com> <84ce626f0602152239t451de6b6of0b7d603a827b510@mail.gmail.com> Message-ID: <43F49D24.60209@limewire.com> Charles Iliya Krempeaux wrote: [snip] >I suppose if the "Link" header is being used to return information, >this could be generalized to make this "conditional get" be a way to >probe for "Link" data. > >So, instead of using "X-If-No-Alternate" it could be "X-If-No-Link" >(and then specify what relation or other meta data you want). > >For example, I could get a blogs RSS feed with: > > X-If-No-Link: rel="alternate"; type="application/rss+xml" > >Although the exact notation may need to be different since using the > > Why add an extra header in this case? It would be less intrusive to the webserver code if the client issues a HEAD request and checks for Link headers, rather than add a completely new X-If-No-Link header. -Karl From supercanadian at gmail.com Thu Feb 16 16:22:04 2006 From: supercanadian at gmail.com (Charles Iliya Krempeaux) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Tighter HTTP and P2P integration?? In-Reply-To: <43F49D24.60209@limewire.com> References: <84ce626f0602140035t608263dar1d63f0eecfa92ccb@mail.gmail.com> <43F20046.4040709@limewire.com> <84ce626f0602141549l2d994826g84b01eafb61da5bd@mail.gmail.com> <43F28B42.8070009@limewire.com> <84ce626f0602151439w2031da3cl966c0d0a08aefa67@mail.gmail.com> <84ce626f0602152239t451de6b6of0b7d603a827b510@mail.gmail.com> <43F49D24.60209@limewire.com> Message-ID: <84ce626f0602160822q24b79acw79d5e09c23b1eb1f@mail.gmail.com> Hello Karl, On 2/16/06, Karl A. Magdsick wrote: > > Charles Iliya Krempeaux wrote: > [snip] > > >I suppose if the "Link" header is being used to return information, > >this could be generalized to make this "conditional get" be a way to > >probe for "Link" data. > > > >So, instead of using "X-If-No-Alternate" it could be "X-If-No-Link" > >(and then specify what relation or other meta data you want). > > > >For example, I could get a blogs RSS feed with: > > > > X-If-No-Link: rel="alternate"; type="application/rss+xml" > > > >Although the exact notation may need to be different since using the > > > > > Why add an extra header in this case? It would be less intrusive to the > webserver > code if the client issues a HEAD request and checks for Link headers, > rather > than add a completely new X-If-No-Link header. > > There's 2 reasons. The first is that it would be for the same reason that we have the other "conditional gets"... speed. The time it takes to "create" and "tear down" a TCP connection is significant. And, from a users point of view, will impact usability, in that it will make things seem slower (if we have to create 2 separate TCP connections for every request we want to do). If you could assume that ever server always had a "persistent connection" (or "keep alive" connection, or whatever you want to call them), then this wouldn't be an issue. But, you can't make that assumption. So that's why "conditional gets" are used. And the second (and probably more important reason) is that I'm making the assumption that the server would NOT always give all these Link headers. (And that some kind of "probing request", like via "X-If-No-Link", is required.) Comments? Suggestions? See ya -- Charles Iliya Krempeaux, B.Sc. charles @ reptile.ca supercanadian @ gmail.com developer weblog: http://ChangeLog.ca/ ___________________________________________________________________________ Make Television http://maketelevision.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: http://zgp.org/pipermail/p2p-hackers/attachments/20060216/9ad58d30/attachment.htm From gbildson at limepeer.com Thu Feb 16 16:56:37 2006 From: gbildson at limepeer.com (Greg Bildson) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Tighter HTTP and P2P integration?? In-Reply-To: <84ce626f0602140035t608263dar1d63f0eecfa92ccb@mail.gmail.com> Message-ID: > -----Original Message----- > From: p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org]On > Behalf Of Charles Iliya Krempeaux > > A "P2P conditional get" (sent by the client) would tell the server NOT > to send the requested document if there is an alternative way of > getting the document via a P2P technology supported by the client. > (The client would tell the server which P2P technologies it supports.) > (This "P2P conditional get" would work similar to the "If-None-Match" > and "If-Modified-Since" "conditional get" headers.). > > For example, it might be something like: > > X-If-No-Alternate: bittorrent, something-else, magnet > > First, this says that the client supports the protocols: "bittorrent", > "something-else", and "magnet". It also says it prefers "bittorrent" > first, then "something-else", and then "magnet". Are these hypothetical options for future web browsers? I take it that there would be no way to make a current browser actually communicate what the computer it's running on supports in this way right now? > > The server might then respond with an "HTTP 302" (Found) and then > redirect the user to a P2P version using a URI for the P2P technology. > This "redirect" would be done via the HTTP "Location" header. For > example: > > HTTP/1.1 302 Found > Location: magnet:?xt=urn:sha1:WRCIRZV5ZO56CWMNHFV4FRGNPWPPDVKT For what it's worth, a Location redirect will trigger magnet handling. We do something similar now with javascript. I wish magnets embedded in emails worked so easily. Thanks -greg From supercanadian at gmail.com Thu Feb 16 19:44:01 2006 From: supercanadian at gmail.com (Charles Iliya Krempeaux) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Tighter HTTP and P2P integration?? In-Reply-To: References: <84ce626f0602140035t608263dar1d63f0eecfa92ccb@mail.gmail.com> Message-ID: <84ce626f0602161144h43b2b13clf42d351ee6389fed@mail.gmail.com> Hello Greg, On 2/16/06, Greg Bildson wrote: > > > > > -----Original Message----- > > From: p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org]On > > Behalf Of Charles Iliya Krempeaux > > > > A "P2P conditional get" (sent by the client) would tell the server NOT > > to send the requested document if there is an alternative way of > > getting the document via a P2P technology supported by the client. > > (The client would tell the server which P2P technologies it supports.) > > (This "P2P conditional get" would work similar to the "If-None-Match" > > and "If-Modified-Since" "conditional get" headers.). > > > > For example, it might be something like: > > > > X-If-No-Alternate: bittorrent, something-else, magnet > > > > First, this says that the client supports the protocols: "bittorrent", > > "something-else", and "magnet". It also says it prefers "bittorrent" > > first, then "something-else", and then "magnet". > > > Are these hypothetical options for future web browsers? I take it that > there would be no way to make a current browser actually communicate what > the computer it's running on supports in this way right now? First let me say that my proposal actually changed a bit. As I discussed things with others on this mailing list (in other messages in this thread/tree of e-mails) I modified my proposal. Right now (in my proposal) the information that I previously had being returned with the "Location" header is now being returned with the "Link" header. (See section 19.6.2.4of RFC 2068 for more information on the "Link" header.) Basically though the "Link" header basically has the same semantics as the HTML element. Given that,... the header I was proposing is now called "X-If-No-Link". So, getting back to you question... Yes, these are ideas for future browsers. My project (that is motivating this work) is being built on the Mozilla application framework (using XULRunner). So I could imagine it very easily getting into Firefox via an extention. (And if people find it useful and it becomes popular I could see it getting into the mainstream web browsers. But as far as my work is concerned it doesn't have to.) Now, as far as current browsers are concerned, if you mandate that you don't want to install anything, then you could do some of this via an XmlHttpRequest (or a GM_xmlHttpRequest is you are using Greasemonkey). You could make a request and set all the "X-If-No-Link" headers that you want. And then when you get the response, you could check for the "Link" headers. See ya -- Charles Iliya Krempeaux, B.Sc. charles @ reptile.ca supercanadian @ gmail.com developer weblog: http://ChangeLog.ca/ ___________________________________________________________________________ Make Television http://maketelevision.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: http://zgp.org/pipermail/p2p-hackers/attachments/20060216/b633c8e8/attachment.html From supercanadian at gmail.com Fri Feb 17 08:09:13 2006 From: supercanadian at gmail.com (Charles Iliya Krempeaux) Date: Sat Dec 9 22:13:10 2006 Subject: [p2p-hackers] Tighter HTTP and P2P integration?? In-Reply-To: <097AC0C1-CFE5-46A4-A49C-42691CA3F69D@locut.us> References: <84ce626f0602140035t608263dar1d63f0eecfa92ccb@mail.gmail.com> <41879D03-2A20-4D3B-80EF-724C4A1B6629@locut.us> <84ce626f0602141638r1550bf0aq8fdd0e0b8114ba7a@mail.gmail.com> <097AC0C1-CFE5-46A4-A49C-42691CA3F69D@locut.us> Message-ID: <84ce626f0602170009m39272f99u5d2c9b98dbadcadb@mail.gmail.com> Hello Ian, Is there a way to get at dijjer's cached files besides going through it's local HTTP server? (I.e., with direct file I/O.) See ya On 2/14/06, Ian Clarke wrote: > > On 14 Feb 2006, at 16:38, Charles Iliya Krempeaux wrote: > > I've been actually considering using a local HTTP server to implement > > things. (Which is something you are doing too.) Although I thought > > some extra "sugar" was warranted, to help with (user) usabiliy and > > developer usability. > > We have added some sugar too. For example, we have javascript > available on our site that can detect whether Dijjer is present on > the user's machine and redirect the user to a download page if it isn't. > > > I skimmed through the stuff on dijje