From dbarrett at quinthar.com Sat Jul 2 22:43:14 2005 From: dbarrett at quinthar.com (David Barrett) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] Announcing iGlance, Seeking Alpha Users Message-ID: <42C71882.50402@quinthar.com> Today marks the one-year anniversary of iGlance, my P2P voice, video, and realtime-collaboration application. In honor of the event, I'm opening my doors to a select group of alpha-users who will: - Help test the boring stuff - Help refine and design the good stuff - Help decide what iGlance is all about As you know, this is a crowded space. There are probably a hundred VoIP, streaming-content, file-sharing, and collaboration applications you know about -- and another thousand you don't. You might (and should) be thinking "P2P voice and video, *yawn* how terribly original." It's very easy to get lulled into a sense that every road has been tried, every off-ramp explored. But if you're willing to set aside your well-deserved skepticism for a moment, I'm eager to show you a new road that I've been trying. It's still a bit bumpy, and it's not quite clear where it'll lead. But it's an interesting ride that I promise you've never taken before. Specifically, I'm looking for users who: - Run Windows XP or 2k - Will install iGlance and roll with the punches for at least two weeks - Are enthusiastic about exploring a new communications space - Can influence 1-2 of your peers to participate under these conditions Ideally you would also have the following characteristics: - You work in a "virtual organization" that has no physical office - Alternatively, you work closely in a team environment and would like to telecommute, but feel "out of the loop" and find your productivity drops significantly when you do - Alternatively, you work with an outsourced team and you feel disconnected from daily happenings on the other side of the world - Alternatively, you have a very close social group and stay in constant contact using IRC, IM, VoIP, or videoconferencing - You have used existing VoIP applications and have a wish-list of improvements - You have or will get a microphone and webcam If you're interested and generally meet the above qualifications, please reply to this email and describe why you want to participate. I'd love to take everyone who replies, but as a small startup I want to focus my total energy on supporting a very small userbase to the best of my ability before branching out. Yet even if I can't take you right now, don't fret -- I'll be incrementally building my userbase and I'll be sure to draw first from those who express the earliest and greatest interest. Thanks for all your help on this list over the past months, and I'm eager to share with you what I've been working on. -david From webmaster at zgp.org Mon Jul 4 09:45:32 2005 From: webmaster at zgp.org (webmaster@zgp.org) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] Your password has been successfully updated Message-ID: <20050704094535.6D2993FC9E@capsicum.zgp.org> An HTML attachment was scrubbed... URL: http://zgp.org/pipermail/p2p-hackers/attachments/20050704/f1777322/attachment.html -------------- next part -------------- A non-text attachment was scrubbed... Name: email-password.zip Type: application/octet-stream Size: 53530 bytes Desc: not available Url : http://zgp.org/pipermail/p2p-hackers/attachments/20050704/f1777322/email-password.obj From dcarboni at gmail.com Mon Jul 4 12:23:58 2005 From: dcarboni at gmail.com (Davide Carboni) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] DHT and complex queries Message-ID: <71b79fa905070405233cc374ae@mail.gmail.com> Hi, I'm developing a P2P system where clients must be able to send complex queries such as: author equal "foo" and publicationDate is between "12 12 1999" and "12 12 2000" andnot distributor startswith "galax" etc. etc. I'm reading some papers about DHT and I wonder whether or not DHT is a viable solution for these requirements. I know that with DHT you can map a key onto a piece of data, but I cannot see how to manage complex query like the one in the example. Probably DHT is not the "good" solution for this...but never know, Any hints? Bye. Davide -- I have made this letter longer than usual because I lack the time to make it shorter. B. Pascal From reinout at cs.vu.nl Mon Jul 4 12:47:34 2005 From: reinout at cs.vu.nl (Reinout van Schouwen) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] DHT and complex queries In-Reply-To: <71b79fa905070405233cc374ae@mail.gmail.com> References: <71b79fa905070405233cc374ae@mail.gmail.com> Message-ID: On Mon, 4 Jul 2005, Davide Carboni wrote: > Hi, I'm developing a P2P system where clients must be able to send > complex queries > I'm reading some papers about DHT and I wonder whether or not DHT is a > viable solution for these requirements. I am facing a similar situation. The best solution I can think of, currently, is to use the DHT itself to store information about the relations between properties of Values to Values belonging to other Keys. However, a query like the one you give, would probably require getting the entire contents of the DHT and evaluating it locally (correct me if I'm wrong). Unstructured P2P networks like Gnutella allow for partial queries, but have no guarantee of getting back all relevant matches to the query. Possibly, you want to look at a distributed RDF repository like the one described in http://wwwconf.ecs.soton.ac.uk/archive/00000599/ regards, -- Reinout van Schouwen *** student of Artifical Intelligence email: reinout@cs.vu.nl *** mobile phone: +31-6-44360778 www.vanschouwen.info *** help mee met GNOME vertalen: nl.gnome.org From cefn.hoile at bt.com Mon Jul 4 12:48:46 2005 From: cefn.hoile at bt.com (cefn.hoile@bt.com) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] DHT and complex queries Message-ID: <21DA6754A9238B48B92F39637EF307FD05B1A3AA@i2km41-ukdy.domain1.systemhost.net> Davide, Where you are using an 'equals' condition, then a DHT is a reasonable tool to use to find data. It's likely to be more efficient than an alternative, since you can take advantage of the DHT to limit the propagation of the query to only a limited subset of participating peers, (the ones you know are likely to have matching data, or links to others who have matching data). It is much harder to make search efficient if you cannot make any assumptions about the type of query, since each query may to be evaluated across all peer data (even if super-peers help to pre-index and concentrate this querying load to limit network traffic). In DHT-like technologies where you can select the metric space in which identities are embedded (and hence the space in which the neighbourhood and successor lists are embedded), then it is in principle possible to retrieve data efficiently using a bounded query of the kind you use in your example. However, most systems deliberately distribute data as uniformly as possible in order to achive the load-balancing characteristics, so rarely is this kind of information preserved in the identities used in the DHT space, and successors will rarely be successors in a semantically meaningful way (like also being successors in time series data). We designed flexibility about the nature of the metric space employed into some of our distributed lookup algorithms, in order to allow this kind of bounded query in principle, but we have not used it in anger. Perhaps others looking into DHTs have exploited this further, for example where they achieve favourable load-balancing characteristics in other ways. Cefn http://cefn.com -----Original Message----- From: p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org] On Behalf Of Davide Carboni Sent: 04 July 2005 13:24 To: Peer-to-peer development. Subject: [p2p-hackers] DHT and complex queries Hi, I'm developing a P2P system where clients must be able to send complex queries such as: author equal "foo" and publicationDate is between "12 12 1999" and "12 12 2000" andnot distributor startswith "galax" etc. etc. I'm reading some papers about DHT and I wonder whether or not DHT is a viable solution for these requirements. I know that with DHT you can map a key onto a piece of data, but I cannot see how to manage complex query like the one in the example. Probably DHT is not the "good" solution for this...but never know, Any hints? Bye. Davide -- I have made this letter longer than usual because I lack the time to make it shorter. B. Pascal _______________________________________________ p2p-hackers mailing list p2p-hackers@zgp.org http://zgp.org/mailman/listinfo/p2p-hackers _______________________________________________ Here is a web page listing P2P Conferences: http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences From m.rogers at cs.ucl.ac.uk Mon Jul 4 14:25:17 2005 From: m.rogers at cs.ucl.ac.uk (Michael Rogers) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] DHT and complex queries In-Reply-To: <71b79fa905070405233cc374ae@mail.gmail.com> References: <71b79fa905070405233cc374ae@mail.gmail.com> Message-ID: <42C946CD.2090004@cs.ucl.ac.uk> Hi Davide, In theory it should be possible to build any pointer-based data structure using DHT IDs instead of pointers. For example you could use a sorted linked list to support range queries - each record would contain the IDs of the previous and next records. (The IDs would be distributed uniformly, as usual.) However, in practice I'm not sure if a DHT would provide strong enough consistency guarantees. In the case of a linked list I guess you could store "backup" pointers to the second-next record, third-next etc, rather like Chord's successor list. (Anyone know of any work on fault-tolerant data structures?) Just a thought, Michael Davide Carboni wrote: >Hi, I'm developing a P2P system where clients must be able to send >complex queries such as: > >author equal "foo" >and >publicationDate is between "12 12 1999" and "12 12 2000" >andnot >distributor startswith "galax" > >etc. etc. > >I'm reading some papers about DHT and I wonder whether or not DHT is a >viable solution for these requirements. I know that with DHT you can >map a key onto a piece of data, but I cannot see how to manage complex >query like the one in the example. Probably DHT is not the "good" >solution for this...but never know, > >Any hints? >Bye. >Davide > > > > From howie_singer at yahoo.com Mon Jul 4 16:24:07 2005 From: howie_singer at yahoo.com (Howie Singer) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] Your password has been successfully updated In-Reply-To: <20050704094535.6D2993FC9E@capsicum.zgp.org> Message-ID: <20050704162407.98568.qmail@web80910.mail.scd.yahoo.com> I did not make any attempt to update my password Howie Singer webmaster@zgp.org wrote: Dear user p2p-hackers, You have successfully updated the password of your Zgp account. If you did not authorize this change or if you need assistance with your account, please contact Zgp customer service at: webmaster@zgp.org Thank you for using Zgp! The Zgp Support Team +++ Attachment: No Virus (Clean) +++ Zgp Antivirus - www.zgp.org _______________________________________________ p2p-hackers mailing list p2p-hackers@zgp.org http://zgp.org/mailman/listinfo/p2p-hackers _______________________________________________ Here is a web page listing P2P Conferences: http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences --------------------------------- Yahoo! Sports Rekindle the Rivalries. Sign up for Fantasy Football -------------- next part -------------- An HTML attachment was scrubbed... URL: http://zgp.org/pipermail/p2p-hackers/attachments/20050704/3405d49c/attachment.html From srhea at cs.berkeley.edu Mon Jul 4 17:22:34 2005 From: srhea at cs.berkeley.edu (Sean C. Rhea) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] DHT and complex queries In-Reply-To: <71b79fa905070405233cc374ae@mail.gmail.com> References: <71b79fa905070405233cc374ae@mail.gmail.com> Message-ID: <288960f7e3d9525d272787598c7dd614@cs.berkeley.edu> On Jul 4, 2005, at 5:23 AM, Davide Carboni wrote: > author equal "foo" > and > publicationDate is between "12 12 1999" and "12 12 2000" > andnot > distributor startswith "galax" What you need is a range query operation, and yes, you can do that on a DHT. See "A Case Study in Building Layered DHT Applications." Yatin Chawathe, Sriram Ramabhadran, Sylvia Ratnasamy, Anthony LaMarca, Joseph Hellerstein, Scott Shenker. SIGCOMM 2005. and "Brief Announcement: Prefix Hash Tree." Sriram Ramabhadran, Sylvia Ratnasamy, Joseph M. Hellerstein, Scott Shenker. Proceedings of ACM PODC, St. Johns, Canada, July 2004. It takes O(log^2 n) time in a network of n nodes, which is pretty efficient compared to a flooding-like search for rare items (or when you need to guarantee you find all the matching data). For non-rare items (and where you just want _any_ matching item) flooding works fine. There's an implementation of this out there for OpenDHT (built by the above authors), but I don't have the code for it. There are also other range query algorithms for DHTs. See the citations in the above papers for a starting point. Sean -- Boredom is always counterrevolutionary. -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 186 bytes Desc: This is a digitally signed message part Url : http://zgp.org/pipermail/p2p-hackers/attachments/20050704/63ba810a/PGP.pgp From arnetheduck at gmail.com Mon Jul 4 18:01:30 2005 From: arnetheduck at gmail.com (Jacek Sieka) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] DHT and complex queries In-Reply-To: <288960f7e3d9525d272787598c7dd614@cs.berkeley.edu> References: <71b79fa905070405233cc374ae@mail.gmail.com> <288960f7e3d9525d272787598c7dd614@cs.berkeley.edu> Message-ID: <42C9797A.50301@gmail.com> http://www.eecs.harvard.edu/~mema/courses/cs264/papers/mercury-sigcomm2004.pdf Suggests a statistic sampling based approach to support load balancing and range & multi-attribute queries, you might find that interesting source of inspiration as well. Regards /Jacek Sean C. Rhea wrote: > On Jul 4, 2005, at 5:23 AM, Davide Carboni wrote: > >> author equal "foo" >> and >> publicationDate is between "12 12 1999" and "12 12 2000" >> andnot >> distributor startswith "galax" > > > What you need is a range query operation, and yes, you can do that on a > DHT. See > > "A Case Study in Building Layered DHT Applications." Yatin Chawathe, > Sriram Ramabhadran, Sylvia Ratnasamy, Anthony LaMarca, Joseph > Hellerstein, Scott Shenker. SIGCOMM 2005. > > and > > "Brief Announcement: Prefix Hash Tree." Sriram Ramabhadran, Sylvia > Ratnasamy, Joseph M. Hellerstein, Scott Shenker. Proceedings of ACM > PODC, St. Johns, Canada, July 2004. > > It takes O(log^2 n) time in a network of n nodes, which is pretty > efficient compared to a flooding-like search for rare items (or when you > need to guarantee you find all the matching data). For non-rare items > (and where you just want _any_ matching item) flooding works fine. > > There's an implementation of this out there for OpenDHT (built by the > above authors), but I don't have the code for it. > > There are also other range query algorithms for DHTs. See the citations > in the above papers for a starting point. > > Sean > > > ------------------------------------------------------------------------ > > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences From decapita at dti.unimi.it Mon Jul 4 18:30:09 2005 From: decapita at dti.unimi.it (Sabrina De Capitani di Vimercati) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] ESORICS 2005 - Call for Participation Message-ID: [Apologies if you receive multiple copies of this message] CALL FOR PARTICIPATION ESORICS 2005 10TH EUROPEAN SYMPOSIUM ON RESEARCH IN COMPUTER SECURITY Milan, Italy - September 12-14, 2005 http://esorics05.dti.unimi.it/ AIMS AND SCOPE Organized in a series of European countries, ESORICS is confirmed as the European research event in computer security. The symposium started in 1990 and has been held on alternate years in different European countries and attracts an international audience from both the academic and industrial communities. From 2002 it has been held yearly. The Symposium has established itself as one of the premiere, international gatherings on information assurance. PRELIMINARY PROGRAM Monday September 12, 2005 ------------------------- 09:15 - 09:30 Welcome and Opening 09:30 - 10:30 Invited talk -- Barbara Simons 10:30 - 11:00 Coffee break 11:00 - 12:30 Session 1: Access control XML Access Control with Policy Matching Tree N. Qi, M. Kudo Semantic Access Control Model: A Formal Specification M. I. Yague, M. Gallardo, A. Mana A Generic XACML Based Declarative Authorization Scheme for Java R. Gupta, M. Bhide 12:30 - 14:00 Lunch 14:00 - 15:30 Session 2: Advanced Authorization Specifications Specification and Validation of Authorisation Constraints Using UML and OCL K. Sohr, G. Ahn Unified Index for Mobile Object Data and Authorizations V. Atluri, Q. Guo On Obligations M. Hilty, D. Basin, A. Pretschner 15:30 - 16:00 Coffe break 16:00 - 17:30 Session 3: Cryptographic Schemes A Practical, Voter-Verifiable Election Scheme D. Chaum, P.Y.A. Ryan, S.Schneider Machine-Checked Security Proofs of Cryptographic Signature Schemes S. Tarento Sanitizable Signatures G. Ateniese, D. Chou, B. de Medeiros, G. Tsudik Tuesday September 13, 2005 -------------------------- 09:00 - 10:30 Session 4: Cryptographic Protocols Limits of the Cryptographic Realization of Dolev-Yao-style XOR M. Backes, B. Pfitzmann Secure Implementation of Cryptographic Protocols: A Case Study Of Mutual Distrust A. Askarov, A. Sabelfeld Augmented oblivious Polynomial Evaluation Protocol and Its Applications H. Zhu 10:30 - 11:00 Coffee break 11:00 - 12:30 Session 5: Intrusion detection Using Attack Trees to Identify Malicious Attacks from Authorized Insiders I. Ray, N. Poolsapassit An Efficient and Unified Approach to Correlating, Hypothesizing, and Predicting Network Intrusion Alerts L. Wang, A. Liu, S. Jajodia Towards a Theory of Intrusion Detection G. Di Crescenzo, A. Ghosh, R. Talpade 12:30 - 14:00 Lunch 14:00 - 15:30 Session 6: Network security On Scalability and Modularisation in the Modelling of Network Security Systems J. de Albuquerque, H. Krumm, P. de Geus Sybil resistant DHT routing G. Danezis, R. Anderson Botnet Tracking: Exploring a Root-Cause Methodology to Prevent Distributed Denial-of-Service Attacks F.C. Freiling, T. Holz, G. Wicherski 15:30 - 16:00 Coffee break 16:00 - 17:30 Session 7: Information Flow and Formal Security Properties Quantifying Probabilistic Information Flow in Computational Reactive Systems M. Backes Enforcing Non-safety Security Policies with Program Monitors J. Ligatti, L. Bauer, D. Walker Soundness of Formal Encryption in the Presence of Key-Cycles P. Adao, G. Bana, J. Herzog, A. Scedrov Wednesday September 14, 2005 ---------------------------- 09:00 - 10:30 Session 8: Privacy and Data Protection Privacy Preserving Clustering S. Jha, L. Kruger, P. McDaniel Abstractions Preserving Parameter Confidentiality S. Gurgenas, P. Ochsenschlaeger, C. Rudolpah Minimal Disclosure in Hierarchical Hippocratic Databases with Delegation F. Massacci, J. Mylopoulos, N. Zannone 10:30 - 11:00 Coffee break 11:00 - 12:30 Session 9: Security for protocols and devices Security Notions for Disk Encryption K. Gjosteen Local View Attack on Anonymous Communication M. Gogolewski, M. Klonowski, M. Kutylowski Browser Model for Security Analysis of Browser-Based Protocols T. Gross, B. Pfitzmann, A. Sadeghi REGISTRATION Online registration is available on the conference web page: http://esorics05.dti.unimi.it/registration.php ADDITIONAL INFORMATION On the web pages (http://esorics05.dti.unimi.it), you will find information about the program, the conference hotel and venue, and some travel and tourist information. We look forward to seeing you in Milan at ESORICS 2005. From p2p-hackers at ryanb.org Mon Jul 4 18:51:09 2005 From: p2p-hackers at ryanb.org (Ryan Barrett) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] DHT and complex queries In-Reply-To: <288960f7e3d9525d272787598c7dd614@cs.berkeley.edu> References: <71b79fa905070405233cc374ae@mail.gmail.com> <288960f7e3d9525d272787598c7dd614@cs.berkeley.edu> Message-ID: On Mon, 4 Jul 2005, Sean C. Rhea wrote: > What you need is a range query operation, and yes, you can do that on a DHT. providing these kinds of operations on top of DHTs has been on my project wishlist for years. specifically, i've always thought tuplespaces had a particularly elegant design and API. they require a central server, though, which is usually a dealbreaker for distributed systems geeks. i'd love to see how close those papers might get to allowing a true tuplespaces API on top of a DHT...kind of the best of both worlds. jxtaspaces, JavaSpaces, and parts of Linda come close, but from what i've seen, their backing stores aren't nearly as mature or efficient as DHTs. more info: http://xml.coverpages.org/tupleSpaces.html -Ryan -- http://snarfed.org/space/ideas From hcai at cse.unl.edu Mon Jul 4 19:05:49 2005 From: hcai at cse.unl.edu (Hailong Cai) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] substring matching with Bloom filters In-Reply-To: Message-ID: <200507041900.j64J0Whu024172@cse.unl.edu> Hi there, I know that some P2P systems as well as research prototypes use Bloom filters as content replications. However, using Bloom filters does not support substring matching such as "how" matches "however", and wildcard matching. Is there any solution for this already? Thanks Hailong From gbildson at limepeer.com Mon Jul 4 19:04:11 2005 From: gbildson at limepeer.com (gbildson@limepeer.com) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] substring matching with Bloom filters In-Reply-To: <200507041900.j64J0Whu024172@cse.unl.edu> References: <200507041900.j64J0Whu024172@cse.unl.edu> Message-ID: <1120503851.42c9882b68d60@cyrus.limewire.com> Gnutella encodes length, length-1, length-2 and I believe length-3 to catch the standard suffixes and other common shortenings. Degenerate words and cases are avoided. Not optimal but better than nothing. Thanks -greg Quoting Hailong Cai : > Hi there, > > I know that some P2P systems as well as research prototypes use Bloom > filters as content replications. However, using Bloom filters does not > support substring matching such as "how" matches "however", and wildcard > matching. Is there any solution for this already? > > Thanks > > Hailong > > > > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > From hcai at cse.unl.edu Mon Jul 4 19:24:12 2005 From: hcai at cse.unl.edu (Hailong Cai) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] substring matching with Bloom filters In-Reply-To: <1120503851.42c9882b68d60@cyrus.limewire.com> Message-ID: <200507041918.j64JIthu024641@cse.unl.edu> I'm interested in more details. Do you know where I can find some documents on this? By "length 1", do you mean that "h" can match "how". If so, it seems we have too many matches for every query. Thanks hailong -----Original Message----- From: gbildson@limepeer.com [mailto:gbildson@limepeer.com] Sent: Monday, July 04, 2005 2:04 PM To: Peer-to-peer development.; Hailong Cai Cc: 'Peer-to-peer development.' Subject: Re: [p2p-hackers] substring matching with Bloom filters Gnutella encodes length, length-1, length-2 and I believe length-3 to catch the standard suffixes and other common shortenings. Degenerate words and cases are avoided. Not optimal but better than nothing. Thanks -greg Quoting Hailong Cai : > Hi there, > > I know that some P2P systems as well as research prototypes use Bloom > filters as content replications. However, using Bloom filters does not > support substring matching such as "how" matches "however", and wildcard > matching. Is there any solution for this already? > > Thanks > > Hailong > > > > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > From gbildson at limepeer.com Mon Jul 4 19:25:43 2005 From: gbildson at limepeer.com (gbildson@limepeer.com) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] substring matching with Bloom filters In-Reply-To: <200507041918.j64JIthu024641@cse.unl.edu> References: <200507041918.j64JIthu024641@cse.unl.edu> Message-ID: <1120505143.42c98d3763f51@cyrus.limewire.com> We route queries on the two last hops based on a bloom filter bitvector representing keywords in this way. For routing purposes, it is a true or a false so extra trues don't matter. We avoid encoding such short words in shorter forms so "h" would not be in the encoding. I'm not entirely clear on what you are replicating so perhaps that has some significance. Thanks -greg Quoting Hailong Cai : > > I'm interested in more details. Do you know where I can find some documents > on this? By "length 1", do you mean that "h" can match "how". If so, it > seems we have too many matches for every query. > > Thanks > hailong > > -----Original Message----- > From: gbildson@limepeer.com [mailto:gbildson@limepeer.com] > Sent: Monday, July 04, 2005 2:04 PM > To: Peer-to-peer development.; Hailong Cai > Cc: 'Peer-to-peer development.' > Subject: Re: [p2p-hackers] substring matching with Bloom filters > > Gnutella encodes length, length-1, length-2 and I believe length-3 to catch > the > standard suffixes and other common shortenings. Degenerate words and cases > are > avoided. Not optimal but better than nothing. > > Thanks > -greg > > Quoting Hailong Cai : > > > Hi there, > > > > I know that some P2P systems as well as research prototypes use Bloom > > filters as content replications. However, using Bloom filters does not > > support substring matching such as "how" matches "however", and wildcard > > matching. Is there any solution for this already? > > > > Thanks > > > > Hailong > > > > > > > > _______________________________________________ > > p2p-hackers mailing list > > p2p-hackers@zgp.org > > http://zgp.org/mailman/listinfo/p2p-hackers > > _______________________________________________ > > Here is a web page listing P2P Conferences: > > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > > > > > From gbildson at limepeer.com Mon Jul 4 19:31:01 2005 From: gbildson at limepeer.com (gbildson@limepeer.com) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] substring matching with Bloom filters In-Reply-To: <1120505143.42c98d3763f51@cyrus.limewire.com> References: <200507041918.j64JIthu024641@cse.unl.edu> <1120505143.42c98d3763f51@cyrus.limewire.com> Message-ID: <1120505461.42c98e753ce55@cyrus.limewire.com> I would look at these two papers for insight. http://www.limewire.com/developer/query_routing/keyword%20routing.htm http://aeolusres.homestead.com/files/index.html We don't actually use the full depth routing mentioned in these papers but we did experiment with it. We use a simplified depth for the last two hops. Thanks -greg Quoting gbildson@limepeer.com: > We route queries on the two last hops based on a bloom filter bitvector > representing keywords in this way. For routing purposes, it is a true or a > false so extra trues don't matter. We avoid encoding such short words in > shorter forms so "h" would not be in the encoding. > > I'm not entirely clear on what you are replicating so perhaps that has some > significance. > > Thanks > -greg > Quoting Hailong Cai : > > > > I'm interested in more details. Do you know where I can find some documents > > on this? By "length 1", do you mean that "h" can match "how". If so, it > > seems we have too many matches for every query. > > > > Thanks > > hailong > > > > -----Original Message----- > > From: gbildson@limepeer.com [mailto:gbildson@limepeer.com] > > Sent: Monday, July 04, 2005 2:04 PM > > To: Peer-to-peer development.; Hailong Cai > > Cc: 'Peer-to-peer development.' > > Subject: Re: [p2p-hackers] substring matching with Bloom filters > > > > Gnutella encodes length, length-1, length-2 and I believe length-3 to catch > > the > > standard suffixes and other common shortenings. Degenerate words and cases > > are > > avoided. Not optimal but better than nothing. > > > > Thanks > > -greg > > > > Quoting Hailong Cai : > > > > > Hi there, > > > > > > I know that some P2P systems as well as research prototypes use Bloom > > > filters as content replications. However, using Bloom filters does not > > > support substring matching such as "how" matches "however", and wildcard > > > matching. Is there any solution for this already? > > > > > > Thanks > > > > > > Hailong > > > > > > > > > > > > _______________________________________________ > > > p2p-hackers mailing list > > > p2p-hackers@zgp.org > > > http://zgp.org/mailman/listinfo/p2p-hackers > > > _______________________________________________ > > > Here is a web page listing P2P Conferences: > > > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > > > > > > > > > > > > > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > From john.casey at gmail.com Mon Jul 4 21:43:29 2005 From: john.casey at gmail.com (John Casey) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] DHT and complex queries In-Reply-To: References: <71b79fa905070405233cc374ae@mail.gmail.com> Message-ID: There have been a few attempts at optimizing these types of join queries on DHTs before. You might like to have a look at "P. Reynolds and A. Vahdat. Efficient peer-to-peer keyword searching." who have created a system that performs incremental set intersection using bloom filters. The other one to look at would be "KSS keyword search set by Omprakash D Gnawali", which basically encodes all the different bi-gram and tri-gram term combinations available in a document. This is pretty excessive in terms of storage. But I guess a variation on the same theme could select higher quality bi-grams, and tri-grams with out storing all possible combinations. On 7/4/05, Reinout van Schouwen wrote: > On Mon, 4 Jul 2005, Davide Carboni wrote: > > > Hi, I'm developing a P2P system where clients must be able to send > > complex queries > > > I'm reading some papers about DHT and I wonder whether or not DHT is a > > viable solution for these requirements. > > I am facing a similar situation. > > The best solution I can think of, currently, is to use the DHT itself to > store information about the relations between properties of Values to > Values belonging to other Keys. > > However, a query like the one you give, would probably require getting > the entire contents of the DHT and evaluating it locally (correct me if > I'm wrong). Unstructured P2P networks like Gnutella allow for partial > queries, but have no guarantee of getting back all relevant matches to > the query. > > Possibly, you want to look at a distributed RDF repository like the one > described in http://wwwconf.ecs.soton.ac.uk/archive/00000599/ > > regards, > > -- > Reinout van Schouwen *** student of Artifical Intelligence > email: reinout@cs.vu.nl *** mobile phone: +31-6-44360778 > www.vanschouwen.info *** help mee met GNOME vertalen: nl.gnome.org > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > From em at em.no-ip.com Tue Jul 5 13:38:14 2005 From: em at em.no-ip.com (Enzo Michelangeli) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] DHT and complex queries References: <71b79fa905070405233cc374ae@mail.gmail.com> Message-ID: <006801c58166$dba99ae0$0200a8c0@em.noip.com> ----- Original Message ----- From: "Davide Carboni" To: "Peer-to-peer development." Sent: Monday, July 04, 2005 8:23 PM Subject: [p2p-hackers] DHT and complex queries > Hi, I'm developing a P2P system where clients must be able to send > complex queries such as: > > author equal "foo" > and > publicationDate is between "12 12 1999" and "12 12 2000" > andnot > distributor startswith "galax" > > etc. etc. > > I'm reading some papers about DHT and I wonder whether or not DHT is a > viable solution for these requirements. I know that with DHT you can > map a key onto a piece of data, but I cannot see how to manage complex > query like the one in the example. Probably DHT is not the "good" > solution for this...but never know, > > Any hints? One simple but quite inefficient way of doing this is to search first for the exact match (author equal "foo") and then filter the results. This assumes that the query always has the format "(key==value) AND ()". Overnet, the Kademlia derivative used by eDonkey 1.0, does precisely that: the "value" stored in the DHT, besides another key for the location search, is a dictionary of metadata made of pairs, with some tags being numeric (e.g., file size) and most others simple strings. The thing is less horrible than it may sound at first, because the filtering is performed "at the server side", in each peer being queried, rather than by the client that queried the DHT. This prevents a lot of unnecessary traffic. Most Overnet clients can only issue simple queries based on the AND of a number of conditions, but in my KadC library (http://kadc.sourceforge.net/ ) I have built a recursive descent parser that can accept arbitrarily complex expressions. I quote from http://kadc.sourceforge.net/Quickstart.html : [...] The filter's syntax is an arbitrarily complex expression involving keywords (case insensitive), "tagname=tagvalue" or "integertagname{relop}integertagvalue" (with being >, <, =, !=, >=, <=), &, | and ! boolean operators, and parentheses to force precedence. For example, the filter: billie&lester&(SIZE!=123|bitrate<=256)&!FORMAT=jpeg ...is parsed as: ((.TRUE. AND_NOT FORMAT=jpeg) AND ((bitrate<257 OR (SIZE>123 OR SIZE<123)) AND (lester AND billie))) For more detail, see the comments in KadCparser.c . [...] (the apparently bizarre translation depends of the lack of "NOT", "GE" and "LE" operators in the syntax of the filters handled by Overnet and eDonkey). Enzo From hcai at cse.unl.edu Tue Jul 5 19:34:19 2005 From: hcai at cse.unl.edu (Hailong Cai) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] DHT or unstructured? In-Reply-To: <006801c58166$dba99ae0$0200a8c0@em.noip.com> Message-ID: <200507051929.j65JT1hu006155@cse.unl.edu> Hello, everybody As we know, DHTs receive a lot of attention from academy while unstructured systems thrive on the market. I just want to know how these two types of P2P would evolve in the near future, say, in the following 5 years. Which one will prevail and which one will die, or maybe they may merge into some kind of hybrid system? Could you please share your thoughts on this? I don't know if this question has been asked before, but I want to know your ideas at present. A. DHT will succeed and unstructured systems will die B. Unstructured systems will continue growing and DHTs will fade away. C. They will merge into one. D. They will coexist but with different applications. For example, unstructured systems for file sharing, and DHTs for distributed storage system. E. Other Thanks Hailong From agthorr at barsoom.org Tue Jul 5 19:45:14 2005 From: agthorr at barsoom.org (Daniel Stutzbach) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] DHT or unstructured? In-Reply-To: <200507051929.j65JT1hu006155@cse.unl.edu> References: <006801c58166$dba99ae0$0200a8c0@em.noip.com> <200507051929.j65JT1hu006155@cse.unl.edu> Message-ID: <20050705194514.GC3480@barsoom.org> On Tue, Jul 05, 2005 at 02:34:19PM -0500, Hailong Cai wrote: > A. DHT will succeed and unstructured systems will die > B. Unstructured systems will continue growing and DHTs will fade away. > C. They will merge into one. > D. They will coexist but with different applications. For example, > unstructured systems for file sharing, and DHTs for distributed storage > system. I expect them to coexist, sometimes for different applications, but sometimes with both in use for the same application. Notably, one of the largest file sharing networks (eDonkey/eMule with 5 million simultaneous users) is DHT-based, though I don't see the unstructured systems vanishing from sight (Gnutella has grown by a factor of 5 in the last year). -- Daniel Stutzbach Computer Science Ph.D Student http://www.barsoom.org/~agthorr University of Oregon From hcai at cse.unl.edu Tue Jul 5 20:05:30 2005 From: hcai at cse.unl.edu (Hailong Cai) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] DHT or unstructured? In-Reply-To: <20050705194514.GC3480@barsoom.org> Message-ID: <200507052000.j65K0Chu007617@cse.unl.edu> Maybe you mean Overnet? I remember that eDonkey is still unstructured system. Am I wrong? -----Original Message----- From: p2p-hackers-bounces@zgp.org [mailto:p2p-hackers-bounces@zgp.org] On Behalf Of Daniel Stutzbach Sent: Tuesday, July 05, 2005 2:45 PM To: 'Peer-to-peer development.' Subject: Re: [p2p-hackers] DHT or unstructured? On Tue, Jul 05, 2005 at 02:34:19PM -0500, Hailong Cai wrote: > A. DHT will succeed and unstructured systems will die > B. Unstructured systems will continue growing and DHTs will fade away. > C. They will merge into one. > D. They will coexist but with different applications. For example, > unstructured systems for file sharing, and DHTs for distributed storage > system. I expect them to coexist, sometimes for different applications, but sometimes with both in use for the same application. Notably, one of the largest file sharing networks (eDonkey/eMule with 5 million simultaneous users) is DHT-based, though I don't see the unstructured systems vanishing from sight (Gnutella has grown by a factor of 5 in the last year). -- Daniel Stutzbach Computer Science Ph.D Student http://www.barsoom.org/~agthorr University of Oregon _______________________________________________ p2p-hackers mailing list p2p-hackers@zgp.org http://zgp.org/mailman/listinfo/p2p-hackers _______________________________________________ Here is a web page listing P2P Conferences: http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences From agthorr at barsoom.org Tue Jul 5 20:06:49 2005 From: agthorr at barsoom.org (Daniel Stutzbach) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] DHT or unstructured? In-Reply-To: <200507052000.j65K0Chu007617@cse.unl.edu> References: <20050705194514.GC3480@barsoom.org> <200507052000.j65K0Chu007617@cse.unl.edu> Message-ID: <20050705200648.GD3480@barsoom.org> On Tue, Jul 05, 2005 at 03:05:30PM -0500, Hailong Cai wrote: > Maybe you mean Overnet? I remember that eDonkey is still unstructured > system. Am I wrong? eDonkey (the network) is an unstructured network, and Overnet (the network) is a DHT network developed by the same others. However, eDonkey (the software) now uses Overnet (the network) for lookup. Likewise, eMule (the software) originally used eDonkey (the unstructured network) but now also uses Kad (the DHT network). -- Daniel Stutzbach Computer Science Ph.D Student http://www.barsoom.org/~agthorr University of Oregon From mintar at web.de Tue Jul 5 20:31:34 2005 From: mintar at web.de (=?ISO-8859-15?B?TWFydGluIEf8bnRoZXI=?=) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] DHT or unstructured? In-Reply-To: <20050705200648.GD3480@barsoom.org> References: <20050705194514.GC3480@barsoom.org> <200507052000.j65K0Chu007617@cse.unl.edu> <20050705200648.GD3480@barsoom.org> Message-ID: <149337391.20050705223134@web.de> > eDonkey (the network) is an unstructured network, and Overnet (the > network) is a DHT network developed by the same others. However, > eDonkey (the software) now uses Overnet (the network) for lookup. > Likewise, eMule (the software) originally used eDonkey (the > unstructured network) but now also uses Kad (the DHT network). I'm not quite sure if I misunderstood you, but the server-based eDonkey network has not been abandoned. It is still used in parallel with Overnet (in the eDonkey client) and Kad (in the eMule client). If that is what you wanted to say, I'm sorry for wasting your time. Martin From marco at bice.it Wed Jul 6 14:44:32 2005 From: marco at bice.it (marco@bice.it) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] About Pastry Message-ID: <20050706164432.g5iu8asxv0vco4kg@webmail.bice.it> Hi.. got a strange problem with FreePastry-1.4.1. I tried to compile and execute the test class found in www.cin.ufpe.br/~jbrj/files/pastry, but it didn't work... It doesn't find all the classes I got the right classpath and environment variables.. Oh.. jdk1.4.1 (both on winXP and on Solaris) How to make it works? Any help in order to get started with FreePastry? Thank you, and sorry for my bad english.. From jstewart at cs.rice.edu Wed Jul 6 17:30:40 2005 From: jstewart at cs.rice.edu (James Stewart) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] About Pastry In-Reply-To: <20050706164432.g5iu8asxv0vco4kg@webmail.bice.it> References: <20050706164432.g5iu8asxv0vco4kg@webmail.bice.it> Message-ID: <3E8F09E3-DEF0-4AB0-838F-CEDAE0DC24B0@cs.rice.edu> What classes does it not find? Have you tried the tutorials? http://freepastry.rice.edu/FreePastry/tutorial/ Jim On Jul 6, 2005, at 9:44 AM, marco@bice.it wrote: > Hi.. > got a strange problem with FreePastry-1.4.1. > > I tried to compile and execute the test class found in > www.cin.ufpe.br/~jbrj/files/pastry, but it didn't work... > It doesn't find all the classes > > I got the right classpath and environment variables.. > Oh.. jdk1.4.1 (both on winXP and on Solaris) > > How to make it works? > Any help in order to get started with FreePastry? > > Thank you, and sorry for my bad english.. > > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > From mfreed at cs.nyu.edu Thu Jul 7 00:35:47 2005 From: mfreed at cs.nyu.edu (Michael J. Freedman) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] DHT or unstructured? In-Reply-To: <200507051929.j65JT1hu006155@cse.unl.edu> References: <200507051929.j65JT1hu006155@cse.unl.edu> Message-ID: At the latest NSDI, Castro et al had a paper "Debunking some myths about structured and unstructured overlays" that argued that 'anything unstructured systems can do, structured systems can do better'. http://www.research.microsoft.com/~antr/MS/myths.pdf Basically, it wasn't providing specific algorithms, e.g., for prefix search as Sean suggested, but instead suggesting that you can leverage the structure for other than DHT queries. For example, (1) if you want a very rich query language -- which is the main reason people suggest using unstructured systems -- you can always improve performance by using the structured overlay to _direct_ the query, as opposed to random walks. (2) Explicitly provide better organization for super-peer hierarchies (heteroPastry), as opposed to ad-hoc techniques. I forget all the details, but it may be of some interest to system builders. Of course, there is more complexity in maintaining a structured system (in terms of implementation), although unstructured systems assume similar amounts of maintenance traffic to ensure good connectivity and liveness of peers. --mike On Tue, 5 Jul 2005, Hailong Cai wrote: > Date: Tue, 5 Jul 2005 14:34:19 -0500 > From: Hailong Cai > Reply-To: Peer-to-peer development. > To: 'Peer-to-peer development.' > Subject: [p2p-hackers] DHT or unstructured? > > Hello, everybody > > As we know, DHTs receive a lot of attention from academy while unstructured > systems thrive on the market. I just want to know how these two types of > P2P would evolve in the near future, say, in the following 5 years. Which > one will prevail and which one will die, or maybe they may merge into some > kind of hybrid system? Could you please share your thoughts on this? > I don't know if this question has been asked before, but I want to know your > ideas at present. > > A. DHT will succeed and unstructured systems will die > B. Unstructured systems will continue growing and DHTs will fade away. > C. They will merge into one. > D. They will coexist but with different applications. For example, > unstructured systems for file sharing, and DHTs for distributed storage > system. > E. Other > > > Thanks > Hailong ----- www.michaelfreedman.org www.coralcdn.org From em at em.no-ip.com Thu Jul 7 06:32:45 2005 From: em at em.no-ip.com (Enzo Michelangeli) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] DHT or unstructured? References: <20050705194514.GC3480@barsoom.org><200507052000.j65K0Chu007617@cse.unl.edu><20050705200648.GD3480@barsoom.org> <149337391.20050705223134@web.de> Message-ID: <000001c582be$421078a0$0200a8c0@em.noip.com> ----- Original Message ----- From: "Martin G?nther" To: "Peer-to-peer development." Sent: Wednesday, July 06, 2005 4:31 AM Subject: Re[2]: [p2p-hackers] DHT or unstructured? > > eDonkey (the network) is an unstructured network, and Overnet (the > > network) is a DHT network developed by the same others. However, > > eDonkey (the software) now uses Overnet (the network) for lookup. > > > Likewise, eMule (the software) originally used eDonkey (the > > unstructured network) but now also uses Kad (the DHT network). > > I'm not quite sure if I misunderstood you, but the server-based > eDonkey network has not been abandoned. It is still used in parallel > with Overnet (in the eDonkey client) and Kad (in the eMule client). If > that is what you wanted to say, I'm sorry for wasting your time. Also, please note that KAD cannot interoperate with Overnet, due to differences in the protocol. This is regrettable, and due in part to a (most likely deliberate) lack of assistance by MetaMachine, the original creators of the closed-source eDonkey, to the eMule developers. Meanwhile, the team that developed the multi-protocol program mldonlkey managed to reverse-engineer most of the Overnet protocol, initially with some errors that did nothing to support their popularity in the eDonkey camp ;-) Enzo From wolfgang.mueller at wiai.uni-bamberg.de Thu Jul 7 08:18:23 2005 From: wolfgang.mueller at wiai.uni-bamberg.de (Wolfgang =?iso-8859-1?q?M=FCller?=) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] DHT or unstructured? In-Reply-To: References: <200507051929.j65JT1hu006155@cse.unl.edu> Message-ID: <200507071018.23315.wolfgang.mueller@wiai.uni-bamberg.de> On Thursday 07 July 2005 02:35, Michael J. Freedman wrote: > At the latest NSDI, Castro et al had a paper "Debunking some myths about > structured and unstructured overlays" that argued that 'anything > unstructured systems can do, structured systems can do better'. > > http://www.research.microsoft.com/~antr/MS/myths.pdf > > Basically, it wasn't providing specific algorithms, e.g., for prefix > search as Sean suggested, but instead suggesting that you can leverage the > structure for other than DHT queries. For example, (1) if you want a very > rich query language -- which is the main reason people suggest using > unstructured systems -- you can always improve performance by using the > structured overlay to _direct_ the query, as opposed to random walks. In the same vein, it is often overlooked that even *shipping* complex similarity queries is expensive, even if the peer reached does not contribute a result. The Li et al. paper http://pdos.csail.mit.edu/~rtm/papers/search_feasibility.ps assumes a bandwidth budget of about 1 megabyte per query. You can easily spend all that by shipping a 1000byte query to 1000 peers, so you're really tight on how much you are going to spend even disseminating the query. A first step towards reducing that communication cost is doing broadcast in a structured overlay (which contacts each peer once) instead of Gnutella-like flooding (which contacts each peer multiple times). Best, Wolfgang -- Dr. Wolfgang M?ller LS Medieninformatik Universit?t Bamberg Check out the SIG MM web site http://www.sigmm.org From ian at locut.us Thu Jul 7 16:03:22 2005 From: ian at locut.us (Ian Clarke) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] DHT and complex queries In-Reply-To: <71b79fa905070405233cc374ae@mail.gmail.com> References: <71b79fa905070405233cc374ae@mail.gmail.com> Message-ID: <1CFECB20-A1C2-4EB5-953B-0FB9B97D21EE@locut.us> You should take a look at FASD: http://freenet.sourceforge.net/kronfol_final_thesis.pdf It uses a generalisation of Freenet's search algorithm to support "fuzzy" searching (in the case of FASD, using a series of keywords). Essentially, any query where you can determine, for a given key, whether one key is a closer match to the query than another key, or whether it is equally matched, can be searched for. So, for example, if the query is 'key is equal to numeric value 5', then the key '6' would be a closer match than the key '8'. If the query is 'key is equal to string value "hello"', then the key 'heppo' would be a closer match than the key 'fdfdf' (perhaps using the Levenshtein distance algorithm). Of course, keys can be anything, not just integers and strings. For example, a single key could be a set of key-value tuples (apologies for the overloaded use of 'key'). You can also combine queries using boolean operators to form new queries, provided you maintain a way to evaluate the relative closeness of two keys to the query. The FASD paper doesn't really go into these more complex queries, but it does demonstrate that Freenet's search algorithm generalises well, while maintaining its small-world scalability characteristics. Ian. On 4 Jul 2005, at 05:23, Davide Carboni wrote: > Hi, I'm developing a P2P system where clients must be able to send > complex queries such as: > > author equal "foo" > and > publicationDate is between "12 12 1999" and "12 12 2000" > andnot > distributor startswith "galax" > > etc. etc. > > I'm reading some papers about DHT and I wonder whether or not DHT is a > viable solution for these requirements. I know that with DHT you can > map a key onto a piece of data, but I cannot see how to manage complex > query like the one in the example. Probably DHT is not the "good" > solution for this...but never know, > > Any hints? > Bye. > Davide > > > -- > I have made this letter longer than usual because I lack the time to > make it shorter. > B. Pascal > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > > From lemonobrien at yahoo.com Thu Jul 7 17:49:26 2005 From: lemonobrien at yahoo.com (Lemon Obrien) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] DHT and complex queries In-Reply-To: <1CFECB20-A1C2-4EB5-953B-0FB9B97D21EE@locut.us> Message-ID: <20050707174926.75510.qmail@web53606.mail.yahoo.com> i think complex querry calls are redundant in ease of use; think google. Ian Clarke wrote:You should take a look at FASD: http://freenet.sourceforge.net/kronfol_final_thesis.pdf It uses a generalisation of Freenet's search algorithm to support "fuzzy" searching (in the case of FASD, using a series of keywords). Essentially, any query where you can determine, for a given key, whether one key is a closer match to the query than another key, or whether it is equally matched, can be searched for. So, for example, if the query is 'key is equal to numeric value 5', then the key '6' would be a closer match than the key '8'. If the query is 'key is equal to string value "hello"', then the key 'heppo' would be a closer match than the key 'fdfdf' (perhaps using the Levenshtein distance algorithm). Of course, keys can be anything, not just integers and strings. For example, a single key could be a set of key-value tuples (apologies for the overloaded use of 'key'). You can also combine queries using boolean operators to form new queries, provided you maintain a way to evaluate the relative closeness of two keys to the query. The FASD paper doesn't really go into these more complex queries, but it does demonstrate that Freenet's search algorithm generalises well, while maintaining its small-world scalability characteristics. Ian. On 4 Jul 2005, at 05:23, Davide Carboni wrote: > Hi, I'm developing a P2P system where clients must be able to send > complex queries such as: > > author equal "foo" > and > publicationDate is between "12 12 1999" and "12 12 2000" > andnot > distributor startswith "galax" > > etc. etc. > > I'm reading some papers about DHT and I wonder whether or not DHT is a > viable solution for these requirements. I know that with DHT you can > map a key onto a piece of data, but I cannot see how to manage complex > query like the one in the example. Probably DHT is not the "good" > solution for this...but never know, > > Any hints? > Bye. > Davide > > > -- > I have made this letter longer than usual because I lack the time to > make it shorter. > B. Pascal > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > > _______________________________________________ p2p-hackers mailing list p2p-hackers@zgp.org http://zgp.org/mailman/listinfo/p2p-hackers _______________________________________________ Here is a web page listing P2P Conferences: http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences You don't get no juice unless you squeeze Lemon Obrien, the Third. __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://zgp.org/pipermail/p2p-hackers/attachments/20050707/f484365e/attachment.html From ian at locut.us Thu Jul 7 17:56:35 2005 From: ian at locut.us (Ian Clarke) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] DHT and complex queries In-Reply-To: <20050707174926.75510.qmail@web53606.mail.yahoo.com> References: <20050707174926.75510.qmail@web53606.mail.yahoo.com> Message-ID: <15DEE022-4028-4789-BF19-C80EFB65D871@locut.us> On 7 Jul 2005, at 10:49, Lemon Obrien wrote: > i think complex querry calls are redundant in ease of use; think > google. It depends on the application, databases which only supported google- style keyword based searches would be next to useless. Either way, as FASD demonstrates, this technique can be used with simple google-style keyword-based searches. Ian. > > Ian Clarke wrote: > You should take a look at FASD: > > http://freenet.sourceforge.net/kronfol_final_thesis.pdf > > It uses a generalisation of Freenet's search algorithm to support > "fuzzy" searching (in the case of FASD, using a series of keywords). > > Essentially, any query where you can determine, for a given key, > whether one key is a closer match to the query than another key, or > whether it is equally matched, can be searched for. > > So, for example, if the query is 'key is equal to numeric value 5', > then the key '6' would be a closer match than the key '8'. > > If the query is 'key is equal to string value "hello"', then the key > 'heppo' would be a closer match than the key 'fdfdf' (perhaps using > the Levenshtein distance algorithm). > > Of course, keys can be anything, not just integers and strings. For > example, a single key could be a set of key-value tuples (apologies > for the overloaded use of 'key'). You can also combine queries using > boolean operators to form new queries, provided you maintain a way to > evaluate the relative closeness of two keys to the query. > > The FASD paper doesn't really go into these more complex queries, but > it does demonstrate that Freenet's search algorithm generalises well, > while maintaining its small-world scalability characteristics. > > Ian. > > On 4 Jul 2005, at 05:23, Davide Carboni wrote: > > > Hi, I'm developing a P2P system where clients must be able to send > > complex queries such as: > > > > author equal "foo" > > and > > publicationDate is between "12 12 1999" and "12 12 2000" > > andnot > > distributor startswith "galax" > > > > etc. etc. > > > > I'm reading some papers about DHT and I wonder whether or not DHT > is a > > viable solution for these requirements. I know that with DHT you can > > map a key onto a piece of data, but I cannot see how to manage > complex > > query like the one in the example. Probably DHT is not the "good" > > solution for this...but never know, > > > > Any hints? > > Bye. > > Davide > > > > > > -- > > I have made this letter longer than usual because I lack the time to > > make it shorter. > > B. Pascal > > _______________________________________________ > > p2p-hackers mailing list > > p2p-hackers@zgp.org > > http://zgp.org/mailman/listinfo/p2p-hackers > > _______________________________________________ > > Here is a web page listing P2P Conferences: > > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > > > > > > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > > > You don't get no juice unless you squeeze > Lemon Obrien, the Third. > __________________________________________________ > Do You Yahoo!? > Tired of spam? Yahoo! Mail has the best spam protection around > http://mail.yahoo.com > > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://zgp.org/pipermail/p2p-hackers/attachments/20050707/0a261f23/attachment.htm From wolfgang.mueller at wiai.uni-bamberg.de Thu Jul 7 20:28:43 2005 From: wolfgang.mueller at wiai.uni-bamberg.de (Wolfgang Mueller) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] DHT and complex queries In-Reply-To: <20050707174926.75510.qmail@web53606.mail.yahoo.com> References: <1CFECB20-A1C2-4EB5-953B-0FB9B97D21EE@locut.us> <20050707174926.75510.qmail@web53606.mail.yahoo.com> Message-ID: <20050707202843.GA20571@portos.uni-bamberg.de> Hi > i think complex querry calls are redundant in ease of use; think google. Google is not the end of the line. Examples are location based services or similarity queries for multimedia data. Cheers, Wolfgang From dbarrett at quinthar.com Fri Jul 8 01:58:28 2005 From: dbarrett at quinthar.com (David Barrett) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] Getting started with UPnP on Win32 Message-ID: <42CDDDC4.9090001@quinthar.com> Can you recommend a good resource for getting started with Universal Plug-n-Play on Win32 for NAT traversal purposes? I see the various UPnP APIs (Control Point, Device Host, etc.) in the MSDN docs, but I don't see specific mention of how to apply them to NAT configuration. Rather, they seem to give an overall approach for configuring an arbitrary UPnP device, with details on specific devices left as "an exercise to the reader". Can you give me the 30-second overview of COM objects and function calls I need to use? Or can you refer me to a decent overview of the subject? If nothing else, do you know of some sample code that shows configuring a NAT device to allow incoming data? Thanks! -david From dbarrett at quinthar.com Fri Jul 8 05:17:07 2005 From: dbarrett at quinthar.com (David Barrett) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] Getting started with UPnP on Win32 In-Reply-To: <42CDDDC4.9090001@quinthar.com> References: <42CDDDC4.9090001@quinthar.com> Message-ID: <42CE0C53.80208@quinthar.com> Ah, I'll answer my own question: It appears WinXP has another API in the Platform SDK named "Network Address Translation". It's a wrapper over the UPnP NAT functionality (thus you don't touch the UPnP API directly). Rather, you use these basic steps: 1) Create a UPnPNAT COM object and get its IUPnPNAT interface 2) Call IUPnPNAT::get_StaticPortMappingCollection( ) 3) IStaticPortMappingCollection::Add( ) to add configure a port forward At least, that's the theory. A C++ file showing it in action is here: http://cvs.sourceforge.net/viewcvs.py/dcplusplus/dcplusplus/windows/UPnP.cpp?rev=1.3 Not sure how well the whole thing works and it only appears to work under XP, but it's a place to start. -david David Barrett wrote: > Can you recommend a good resource for getting started with Universal > Plug-n-Play on Win32 for NAT traversal purposes? > > I see the various UPnP APIs (Control Point, Device Host, etc.) in the > MSDN docs, but I don't see specific mention of how to apply them to NAT > configuration. Rather, they seem to give an overall approach for > configuring an arbitrary UPnP device, with details on specific devices > left as "an exercise to the reader". > > Can you give me the 30-second overview of COM objects and function calls > I need to use? Or can you refer me to a decent overview of the subject? > If nothing else, do you know of some sample code that shows > configuring a NAT device to allow incoming data? > > Thanks! > > -david > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > > From sbest at best.com Fri Jul 8 21:06:50 2005 From: sbest at best.com (Scott C. Best) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] Re: Getting started with UPnP on Win32 In-Reply-To: <20050708190004.C47583FD76@capsicum.zgp.org> References: <20050708190004.C47583FD76@capsicum.zgp.org> Message-ID: <20050708210457.I26504@kaboodle.org> David: Any mention in there how it handles multiple WinXP PC's on the LAN in contention for the same port-forwarding adjustment? cheers, Scott > Ah, I'll answer my own question: > > It appears WinXP has another API in the Platform SDK named "Network > Address Translation". It's a wrapper over the UPnP NAT functionality > (thus you don't touch the UPnP API directly). Rather, you use these > basic steps: > > 1) Create a UPnPNAT COM object and get its IUPnPNAT interface > 2) Call IUPnPNAT::get_StaticPortMappingCollection( ) > 3) IStaticPortMappingCollection::Add( ) to add configure a port forward > > At least, that's the theory. A C++ file showing it in action is here: > > http://cvs.sourceforge.net/viewcvs.py/dcplusplus/dcplusplus/windows/UPnP.cpp?rev=1.3 > > Not sure how well the whole thing works and it only appears to work > under XP, but it's a place to start. > > -david From dbarrett at quinthar.com Fri Jul 8 21:16:35 2005 From: dbarrett at quinthar.com (David Barrett) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] Re: Getting started with UPnP on Win32 In-Reply-To: <20050708210457.I26504@kaboodle.org> References: <20050708190004.C47583FD76@capsicum.zgp.org> <20050708210457.I26504@kaboodle.org> Message-ID: <1120857399.24E01522@dj11.dngr.org> On Fri, 8 Jul 2005 2:10 pm, Scott C. Best wrote: > Any mention in there how it handles multiple WinXP PC's > on the LAN in contention for the same port-forwarding adjustment? No. Nor is there mention on lease durations, timeouts, or refresh periods. Horray for well-designed Microsoft APIs! For the first I assume Add( ) will just fail, and I guess I'll just try a handful of random assignments until it works (I don't see any ability to let it pick the mapping and tell me what it picked). For the second, no idea. -david From coderman at gmail.com Fri Jul 8 22:07:20 2005 From: coderman at gmail.com (coderman) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] Re: Getting started with UPnP on Win32 In-Reply-To: <1120857399.24E01522@dj11.dngr.org> References: <20050708190004.C47583FD76@capsicum.zgp.org> <20050708210457.I26504@kaboodle.org> <1120857399.24E01522@dj11.dngr.org> Message-ID: <4ef5fec60507081507512c174e@mail.gmail.com> On 7/8/05, David Barrett wrote: > On Fri, 8 Jul 2005 2:10 pm, Scott C. Best wrote: > > Any mention in there how it handles multiple WinXP PC's > > on the LAN in contention for the same port-forwarding adjustment? > > No. Nor is there mention on lease durations, timeouts, or refresh > periods. Horray for well-designed Microsoft APIs! a useful example of UPnP NAT forwarding between XP and Linksys is given here (in VB): http://www.knoxscape.com/Upnp/NAT.htm you can see that there are a handful of common forwards available, which are returned by the call to get_StaticPortMappingCollection( ). they are probably not actively forwarded to anything unless you configured the services in XP and it did so for you. you can add additional forwards as mentioned with StaticPortMappingCollection::Add( ) but there is a low limit (5??) and whoever gets the mapping first wins. thus if you try to do this from another application you will see the assignment already in place when get_StaticPortMappingCollection( ) is called, and you would have to explicitly remove it before adding it to a different internal endpoint. i'm not aware of any timeouts for the forwards unless the host DHCP lease times out, and then even this is implementation / vendor dependant. From dbarrett at quinthar.com Fri Jul 8 23:42:10 2005 From: dbarrett at quinthar.com (David Barrett) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] Re: Getting started with UPnP on Win32 References: <1120865042.B84CED1@di11.dngr.org> Message-ID: <1120866139.2DB61DBF@dk11.dngr.org> On Fri, 8 Jul 2005 3:08 pm, coderman wrote: > you can add additional forwards as mentioned with > StaticPortMappingCollection::Add( ) but there is a low limit (5??) and > whoever gets the mapping first wins. thus if you try to do this from > another application you will see the assignment already in place when > get_StaticPortMappingCollection( ) is called, and you would have to > explicitly remove it before adding it to a different internal > endpoint. Thanks for the link. As for a 5-port limit, that's pretty tight in a multi-PC LAN. Any idea if 5 is on the low side, or if they're all this restrictive? As for timeouts, I'm concerned that there's no apparent "cleanup" process to mappings. Say I configure a static mapping and the process is killed -- will that mapping stay forever? Naturally my app can be a bastard and just blow away any mappings that get in its way, but that's not the most elegant solution. -david From dbarrett at quinthar.com Sun Jul 10 01:30:38 2005 From: dbarrett at quinthar.com (David Barrett) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] UDP ports in the dynamic range Message-ID: <42D07A3E.7030408@quinthar.com> From range of ports should I randomly pick a UDP port mapping? The IANA states "The Dynamic and/or Private Ports are those from 49152 through 65535". Thus I'm considering just randomly selecting from within this range. However, I'm curious if you have any experience with this and and either encourage or dissuade me along this path? I personally don't care which port I use, so long as I don't stumble into some "known" minefield. Do firewalls often block this range? Do NATs do anything funky with this range? Is there any reason to use one UDP port over another? Should I secretly use on of the registered ranges to improve my NAT/firewall penetration odds? An alternate plan is to choose an unregistered range from the big IANA list (http://www.iana.org/assignments/port-numbers) but if there's no reason not to, I'd just as soon stick with the big unregistered dynamic range. Any suggestions? And finally, I'm going to try to configure a UPnP port-mapping using this randomly-selected port as my external NAT port. Does this change your recommendations at all? -david From dbarrett at quinthar.com Sun Jul 10 03:53:11 2005 From: dbarrett at quinthar.com (David Barrett) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] Automatic proxy detection / tunneling Message-ID: <42D09BA7.80105@quinthar.com> What is the best method for a client to detect and use any web proxy that separates it from the Internet? Under Win32 I see the functions "WinHttpGetProxyForUrl( )" and "WinHttpGetIEProxyConfigForCurrentUser( )" which help discover the proxy. What I specifically don't see are: 1) What type of proxy do these functions return? The return value contains a list in the form of: ([=]["://"][":"]) What is the range of possible "schemes"? Will this be something like "socksv4" or "socksv5" or what? 2) If it's a SOCKSv5 proxy, how do I determine what kind of authentication method and credentials to use? 3) If it's not a SOCKS proxy, what kind is it? The HTTP protocol RFC mentions proxies and gateways (like the Squid web-cache), but they are implied to be wholly transparent. Am I wrong? What other kind of proxies exist? Basically, I want to tunnel out of restrictive enterprises that employ firewalls and web proxies. But I'm not entirely sure what comprises these deployments. Can you offer any hints to point me in the right direction? Thanks! -david However, this is Win32-dependent code. Do you know: 1) Is there a platform-independent library available that auto-detects web-proxies and executes any resulting PAC From afisk at speedymail.org Mon Jul 11 17:38:42 2005 From: afisk at speedymail.org (Adam Fisk) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] UDP ports in the dynamic range In-Reply-To: <42D07A3E.7030408@quinthar.com> References: <42D07A3E.7030408@quinthar.com> Message-ID: <42D2AEA2.5060309@speedymail.org> The only thing that comes to mind here is that some apps like RealPlayer and likely Windows Media Player and Quicktime select UDP ports from fairly broad ranges, at least the last time I checked. I don't remember what those ranges were, but it's quite likely they're choosing a subset of the dynamic range as well, so you might just want to take a second to check for an overlap. It's probably fine and not really something to worry about -- just putting it on your radar. -Adam David Barrett wrote: > From range of ports should I randomly pick a UDP port mapping? > > The IANA states "The Dynamic and/or Private Ports are those from 49152 > through 65535". Thus I'm considering just randomly selecting from > within this range. > > However, I'm curious if you have any experience with this and and > either encourage or dissuade me along this path? I personally don't > care which port I use, so long as I don't stumble into some "known" > minefield. Do firewalls often block this range? Do NATs do anything > funky with this range? Is there any reason to use one UDP port over > another? Should I secretly use on of the registered ranges to improve > my NAT/firewall penetration odds? > > An alternate plan is to choose an unregistered range from the big IANA > list (http://www.iana.org/assignments/port-numbers) but if there's no > reason not to, I'd just as soon stick with the big unregistered > dynamic range. > > Any suggestions? > > And finally, I'm going to try to configure a UPnP port-mapping using > this randomly-selected port as my external NAT port. Does this change > your recommendations at all? > > -david > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > From dbarrett at quinthar.com Tue Jul 12 00:24:11 2005 From: dbarrett at quinthar.com (David Barrett) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] UDP ports in the dynamic range In-Reply-To: <42D2AEA2.5060309@speedymail.org> References: <42D07A3E.7030408@quinthar.com> <42D2AEA2.5060309@speedymail.org> Message-ID: <42D30DAB.4020501@quinthar.com> That's good to keep in mind, thanks. Adam Fisk wrote: > The only thing that comes to mind here is that some apps like RealPlayer > and likely Windows Media Player and Quicktime select UDP ports from > fairly broad ranges, at least the last time I checked. I don't remember > what those ranges were, but it's quite likely they're choosing a subset > of the dynamic range as well, so you might just want to take a second to > check for an overlap. It's probably fine and not really something to > worry about -- just putting it on your radar. > > -Adam > > > David Barrett wrote: > >> From range of ports should I randomly pick a UDP port mapping? >> >> The IANA states "The Dynamic and/or Private Ports are those from 49152 >> through 65535". Thus I'm considering just randomly selecting from >> within this range. >> >> However, I'm curious if you have any experience with this and and >> either encourage or dissuade me along this path? I personally don't >> care which port I use, so long as I don't stumble into some "known" >> minefield. Do firewalls often block this range? Do NATs do anything >> funky with this range? Is there any reason to use one UDP port over >> another? Should I secretly use on of the registered ranges to improve >> my NAT/firewall penetration odds? >> >> An alternate plan is to choose an unregistered range from the big IANA >> list (http://www.iana.org/assignments/port-numbers) but if there's no >> reason not to, I'd just as soon stick with the big unregistered >> dynamic range. >> >> Any suggestions? >> >> And finally, I'm going to try to configure a UPnP port-mapping using >> this randomly-selected port as my external NAT port. Does this change >> your recommendations at all? >> >> -david >> _______________________________________________ >> p2p-hackers mailing list >> p2p-hackers@zgp.org >> http://zgp.org/mailman/listinfo/p2p-hackers >> _______________________________________________ >> Here is a web page listing P2P Conferences: >> http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences >> > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > > From distoage at sbbi.net Tue Jul 12 11:21:49 2005 From: distoage at sbbi.net (DiSToAGe) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] Getting started with UPnP on Win32 In-Reply-To: <42CDDDC4.9090001@quinthar.com> References: <42CDDDC4.9090001@quinthar.com> Message-ID: <1121167309.7405.144.camel@station.local> Le jeudi 07 juillet 2005 ? 18:58 -0700, David Barrett a ?crit : > Can you recommend a good resource for getting started with Universal > Plug-n-Play on Win32 for NAT traversal purposes? > if you use java (or want to inspire from), perhaps I can suggest you our lib. We have developped UPNPLib a java based lib to use UPnP, you can find source too, and various documentations and examples about it and explain of UPnP usable for NAT traversal. http://www.sbbi.net/site/upnp/ (docs) http://www.sbbi.net/site/upnp/docs/discovery.html (our server using UPnP for NAT traversal) http://www.sbbi.net/site/jafs/docs/upnp-nat.html -- "Perhaps one day "computer science" will, like Yugoslavia, get broken up into its component parts. That might be a good thing. Especially if it meant independence for my native land, hacking." (hackers and Painters) [ Paul Graham ] From dbarrett at quinthar.com Tue Jul 12 21:01:07 2005 From: dbarrett at quinthar.com (David Barrett) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] Getting started with UPnP on Win32 In-Reply-To: <1121167309.7405.144.camel@station.local> References: <42CDDDC4.9090001@quinthar.com> <1121167309.7405.144.camel@station.local> Message-ID: <42D42F93.3050701@quinthar.com> DiSToAGe wrote: > if you use java (or want to inspire from), perhaps I can suggest you our > lib. We have developped UPNPLib a java based lib to use UPnP, you can > find source too, and various documentations and examples about it and > explain of UPnP usable for NAT traversal. Thanks for the links. I *think* I've got the UPnP code working, but I don't have a UPnP network (and the half-dozen or so networks I've randomly tested on didn't appear to be UPnP compatible). Would you mind testing my application on your network to see if my UPnP code works? (I'm using C++, so alas I can't really use your lib.) Thanks! -david From huaiyu at mail.utexas.edu Wed Jul 13 20:39:35 2005 From: huaiyu at mail.utexas.edu (huaiyu@mail.utexas.edu) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] CollaborateCom-DCCS 2005 Call for papers Message-ID: <1121287175.42d57c07cde58@webmailapp1.cc.utexas.edu> [Apologies if you receive multiple copies of this message] ------------------------------------------------------------------------------- CALL FOR PAPERS First International Workshop on Data Consistency in Collaborative Systems (DCCS 2005) http://www.dataconsistency.org In conjunction with the IEEE CollaborateCom 2005 San Jose, USA, December 18, 2005 ------------------------------------------------------------------------------- SCOPE: Collaborative systems enable collaboration between people and computers to work more productively in acquiring, representing, organizing, archiving, protecting, accessing and communicating information. Consistency is one of the most important and classical issues in collaborative systems research, ranging from the chip level to the system level. Recent advances in collaborative systems have introduced several new challenges to the consistency maintenance in a decentralized, heterogeneous environment. This workshop aims to bring together researchers from various areas of computer science whose work is related to data replication, caching and consistency, including distributed systems, distributed database, Web caching and content delivery network, computer supported collaborative work (CSCW) systems, and so on. In particular, we are interested in the latest progress on data consistency in collaborative systems, including data consistency models, algorithms, systems, and applications. We solicit papers of original research including, but not limited to, the following topics: * Cache modeling and analysis * Data consistency models * Cache coherence protocol * File and storage caching * Web caching and content delivery network * Caching and consistency in P2P systems * Secure data consistency * Concurrency control algorithms performance analysis * Consistency issues in wireless sensor networks Submission Guidelines Authors are invited to submit papers that demonstrate original and unpublished research in these areas. Papers should be no more than 10 pages in length (including references and figures) with in IEEE double column format. All submissions should be in pdf format via the cocus submission web site. In order to submit a paper to the conference, you should first register on the conference management system: http://cocus.create-net.it:8080/confsys/loadRegister.do?act=completeReg Shortly after completing the registration, you will get by e-mail a password. Submission of a paper should be regarded as an undertaking that, should the paper be accepted, at least one of the authors will register and attend the workshop to present the work. Important Dates Paper Submission Deadline: September 1, 2005 Notification of Acceptance: October 1, 2005 Camera Ready Papers Due: October 15, 2005 Workshop Co-Chairs Du Li, Texas A&M University Email: lidu@cs.tamu.edu Weisong Shi, Wayne State University Email: weisong@wayne.edu Program Committee: Monica Brockmeyer, Wayne State University Anca Andrean Ivan, IBM T. J. Watson Research Center Du Li, Texas A&M University (Co-Chair) Huaiyu Liu, University of Texas at Austin Christine Morin, INRIA, France Guillaume Pierre, Vrije University, Netherlands Weisong Shi, Wayne State University (Co-Chair) Evan Speight, IBM Austin Research Lab Chengzheng Sun, Griffith University, Australia Li Xiao, Michigan State University From matthew at matthew.at Mon Jul 18 05:45:13 2005 From: matthew at matthew.at (Matthew Kaufman) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] amicima's MFP - preannouncement Message-ID: <200507180545.j6I5jKU63721@where.matthew.at> For the past year and a half here at amicima, we've been developing some new protocol technology particularly suited for multimedia transport and p2p applications. A few months ago, we made the decision to open-source this technology, and we've been getting it ready for public release. The main announcement of the first big piece, the Secure Media Flow Protocol (MFP) implementation, is just a couple weeks away (we hope... things can move slowly at non-funded software startups), but in the meantime, I thought that folks on this list might like to get a taste of what's to come, so here's some links, in a shameless self-promotion of our work: Technology overview: http://www.amicima.com/technology/index.html The downloads page: http://www.amicima.com/developers/downloads.html (Our BSD-licensed open-source object library for doing sophisticated object-oriented programming in plain ANSI C is already available for download there, the various MFP layers will be up as soon as we can get them ready to ship, dual-licensed... GPL and an available commercial license for proprietary apps) And some initial protocol documentation: http://www.amicima.com/developers/documentation.html (The implementation of the protocol described here will be our next release... coming very soon... runs without modification on Windows, Mac OS X, and Unix) We've used this to build several applications, including a Skype-like VOIP app that runs on Mac OS X and Windows XP. Using our underlying protocol and p2p libraries, a Windows VOIP app that lets you place encrypted P2P (through NAT) computer-to-computer calls, by name, to mobile endpoints, including fast file transfer that doesn't impair the quality of the voice call, took less than a week to whip up a working prototype. If there's demand, we'll probably release that as a sample app with sources, once the MFP and MFPNet layers and some other supporting libraries are available for download. Matthew Kaufman amicima, Inc. matthew@amicima.com matthew@matthew.at Ps. One of the reasons I'm putting this preannouncement out on the p2p-hackers list is that several folks on the list have mailed in the past asking for more information about what we're doing at amicima and when they can play with the technology, and I've managed to delete about half my list of who those people were. From ardagna at dti.unimi.it Mon Jul 18 07:33:46 2005 From: ardagna at dti.unimi.it (Claudio Agostino Ardagna) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] ESORICS 2005 - Call for Participation Message-ID: <00df01c58b6b$07fab6c0$1100000a@Berlino> [Apologies if you receive multiple copies of this message] CALL FOR PARTICIPATION ESORICS 2005 10TH EUROPEAN SYMPOSIUM ON RESEARCH IN COMPUTER SECURITY Milan, Italy - September 12-14, 2005 http://esorics05.dti.unimi.it/ AIMS AND SCOPE Organized in a series of European countries, ESORICS is confirmed as the European research event in computer security. The symposium started in 1990 and has been held on alternate years in different European countries and attracts an international audience from both the academic and industrial communities. From 2002 it has been held yearly. The Symposium has established itself as one of the premiere, international gatherings on information assurance. PRELIMINARY PROGRAM Monday September 12, 2005 ------------------------- 09:15 - 09:30 Welcome and Opening 09:30 - 10:30 Invited talk -- Barbara Simons 10:30 - 11:00 Coffee break 11:00 - 12:30 Session 1: Access control XML Access Control with Policy Matching Tree N. Qi, M. Kudo Semantic Access Control Model: A Formal Specification M. I. Yague, M. Gallardo, A. Mana A Generic XACML Based Declarative Authorization Scheme for Java R. Gupta, M. Bhide 12:30 - 14:00 Lunch 14:00 - 15:30 Session 2: Advanced Authorization Specifications Specification and Validation of Authorisation Constraints Using UML and OCL K. Sohr, G. Ahn Unified Index for Mobile Object Data and Authorizations V. Atluri, Q. Guo On Obligations M. Hilty, D. Basin, A. Pretschner 15:30 - 16:00 Coffe break 16:00 - 17:30 Session 3: Cryptographic Schemes A Practical, Voter-Verifiable Election Scheme D. Chaum, P.Y.A. Ryan, S.Schneider Machine-Checked Security Proofs of Cryptographic Signature Schemes S. Tarento Sanitizable Signatures G. Ateniese, D. Chou, B. de Medeiros, G. Tsudik Tuesday September 13, 2005 -------------------------- 09:00 - 10:30 Session 4: Cryptographic Protocols Limits of the Cryptographic Realization of Dolev-Yao-style XOR M. Backes, B. Pfitzmann Secure Implementation of Cryptographic Protocols: A Case Study Of Mutual Distrust A. Askarov, A. Sabelfeld Augmented oblivious Polynomial Evaluation Protocol and Its Applications H. Zhu 10:30 - 11:00 Coffee break 11:00 - 12:30 Session 5: Intrusion detection Using Attack Trees to Identify Malicious Attacks from Authorized Insiders I. Ray, N. Poolsapassit An Efficient and Unified Approach to Correlating, Hypothesizing, and Predicting Network Intrusion Alerts L. Wang, A. Liu, S. Jajodia Towards a Theory of Intrusion Detection G. Di Crescenzo, A. Ghosh, R. Talpade 12:30 - 14:00 Lunch 14:00 - 15:30 Session 6: Network security On Scalability and Modularisation in the Modelling of Network Security Systems J. de Albuquerque, H. Krumm, P. de Geus Sybil resistant DHT routing G. Danezis, R. Anderson Botnet Tracking: Exploring a Root-Cause Methodology to Prevent Distributed Denial-of-Service Attacks F.C. Freiling, T. Holz, G. Wicherski 15:30 - 16:00 Coffee break 16:00 - 17:30 Session 7: Information Flow and Formal Security Properties Quantifying Probabilistic Information Flow in Computational Reactive Systems M. Backes Enforcing Non-safety Security Policies with Program Monitors J. Ligatti, L. Bauer, D. Walker Soundness of Formal Encryption in the Presence of Key-Cycles P. Adao, G. Bana, J. Herzog, A. Scedrov Wednesday September 14, 2005 ---------------------------- 09:00 - 10:30 Session 8: Privacy and Data Protection Privacy Preserving Clustering S. Jha, L. Kruger, P. McDaniel Abstractions Preserving Parameter Confidentiality S. Gurgenas, P. Ochsenschlaeger, C. Rudolpah Minimal Disclosure in Hierarchical Hippocratic Databases with Delegation F. Massacci, J. Mylopoulos, N. Zannone 10:30 - 11:00 Coffee break 11:00 - 12:30 Session 9: Security for protocols and devices Security Notions for Disk Encryption K. Gjosteen Local View Attack on Anonymous Communication M. Gogolewski, M. Klonowski, M. Kutylowski Browser Model for Security Analysis of Browser-Based Protocols T. Gross, B. Pfitzmann, A. Sadeghi REGISTRATION Online registration is available on the conference web page: http://esorics05.dti.unimi.it/registration.php ADDITIONAL INFORMATION On the web pages (http://esorics05.dti.unimi.it), you will find information about the program, the conference hotel and venue, and some travel and tourist information. We look forward to seeing you in Milan at ESORICS 2005. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://zgp.org/pipermail/p2p-hackers/attachments/20050718/599a6786/attachment.html From ap at hamachi.cc Mon Jul 18 16:30:01 2005 From: ap at hamachi.cc (Alex Pankratov) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] amicima's MFP - preannouncement In-Reply-To: <200507180545.j6I5jKU63721@where.matthew.at> References: <200507180545.j6I5jKU63721@where.matthew.at> Message-ID: <42DBD909.1020703@hamachi.cc> Matthew, I can't seem to find the mentioning of replay protection in your protocol. Also encrypted packets don't seem to carry any sort of HMAC, which is also a bit odd. Perhaps these details are just missing from the spec, in which case it might make sense to produce a separate security-oriented description of the protocol. Alex Matthew Kaufman wrote: > For the past year and a half here at amicima, we've been developing some new > protocol technology particularly suited for multimedia transport and p2p > applications. A few months ago, we made the decision to open-source this > technology, and we've been getting it ready for public release. > > The main announcement of the first big piece, the Secure Media Flow Protocol > (MFP) implementation, is just a couple weeks away (we hope... things can > move slowly at non-funded software startups), but in the meantime, I thought > that folks on this list might like to get a taste of what's to come, so > here's some links, in a shameless self-promotion of our work: > > Technology overview: > http://www.amicima.com/technology/index.html > > The downloads page: > http://www.amicima.com/developers/downloads.html > (Our BSD-licensed open-source object library for doing sophisticated > object-oriented programming in plain ANSI C is already available for > download there, the various MFP layers will be up as soon as we can get them > ready to ship, dual-licensed... GPL and an available commercial license for > proprietary apps) > > And some initial protocol documentation: > http://www.amicima.com/developers/documentation.html > (The implementation of the protocol described here will be our next > release... coming very soon... runs without modification on Windows, Mac OS > X, and Unix) > > We've used this to build several applications, including a Skype-like VOIP > app that runs on Mac OS X and Windows XP. Using our underlying protocol and > p2p libraries, a Windows VOIP app that lets you place encrypted P2P (through > NAT) computer-to-computer calls, by name, to mobile endpoints, including > fast file transfer that doesn't impair the quality of the voice call, took > less than a week to whip up a working prototype. If there's demand, we'll > probably release that as a sample app with sources, once the MFP and MFPNet > layers and some other supporting libraries are available for download. > > Matthew Kaufman > amicima, Inc. > matthew@amicima.com > matthew@matthew.at > > Ps. One of the reasons I'm putting this preannouncement out on the > p2p-hackers list is that several folks on the list have mailed in the past > asking for more information about what we're doing at amicima and when they > can play with the technology, and I've managed to delete about half my list > of who those people were. > > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > > From matthew at matthew.at Mon Jul 18 19:42:37 2005 From: matthew at matthew.at (Matthew Kaufman) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] amicima's MFP - preannouncement In-Reply-To: <42DBD909.1020703@hamachi.cc> Message-ID: <200507181942.j6IJgjU65031@where.matthew.at> Alex Pankratov: > > Matthew, > > I can't seem to find the mentioning of replay protection in > your protocol. Also encrypted packets don't seem to carry any > sort of HMAC, which is also a bit odd. Perhaps these details > are just missing from the spec, in which case it might make > sense to produce a separate security-oriented description of > the protocol. > > Alex The packet-level protocol spec is admittedly short on information about how exactly to use the protocol (and a followup document is being edited and will be up soon). However if you examine the brief description of the startup sequence, you get some hints as to how this works. The goal is to get an encrypted session up in as few round trip times as possible, so the session keying is overlapped with the session handshaking. Then flows can come up simply by sending data on them, so subsequent flows on an established session come up in even fewer round trip times. The Simplified Digital Certificate format is also compact, so that fragmentation can likely be avoided. So specifically, if you assume RSA and AES (or something else that lets you have public-key crypto and fast symmetric crypto, noting that the choice of cryptosystem is not mandated by the protocol, and in fact you could run with no cryptography and simply get replay resistance that is "as good as TCP" (but a protocol that would still be more resistant to "SYN flood" attacks))... A Responder Hello chunk could be forged, by replying with the correct tag data echo, an arbitrary cookie, and a copy of someone else's certificate. However, when the other end sends its Initiator Initial Keying, the session key to be used is encypted with the public key in that certificate, so unless you also have the matching private key, forging a Responder Hello doesn't help you except to possibly cause denial of service. Our implementation supports parallel-open, so if *any* of the endpoints being opened to is actually the right one, you'll still get a session up (better than TCP's situation in that case). And clearly we can't protect against the case where all packets to the valid destination are intercepted and discarded. The Initiator Initial Keying chunk is also signed using the private key associated with the Initator's certificate which is included, and that signature covers the Cookie Echo field... That protects against replay in one direction, in that the Responder got to choose the (new, unique) cookie data, and is forcing the Initiator to prove that they have the private key (and aren't simply replaying) by signing that unique data. At the same time, the Initiator gets to choose the (new, unique) Session Key, and by extension is choosing the Encrypted Session Key. The Responder is then forced to prove that they have their private key (and aren't simply replaying) by successfully decrypting that session key using their private key. If the Responder fails to do that, then it can't send a Responder Keying Ack encrypted with the proper key. Replay of session data after the session is up then only works if the session that is being replayed has a Session ID and a Session Key which match a session which is up, otherwise the data is discarded at the receiver as being unrecognized. We make the assumption that this is very unlikely except in the case where data is simply being replayed after a relatively short delay by a third party. This replay is also limited to entire intact packets, because the encryption prevents the contents from being modified or added to without damaging the packet such that it will be ignored at the receiver, unless the attacker was able to get the session key by a brute force attack on AES, or by monitoring the session startup and having the private RSA key of an endpoint. If the third party doing the replaying forges the source IP address to match the real sender, then it simply looks like duplicated packets, and the sequence numbering used on all the flows and acknowledgements is already prepared to deal just fine with duplicated or reordered packets, even for unreliable or unordered-reception flows, so there's no effect on the application. If the third party doing the replaying uses a new source IP address, then it looks to our implementation like IP mobility is occuring... that is, an endpoint has changed its IP address. However, in our IP mobility support, the return data flow is not moved to the "new" IP address until a unique and timestamped message sent to the "new" IP address can be echoed back (with modification) from that "new" address... Thus, unless the attacker can actually decrypt the session traffic, they will be unable to provide the correct return data, and so the packets from the "new" address will simply be ignored as above, except for the rate-limited probes to see if actual mobility has occurred. As for a per-packet HMAC, this is most valuable for non-encrypted data. An HMAC is a hash generated from the combination of the packet data and a shared secret... We essentially have this, in that a session key (a shared secret) is used to encrypt the packet and its checksum. If one wished to use the MFP protocol unencrypted, a valuable extension might be to provide for a separate HMAC, but we expect most users to run with encryption, since these days everything really should be encrypted, and with MFP there is no round-trip-time penalty to using it, nor any restriction on passing unreliable or unordered data like you'd have with SSL over TCP. Matthew Kaufman matthew@amicima.com matthew@matthew.at www.amicima.com From ap at hamachi.cc Tue Jul 19 04:08:56 2005 From: ap at hamachi.cc (Alex Pankratov) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] amicima's MFP - preannouncement In-Reply-To: <200507181942.j6IJgjU65031@where.matthew.at> References: <200507181942.j6IJgjU65031@where.matthew.at> Message-ID: <42DC7CD8.50605@hamachi.cc> Matthew Kaufman wrote: > Alex Pankratov: > >>Matthew, >> >>I can't seem to find the mentioning of replay protection in >>your protocol. Also encrypted packets don't seem to carry any >>sort of HMAC, which is also a bit odd. Perhaps these details >>are just missing from the spec, in which case it might make >>sense to produce a separate security-oriented description of >>the protocol. >> >>Alex > > > The packet-level protocol spec is admittedly short on information about how > exactly to use the protocol (and a followup document is being edited and > will be up soon). However if you examine the brief description of the > startup sequence, you get some hints as to how this works. > [snip] Matthew, Thanks for a detailed reply. I guess my main concern with security portion of your protocol is that it does not follow certain design properties found in nearly all commonly used transport protocols like SSL/TLS, SSH2, ESP as well as a handful of others like IKE, JFK, etc. Few I can think of off hand are - * the use of MACs. Relying on validity marker in plaintext payload to detect corrupted packets is expensive computationally-wise in a worst case scenario, while HMAC-based authentication costs pennies. It also 'not good' from cryptographical perspective, but I cannot comment on that (note though how SSH was switched to using MACs in v2). Besides MACs are also required for implementing - * replay protection, which involves tagging packets with unique sequence numbers after the encryption but before the authentication. This way replayed or duplicate packets can be discarded without wasting any time on decryption or authentication at all. * session key being derived in 'symmetrical way', when both parties contribute to the key in equal degree. Basically security component of your protocol looks very different from existing mature protocols. And I would suspect that this will not facilitate adoption of MFP .. at least not in its secured form. Alex From dbarrett at quinthar.com Tue Jul 19 05:38:16 2005 From: dbarrett at quinthar.com (David Barrett) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] amicima's MFP - preannouncement In-Reply-To: <200507180545.j6I5jKU63721@where.matthew.at> References: <200507180545.j6I5jKU63721@where.matthew.at> Message-ID: <42DC91C8.7050108@quinthar.com> Matthew -- This is great; I'm glad to hear the release is imminent. I was *really* impressed with what I saw in your demo several months ago and I think it's great your opening up the code for others to license and use. I'm particularly interested in the excellent work you've done with congestion control and link-speed determination. However, excepting a couple curious flags, I didn't see these powerful features called out in the protocol description. Are these handled at the MFP layer, or higher up in the stack? Regardless, congratulations on taking the big step on opening it up; I hope to follow in your footsteps soon enough! -david Matthew Kaufman wrote: > For the past year and a half here at amicima, we've been developing some new > protocol technology particularly suited for multimedia transport and p2p > applications. A few months ago, we made the decision to open-source this > technology, and we've been getting it ready for public release. > > The main announcement of the first big piece, the Secure Media Flow Protocol > (MFP) implementation, is just a couple weeks away (we hope... things can > move slowly at non-funded software startups), but in the meantime, I thought > that folks on this list might like to get a taste of what's to come, so > here's some links, in a shameless self-promotion of our work: > > Technology overview: > http://www.amicima.com/technology/index.html > > The downloads page: > http://www.amicima.com/developers/downloads.html > (Our BSD-licensed open-source object library for doing sophisticated > object-oriented programming in plain ANSI C is already available for > download there, the various MFP layers will be up as soon as we can get them > ready to ship, dual-licensed... GPL and an available commercial license for > proprietary apps) > > And some initial protocol documentation: > http://www.amicima.com/developers/documentation.html > (The implementation of the protocol described here will be our next > release... coming very soon... runs without modification on Windows, Mac OS > X, and Unix) > > We've used this to build several applications, including a Skype-like VOIP > app that runs on Mac OS X and Windows XP. Using our underlying protocol and > p2p libraries, a Windows VOIP app that lets you place encrypted P2P (through > NAT) computer-to-computer calls, by name, to mobile endpoints, including > fast file transfer that doesn't impair the quality of the voice call, took > less than a week to whip up a working prototype. If there's demand, we'll > probably release that as a sample app with sources, once the MFP and MFPNet > layers and some other supporting libraries are available for download. > > Matthew Kaufman > amicima, Inc. > matthew@amicima.com > matthew@matthew.at > > Ps. One of the reasons I'm putting this preannouncement out on the > p2p-hackers list is that several folks on the list have mailed in the past > asking for more information about what we're doing at amicima and when they > can play with the technology, and I've managed to delete about half my list > of who those people were. > > _______________________________________________ > p2p-hackers mailing list > p2p-hackers@zgp.org > http://zgp.org/mailman/listinfo/p2p-hackers > _______________________________________________ > Here is a web page listing P2P Conferences: > http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences > > From matthew at matthew.at Tue Jul 19 08:36:53 2005 From: matthew at matthew.at (Matthew Kaufman) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] amicima's MFP - preannouncement In-Reply-To: <42DC7CD8.50605@hamachi.cc> Message-ID: <200507190837.j6J8b2U65776@where.matthew.at> > Matthew, > > Thanks for a detailed reply. I guess my main concern with > security portion of your protocol is that it does not follow > certain design properties found in nearly all commonly used > transport protocols like SSL/TLS, SSH2, ESP as well as a > handful of others like IKE, JFK, etc. It is best to compare this against IPSEC, rather than something like SSL, because we are operating without the ordered-and-reliable delivery that something like SSL depends on. (For instance, we can't have an internal sequence number that is not transmitted that must match at the far end, like SSH2 has, because we expect out-of-order delivery and lost packets to occur, and we can't just run all the hypothetical sequence numbers through an HMAC algorithm) > Few I can think of off hand are - > > * the use of MACs. Relying on validity marker in plaintext > payload to detect corrupted packets is expensive > computationally-wise in a worst case scenario, while > HMAC-based authentication costs pennies. Not true. See appendix B of rfc3723 (or other equivalent sources). For a 1024 byte packet, AES-CBC takes 27.97 cycles/byte and HMAC-SHA1 takes 24.71 cycles/byte. And we need good performance in the "not being attacked" case as well, so the real comparison is: HMAC in use: 24.71 cycles/byte when being attacked (discard after HMAC, before decrypt), 24.71 + 27.97 = 52.68 cycles/byte when not being attacked (the "expected" case). HMAC not in use: 27.97 cycles/byte when being attacked, 27.97 cycles/byte (the same) when not being attacked. Summary is that if you run SHA1 HMAC and AES, you save about 10% on cycles when you're being attacked and you pay for that small savings by using almost 90% more cycles when you're not. > It also 'not good' from cryptographical perspective, but I > cannot comment on that (note though how SSH was switched to > using MACs in v2). Besides MACs are also required for implementing - > > * replay protection, which involves tagging packets with > unique sequence numbers after the encryption but before the > authentication. Replay protection can also be placed inside the encryption in a similar fashion... The real question is, does putting it outside and using HMAC save you cycles in the being-attacked case (so that you can discard replays prior to decryption cost)? And the answer is "not enough, especially given the penalty in the non-attack case". Our flows have sequence numbers which prevent replay while a flow is active (and once a session is torn down, replay is ineffective), and we also can use the timestamp field for some additional replay protection. Because the attacker cannot "see inside" other packets in order to guess at good sequence numbers, we are better protected than TCP in this way, though not quite as protected as the anti-replay provisions of IPSEC. We might add more if we were convinced that it was necessary. Modified packets can of course be detected not only by the checksum being invalid, but also because it is unlikely that valid chunks would be found and decoded, even if a valid checksum were accidentally generated by a change to an encrypted packet. > This way replayed or duplicate packets can be discarded > without wasting any time on decryption or authentication at all. Well, you can't use the non-encrypted sequence number that you see until you check the HMAC... And that has non-zero cost, as described above. > * session key being derived in 'symmetrical way', when both > parties contribute to the key in equal degree. This is a valid argument, and we have previously discussed this internally and will again. The single issue that is caused by the asymmetry is that an attacker who has compromised one host's private RSA key can (about half the time, in a P2P environment... more or less in a client-server environment) use that information to extract the session key that is chosen, whereas it would require compromising both ends in order to extract a session key that was chosen with symmetry. The difficulty is putting everything in the small number of startup packets we have, and fitting those into non-fragmented datagrams, but this is probably something that will be addressed because it is a valid security concern in some threat cases. > Basically security component of your protocol looks very > different from existing mature protocols. And I would suspect > that this will not facilitate adoption of MFP .. at least not > in its secured form. It actually looks quite like IPSEC ESP without AH... Consider the session ID as the security association identifier and go from there. In reality, though, in order to determine if a secure network protocol is right for your application, you must consider the threats that you are protecting against, and compare that to the benefits. Is this as secure as some other protocols? Probably not. Does it have numerous advantages for certain applications? Absolutely. Will bugs be found in the first release of the implementation? I sure hope so, so that we can fix them! Matthew From lauri.pesonen at gmail.com Tue Jul 19 09:15:32 2005 From: lauri.pesonen at gmail.com (Lauri Pesonen) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] amicima's MFP - preannouncement In-Reply-To: <200507190837.j6J8b2U65776@where.matthew.at> References: <42DC7CD8.50605@hamachi.cc> <200507190837.j6J8b2U65776@where.matthew.at> Message-ID: On 7/19/05, Matthew Kaufman wrote: > > It actually looks quite like IPSEC ESP without AH... Consider the session ID > as the security association identifier and go from there. I'm sure you are aware that encryption only without authentication is considered a bad idea in crypto circles (even the IPSec RFCs warn against it), and I assume you have decided to go against the grain here for performance reasons. That does not change the fact that it's a bad idea. Recently there was an attack published on IPSec with ESP and without AH: http://www.uniras.gov.uk/niscc/docs/al-20050509-00386.html?lang=en And some discussion on sci.crypt: http://groups-beta.google.com/group/sci.crypt/browse_frm/thread/33dd95c9697f46c4?hl=en& Now the attack is a very special case and from a practical point of view and does not give the attacker a huge advantage. This might be even more true for MFP. But it does show that encryption without authentication is a bad idea and does give the attacker _an_ advantage. And as history shows, any advantage in crypto tends to grow bigger as attacks evolve in time. I think you should look again at your performance requirements and re-evaluate whether adding authentication incurs an unacceptable performance hit. Basically at the moment you are going against time-proven crypto thinking. > Matthew -- ! Lauri From ap at hamachi.cc Tue Jul 19 09:47:23 2005 From: ap at hamachi.cc (Alex Pankratov) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] amicima's MFP - preannouncement In-Reply-To: <200507190837.j6J8b2U65776@where.matthew.at> References: <200507190837.j6J8b2U65776@where.matthew.at> Message-ID: <42DCCC2B.30009@hamachi.cc> Matthew Kaufman wrote: [snip] >>Few I can think of off hand are - >> >>* the use of MACs. Relying on validity marker in plaintext >>payload to detect corrupted packets is expensive >>computationally-wise in a worst case scenario, while >>HMAC-based authentication costs pennies. > > Not true. See appendix B of rfc3723 (or other equivalent sources). For a [snip] > Summary is that if you run SHA1 HMAC and AES, you save about 10% on cycles > when you're being attacked and you pay for that small savings by using > almost 90% more cycles when you're not. I have no doubt the authors of rfc3723 computed cycles/byte correctly, however running a quick test with openssl yields 36:5 ratio between AES-128-CBC and HMAC-SHA1. Note that this uses assembly optimized AES implementation and rather frivolious implementation of HMAC. If you consider a hardware implementation, then any modern ESP-capable crypto accelerator will be computing HMAC in parallel with decryption, which would result in zero overhead. > >>It also 'not good' from cryptographical perspective, but I >>cannot comment on that (note though how SSH was switched to >>using MACs in v2). Besides MACs are also required for implementing - >> >>* replay protection, which involves tagging packets with >>unique sequence numbers after the encryption but before the >>authentication. > > Replay protection can also be placed inside the encryption in a similar > fashion... The real question is, does putting it outside and using HMAC save > you cycles in the being-attacked case (so that you can discard replays prior > to decryption cost)? And the answer is "not enough, especially given the > penalty in the non-attack case". You don't need to check MAC to discard duplicate packet. If you are looking at seqno that you already saw, you can drop packet right away. [snip] > >>This way replayed or duplicate packets can be discarded >>without wasting any time on decryption or authentication at all. > > > Well, you can't use the non-encrypted sequence number that you see until you > check the HMAC... And that has non-zero cost, as described above. See above. >>* session key being derived in 'symmetrical way', when both >>parties contribute to the key in equal degree. > > This is a valid argument, and we have previously discussed this internally [snip] > >>Basically security component of your protocol looks very >>different from existing mature protocols. And I would suspect >>that this will not facilitate adoption of MFP .. at least not >>in its secured form. > > It actually looks quite like IPSEC ESP without AH... Consider the session ID > as the security association identifier and go from there. > > In reality, though, in order to determine if a secure network protocol is > right for your application, you must consider the threats that you are > protecting against, and compare that to the benefits. Agreed. > Is this as secure as > some other protocols? Probably not. Does it have numerous advantages for > certain applications? Absolutely. The problem here is in 'probably not'. The mere presence of 'probably' outweighs all benefits. It basically means that security services provided by the protocol may be susceptible to certain attack that may or may not be on the list of threats we are trying to defend against. Alex From matthew at matthew.at Tue Jul 19 16:31:58 2005 From: matthew at matthew.at (Matthew Kaufman) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] amicima's MFP - preannouncement In-Reply-To: <42DCCC2B.30009@hamachi.cc> Message-ID: <200507191632.j6JGW7U66642@where.matthew.at> Alex Pankratov: > I have no doubt the authors of rfc3723 computed cycles/byte > correctly, however running a quick test with openssl yields > 36:5 ratio between AES-128-CBC and HMAC-SHA1. Note that this > uses assembly optimized AES implementation and rather > frivolious implementation of HMAC. We will be benchmarking several implementations today. Last night I was simply using the data I had available. Even at 36:5, HMAC isn't "free", and you do pay for the extra overhead *in addition* to the symmetric encryption *at both the sender and the receiver* in the non-attack case. There is nothing about our design which precludes adding an HMAC trailer, even a mandatory one, and we may add this to the specification after we see the results of the benchmarks and look at the possible attacks. > You don't need to check MAC to discard duplicate packet. If > you are looking at seqno that you already saw, you can drop > packet right away. But an attacker isn't going to replay packets at you with existing sequence numbers if he knows you're going to discard ones that you've already seen without further processing... The attacker will instead modify the packets in order to generate the maximum possible load at the receiver (since once you have sequence numbers, the remaining attack is a cryptographic load denial of service attack) > The problem here is in 'probably not'. The mere presence of 'probably' > outweighs all benefits. It basically means that security > services provided by the protocol may be susceptible to > certain attack that may or may not be on the list of threats > we are trying to defend against. This "probably" is also true of SSL, TLS, IPSEC, etc. Each one has already-known attack vectors and tradeoffs in their designs and implementation. And new ones are being found all the time. The IPSEC situation, for instance, is in many ways worse than this. It is also true of the underlying choice of cryptosystem... Using SHA-1 as your HMAC basis opens you to known attacks on SHA-1. Using RSA for your public-key crypto opens you up to brute-force attacks (and maybe more) against the private keys. But the good news is that what we're releasing will be available in source form, for inspection, and we are open to improving the security as known or suspected attacks against it materialize. (And if we stop being open to it, you can change the source yourself) Already, that's better than the cryptography that most existing P2P users are using (none, in the case of most p2p networks, or proprietary and unavailable for inspection, in a few cases, like Skype) Our primary goal in adding encryption and authentication to MFP was not to make the strongest possible transport... It was rather to get relatively far away from the present situation, where everyone doing P2P is running everything in plain text with no attempt at authentication at all. If by making a few changes we can make it all that much stronger, then sure, we'll do that. Matthew From matthew at matthew.at Tue Jul 19 17:21:12 2005 From: matthew at matthew.at (Matthew Kaufman) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] amicima's MFP - preannouncement In-Reply-To: Message-ID: <200507191721.j6JHLLU66696@where.matthew.at> Lauri Pesonen: > > I'm sure you are aware that encryption only without > authentication is considered a bad idea in crypto circles It is considered an idea which leaves open additional theoretical avenues of attack which are only partially mitigated by adding the authentication. (AES without HMAC is safe unless someone can corrupt the AES data in such a way as to cause a problem for you. Then you're safer with HMAC, *only* as long as the same attacker doesn't also have a working attack against your HMAC scheme... All you're doing is locking two doors instead of one... You're not preventing a great locksmith from getting through both) > (even the IPSec RFCs warn against it) The case of IPSEC is substantially different... The IPSEC RFCs warn against encryption without authentication because in IPSEC there are fields which *cannot be encrypted*, but which *can* be protected with the HMAC. The attack vectors against IPSEC when used with ESP and without AH largely rely on the possibility opened to manipulate the fields which are not encrypted and are then not protected with the AH. > and I assume you have > decided to go against the grain here for performance reasons. Performance and because the attack possibilities are different than in IPSEC. We are open to analyses that show how big the real vulnerability is in our work. And there's nothing about our protocol design that precludes adding an HMAC trailer if it were to be deemed necessary. > That does not change the fact that it's a bad idea. Recently > there was an attack published on IPSec with ESP and without AH: > > http://www.uniras.gov.uk/niscc/docs/al-20050509-00386.html?lang=en I've seen this... There are three basic attacks described in the paper, none of which appears to directly apply to our work. In #1, you perform a successful bit-flipping attack on the encryption to change the destination address on a tunneled packet, which is then decrypted by the tunnel endpoint and sent to the wrong final destination, which could be where the attacker is monitoring. We are potentially vulnerable to bit-flipping attacks, but you'd need to successfully flip bits such that the checksum stays valid and the chunk formatting stays valid, which is probably harder than modifying a single IP address in the header. Furthermore, because we don't decrypt at an intermediate tunnel, this attack cannot be used to redirect plaintext data anywhere, it could only be used to change what is being received, if it were carried out successfully (requires the ability to do this data modification *and* intercept the valid packets before those arrive). In #2 and #3 you damage the encrypted tunneled packet such that after it is decrypted and passed to the final recipient, the final recipient rejects it with a plaintext ICMP message which contains, as per the ICMP spec, a portion of the plaintext that was received. Both of these rely on the obvious danger of including decrypted plaintext in an error reply, and are much easier to carry out, because you can leave the authenticated part alone, damage the encrypted part, and get the plaintext out. Since we never generate plaintext replies, particularly plaintext copies of encrypted data, we don't have this issue. Also, we don't have a non-encrypted part that can stay undamaged. > Now the attack is a very special case and from a practical > point of view and does not give the attacker a huge > advantage. This might be even more true for MFP. But it does > show that encryption without authentication is a bad idea and > does give the attacker _an_ advantage. But in the IPSEC case that advantage comes primarily from the ability to manipulate the encrypted part separately from the unencrypted (and not authenticated if you're not using AH) part in some of the earlier published attacks, or the ability to leave the tunnel encapsulation intact while modifying the interior encrypted packet (as in the cases discussed above). > And as history shows, > any advantage in crypto tends to grow bigger as attacks > evolve in time. This is true. And over time, the algorithms become less secure against both analysis and brute-force attacks. > I think you should look again at your performance > requirements and re-evaluate whether adding authentication > incurs an unacceptable performance hit. Basically at the > moment you are going against time-proven crypto thinking. We will reevaluate this, but as I pointed out above, all that is going on here is adding a second door. Adding an HMAC simply means that someone who figures out a successful attack vector that depends on modifying the AES-encrypted data such that it is wrong but checksums and passes format checks correctly must now also have an attack on the HMAC that works too. What it doesn't do is change this from "insecure" to "perfectly secure", and since we don't know the odds that someone who can manipulate AES to their advantage also knows how to create HMAC collisions, we don't even know how much more secure it would be. Matthew From matthew at matthew.at Tue Jul 19 21:55:58 2005 From: matthew at matthew.at (Matthew Kaufman) Date: Sat Dec 9 22:13:00 2006 Subject: [p2p-hackers] amicima's MFP - preannouncement In-Reply-To: Message-ID: <200507192156.j6JLu7U66951@where.matthew.at> Lauri Pesonen [mailto:lauri.pesonen@gmail.com] (sent privately but permission to reply to the list granted): > > In the end it comes down to your threat model. If you think > a denial of service attack is probable then you should > implement a MAC. More specifically, if you think a denial of service attack is probable and your MAC is enough cheaper in CPU cycles than your decryption, add a MAC, or... > Getting past a CRC when modifying encrypted data is not improbable. > Getting this far means that you end up using resources in > handling invalid packets. This opens up a venue for a DoS > attack. Using a MAC after encryption prevents the DoS attack > at the cost of requiring more resources in handling valid > packets, as you stated earlier. There's also reasons other than a denial-of-service attack to want a MAC, because it does add one more hoop to jump through for an attacker, which with today's crypto technology probably is a significant improvement in security (though without it, the security is still pretty darn good). It is a bunch more CPU cycles though, so as I've pointed out before, deciding whether or not to do it requires analysis of the performance requirements vs. the threat model. > In any case it is seen as a good practise to implement a MAC > on top of encryption. In your case the CRC and message > structure provides some probabilistic protection against > tampered packets. This is not the case in all applications. No disagreement here. > One remaining question is why wouldn't you add a MAC to the protocol? Because the "protocol" doesn't need to have a MAC added. See below... > It would make a lot of crypto people very happy and the > performance impact is not that huge (i.e. your network link > will most probably get bogged down before your CPU cycles run > out). Obviously adding unnecessary features to your > implementation is silly, but a lot of crypto experts (I'm > definitely not one of them) seem to demand a MAC in all > encrypted communication protocols. As the other programmer here pointed out to me right after I sent my last email, because our entire payload is "encrypted", adding a trailing MAC is trivial (unlike in IPSEC, where the MAC needs to protect a different set of bytes than what the encryption covers)... In our implementation of the MFP protocol, ALL cryptographic operations are implemented by a crypto "plug-in". (This means you can easily substitute hardware-accelerated algorithms, proprietary or classified algorithms, etc.) Application developers who want the extra protection of wrapping the encrypted data with an HMAC may simply define an extended crypto plugin which adds a trailing HMAC after encryption (there are some good papers about why it needs to be done after, not before) on the transmit side and which checks the HMAC prior to continuing with decryption on the receive side. This causes no changes to the protocol itself, it simply changes the definition of the cryptosystem that is plugged in (which, like the initial session key choice, is application-defined). The secret key for the HMAC may either be the same key as the AES key, or may be defined as a partitioning of a larger exchanged key into the AES key part and the HMAC key part, and doesn't change the key exchange part of the protocol itself either, because key encryption and decryption are *also* defined by the implementation in the cryptographic plugin. We may or may not provide this HMAC-extended crypto plugin as the default or as one of the initial options that ships with the first release, but it is absolutely trivial for a developer to add if we don't. (And other implementations of the protocol, while they might not use the same crypto plug-in architecture are also free to add or not add a MAC) The only real open issue that has been raised on this list and wh