[linux-elitists] Cluster filesystems
Thu Jan 1 12:43:51 PST 2004
On Thu, 2004-01-01 at 23:05, Nathan Poznick wrote:
> Thus spake Jeff Waugh:
> > <quote who="Jason Spence">
> > > The last time I tried Intermezzo, GFS, Coda, OCFS, etc, I found them
> > > to be lacking in stability and documentation, and recovery from the
> > > simultaneous update problem was really spotty and labor intensive.
> > Add lustre <http://www.lustre.org/> to your list to try out next time. Plus,
> > GFS may be back on the cards since RH bought Sistina. And I hear that SGI
> > have considered releasing CXFS a number of times.
> IMHO, Lustre also suffers from the documentation. Oh sure it's there,
> but the design of the system seems to have been conceived with forcing
> you to pay ClusterFS for support by making it obscenely complicated.
> (Something like 13 or 14 kernel modules which must be in place for a
Let me chime in and say that I have a non-trivial amount of experience
with lustre. It is quite honestly great. We use it extensively here at
LLNL. If you have any questions about it, I probably will not know the
answer but I have very ready access to all of the CFS people and can
find out very quickly.
As one who has been right there watching CFS develop lustre for the past
2 years. I can say that your belief that CFS intentionally designed
Lustre to be complicated so that they would get support money is
completely 100% and unequivocally wrong.
1) The lack of documentation is basically due to the fact that the
people paying for lustre are paying to get it working and the developers
are working closely with the people actually using it and the
consequence is that much of the understanding of how things need to be
setup is passed person to person rather than through documentation.
2) Also the lack of documentation is partly due to the fact that things
have been changing too quickly to be written down. Just a couple of
weeks ago, they finally got it up to the place where you can mount a
lustre file system using the mount command. Before this, it took a
special script to get things working. The overall goal, is to make it so
that lustre works more like normal networked file systems. It is just
that we are not there yet and it will take time. The good thing is there
is money behind this project pushing it forward.
3) The lustre developers at CFS want absolutely nothing to do with doing
support. They want to get paid to code and don't want to do support at
all. They have been trying to convince Red Hat to productize and support
CFS and just give them some money to continue developing it.
> Also, can't leave PVFS and PVFS2 <http://www.pvfs.org/> off the list.
> If you want to go really fast, and don't mind sacrificing some of the
> guarantees of the heftier filesystems, they're wonderful (PVFS2 is still
> in early development, but is testable).
One more thing I can say about lustre. Since this is public money that
we are spending here. We constantly are reevaluating the various cluster
file systems and every single time we do this we find that nothing meets
our needs other than lustre. PVFS is one that we just evaluated and it
didn't do very well on our tests. I can probably dig up the report.
However, all of this info is based upon our needs. Your needs might be
More information about the linux-elitists