[Gluster-devel] Config advice?

[David Braginsky]

Hey guys,

I am trying to set up glusterfs on a few hundred nodes and was hoping for some 
advice.

I have a few hundred machines, spread across two datacenters, in racks of 40 
nodes. These machines act as in-memory servers, but during startup need to load 
their data off disk. Each server is replicated, so several machines serve the 
same data. Therefore, on startup, each file will be loaded by several machines 
at once. The data is written via appends to a set of log files, which are 
periodically rotated. 

I don't know how well glusterfs would handle cross-datacenter writes, so my 
plan is to have a separate fs on in each datacenter. I plan on using the same 
server machines to run the fs. My naïve approach is to pick a replication 
factor (4), generate a bunch of AFR clusters (numNodes / replicationFactor), 
then use DHT to map my data onto the AFT clusters. That way each file will get 
mapped onto 4 machines, which should handle the read throughput as well as 
failures. Does that make sense?

Or would it be better to use AFT over DHT? Or HA in some capacity? Is there any 
way to achieve rack affinity? It'd be nice if reads were done from the local 
rack when possible.

I can imagine all of these setups, but some require more complicated config 
files than others. Any advice would be appreciated.