address@hidden 049891002526]# pwd
/intstore/intstore01c/gcdata/data/tracks/tmg/2008_02_05/049891002526
address@hidden 049891002526]# ls -al 049891002526_01_09.wma.sigKey01.k
-rw-rw-rw- 1 10015 root 19377712 Feb 6 2008 049891002526_01_09.wma.sigKey01.k
address@hidden 049891002526]# attr -l 049891002526_01_09.wma.sigKey01.k
Attribute "glusterfs.createtime" has a 10 byte value for 049891002526_01_09.wma.sigKey01.k
Attribute "glusterfs.version" has a 1 byte value for 049891002526_01_09.wma.sigKey01.k
Attribute "selinux" has a 24 byte value for 049891002526_01_09.wma.sigKey01.k
address@hidden 049891002526]# attr -l .
Attribute "glusterfs.createtime" has a 10 byte value for .
Attribute "glusterfs.version" has a 1 byte value for .
Attribute "glusterfs.dht" has a 16 byte value for .
Attribute "selinux" has a 24 byte value for .
Nothing in both the client and server logs. I've tried all the normal replication checks and self-heal such as ls -alR. If i copy the file back from one of the bricks into the volume it will show up again however it has a 1/3 chance of getting written to the files original location. So then i end up with two identical files on two different bricks.
This volume has over 40 million files and directories so it can be very tedious to find anomalies. I wrote a quick perl script to search 1/25 of our total files in the volume for missing files and md5 checksum differences and as of now its about 15% (138,500 files) complete and has found ~7000 missing files and 0 md5 checksum differences.
How could i debug this? I'd image it has something to do with the extended attributes on either the file or parent directory...but as far as i can tell that all looks fine.
thanks,
liam
client glusterfs.vol:
volume brick1a
type protocol/client
option transport-type tcp
option remote-host server1
option remote-subvolume brick1a
end-volume
volume brick1b
type protocol/client
option transport-type tcp
option remote-host server1
option remote-subvolume brick1b
end-volume
volume brick1c
type protocol/client
option transport-type tcp
option remote-host server1
option remote-subvolume brick1c
end-volume
volume brick2a
type protocol/client
option transport-type tcp
option remote-host server2
option remote-subvolume brick2a
end-volume
volume brick2b
type protocol/client
option transport-type tcp
option remote-host server2
option remote-subvolume brick2b
end-volume
volume brick2c
type protocol/client
option transport-type tcp
option remote-host server2
option remote-subvolume brick2c
end-volume
volume bricks1
type cluster/replicate
subvolumes brick1a brick2a
end-volume
volume bricks2
type cluster/replicate
subvolumes brick1b brick2b
end-volume
volume bricks3
type cluster/replicate
subvolumes brick1c brick2c
end-volume
volume distribute
type cluster/distribute
subvolumes bricks1 bricks2 bricks3
end-volume
volume writebehind
type performance/write-behind
option block-size 1MB
option cache-size 64MB
option flush-behind on
subvolumes distribute
end-volume
volume cache
type performance/io-cache
option cache-size 2048MB
subvolumes writebehind
end-volume
server glusterfsd.vol:
volume intstore01a
type storage/posix
option directory /intstore/intstore01a/gcdata
end-volume
volume intstore01b
type storage/posix
option directory /intstore/intstore01b/gcdata
end-volume
volume intstore01c
type storage/posix
option directory /intstore/intstore01c/gcdata
end-volume
volume locksa
type features/posix-locks
option mandatory-locks on
subvolumes intstore01a
end-volume
volume locksb
type features/posix-locks
option mandatory-locks on
subvolumes intstore01b
end-volume
volume locksc
type features/posix-locks
option mandatory-locks on
subvolumes intstore01c
end-volume
volume brick1a
type performance/io-threads
option thread-count 32
subvolumes locksa
end-volume
volume brick1b
type performance/io-threads
option thread-count 32
subvolumes locksb
end-volume
volume brick1c
type performance/io-threads
option thread-count 32
subvolumes locksc
end-volume
volume server
type protocol/server
option transport-type tcp
option auth.addr.brick1a.allow 192.168.12.*
option auth.addr.brick1b.allow 192.168.12.*
option auth.addr.brick1c.allow 192.168.12.*
subvolumes brick1a brick1b brick1c
end-volume