Re: [Gluster-devel] Major slowdown in cp performance in 1.4 branch

[Krishna Srinivas]

Hi Brent,

Yes it is due to the recent AFR change. We made the change to hold locks during
write. This is to avoid race condition that is seen when two afrs are
writing to the
same region. We are seeing how we can improve the performance.

What we do now is:
hold locks on all subvols (till we succeed on all subvols)
write()
release locks on all subvols.

This is fool proof but is very slow.

Thanks
Krishna

On Tue, Aug 26, 2008 at 11:58 PM, Brent A Nelson <address@hidden> wrote:
> Without the AFRs in the setup, writes are very fast.  56-58MBps for dd, 6
> minutes for my "cp -a" (normally takes ~9 minutes with AFRs in place,
> currently takes > 1hr with AFRs).
>
> It appears that a recent AFR change (perhaps over the weekend?) is killing
> write performance.
>
> Thanks,
>
> Brent
>
> On Tue, 26 Aug 2008, Brent A Nelson wrote:
>
>> No, I noticed the trace translator and removed it yesterday, just before
>> Amar warned me that it was there; no effect.  The crash is gone now, but the
>> slowdown on writes is still an issue.  Avati's suggestion sounds good; I'll
>> try keeping my setup intact, but eliminate the AFRs and see what happens.
>>
>> Thanks,
>>
>> Brent
>>
>> On Tue, 26 Aug 2008, Anand Babu Periasamy wrote:
>>
>>> Avati, trace translator is loaded in this setup. It could have the
>>> slowdown or crash!
>>> --
>>> Anand Babu Periasamy
>>> GPG Key ID: 0x62E15A31
>>> Blog [http://ab.freeshell.org]
>>> The GNU Operating System [http://www.gnu.org]
>>> Z RESEARCH Inc [http://www.zresearch.com]
>>>
>>>
>>>
>>> Anand Avati wrote:
>>>>
>>>> Brent,
>>>>  do you have a non-AFR volume to compare the write speeds against?
>>>>
>>>> avati
>>>>
>>>> 2008/8/26 Brent A Nelson <address@hidden>
>>>>
>>>>> I don't seem to be getting this crashing namespace issue anymore with a
>>>>> tla
>>>>> from today and adding posix-locks to my namespace exports (which are
>>>>> AFRed,
>>>>> and AFR now needs locking, which sounds like a good thing).
>>>>>
>>>>> Writes are still just as slow (3.3MBps), but dd reads are very fast
>>>>> (~117-118MBps, apparently saturating my gigabit link).
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Brent
>>>>>
>>>>> On Mon, 25 Aug 2008, Brent A Nelson wrote:
>>>>>
>>>>>  I went to try to test the read performance of that file and got:
>>>>>>
>>>>>> dd if=/beast/blah0 of=/dev/null bs=1M
>>>>>> dd: opening `/beast/blah0': Input/output error
>>>>>> ls -al /beast
>>>>>> ls: cannot access /beast: No such file or directory
>>>>>> df
>>>>>> ...
>>>>>> glusterfs            5697753088   4054016 5406548992   1% /beast
>>>>>> df /beast
>>>>>> df: `/beast': No such file or directory
>>>>>> df: no file systems processed
>>>>>>
>>>>>> Looking through the processes, I see the namespace volumes both died.
>>>>>>
>>>>>> Here's the tail end of one log:
>>>>>>
>>>>>> 2008-08-25 17:57:43 N [trace.c:1117:trace_lookup] ns0: callid: 420040
>>>>>> (*this=0x8052ff8, loc=0xbf99a9a0 {path=/blah0, inode=0x80a2070} )
>>>>>> 2008-08-25 17:57:43 N [trace.c:535:trace_lookup_cbk] ns0: callid:
>>>>>> 420040
>>>>>> (*this=0x8052ff8, op_ret=0, op_errno=61, inode=0x80a2070,
>>>>>> *buf=0xbf99a864
>>>>>> {st_dev=65031, st_ino=28855, st_mode=33188, st_nlink=1, st_uid=0,
>>>>>> st_gid=0,
>>>>>> st_rdev=0, st_size=0, st_blksize=4096, st_blocks=0})
>>>>>> 2008-08-25 17:57:43 N [trace.c:1505:trace_open] ns0: (*this=0x8052ff8,
>>>>>> loc=0x80adf20 {path=/blah0, inode=0x80a2070}, flags=32768,
>>>>>> fd=0x80988e8)
>>>>>> 2008-08-25 17:57:43 N [trace.c:150:trace_open_cbk] ns0:
>>>>>> (*this=0x8052ff8,
>>>>>> op_ret=0, op_errno=0, *fd=0x80988e8)
>>>>>> 2008-08-25 17:57:43 W [common-utils.c:156:gf_print_bytes] glusterfs:
>>>>>> Total
>>>>>> data (in bytes): transfered (47460701), received (41736499)
>>>>>> pending frames:
>>>>>> frame : type(1) op(40)
>>>>>>
>>>>>> Signal received: 11/lib/tls/i686/cmov/libc.so.6[0xb7d99128]
>>>>>> /usr/lib/libglusterfs.so.0(default_gf_lk+0xb4)[0xb7ee8724]
>>>>>>
>>>>>>
>>>>>> /usr/lib/glusterfs/1.4.0qa34/xlator/protocol/server.so(server_lk_common+0x6ff)[0xb7cf93df]
>>>>>>
>>>>>>
>>>>>> /usr/lib/glusterfs/1.4.0qa34/xlator/protocol/server.so(server_gf_lk+0x48)[0xb7cf95d8]
>>>>>>
>>>>>>
>>>>>> /usr/lib/glusterfs/1.4.0qa34/xlator/protocol/server.so(protocol_server_interpret+0xd6)[0xb7cf41e6]
>>>>>>
>>>>>>
>>>>>> /usr/lib/glusterfs/1.4.0qa34/xlator/protocol/server.so(protocol_server_pollin+0xb3)[0xb7cf43e3]
>>>>>>
>>>>>>
>>>>>> /usr/lib/glusterfs/1.4.0qa34/xlator/protocol/server.so(notify+0x51)[0xb7cf44e1]
>>>>>> /usr/lib/glusterfs/1.4.0qa34/transport/socket.so[0xb74e7f3a]
>>>>>> /usr/lib/libglusterfs.so.0[0xb7ef6a45]
>>>>>> /usr/lib/libglusterfs.so.0(event_dispatch+0x21)[0xb7ef5661]
>>>>>> glusterfsd(main+0x953)[0x804a163]
>>>>>> /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe0)[0xb7d84450]
>>>>>> glusterfsd[0x80497a1]
>>>>>> ---------
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Brent
>>>>>>
>>>>>> On Mon, 25 Aug 2008, Brent A Nelson wrote:
>>>>>>
>>>>>>  3.2 MBps, which is far less than previous, on a "dd if=/dev/zero
>>>>>>>
>>>>>>> of=/beast/blah0 bs=1M count=10000"
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Brent
>>>>>>>
>>>>>>> On Mon, 25 Aug 2008, Amar S. Tumballi wrote:
>>>>>>>
>>>>>>>  Brent,
>>>>>>>>
>>>>>>>> Can you check the dd speed too? One user on irc reported very slow
>>>>>>>> cp
>>>>>>>> performance too, but his dd performance was good.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>>
>>>>>>>> 2008/8/25 Brent A Nelson <address@hidden>
>>>>>>>>
>>>>>>>>  A checkout today from the 1.4 branch seems to give terrible
>>>>>>>> performance
>>>>>>>>>
>>>>>>>>> on
>>>>>>>>> "cp -a".  Something that took a little over 9 minutes from a
>>>>>>>>> checkout
>>>>>>>>> last
>>>>>>>>> week now takes over an hour (it hasn't finished yet).  CPU time
>>>>>>>>> consumed by
>>>>>>>>> glusterfs and all the glusterfsd processes is quite a bit smaller
>>>>>>>>> than
>>>>>>>>> it
>>>>>>>>> used to be; it looks like a recent patch is causing a substantial
>>>>>>>>> performance issue...
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> Brent
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Gluster-devel mailing list
>>>>>>>>> address@hidden
>>>>>>>>> http://lists.nongnu.org/mailman/listinfo/gluster-devel
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Amar Tumballi
>>>>>>>> Gluster/GlusterFS Hacker
>>>>>>>> [bulde on #gluster/irc.gnu.org]
>>>>>>>> http://www.zresearch.com - Commoditizing Super Storage!
>>>>>>>>
>>>>>>>>
>>>>> _______________________________________________
>>>>> Gluster-devel mailing list
>>>>> address@hidden
>>>>> http://lists.nongnu.org/mailman/listinfo/gluster-devel
>>>>>
>>>>
>>>>
>>>>
>>>
>>
>
>
> _______________________________________________
> Gluster-devel mailing list
> address@hidden
> http://lists.nongnu.org/mailman/listinfo/gluster-devel
>