Re: [Qemu-devel] [PATCH v3 04/25] tcg-ppc64: Relax register restrictions in tcg_out_mem_long

[Greg Kurz]

On Fri, 20 Jun 2014 07:13:20 -0700
Richard Henderson <address@hidden> wrote:
> In order to be able to use tcg_out_ld/st sensibly with scratch
> registers, assert only when we'd incorrectly clobber a scratch.
> 
> Signed-off-by: Richard Henderson <address@hidden>
> ---

Hi,

While testing various guest/host combinations for virtio, Cedric hit the 
following crash with
a x86_64 fedora 20 TCG guest run by a ppc64 or ppc64le upstream QEMU:

[    0.946484] Unpacking initramfs...
[    2.371827] Freeing initrd memory: 15620K (ffff88007f0be000 - 
ffff88007ffff000)
[    2.372459] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[    2.372818] software IO TLB [mem 0xbbffe000-0xbfffe000] (64MB) mapped at 
[ffff8800bbffe000-ffff8800bfffdfff]
[    2.389534] futex hash table entries: 256 (order: 2, 16384 bytes)
[    2.392753] ------------[ cut here ]------------
[    2.393213] WARNING: CPU: 0 PID: 25 at kernel/pid.c:278 
free_pid+0x14b/0x150()
[    2.393310] Modules linked in:
[    2.393310] CPU: 0 PID: 25 Comm: cryptomgr_test Not tainted 
3.14.8-200.fc20.x86_64 #1
[    2.393310] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 
rel-1.7.5-0-ge51488c-20140602_164612-nilsson.home.kraxel.org 04/01/2014
[    2.393310]  0000000000000000 00000000a7a5d6ef ffff880138e47d18 
ffffffff816f0502
[    2.393310]  0000000000000000 ffff880138e47d50 ffffffff8108a1cd 
ffff8800bb599700
[    2.393310]  0000000000000000 0000000000000046 ffffffff81c444e0 
0000000000000000
[    2.393310] Call Trace:
[    2.393310]  [<ffffffff816f0502>] dump_stack+0x45/0x56
[    2.393310]  [<ffffffff8108a1cd>] warn_slowpath_common+0x7d/0xa0
[    2.393310]  [<ffffffff8108a2fa>] warn_slowpath_null+0x1a/0x20
[    2.393310]  [<ffffffff810aa7cb>] free_pid+0x14b/0x150
[    2.393310]  [<ffffffff810aa82a>] __change_pid+0x5a/0x60
[    2.393310]  [<ffffffff810aad90>] detach_pid+0x10/0x20
[    2.393310]  [<ffffffff8108b393>] release_task+0x353/0x470
[    2.393310]  [<ffffffff8108ca9a>] do_exit+0x5ea/0xa30
[    2.393310]  [<ffffffff81313df0>] ? crypto_unregister_pcomp+0x20/0x20
[    2.393310]  [<ffffffff81100b4f>] __module_put_and_exit+0x2f/0x30
[    2.393310]  [<ffffffff81313e23>] cryptomgr_test+0x33/0x50
[    2.393310]  [<ffffffff810ae2d1>] kthread+0xe1/0x100
[    2.393310]  [<ffffffff810ae1f0>] ? insert_kthread_work+0x40/0x40
[    2.393310]  [<ffffffff8170083c>] ret_from_fork+0x7c/0xb0
[    2.393310]  [<ffffffff810ae1f0>] ? insert_kthread_work+0x40/0x40
[    2.393310] ---[ end trace c82ee4daf4a04f19 ]---
[    2.393310] ------------[ cut here ]------------
[    2.393310] WARNING: CPU: 0 PID: 25 at kernel/workqueue.c:1393 
__queue_work+0x2ad/0x310()
[    2.393310] Modules linked in:
[    2.393310] CPU: 0 PID: 25 Comm: cryptomgr_test Tainted: G        W    
3.14.8-200.fc20.x86_64 #1
[    2.393310] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 
rel-1.7.5-0-ge51488c-20140602_164612-nilsson.home.kraxel.org 04/01/2014
[    2.393310]  0000000000000000 00000000a7a5d6ef ffff880138e47cb8 
ffffffff816f0502
[    2.393310]  0000000000000000 ffff880138e47cf0 ffffffff8108a1cd 
ffff88013fc17e00
[    2.393310]  ffffffff81c44d40 0000000000000400 ffff88013b005a00 
0000000000010368
[    2.393310] Call Trace:
[    2.393310]  [<ffffffff816f0502>] dump_stack+0x45/0x56
[    2.393310]  [<ffffffff8108a1cd>] warn_slowpath_common+0x7d/0xa0
[    2.393310]  [<ffffffff8108a2fa>] warn_slowpath_null+0x1a/0x20
[    2.393310]  [<ffffffff810a484d>] __queue_work+0x2ad/0x310
[    2.393310]  [<ffffffff810a4d67>] queue_work_on+0x27/0x50
[    2.393310]  [<ffffffff810aa6d1>] free_pid+0x51/0x150
[    2.393310]  [<ffffffff810aa82a>] __change_pid+0x5a/0x60
[    2.393310]  [<ffffffff810aad90>] detach_pid+0x10/0x20
[    2.393310]  [<ffffffff8108b393>] release_task+0x353/0x470
[    2.393310]  [<ffffffff8108ca9a>] do_exit+0x5ea/0xa30
[    2.393310]  [<ffffffff81313df0>] ? crypto_unregister_pcomp+0x20/0x20
[    2.393310]  [<ffffffff81100b4f>] __module_put_and_exit+0x2f/0x30
[    2.393310]  [<ffffffff81313e23>] cryptomgr_test+0x33/0x50
[    2.393310]  [<ffffffff810ae2d1>] kthread+0xe1/0x100
[    2.393310]  [<ffffffff810ae1f0>] ? insert_kthread_work+0x40/0x40
[    2.393310]  [<ffffffff8170083c>] ret_from_fork+0x7c/0xb0
[    2.393310]  [<ffffffff810ae1f0>] ? insert_kthread_work+0x40/0x40
[    2.393310] ---[ end trace c82ee4daf4a04f1a ]---
[    2.411147] Initialise system trusted keyring
[    2.412887] audit: initializing netlink subsys (disabled)
[    2.414491] audit: type=2000 audit(1403786361.413:1): initialized
[    2.510453] ------------[ cut here ]------------
[    2.510737] kernel BUG at mm/vmscan.c:3401!
[    2.511000] invalid opcode: 0000 [#1] SMP 
[    2.511056] Modules linked in:
[    2.511056] CPU: 0 PID: 1 Comm: swapper/0 Tainted: G        W    
3.14.8-200.fc20.x86_64 #1
[    2.511056] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 
rel-1.7.5-0-ge51488c-20140602_164612-nilsson.home.kraxel.org 04/01/2014
[    2.511056] task: ffff880139b00000 ti: ffff880139a9e000 task.ti: 
ffff880139a9e000
[    2.511056] RIP: 0010:[<ffffffff81188711>]  [<ffffffff81188711>] 
kswapd_run+0xc1/0xd0
[    2.511056] RSP: 0000:ffff880139a9fe08  EFLAGS: 00000246
[    2.511056] RAX: fffffffffffffff4 RBX: 0000000000000000 RCX: 0000000000000000
[    2.511056] RDX: 00000000000006ca RSI: ffff880139b00000 RDI: ffff88013b001b00
[    2.511056] RBP: ffff880139a9fe28 R08: 00000000000173e0 R09: ffff88013fc173e0
[    2.511056] R10: ffffea0004e4df00 R11: ffffffff810ae161 R12: ffff88013ffe9000
[    2.511056] R13: 0000000000000000 R14: fffffffffffffff4 R15: 0000000000000000
[    2.511056] FS:  0000000000000000(0000) GS:ffff88013fc00000(0000) 
knlGS:0000000000000000
[    2.511056] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[    2.511056] CR2: 0000000000000000 CR3: 0000000001c0c000 CR4: 00000000000006f0
[    2.511056] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[    2.511056] DR3: 0000000000000000 DR6: 0000000000000000 DR7: 0000000000000000
[    2.511056] Stack:
[    2.511056]  0000000000000001 0000000000000200 00000000000000fe 
0000000000000000
[    2.511056]  ffff880139a9fe48 ffffffff81d5065e 0000000000000000 
ffffffff81d5061d
[    2.511056]  ffff880139a9fec0 ffffffff8100216a 0000000000000200 
ffff880139a9fec0
[    2.511056] Call Trace:
[    2.511056]  [<ffffffff81d5065e>] kswapd_init+0x41/0x75
[    2.511056]  [<ffffffff81d5061d>] ? 
ftrace_define_fields_mm_vmscan_lru_shrink_inactive+0x138/0x138
[    2.511056]  [<ffffffff8100216a>] do_one_initcall+0xfa/0x1b0
[    2.511056]  [<ffffffff810ac225>] ? parse_args+0x225/0x3f0
[    2.511056]  [<ffffffff81d261a3>] kernel_init_freeable+0x1ab/0x247
[    2.511056]  [<ffffffff81d25926>] ? do_early_param+0x88/0x88
[    2.511056]  [<ffffffff816e1690>] ? rest_init+0x80/0x80
[    2.511056]  [<ffffffff816e169e>] kernel_init+0xe/0xf0
[    2.511056]  [<ffffffff8170083c>] ret_from_fork+0x7c/0xb0
[    2.511056]  [<ffffffff816e1690>] ? rest_init+0x80/0x80
[    2.511056] Code: 2a 44 89 ee 48 c7 c7 20 58 a2 81 31 c0 e8 ad 42 56 00 49 
8b 9c 24 d8 3d 01 00 49 c7 84 24 d8 3d 01 00 00 00 00 00 e9 6a ff ff ff <0f> 0b 
66 66 66 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 
[    2.511056] RIP  [<ffffffff81188711>] kswapd_run+0xc1/0xd0
[    2.511056]  RSP <ffff880139a9fe08>
[    2.525816] ---[ end trace c82ee4daf4a04f1b ]---
[    2.526424] Kernel panic - not syncing: Attempted to kill init! 
exitcode=0x0000000b
[    2.526424] 
[    2.527124] Kernel Offset: 0x0 from 0xffffffff81000000 (relocation range: 
0xffffffff80000000-0xffffffff9fffffff)
[    2.527124] general protection fault: fff2 [#2] SMP 
[    2.527124] Modules linked in:
[    2.527124] CPU: 0 PID: 1 Comm: swapper/0 Tainted: G      D W    
3.14.8-200.fc20.x86_64 #1
[    2.527124] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 
rel-1.7.5-0-ge51488c-20140602_164612-nilsson.home.kraxel.org 04/01/2014
[    2.527124] task: ffff880139b00000 ti: ffff880139a9e000 task.ti: 
ffff880139a9e000
[    2.527124] RIP: 0010:[<ffffffff816ec356>]  [<ffffffff816ec356>] 
panic+0x1a3/0x1e7
[    2.527124] RSP: 0000:ffff880139a9faf8  EFLAGS: 00000246
[    2.527124] RAX: 000000000c1f0c1f RBX: ffffffff81a12e20 RCX: 00000000000004ea
[    2.527124] RDX: 0000000000000c1f RSI: 0000000000000000 RDI: 0000000000000046
[    2.527124] RBP: ffff880139a9fb68 R08: 0000000000000001 R09: 0000000000000187
[    2.527124] R10: 0720072007200720 R11: 0720072007200720 R12: 0000000000000000
[    2.527124] R13: 0000000000000000 R14: 0000000000000000 R15: ffff880139b00000
[    2.527124] FS:  0000000000000000(0000) GS:ffff88013fc00000(0000) 
knlGS:0000000000000000
[    2.527124] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[    2.527124] CR2: 0000000000000000 CR3: 0000000001c0c000 CR4: 00000000000006f0
[    2.527124] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[    2.527124] DR3: 0000000000000000 DR6: 0000000000000000 DR7: 0000000000000000
[    2.527124] Stack:
[    2.527124]  ffff880100000010 ffff880139a9fb78 ffff880139a9fb18 
00000000c997aaf8
[    2.527124]  ffff880139b00000 000000000000000b ffff880139b00408 
0000000000000019
[    2.527124]  ffffffff81ed9b40 0000000000000099 ffffffff81c444e0 
0000000000000000
[    2.527124] Call Trace:
[    2.527124]  [<ffffffff8108ced1>] do_exit+0xa21/0xa30
[    2.527124]  [<ffffffff816eca1c>] ? printk+0x77/0x8e
[    2.527124]  [<ffffffff816f890c>] oops_end+0x9c/0xe0
[    2.527124]  [<ffffffff81017fdb>] die+0x4b/0x70
[    2.527124]  [<ffffffff816f81a0>] do_trap+0x60/0x170
[    2.527124]  [<ffffffff810150aa>] do_invalid_op+0xaa/0xe0
[    2.527124]  [<ffffffff81188711>] ? kswapd_run+0xc1/0xd0
[    2.527124]  [<ffffffff8118bac0>] ? mem_cgroup_shrink_node_zone+0x160/0x160
[    2.527124]  [<ffffffff816f4579>] ? _cond_resched+0x29/0x40
[    2.527124]  [<ffffffff816f5239>] ? wait_for_completion_killable+0x39/0x180
[    2.527124]  [<ffffffff810bf6a6>] ? try_to_wake_up+0x1e6/0x290
[    2.527124]  [<ffffffff8170201e>] invalid_op+0x1e/0x30
[    2.527124]  [<ffffffff810ae161>] ? kthread_create_on_node+0x141/0x190
[    2.527124]  [<ffffffff81188711>] ? kswapd_run+0xc1/0xd0
[    2.527124]  [<ffffffff811886b0>] ? kswapd_run+0x60/0xd0
[    2.527124]  [<ffffffff81d5065e>] kswapd_init+0x41/0x75
[    2.527124]  [<ffffffff81d5061d>] ? 
ftrace_define_fields_mm_vmscan_lru_shrink_inactive+0x138/0x138
[    2.527124]  [<ffffffff8100216a>] do_one_initcall+0xfa/0x1b0
[    2.527124]  [<ffffffff810ac225>] ? parse_args+0x225/0x3f0
[    2.527124]  [<ffffffff81d261a3>] kernel_init_freeable+0x1ab/0x247
[    2.527124]  [<ffffffff81d25926>] ? do_early_param+0x88/0x88
[    2.527124]  [<ffffffff816e1690>] ? rest_init+0x80/0x80
[    2.527124]  [<ffffffff816e169e>] kernel_init+0xe/0xf0
[    2.527124]  [<ffffffff8170083c>] ret_from_fork+0x7c/0xb0
[    2.527124]  [<ffffffff816e1690>] ? rest_init+0x80/0x80
[    2.527124] Code: 00 00 49 ff cc 74 0c bf 58 89 41 00 e8 54 4e c7 ff eb ef 
48 83 c3 64 eb b1 83 3d 75 8f 7e 00 00 74 05 e8 6e 7b 9c ff fb 66 66 90 <66> 66 
90 45 31 e4 e8 1f 4d a4 ff 4d 39 ec 7c 18 41 83 f6 01 44 
[    2.527124] RIP  [<ffffffff816ec356>] panic+0x1a3/0x1e7
[    2.527124]  RSP <ffff880139a9faf8>
[    2.527124] ---[ end trace c82ee4daf4a04f1c ]---
[    2.527124] Fixing recursive fault but reboot is needed!

Bisect leads to commit:

commit de7761a39d341ab322f0c2f47ec3ec59a4a6f2a2
Author: Richard Henderson <address@hidden>
Date:   Tue Mar 25 12:22:18 2014 -0700

    tcg-ppc64: Relax register restrictions in tcg_out_mem_long

Indeed, I could revert the commit and the crash no longer happens.

Unfortunately, if I pass --enable-debug-tcg to configure, qemu-system-x86_64 
always abort , no matter the revert.

$ qemu-system-x86_64 -m 4G -serial mon:stdio -nographic -nodefaults 
-no-shutdown -snapshot -hda 
/home/legoater/work/qemu/images/fedora20-x86_64.qcow2
qemu-system-x86_64: 
/home/greg/Work/qemu/qemu-upstream/tcg/ppc/tcg-target.c:808: tcg_out_mem_long: 
Assertion `rs != base && (!is_store || rs != rt)' failed.
Aborted

Can a TCG wizard have a look at this ?

Cheers.

--
Greg

>  tcg/ppc64/tcg-target.c | 12 +++++++-----
>  1 file changed, 7 insertions(+), 5 deletions(-)
> 
> diff --git a/tcg/ppc64/tcg-target.c b/tcg/ppc64/tcg-target.c
> index 951a392..dbe9c5c 100644
> --- a/tcg/ppc64/tcg-target.c
> +++ b/tcg/ppc64/tcg-target.c
> @@ -714,10 +714,9 @@ static void tcg_out_mem_long(TCGContext *s, int opi, int 
> opx, TCGReg rt,
>                               TCGReg base, tcg_target_long offset)
>  {
>      tcg_target_long orig = offset, l0, l1, extra = 0, align = 0;
> +    bool is_store = false;
>      TCGReg rs = TCG_REG_R2;
> 
> -    assert(rt != TCG_REG_R2 && base != TCG_REG_R2);
> -
>      switch (opi) {
>      case LD: case LWA:
>          align = 3;
> @@ -725,19 +724,22 @@ static void tcg_out_mem_long(TCGContext *s, int opi, 
> int opx, TCGReg rt,
>      default:
>          if (rt != TCG_REG_R0) {
>              rs = rt;
> +            break;
>          }
>          break;
>      case STD:
>          align = 3;
> -        break;
> +        /* FALLTHRU */
>      case STB: case STH: case STW:
> +        is_store = true;
>          break;
>      }
> 
>      /* For unaligned, or very large offsets, use the indexed form.  */
>      if (offset & align || offset != (int32_t)offset) {
> -        tcg_out_movi(s, TCG_TYPE_PTR, TCG_REG_R2, orig);
> -        tcg_out32(s, opx | TAB(rt, base, TCG_REG_R2));
> +        tcg_debug_assert(rs != base && (!is_store || rs != rt));
> +        tcg_out_movi(s, TCG_TYPE_PTR, rs, orig);
> +        tcg_out32(s, opx | TAB(rt, base, rs));
>          return;
>      }
> 



-- 
Gregory Kurz                                     address@hidden
                                                 address@hidden
Software Engineer @ IBM/Meiosys                  http://www.ibm.com
Tel +33 (0)562 165 496

"Anarchy is about taking complete responsibility for yourself."
        Alan Moore.