Re: ARM Snapshots Not Backwards-Compatible

[Aaron Lindsay]

On Feb 03 10:01, Peter Maydell wrote:
> On Wed, 3 Feb 2021 at 04:01, Aaron Lindsay <aaron@os.amperecomputing.com> 
> wrote:
> > Note that I'm using qemu-system-aarch64 with -loadvm.
> 
> You don't say what machine type and command line you're using. Strictly,
> Strictly speaking, the intended guarantee only covers versioned
> machines, eg "virt-4.1" on QEMU 4.1 to "virt-4.1" on QEMU 5.2.

Sorry for the omission, `-M virt` was used to capture the snapshots on
4.1.0 and I'm using `-M virt-4.1` to restore it on 5.2.0. You get a
fairly helpful error message about that one if you screw it up:

"qemu-system-aarch64: Machine type received is 'virt-4.1' and local is 
'virt-5.2'"

I'm restoring using the following command-line on 5.2.0. Snapshots were taken
with the same, except `-M virt` and without `-loadvm` on 4.1.0:

$ qemu-system-aarch64 -smp 1 -M virt-4.1 -cpu max -m 8G -icount 0 -nographic 
-semihosting -kernel ./Image -append 'console=ttyAMA0 root=/dev/vda3 rw 
earlycon=smh' -drive if=none,id=image,file=rootfs.qcow2,format=qcow2 -device 
virtio-blk-pci,drive=image -loadvm "snapshot-name"

I suspect that some of the differences I raised in this section:

> > These hit errors detected in `write_list_to_cpustate` in
> > target/arm/helper.c.

may be from the fact that I'm running on -max, which understandably may cause
changes in the ID registers between QEMU versions. I'm less concerned about
issues related to my usage of -max than I am about the others.

> > The third is that meanings of the bits in env->features (as defined by
> > `enum arm_features` in target/arm/cpu.h) has shifted. For example,
> > ARM_FEATURE_PXN, ARM_FEATURE_CRC, ARM_FEATURE_VFP, ARM_FEATURE_VFP3,
> > ARM_FEATURE_VFP4 have all been removed and ARM_FEATURE_V8_1M has been
> > added since 4.1.0. Heck, even I have added a field there in the past.
> > Unfortunately, these additions/removals mean that when env->features is
> > saved on one version and restored on another the bits can mean different
> > things. Notably, the removal of the *VFP features means that a snapshot
> > of a CPU reporting it supports ARM_FEATURE_VFP3 on 4.1.0 thinks it's now
> > ARM_FEATURE_M on 5.2.0!
> 
> Ow. I didn't realize the env->features was in the migration state :-(
> There is no reason for it to be, because it's a constant property
> of the CPU. The easy fix is to replace
>        VMSTATE_UINT64(env.features, ARMCPU),
> in target/arm/machine.c with whatever the syntax is for "ignore
> 64 bits of data here". Then we'll ignore whatever is coming in
> from the source, which we don't need, and we'll stop sending it
> out if we're the destination.

I'll look into this.

-Aaron