Re: [PATCH][RESEND v3 1/3] hapvdimm: add a virtual DIMM device for memory hot-add protocols

[Maciej S. Szmigiero]

On 1.03.2023 18:24, David Hildenbrand wrote:
(...)
> > With virtio-mem one can simply have per-node virtio-mem devices.
> >
> > 2) I'm not sure what's the overhead of having, let's say, 1 TiB backing
> > memory device mostly marked madvise(MADV_DONTNEED).
> > Like, how much memory + swap this setup would actually consume - that's
> > something I would need to measure.
>
> There are some WIP items to improve that (QEMU metadata (e.g., bitmaps), KVM 
> metadata (e.g., per-memslot), Linux metadata (e.g., page tables).
> Memory overcommit handling also has to be tackled.
>
> So it would be a "shared" problem with virtio-mem and will be sorted out 
> eventually :)

Yes, but this might take a bit of time, especially if kernel-side changes
are involved - that's why I will check how this setup works in practice
in its current shape.

(...)
> > > Reboot? Logically unplug all memory and as the guest boots up, re-add the 
> > > memory after the guest booted up.
> > >
> > > The only thing we can't do is the following: when going below 4G, we cannot 
> > > resize boot memory.
> > >
> > >
> > > But I recall that that's *exactly* how the HV version I played with ~2 years ago worked: always 
> > > start up with some initial memory ("startup memory"). After the VM is up for some 
> > > seconds, we either add more memory (requested > startup) or request the VM to inflate memory 
> > > (requested < startup).
> >
> > Hyper-V actually "cleans up" the guest memory map on reboot - if the
> > guest was effectively resized up then on reboot the guest boot memory is
> > resized up to match that last size.
> > Similarly, if the guest was ballooned out - that amount of memory is
> > removed from the boot memory on reboot.
>
> Yes, it cleans up, but as I said last time I checked there was this concept of 
> startup vs. minimum vs. maximum, at least for dynamic memory:
>
> https://www.fastvue.co/tmgreporter/blog/understanding-hyper-v-dynamic-memory-dynamic-ram/
>
> Startup RAM would be whatever you specify for "-m xG". If you go below min, you 
> remove memory via deflation once the guest is up.


That article was from 2014, so I guess it pertained Windows 2012 R2.

The memory settings page in more recent Hyper-V versions looks like on
the screenshot at [1].

It no longer calls that main memory amount value "Startup RAM", now it's
just "RAM".

Despite what one might think the "Enable Dynamic Memory" checkbox does
*not* control the Dynamic Memory protocol availability or usage - the
protocol is always available/exported to the guest.

What the "Enable Dynamic Memory" checkbox controls is some host-side
heuristics that automatically resize the guest within chosen bounds
based on some metrics.

Even if the "Enable Dynamic Memory" checkbox is *not* enabled the guest
can still be online-resized via Dynamic Memory protocol by simply
changing the value in the "RAM" field and clicking "Apply".

At least that's how it works on Windows 2019 with a Linux guest.

> > So it's not exactly doing a hot-add after the guest boots.
>
> I recall BUG reports in Linux, that we got hv-balloon hot-add requests ~1 
> minute after Linux booted up, because of the above reason of startup memory [in 
> these BUG reports, memory onlining was disabled and the VM would run out of 
> memory because we hotplugged too much memory]. That's why I remember that this 
> approach once was done.
>
> Maybe there are multiple implementations noways. At least in QEMU you could 
> chose whatever makes most sense for QEMU.

Right, it seems that the Hyper-V behavior evolved with time, too.

> > This approach (of resizing the boot memory) also avoids problems if the
> > guest loses hot-add / ballooning capability after a reboot - for example,
> > rebooting into a Linux guest from Windows with hv-balloon.
>
> TBH, I wouldn't be too concerned about that scenario ("hotplugged memory to a guest, guest reboots 
> into a weird OS, weird OS isn't able to use hotplugged memory). For virtio-mem, the important part was 
> that you always "know" how much memory the VM is aware about. If you always start with 
> "Startup memory" and hotadd later (only if you detected guest support after a bootup), you can 
> handle that scenario.

I'm not *that* concerned with cross-guest-type scenario either,
but if it can be made more smooth then I wouldn't mind.

Thanks,
Maciej

[1]: 
https://www.tenforums.com/performance-maintenance/38478-windows-10-hyper-v-dynamic-memory.html#post544905