Re: constant_tsc support for SVM guest

[Marcelo Tosatti]

On Sun, Apr 25, 2021 at 12:19:11AM -0500, Wei Huang wrote:
> 
> 
> On 4/23/21 4:27 PM, Eduardo Habkost wrote:
> > On Fri, Apr 23, 2021 at 12:32:00AM -0500, Wei Huang wrote:
> > > There was a customer request for const_tsc support on AMD guests. Right 
> > > now
> > > this feature is turned off by default for QEMU x86 CPU types (in
> > > CPUID_Fn80000007_EDX[8]). However we are seeing a discrepancy in guest VM
> > > behavior between Intel and AMD.
> > > 
> > > In Linux kernel, Intel x86 code enables X86_FEATURE_CONSTANT_TSC based on
> > > vCPU's family & model. So it ignores CPUID_Fn80000007_EDX[8] and guest VMs
> > > have const_tsc enabled. On AMD, however, the kernel checks
> > > CPUID_Fn80000007_EDX[8]. So const_tsc is disabled on AMD by default.
> > 
> > Oh.  This seems to defeat the purpose of the invtsc migration
> > blocker we have.
> > 
> > Do we know when this behavior was introduced in Linux?
> 
> This code has existed in the kernel for a long time:
> 
>   commit 2b16a2353814a513cdb5c5c739b76a19d7ea39ce
>   Author: Andi Kleen <ak@linux.intel.com>
>   Date:   Wed Jan 30 13:32:40 2008 +0100
> 
>      x86: move X86_FEATURE_CONSTANT_TSC into early cpu feature detection
> 
> There was another related commit which might explain the reasoning of
> turning on CONSTANT_TSC based on CPU family on Intel:
> 
>   commit 40fb17152c50a69dc304dd632131c2f41281ce44
>   Author: Venki Pallipadi <venkatesh.pallipadi@intel.com>
>   Date:   Mon Nov 17 16:11:37 2008 -0800
> 
>      x86: support always running TSC on Intel CPUs
> 
> According to the commit above, there are two kernel features: CONSTANT_TSC
> and NONSTOP_TSC:
> 
>   * CONSTANT_TSC: TSC runs at constant rate
>   * NONSTOP_TSC: TSC not stop in deep C-states
> 
> If CPUID_Fn80000007_EDX[8] == 1, both CONSTANT_TSC and NONSTOP_TSC are
> turned on. This applies to all x86 vendors. For Intel CPU with certain CPU
> families (i.e. c->x86 == 0x6 && c->x86_model >= 0x0e), it will turn on
> CONSTANT_TSC (but no NONSTOP_TSC) with CPUID_Fn80000007_EDX[8]=0.
> 
> I believe the migration blocker was created for the CONSTANT_TSC case: if
> vCPU claims to have a constant TSC rate, we have to make sure src/dest are
> matched with each other (having the same CPU frequency or have tsc_ratio).
> NONSTOP_TSC doesn't matter in this scope.
>
> > > I am thinking turning on invtsc for EPYC CPU types (see example below). 
> > > Most
> > > AMD server CPUs have supported invariant TSC for a long time. So this 
> > > change
> > > is compatible with the hardware behavior. The only problem is live 
> > > migration
> > > support, which will be blocked because of invtsc. 

It should be blocked, if performed to a host with a different frequency
or without TscRateMsr, if one desires the "constant TSC rate" meaning
to be maintained.

> > > However this problem
> > > should be considered very minor because most server CPUs support 
> > > TscRateMsr
> > > (see CPUID_Fn8000000A_EDX[4]), allowing VMs to migrate among CPUs with
> > > different TSC rates. This live migration restriction can be lifted as long
> > > as the destination supports TscRateMsr or has the same frequency as the
> > > source (QEMU/libvirt do it).
> > > 
> > > [BTW I believe this migration limitation might be unnecessary because it 
> > > is
> > > apparently OK for Intel guests to ignore invtsc while claiming const_tsc.
> > > Have anyone reported issues?]

Not as far as i know.

Fact is that libvirt will set the TSC_KHZ (from the value of
KVM_GET_TSC_KHZ ioctl).

That could be done inside QEMU itself, maybe by specifying -cpu
AAA,cpu-freq=auto ?

https://www.spinics.net/linux/fedora/libvir/msg141570.html