Re: [lmi] perf

[Greg Chicares]

On 2020-09-28 13:41, Vadim Zeitlin wrote:
> On Mon, 28 Sep 2020 12:02:19 +0000 Greg Chicares <gchicares@sbcglobal.net> 
> wrote:
> 
> GC> Perhaps chroots are just incompatible with 'perf'.
> 
>  I don't think so, a quick test using just chroot shows that it works fine.
> However using chroot requires being root, of course,

How interesting that you say "of course". [Spoiler alert: I'm glad you
did, because it leads to an important breakthrough below.]

The crucial question is: "of course" in what sense?
 - chroot(1)? Okay, sure--without being root, I get "Operation not permitted".
 - chroot(2)? The manpage says that requires CAP_SYS_CHROOT. But I'd rather
     avoid compiling a program for this if there's an easier way.
 - schroot(1)? Well, "no", kind of--that's its purpose. But here, "yes".
     IOW, the whole idea of schroot is to use chroots without being root,
     but running schroot as root confers useful extra powers in this case.

> so it's not really a
> solution. Maybe you could (temporarily) make /usr/sbin/chroot sgid root for
> some group and add yourself to it and then run it (without forgetting
> --userspec argument to also avoid being root in the chroot). This would be
> definitely hacky, but I don't see why it wouldn't work.

Favoring brutality over hackishness, at least in the experimental short term...

> GC> This works:
> GC> 
> GC> $perf record -e cpu-clock sleep 1                        
> GC> [ perf record: Woken up 1 times to write data ]
> GC> [ perf record: Captured and wrote 0.017 MB perf.data (3 samples) ]
> GC> $perf report --stdio             
> GC> 
> GC> (the output is valid, even though it has little to say), but this fails:
> GC> 
> GC> $perf record -e cpu-clock -- schroot -c bullseye0 sleep 1
> GC> [ perf record: Woken up 1 times to write data ]
> GC> [ perf record: Captured and wrote 0.017 MB perf.data ]
> GC> zsh: terminated  perf record -e cpu-clock -- schroot -c bullseye0 sleep 1
> 
>  Unfortunately, I have no idea what's going on here, perhaps I'm finally
> going to need to install schroot to really understand it. But this
> definitely doesn't seem normal or expected.

That's the crucial insight--I didn't know it was abnormal or unexpected,
and that emboldened me to try:

/home/greg[0]$su
[...password...]
/home/greg[0]#perf record -e cpu-clock sleep 1
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.017 MB perf.data (4 samples) ]
/home/greg[0]#perf record -e cpu-clock -- schroot -c bullseye0 sleep 1
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.024 MB perf.data (69 samples) ]

Wow--it works.

/home/greg[0]#perf record -- schroot -c bullseye0 -- sh -c 
'LD_LIBRARY_PATH=/opt/lmi/gcc_x86_64-pc-linux-gnu/build/ship 
/opt/lmi/gcc_x86_64-pc-linux-gnu/build/ship/lmi_cli_shared --accept 
--data_path=/opt/lmi/data --selftest' 
Test speed:
  naic, no solve      : 1.936e-02 s mean;      19104 us least of  52 runs
  naic, specamt solve : 3.370e-02 s mean;      33363 us least of  30 runs
  naic, ee prem solve : 3.097e-02 s mean;      30655 us least of  33 runs
  finra, no solve     : 7.698e-03 s mean;       7413 us least of 100 runs
  finra, specamt solve: 2.080e-02 s mean;      20415 us least of  49 runs
  finra, ee prem solve: 1.940e-02 s mean;      19035 us least of  52 runs
[ perf record: Woken up 4 times to write data ]
[ perf record: Captured and wrote 0.963 MB perf.data (24622 samples) ]
/home/greg[0]#perf report 2>&1 |less

There's a slight problem:

Failed to open /usr/lib/x86_64-linux-gnu/ld-2.29.so, continuing without symbols
Failed to open /usr/lib/x86_64-linux-gnu/libc-2.29.so, continuing without 
symbols
Failed to open /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28, continuing 
without symbols
Failed to open /opt/lmi/gcc_x86_64-pc-linux-gnu/build/ship/liblmi.so, 
continuing without symbols

     3.95%  lmi_cli_shared  liblmi.so            [.] 0x0000000000363378
     1.94%  lmi_cli_shared  libm-2.29.so         [.] 0x000000000001b1c9
     1.86%  lmi_cli_shared  libm-2.29.so         [.] 0x000000000001b1cf
     1.70%  lmi_cli_shared  libm-2.29.so         [.] 0x000000000001b1c5
     ...

Let's try to help it find debug symbols, by copying the libraries into a
local directory where they can do no harm [you foresee the outcome, but
I didn't]:

/home/greg[0]#pushd /tmp
/tmp /home/greg
/tmp[0]#mkdir perf-chroot-symbols
/tmp[0]#cd perf-chroot-symbols   
/tmp/perf-chroot-symbols[0]#cp -av 
/srv/chroot/bullseye0/opt/lmi/gcc_x86_64-pc-linux-gnu/build/ship/liblmi.so* .
'/srv/chroot/bullseye0/opt/lmi/gcc_x86_64-pc-linux-gnu/build/ship/liblmi.so' -> 
'./liblmi.so'
'/srv/chroot/bullseye0/opt/lmi/gcc_x86_64-pc-linux-gnu/build/ship/liblmi.so.map'
 -> './liblmi.so.map'

That's not successful: 'perf' still says, e.g.:
  Failed to open /opt/lmi/gcc_x86_64-pc-linux-gnu/build/ship/liblmi.so, 
continuing without symbols
but there's nothing vital in the host's /opt/lmi/ , so we can
experimentally pollute it without fear:

/tmp/perf-chroot-symbols[0]#cp -av 
/srv/chroot/bullseye0/opt/lmi/gcc_x86_64-pc-linux-gnu/build/ship/liblmi.so* 
/opt/lmi/gcc_x86_64-pc-linux-gnu/build/ship/        
'/srv/chroot/bullseye0/opt/lmi/gcc_x86_64-pc-linux-gnu/build/ship/liblmi.so' -> 
'/opt/lmi/gcc_x86_64-pc-linux-gnu/build/ship/liblmi.so'
'/srv/chroot/bullseye0/opt/lmi/gcc_x86_64-pc-linux-gnu/build/ship/liblmi.so.map'
 -> '/opt/lmi/gcc_x86_64-pc-linux-gnu/build/ship/liblmi.so.map'

And now 'perf' informs us:

     6.21%  lmi_cli_shared  liblmi.so                           [.] 
AccountValue::DecrementAVProportionally
     5.02%  lmi_cli_shared  liblmi.so                           [.] 
Irc7702A::DetermineLowestBft
     4.57%  lmi_cli_shared  liblmi.so                           [.] 
AccountValue::TxSetDeathBft
     2.83%  lmi_cli_shared  liblmi.so                           [.] 
AccountValue::ChangeSpecAmtBy
     2.76%  lmi_cli_shared  liblmi.so                           [.] 
AccountValue::TxCreditInt
     2.60%  lmi_cli_shared  liblmi.so                           [.] 
AccountValue::SurrChg
     1.98%  lmi_cli_shared  liblmi.so                           [.] 
AccountValue::DoMonthDR
     1.97%  lmi_cli_shared  libm-2.29.so                        [.] 
0x000000000001b1c9

[This is origin/master, so it doesn't look exactly like the valyuta/002
results you had shared earlier--but it's similar.]

So I guess 'perf' is doing something like 'ldd lmi_cli_shared', finding
shared objects relative to the chroot's '/', and trying to look them up
relative to the host's real '/'. If I were retired, I might work on a
'perf' patch to add a "--find-so-relative-to-chroot" option.

It still doesn't find libraries like 'libm-2.29.so' above (but we can
assume 0x000000000001b1c9 must be some transcendental like log1p()).
On the host system, we have:

/tmp/perf-chroot-symbols[0]#ls -l /usr/lib/x86_64-linux-gnu/libm-*
-rw-r--r-- 1 root root 1579448 May  1  2019 
/usr/lib/x86_64-linux-gnu/libm-2.28.so

and how dangerous would it be if I copied the chroot's 2.29 into
the host's directory? Stop me before I do something really foolish.

> GC> My next step will be to compile lmi on the host. I haven't
> GC> installed gcc there yet, but 'buster' has gcc-8.3.0-1, which
> GC> should be good enough for now. For std::filesystem, we'll need
> GC> gcc-10, which is in 'bullseye', and I'm sure it can't be too
> GC> hard to install that in 'buster'.
> 
>  I'm afraid it's going to be difficult, i.e. installing the new package
> will require upgrading so much that you will basically end up with a half
> Buster/half Bullseye system, and mostly with Bullseye, I think.

I'm so glad you said that. You probably saved me days of agony.
My 'buster' chroots are chimerical:
  $ cat /etc/os-release
  PRETTY_NAME="Debian GNU/Linux bullseye/sid"
and I don't care because I can easily regenerate them; but
I want to keep my host system stable.

What would be better? Compile gcc-10 from sources? Last I heard,
that took days, but that was probably a couple decades ago.
Or just upgrade my host to 'bullseye', though blind faith is not
in my nature?