[RFC PATCH] configure: Enable -fno-omit-frame-pointer for backtrace module

[Glenn Washburn]

The backtrace module is written assuming that the frame pointer is in %ebp.
By default, -Os optimization level is used, which enables the gcc option
-fomit-frame-pointer. This break the backtrace functionality. Enabling
this may cause an unnoticeable performance cost and virtually no size
increase.

Signed-off-by: Glenn Washburn <development@efficientek.com>
---
The backtrace command on x86_64 and probably i386 is broken due to the
above rationale. I've not verified, but presumably the backtrace that used
to be printed for an unhandled CPU exception is also broken. Do any distros
handle this?

Considering that (to my knowledge) no one has complained about this in the
over 13 years that -Os has been used, has this code actually been useful?
Is it worth disabling -fomit-frame-pointer? Though, I don't see much downside
right now in disabling it. Alternatively, we could disable/remove the
backtrace code. I think it would be nice to keep it and have it working.

Nowadays, presumably QEMU makes the GDB stub rarely used as I imagine most
are developing in a virual machine. Also, the GDB stub does not work in UEFI
so if anyone is using it on real hardware, they are doing so on pretty old
machines. The lack of a GDB stub does not seem to be a pain point because
no one has got it working on UEFI.

This patch gets the backtrace command working on x86_64-efi in QEMU for me.
However, it hangs when run on my laptop. Not sure what's going on there.

Also, I've noticed that recents builds have caused a big slow down in the
already slow LUKS unlock code on real hardware, but not noticeable in QEMU.
It seems to occur with and without this patch. I mention this if others
would be interested in testing this.

Glenn
---
 configure.ac | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/configure.ac b/configure.ac
index 278e5a81f805..545cf69c272a 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1020,6 +1020,19 @@ if test x"$target_cpu" = xsparc64 ; then
   TARGET_LDFLAGS="$TARGET_LDFLAGS $grub_cv_target_cc_mno_relax"
 fi
 
+# The backtrace module relies on frame pointers and the default optimization
+# level, -Os, omits them. Make sure they are enabled.
+AC_CACHE_CHECK([whether -fno-omit-frame-pointer works], 
[grub_cv_cc_fno_omit_frame_pointer], [
+  CFLAGS="$TARGET_CFLAGS -fno-omit-frame-pointer"
+  AC_COMPILE_IFELSE([AC_LANG_PROGRAM([[]], [[]])],
+      [grub_cv_cc_fno_omit_frame_pointer=yes],
+      [grub_cv_cc_fno_omit_frame_pointer=no])
+])
+
+if test "x$grub_cv_cc_fno_omit_frame_pointer" = xyes; then
+  TARGET_CFLAGS="$TARGET_CFLAGS -fno-omit-frame-pointer"
+fi
+
 # By default, GCC 4.4 generates .eh_frame sections containing unwind
 # information in some cases where it previously did not. GRUB doesn't need
 # these and they just use up vital space. Restore the old compiler
-- 
2.34.1