RE : RE : grub-emu state of the art

Hi folks,

This email to explain how I worked on grub-emu module loading, the choices I have made (for now) and the diff from grub-1.93 delivery. Marco asked me for details (well, that will make it a pretty long email :-)).

I worked only on the i386-pc architecture but I believe same steps will lead to a working grub-emu with module loading on powerpc architecture (though I can't do that myself).

I’ve got results : My modules are loading fine, but That may be by chance J

(I still have great doubts about memory management).

So take a close look at what I’ve done before agreeing/committing changes and yell if I’ve done anything foolish.

There is still work to do, as you’ll see :

Have it compile

:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

First, I had to deal with the main problem on PC:

Current delivery of grub-1.93 requires kernel and modules to be compiled with -mregparm=3 -mrtd options.

Parameters are passed to functions using registers instead of using memory in stack. Assembler code presume that asm functions are called that way too.

This is not the standard way libc functions are used to.

Also, note that even with -mregparm=3 and -mrtd options, parameters are passed using stack to functions with an undefined number of parameters (such as grub_printf : Those functions in turn expected their parameters in stack). If you don't provide good prototype to functions in your modules that leads to unexpected results.

To have a grub-emu compile with these options :

a. Either you need to compile a new "libc" library using same options

b. Either you can provide grub-emu source files with a "foolyou" prototype include file for all C functions called in libc (there are currently not too many), saying for example :

off_t lseek(int fildes, off_t offset, int whence, ...);

The ending ... will ask the C compiler to pass all parameters in stack (the way the libc wants it for all functions).

And try to compile (I really didn't try that, I was too afraid the compiler would voice me about mismatchs in prototype between Linux standard include files and my "foolme" prototype file.

I believe that things would be worst with time, since other modules to come would like to use more library calls (in libc and in other standard libs).

I'd rather made the following choices :

---------------------------------------

1. Provide new asm code that support for passing parameters in stack, instead of from registers.

2. Change compile options so that the all Grub 2 delivery would now be using passing parameters using stack (and thus I get ride of libc incompatibility)

3. Keep support for "optimized" code (using -mregparm=3 -mrtd) just in case someone feels like he can't live without it ;-).

New Makefile (i386-pc.mk) shows :

#COMMON_ASFLAGS = -nostdinc -fno-builtin -DREGPARM

#COMMON_CFLAGS = -fno-builtin -mrtd -mregparm=3 -m32

COMMON_ASFLAGS = -nostdinc -fno-builtin

COMMON_CFLAGS = $(CFLAGS) -m32

Using the first two lines does create optimized and shorter code for kernel and modules (but you won’t be able to launch grub-emu).

The difference is about 210316 bytes against 212684 (for all the modules I have) : 1 % shorter.

Using the last two lines, the good news is you can use same modules for grub 2 and for grub-emu, and it compiles fine.

Don’t forget to add the following lines also, to be sure grub-emu is compiled with the same options as the (COMMON) modules (that’s only for ease of mind) :

grub_emu_CFLAGS = $(COMMON_CFLAGS)

grub_emu_LDFLAGS = $(COMMON_LDFLAGS) $(LIBCURSES)

I provided a new kern/i386/pc/startup.S :

#ifdef REGPARM

#include "startup.S.regparm"

#else

#include "startup.S.memparm"

#endif

startup.S.regparm is the « old » startup.S file.

Startup.S.memparm is the new one passing parameters using stack for global functions called by C. That’s a “first trial” quickly written that is not fully optimized but seems to be bug free.

Had also to do little changes in normal/i386/pc/setjmp.S (easy to read from the diff I attached to this email) and I guess that this is the only (similar) change that will have to be made in powerpc related files.

Include a complete dynamic loader for grub-emu

:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

I didn’t want to use the standard libld.so to load modules :

a) modules are “relocated” ELF files, while dynamic loader on Linux deals with “shared” .so ELF files.

b) emulation means we better try to use same code than the one used in Grub 2.

So I just let included kern/dl.c to the grub-emu code.

Had to get ride of template functions in util/grub-emu.c such as

grub_arch_dl_check_header (void *ehdr)

grub_arch_dl_relocate_symbols (grub_dl_t mod, void *ehdr)

And use the real ones (for i386-pc !!) from kern/i386/dl.c.

I am also using grub_register_exported_symbols from Grub 2’s instead of a null function.

Beware : I didn’t know what to do about grub_arch_modules_addr (void)

so I let it return 0 not knowing the consequences. I guess I’ll have to come back to that point later on.

Makefiles : I don’t have Ruby running on my pc, so I provided conf/i386-pc.mk diff, but the main thing to be retained is, I just added two dependencies :

kern/i386/dl.c and symlist2.c to be compiled with grub-emu.

Have a Complete grub-emu (no undefined symbols when loading modules)

:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

Well, I guess I was very loosy on that part.

The main purpose of my work was to get grub-emu load modules, whether or not it’s useful somehow.

So I kind of “skip” that part and “gracefully” ;-) ended using a symlist2.c file that shows all the work that remains to do :

I added two “null” items (function / var), in util/misc.c (but they may better be in util/grub-emu.c) :

int grub_null()

{

return 0;

}

int grub_null_var = 0;

And I just “connected” every undefined symbol in grub-emu to those two items, using a “special” symlist2.c file (yes I can do such messy work) :

{"grub_vga_set_mode", grub_null},

{"grub_vga_get_font", grub_null},

{"grub_vbe_get_controller_info", grub_null},

{"grub_vbe_get_mode_info", grub_null},

{"grub_vbe_set_mode", grub_null},

{"grub_vbe_get_mode", grub_null},

{"grub_vbe_set_memory_window", grub_null},

{"grub_vbe_get_memory_window", grub_null},

{"grub_vbe_set_scanline_length", grub_null},

{"grub_vbe_get_scanline_length", grub_null},

{"grub_vbe_set_display_start", grub_null},

{"grub_vbe_get_display_start", grub_null},

{"grub_vbe_set_palette_data", grub_null},

I guess that vbe module won’t be of any use with that “fixup”.

In fact vbetest command will crash grub-emu.

But watch this :

{"grub_lower_mem", &grub_null_var},

{"grub_upper_mem", &grub_null_var},

{"grub_os_area_addr", &grub_null_var},

{"grub_os_area_size", &grub_null_var},

{"grub_linux_prot_size", &grub_null_var},

{"grub_linux_tmp_addr", &grub_null_var},

{"grub_linux_real_addr", &grub_null_var},

{"grub_linux_boot_zimage", grub_null},

{"grub_linux_boot_bzimage", grub_null},

{"grub_chainloader_real_boot", grub_null},

{"grub_multiboot_real_boot", grub_null},

Did you say grub-emu main purpose was to boot some kind of OS ?

LeftToDo : I guess the makefile for grub-emu has to be changed (a lot ?) and that emulation needs to be completed else have a look of what get me into this and tell me a better fixup.

Some functions are only (from I have seen in Grub 2 code) coded in kern/i386/pc/startup.S, and I can’t include that file into grub-emu.

So, either some of the useful functions have to be “relocated” in some other source file, either these functions needs to have a coding to be used in grub-emu (emulation mode) somewhere within util/.

Anyway, for now, modules are loading fine with this “fixup”.

Loading Modules

:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

I thought I will have to change memory allocation to allow for module loading.

Malloc is not what is expected to load code into memory and run it.

mmap is the normal way to do that under Linux.

In fact, one of my modules loads a sub-module in a very special way (specific dynamic loader), so I needed to code a new function :

Grub_mmap(int size) (that takes only one parameter for now) :

static char * heapstart=(char*) 0x01200000;

void *

grub_mmap (grub_size_t size)

{

void * heap;

heap=(void *)mmap(heapstart, size,

PROT_EXEC|PROT_READ|PROT_WRITE, MAP_ANON|MAP_PRIVATE, -1, 0);

heapstart+=size;

printf("Using mmap (size %d)\n", size);

return heap;

}

But surprisingly enough, I didn’t need to use it to load modules with kern/dl.c.

Beware : I am still wondering if I am not loading modules on top of the kernel code in grub-emu !?!, though it’s been a week that I am using loading module with grub-emu and didn’t crash yet.

Getting Crazy enough to fix some code in Grub 2, just in case.

:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

Well, at that point, every thing is setup to “work”.

But it didn’t at first trial.

Had numerous SIGVSEG failed and mess up in malloc/free grub’s routines.

I tried to debug, finding out where the bug would be located, and finally used a different compiler (went to gcc-4.0), but kept some changes in grub 2’s code just in case (not knowing if these changes really matter for your compiler).

The mess came from two major points of the code (from what I’ve found out) :

Call to (mod->init) (mod); to initiate modules after loading

Well, I guess that this is important, especially when you compile with –mregparm=3 –mrtd to specify if the function you are calling takes a fixed number of parameters (to be passed by registers) or an undefined number of parameters (to be passed in the stack). If you don’t specify that, you let the compiler choose his way of doing it … and that might well be the wrong way. The function you called should (has to) be prototyped exactly the same.

It’s crucial for this line of code, because the function you call relies on the fact the parameter mod is a pointer to a module structure.

This structure doesn’t contain a MAGIC number (I believe it should !) to check on, before doing anything foolish such as … doing things on mod->dep.

I fixed :

static void grub_mod_init (grub_dl_t mod) __attribute__ ((used));

(I suppressed __attribute__ “unused” on parameter mod)

I gave a patch for this kind of call :

typedef void FINIT(void * );

typedef FINIT * PFINIT;

((PFINIT)mod->init) (mod);

(just in case, though the compiler should have correctly guessed it from struct grub_dl).

but anyway that sort of patch should be applied everywhere you make a *fct() call to something if *fct is not properly prototyped in include files.

Call to setjmp(grub_exit_env)

In some cases (I didn’t try to reproduce since I’ve moved to gcc 4.0) the setjmp function received *grub_exit_env instead of grub_exit_env. So I’ve coded this patch :

if (grub_setjmp (&grub_exit_env[0]) == 0)

instead of

if (grub_setjmp (grub_exit_env) == 0)

though it is less portable.

Up to you to see if you keep these “patches”. I didn’t tried to reproduce the errors without those patches since I went to gcc 4.0.

Some other changes

:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

Added a grub_break() function : That’s temporary to let me set easy breakpoints with gdb, from modules.

Added grub_dprintf("symbols", …) lines to let me see the loading process and have a table of which section is loaded at where.

I’d like to talk about how to let gdb know about newly loaded modules, but I’ll answer to Tomas 'Ebi' Ebenlendr’s email separately.

I don’t like the diff format I sent, so don’t hesitate to give me the right command to send you a diff (“patch”) with the format you like best.

Best regards,

_________________________________________

Eric Salomé - Paris, France

> -----Message d'origine-----

> De : address@hidden [mailto:grub-devel-

> address@hidden De la part de Marco Gerards

> Envoyé : vendredi 5 mai 2006 13:39

> À : The development of GRUB 2

> Objet : Re: RE : grub-emu state of the art

> Eric Salomé <address@hidden> writes:

> > GRUB 2 Wiki

> > ToDoList : Implement ModuleLoading in grub-emu

> Right, and I even added this item. Sorry, I just misunderstood your

> previous email.

> > I worked on that : Architecture i386/pc - Emulation on Linux/Ubuntu

> > grub-emu is loading all current modules, though some won't do anything

> > in emulation mode. The modules .mod are the same as those used for boot

> > time.

> Nice!

> > Little changes in source code, though I had to adapt asm files to

> > passing parameters in stack (I compile without -mregparm=3 -mrtd options

> > as I didn't want to recompile the whole C library with those passing

> > parameters conventions).

> It would be nice to get this to work for the regular modules too. In

> that case we do not need two set of modules. Besides that, it should

> also work on the PPC, for example. Which is a lot harder.

> BTW, how did you implement this? I once wrote a hack to do exactly

> the same by using mmap to allocate executable memory. On the PPC this

> memory needs to be allocated close to the grub-emu binary because

> otherwise the relocations would fail.

> > ... when I be done with grub 2 debugging I'll just have to recompile

> > with these options (if you like them a lot) and it will work just fine

> > (boot time only).

> It's not only that we like them. They make the size of the binaries

> smaller. And it is a requirement for the assembler code.

> > ToBeAwardedOf:

> > Don't use gcc 3.3 with grub 2's current source files : compiled code is

> > wrong and subject to unattended results.

> > Runs fine with gcc 4.0. I have still a doubt about gcc 3.4.

> Wrong in what way? We have some workaround for specific gcc bugs.

> Although these are sometimes hard to locate, sometimes it might be

> worth the effort.

> > LeftToDo :

> >

> > - see if every modification is useful, as I tried a few things before I

> > got a good result, and send a diff patch.

> Please describe how things work in detail so we can discuss possible

> issues.

> > - I need a "patched" symlist.c to compile with grub-emu (could be

> > generated by a script, but I am too lazy).

> :-)

> > - let gdb know about dynamically loaded modules ;-), but I guess it

> > won't be the easy part.

> Yeah, I can imagine that. But in case specific modules have to be

> debugged, we can just compile them in as happens currently. This is

> what I do now.

> The big advantage of having module support in grub-emu is as a

> debugging tool for module loading itself. It will make porting GRUB

> to another architecture a lot easier. And besides that, debugging

> will be a lot easier too.

> Thanks,

> Marco

> _______________________________________________

> Grub-devel mailing list

> address@hidden

> http://lists.gnu.org/mailman/listinfo/grub-devel

From:	Eric Salomé
Subject:	RE : RE : grub-emu state of the art
Date:	Tue, 9 May 2006 07:51:32 +0200