[fluid-dev] FluidSynth backend for SDL_mixer

[James Le Cuirot]

Hi guys,

First, some good news. I've almost finished implementing a FluidSynth
backend for SDL_mixer. This will bring long-overdue SoundFont support
to SDL_mixer as its bundled version of Timidity is very old and doesn't
support SoundFonts. I just need some help to complete it.

I'm working with FluidSynth at the sequencer level. Fortunately, some
of the "native MIDI" code already present in SDL_mixer does the hard
work of turning the raw MIDI data into a list of event structs.

Because the fluid_event_t instances don't get scheduled when they are
created, I need to store the times somewhere. This struct does have a
time field but the struct implementation is hidden and the
fluid_event_get_time and fluid_event_set_time functions are private.
Wrapping this in another struct would be a PITA so I've cheated and
declared those private functions to make them available. It would be
useful if these functions weren't private. I think there are
potentially other situations where you might want to store the event
time without actually scheduling the event immediately.

That isn't the end of the story though because when I do finally
schedule the event with fluid_sequencer_send_at, the event's time
property gets overwritten. This is a problem because if the song loops,
I have to schedule the event again and what was previously a relative
time has now become an absolute time. I don't think it is necessary to
overwrite this property because the given event gets copied in the
_fluid_seq_queue_pre_insert function anyway. With a small adjustment,
the modification could just be made to the copy instead. For now, I'm
storing the time before calling fluid_sequencer_send_at and then
setting it again afterwards.

I haven't yet dealt with SYSEX events because I've never coded around
MIDI before and this area is a little more complicated so help would be
appreciated here. However, I've noticed that one of the other MIDI
backends deals with SYSEX tempo changes and there doesn't seem to be a
FluidSynth event function for that.

The basic tempo isn't right yet because I haven't got a clue what to
pass to fluid_sequencer_set_time_scale. I have the PPQN and I've tried
passing this as well as variations on this but nothing seems to work
right for both the games I'm testing, D1X-Rebirth and Rise of the Triad.

In the case of Rise of the Triad, there is an additional problem
because SDL_mixer is being opened at 11025Hz, which is below what
FluidSynth supports. As a result, FluidSynth outputs the sound at
22050Hz instead and it gets played at half the speed and pitch. I tried
simply changing the minimum rate in FluidSynth and that worked fine so I
am politely asking if you could make this change as many games are
sampled at 11025Hz.

D1X-Rebirth's pitch is also out but it's only slightly lower. If I
hadn't compared the game's output with standalone FluidSynth and
Timidity, I wouldn't have noticed. In this case, I'm not sure what the
problem is. It seems to be outputting at 44100Hz as it should. Any
ideas?

There is also the issue of sample format. SDL_mixer is supposed to be
able to output sound in signed/unsigned 8/16-bit litte/big integer
format. FluidSynth only seems to support signed 16-bit integer and
float format. The Timidity backend code has some functions to convert
from signed 32-bit integer to the aforementioned formats but they're
greek to me and I don't know whether they'd be useful here.

This is complicated by the fact that SDL_mixer is also supposed to
support multi-channel output. According to the docs, FluidSynth only
supports this when outputting in float format. It also only outputs as
stereo pairs but what if SDL_mixer requests mono? Or 5.1? Does 5.1
count as 2 stereo pairs or 3 and do I have to handle this in some
special way?

Sorry for the barrage of questions but I'm keen to get this going. :)
If you want to try this stuff for yourself, apply the attached patch to
the latest SDL_mixer and run some game with the SDL_SOUNDFONTS
environment variable pointing to some SoundFont. Spaces are allowed,
but semi-colons aren't as they are used to specify more than one
SoundFont. Note that I have only tested this stuff on Linux.

I'll post to the SDL mailing list with a couple of other questions
later but I'm CC'ing the main SDL_mixer authors now as I think they
deserve an early heads up.

Cheers,
James

sdl-mixer-fluidsynth.patch
Description: Text Data