Re: Feature Request Idea - Server Sent Musical Scores

[Ben Sturmfels]

Hi jgart,

Sounds like a really interesting idea! A bit like https://musescore.com/
but using video rather than a JS player that syncs the audio and
notation.

Is there anyone interested in doing some initial exploration on this?

(Just for context, I know a few readers are strongly against the use of
JavaScript in web applications for security reasons. I appreciate this
perspective, but that's *not* the reason we've been working to remove
jQuery and other third party JS. That work is specifically to make it
easier to package MediaGoblin as an OS package. I think it's useful if
MediaGoblin works acceptably without JS, but our aim is not to remove JS
outright, since its necessary to provide the interactivity that people
expect from web apps.)

Regards,
Ben

On Thu, 16 Sep 2021, hjenkins wrote:

> This sounds like an excellent feature.
>
> Apart from ly2video, lilypondToBandVideoConverter and a third approach called
> mkvideo seem to exist.
>
> I realize this feature might add a lot of complexity, probably in the way in
> which Lilypond renders scores, but it would be very nice to have a way of
> marking the portion of music currently playing. Say, highlighting the notes
> while they are being sounded, or having a cursor passing along the staff (at
> least hitting the end of each bar on-cue, maybe a ball bouncing along if it's
> too hard to interpolate correctly). This would be useful for teaching
> sight-reading of music, just as a pointer or finger is useful in learning to
> read text.
>
> And this is even more complex, but there would be a lot of utility in a method
> for controlling the speed (either a manual paging feature like W3C Slidy, or a
> continuous system that counts beats from a microphone or conductor's baton). 
> If
> you're having a music group run through a new piece, scrolling scores on a
> overhead display would save everyone from carrying around paper scores (which
> get lost and blow away and rained upon and so on) and music stands (which are
> cumbrous and fall over with a clatter). It would also improve attention to a
> conductor standing near the screen, and improve posture.
>
>
> Given that an increasing number of illiterate people are online with mobiles, 
> a
> simpler but parallel functionality might simply convert a text into a video 
> with
> a synthesized voice and scrolling text. It could mark the future, current, and
> past text in three different styles and colours, and optionally add concurrent
> visual media. Even if this only worked in English, it would aid literacy and
> access to information in places like India.
>
>
>
> On 2021-09-16 10:21, jgart wrote:
>> Hi Goblineers,
>> I think GNU MediaGoblin should sit down and enjoy a coffee with GNU 
>> LilyPond.
>> I wanted to share some thoughts and ask for feedback on the possibility 
>> of
>> adding support for musical scores?
>> I was thinking of integrating ly2video: 
>> https://github.com/aspiers/ly2video
>> ly2video converts lilypond files into a video containing a horizontally
>> scrolling musical staff which is synchronized with a MIDI-generated audio
>> rendering of the music.
>> A goblineer could upload their lilypond musical score (.ly file) to 
>> mediagoblin
>> and mediagoblin would send it to the backend, process it with ly2video, and
>> return a video of their score synchronized with the MIDI and horizontal
>> scrolling bar.
>> Here are some examples of the output of ly2video in action (invidious 
>> link):
>> https://youtube.076.ne.jp/playlist?list=PLfRwjd606WZlxRU_kaUPagX3-Uv-SYRMH
>> I imagine we could also add batch uploading of musical scores via the 
>> cli as
>> well as offer multiple formats of the score for download such as the original
>> source file (.ly), MIDI, pdf, svg, flac, ogg, etc...
>> We could even offer musical score transposition before sending to 
>> ly2video as
>> well as an alternative color coded static html score view courtesy of
>> python-ly:
>> https://github.com/frescobaldi/python-ly
>> python-ly is already packaged in guix and is used extensively by the
>> Frescobaldi
>> LilyPond IDE.
>> I think this would be an awesome feature to add and would attract 
>> attention for
>> GMG as I know of no other media forge that supports this.
>> Taking this approach would also not introduce any javascript 
>> dependencies since
>> the musical score creation is being strictly handled in the backend and
>> a video, html, etc... is produced and sent to the client.
>> WDYT?
>> https://invidious.io/
>> https://frescobaldi.org/