\set fontSize ignored?

[lily]

I'm trying to typeset some optional notes a bit smaller.
But Lily seems to ignore the font change size in this context.

Here's a sample:


----
Bassus = \relative c {
   << bes'2 
     { \context Voice = "foo" \set fontSize = #-5 g \unset fontSize } 
   >>  f4 d |
}

Tenor = \relative c' {
          d2 c4 d |
}

\score {
   \context Staff { \clef "F" <<
     \context Voice = Tenor {\voiceOne\Tenor}
     \context Voice = Bass {\voiceTwo\Bassus}
   >>
   }
}