I just signed up for the devel list to chime in: I'm a user with a real use case for line numbers, and it's not a disease ;)
I have RSI in my hands. Myself and some other emacsers use speech recognition driving emacs to code. The latency for speech recognition is not great though, and so your best bet for speed is to utter a series of commands all at once, anticipating the outcome from doing all of those actions in sequence. For this it's essential to have *relative* line numbers, so that you can say things like, "up 5, delete 3, down 2". Which is pretty much the scenario you describe, it's just there's a reason to do it still in 2017.
So I'd welcome any new code that speeds up line number calculation, although it'd be nice if it had a relative mode. What about absolute line #s? I have no idea what those heretics use it for ;)
Joe G.