Do you mean the equalization would be performed with another linear filter?
matched filter -> equalizing filter -> clock recovery -> [...]
Wouldn't that be equivalent of convolving the input responses of the two cascaded FIRs, with the net result being an unmatched filter?
Existing implementations of this mode take that approach, using a filter which minimizes the ISI while attempting to be close to matched. I wrote about some of them at
https://ham.stackexchange.com/a/7744/218. I was hoping to try something new, at least for educational purposes.
I'm beyond what I comfortably know at this point, but if someone can confirm I'm on the right track I can continue researching the right things. I think part of the problem of the approach above is after filtering, the noise is no longer uncorrelated. Maybe the canonical solution is something like this?
matched filter -> equalizing filter -> clock recovery -> decimate to 1 sample per symbol -> "whitening filter" -> Costas loop -> Viterbi detector
I haven't yet studied whitening filters in depth, but my guess is it reverses the effects of the equalization filter so that the noise is again uncorrelated. At that point we're back to having ISI, but now with uncorrelated noise the Viterbi detector can deal with that optimally. Am I on the right track?