This morning I suddenly discovered the technique for doing it more efficiently: clicking on the rewind button to go back to precisely the point I want to listen from, rather than using the slider and going back and forth all over the place.

I feel like an idiot now for not having thought of this before :) But, hey, suddenly everything's much faster!