You don't have to drop ice if you don't want to, but leave it there until you get to a much higher level. The cost of deleveling an entire mastery is huge, and it doesn't eat up your mastery levels until you actually get to the level cap. I find that frost nova is one of the most fun moves to use in PVP if they don't have ice immunity
I like to play the "freeze game" with people that piss me off. It goes:
Frost nova > Ice imbue and Cold Nuke > Three arrow combo > Cold Wave Arrest > Cold Nuke > Frost nova