It's entirely situational. I can bushwhack through the Heike, but if we talk about something I have no vocabulary for, I'm screwed. Also, the only dialects I can reliably understand are Kansai- and Kyoto-ben.

All of which is to say, ack! Fwiw, I feel the same way around people who can speak better than I do, such as some of my classmates. My accent in particular has been slipping, I need to try to remember to get it right when I speak.