Reading the headnote, I suppose the flowers are actually the one lady in particular, and the poem is a playful explanation for his going so far off the beaten path--my dictionary has 'avoid', or 'walk along' when it's specifically used with michimo, so I'd say

I don't walk along the road

or

I avoid the road

which without the headnote is a little opaque, and not much different from your version except for the potential ('cannot') connotation.