The answer to this question cannot be found straightforwardly in the language of the poem. There is no direct indication (e.g. gender-related pronouns, references to sexual parts) that the addresser is male and the addressee female. Indeed, in these sexually liberated days, it is possible to imagine any combination (M:F; F:M, M:M; F:F). But our prototypical, or schematic assumptions tend to push us towards the first combination. The addressee wears a dress, and so to assume a male addressee means that we have to take on a rather specific extra assumption not indicated at all in the text, namely that the addressee is a cross-dresser. This is not impossible, of course, but is statistically unlikely. If we assume the addressee to be female then it is most likely that the addresser will be of the opposite sex, and again there is no specific information in the poem to counteract this schematic assumption. You could still come up with a gay and/or cross-dressing reading for the poem, of course. But the assumptions we bring along about how the world works in general terms makes such readings less likely.