You're right, there is such a feature, but i had it unchecked anyway. It must be something else.
I've tried some of the rotation settings in MAME too, but they only force the info window (press ok to continue) to load normally, the game is still played sideways.
D: