In the poem 'For Anne Gregory', the 'great honey-coloured ramparts at your ear' refers to Anne Gregory’s beautiful golden hair. The young man means that her physical beauty, particularly her hair, is so attractive that it captivates and distracts men. He says young men are 'thrown into despair' because their admiration is superficial, focused solely on her external appearance rather than her true inner self.