Over recent decades, the Southern Ocean (SO) has experienced multi-decadal surface cooling despite global warming. Earlier studies have proposed that recent SO cooling has been caused by the strengthening of surface westerlies associated with a positive trend of the Southern Annular Mode (SAM) forced by ozone depletion. Here we revisit this hypothesis by examining the relationships between the SAM, zonal winds and SO sea-surface temperature (SST). Using a low-frequency component analysis, we show that while positive SAM anomalies can induce SST cooling as previously found, this seasonal-to-interannual modulation makes only a small contribution to the observed long-term SO cooling. Global climate models well capture the observed interannual SAM-SST relationship, and yet generally fail to simulate the observed multi-decadal SO cooling. The forced SAM trend in recent decades is thus unlikely the main cause of the observed SO cooling, pointing to a limited role of the Antarctic ozone hole.