Brain age estimation is crucial for identifying brain disorders and developing biomarkers. While deep learning methods have been proposed for high-accuracy age estimation from T1-weighted images, further improvements are still needed. This paper proposes a novel brain age estimation method called SQET-MoE, utilizing a two-stage estimation framework. In the 1st stage, the Squeeze-and-Excitation Transformer (SQET), which fuses the benefits of CNNs and Transformers, performs coarse age estimation and feature extraction. The 2nd stage employs a Mixture-of-Experts (MoE) module, which uses SQET's output to estimate and correct the residual associated with systematic errors and complex non-linearity. This task division stabilizes training and maximizes estimation accuracy. Through a set of experiments using large-scale datasets, we demonstrate that the proposed SQET-MoE achieves the highest estimation accuracy compared to conventional methods.