Magnetic Resonance (MR) imaging is essential in clinical diagnostics due to its ability to capture detailed soft tissue structures. However, acquiring high-resolution MR images is expensive and often leads to reduced signal-to-noise ratios. To address this, MR image super-resolution aims to generate high-resolution images from low-resolution inputs. While deep neural networks have been widely applied for MR image superresolution, they struggle to effectively utilize structural information critical for accurate reconstruction. This paper introduces a novel Transformer-based framework for super-resolving T2-weighted MR image which is a critical MR imaging modality. This framework excels in leveraging both intra-modality and inter-modality dependencies to enhance the structural information. The innovative component of our proposed architecture is termed as High-frequency Structure Transformer (HFST) which operates on the gradients of input images, leveraging the high-frequency structure prior. It also employs high-resolution T1-weighted images which is a more efficient MR imaging modality to provide substantial inter-modality structure priors for the processing of low-resolution T2-weighted images. HFST is featured by parallel intra-modality and inter-modality context exploration and window-based self-attention modules. Notably, both intra-head and inter-head correlations are incorporated to build up the self-attention modules, amplifying the relation extraction capacity. Rigorous evaluations on three benchmarks including IXI, BraTS2018, and fastMRI reveal that our method sets a new state of the art in MR image super-resolution. Especially, our method increases the PSNR metric by up to 1.28 dB under the 4× super-resolution setting. Our codes are available at https://github.com/dummerchen/HFST.