Abstract: Based on breast ultrasound images, a deep learning automatic classification algorithm TDS-Net is studied, which is utilized to realize the five classification levels of 3, 4a, 4b, 4c and 5 in Breast Image Reporting and Data System (BI-RADS). TDS-Net designs a double branch structure. Firstly, the first branch is composed of the superposition of the DD Module proposed in this paper, which can reduce artifacts in ultrasonic images and extract rich local detail features. Secondly, the second branch is composed of convolution blocks, which is mainly used to extract the global feature information of the image as the information supplement of the first branch. Then, the two branches are fused to obtain a fused feature map containing rich feature information, and the information in the map is further extracted by deep separable convolution and SENet. Among them, deep separable convolution can reduce parameters and increase the nonlinearity of the network, to enhance its ability to extract features, and SENet attention mechanism can enhance the extraction of high-order feature information. To verify the algorithm, this paper utilizes the data provided by Yunnan Cancer Hospital for experiments. The results show that the accuracy, precision, and F1 values are 94.67%, 94.81%, and 94.69% respectively, which are higher than the comparison algorithm, reflecting the superiority of the algorithm. At the same time, to verify the robustness and universality of the algorithm, we experiment with benign and malignant binary classification based on two public datasets. The experimental results are also higher than the comparison algorithm. These results show that the proposed algorithm TDS-Net has a strong recognition ability for breast ultrasound images, and is expected to be applied in clinical medicine.