Abstract: In this paper,a WMF-CNN (Convolutional neural network with weighted multi-feature fusion,WMF-CNN) model based on deep learning is proposed to solve the problem of the recognition on street view house number images in natural scene.The model adopts the idea of weighted multi-layer feature fusion.The PCA method is used to reduce the dimensions of each fusion feature map and then corresponding weights are computed according to their contributions to recognition results.The weighted feature maps representing detailed information are fused with global approximation information provided by the fully connected layer.Finally,the fused features are input to the SoftMax classifier to get a more reasonable recognition result.Our experimental results on SVHN dataset indicate that the proposed WMF-CNN model could be fully trained within 2.2 hours and achieve the recognition rates of 95.6%.Compared with some other methods or models,the suggested WMF-CNN model not only can obtain higher accuracy,but also may meet some the requirements of real-time applications since it takes an average of about 0.38 milliseconds to recognize an image.