The conventional image processing algorithms are not found to perform well in detecting crack problems. Generously the crack classification performance is also not clear with traditional deep learning neural network. To mitigate these issues, a convolutional neural network-based detection of bridge crack is presented in this work. The arous space pyramid pool (ASPP)-based feature extraction with depthwise separable convolution is modeled. With ASPP, the multiscale information of image features can be obtained, and the proposed convolution model provides large reception field so as to effectively fuse huge amount of contextual data on feature maps. Hence, the computational complexity of the model is greatly reduced. The results are verified in simulation which shows that the proposed method has achieved a highest detection accuracy of 96.68% which is higher than the conventional deep learning model.