The tomato crop is more susceptible to disease than any other vegetable, and it can be infected with over 200 diseases caused by different pathogens worldwide. Tomato plant diseases have become a challenge to food security globally. Currently, diagnosing and preventing tomato plant diseases is a challenge due to the lack of essential methods or tools. The traditional techniques of detecting plant disease are arduous and error-prone. Utilizing precise or automatic detection methods in spotting early plant disease can improve the quality of food production and reduce adverse effects. Deep learning has significantly increased the recognition accuracy of image classification and object detection systems in recent years. In this study, a 15-layer convolutional neural network is proposed as the backbone for single shot detector (SSD) to improve the detection of healthy, and three classes of tomato fruit diseases. The proposed model performance is compared with ResNet-50, AlexNet, VGG 16, and VGG19 as the backbone for Single shot detector. The findings of the experiment showed that the proposed CNN-SDD achieved 98.87% higher detection accuracy, which outperformed state-of-the-art models.