Manipulating digital images to present a false version of reality is known as image forgery. This modification is performed to deceive the viewers by removing, adding, or altering the elements in an image. Advanced Deep Learning (DL) approaches have been extensively used for detecting image forgery in recent years, as they enable automated analysis of subtle inconsistencies and complex patterns in images. These approaches, however, frequently experience overfitting issues and tend to miss detections due to a significant number of false negatives. Therefore, a new method called the Taylor Loss Xception Network (TayLoXNet) is introduced to detect image forgery. Firstly, the accumulated image is preprocessed by employing a median filter. After extracting the relevant features, the TayLoXNet method is employed to detect image forgery. In this approach, TayLoXNet is implemented by modifying XceptionNet’s learning rule with the Taylor Softmax Mean Square (TaylorSMS) loss. Besides, the TaylorSMS loss function is developed by merging the Mean Squared Error (MSE) and SoftMax loss using the Taylor series. Furthermore, the newly devised TayLoXNet method is evaluated using maximal True Negative Rate (TNR), accuracy, and True Positive Rate (TPR) and obtained superior values of 97.227%, 97.366%, and 98.357%, respectively.