In this work, application of machine learning model is used to estimate the thermal performance and friction factor of a cylindrical tube heat exchanger equipped with perforated conical ring turbulators. Ternary hybrid nanofluid is used as a coolant with appropriate volume fraction and proposition. Interestingly, it is noted that lowest and highest volume fraction of the coolants acts as similar in the case of friction factor. Therefore, the optimization of volume fraction of nanofluids is very crucial to estimate. For this purpose, three different machine learning models are employed: XGBoost (Extreme Gradient Boosting), Artificial Neural Network (ANN), and Recurrent Neural Network (RNN) model. The grid search algorithm enhances model performance exhaustively by searching for the best parameters for the XGBoost algorithm, thereby optimizing its performance. Several performance metrics, including correlation coefficient, mean absolute error (MAE), mean squared error (MSE), root mean square error (RMSE), and coefficient of determination, and are used to compare the performance of the three models. According to the MSE metric, XGBoost outperforms ANN by around 54.84% and RNN by about 22.22%. The proposed XGBoost model consistently performs better than ANN and RNN.