Surface roughness prediction ensures high product quality, boosts manufacturing efficiency, and aids in effective tool life management. In this study, surface roughness optimization and prediction in the end milling of AISI H11 die steel were examined under three cooling conditions: dry milling, Minimum Quantity Lubrication (MQL), and Nano Fluid Minimum Quantity Lubrication (NMQL). The experiments were designed using a Taguchi L27 orthogonal array, with cutting speed, feed, and cooling environments as the variables. Surface roughness, the performance output parameter, was analyzed through Taguchi Signal-to-Noise (S/N) analysis. The dataset's diversity and robustness were further enhanced using the Gaussian Data Augmentation (GDA) technique, ensuring improved predictive accuracy of the Machine Learning (ML) models. Advanced machine ML models, including Decision Tree(DT), XGBoost (XGB), Support Vector Regression (SVR), CATBoost, AdaBoost Regression (ABR), and Random Forest Regression (RFR), were developed, with hyperparameters optimized using Grid Search Cross Validation. The ideal cutting parameters were identified as a cutting speed of 40 m/min, a feed rate of 0.01 mm/rev, and utilization of the NMQL cooling environment. The ML models, including DT, ABR, RFR, and CATBoost, demonstrate exceptional performance by achieving accuracy rates above 90 % and determinant coefficient (R2) greater than 0.9. Remarkably, the CATBoost model exhibited heightened precision, boasting 90.8 % accuracy, a R2 of 0.94, a mean absolute error (MAE) of 0.05, a mean squared error (MSE) of 0.005, a root mean squared error (RMSE) of 0.07, and a mean absolute percentage error (MAPE) of 9.17.