The safety, navigation, and decision-making capabilities of autonomous vehicles are all significantly influenced by accurate object detection. Recent advances in computer vision have resulted in models such as faster R-CNN, which offer robust feature extraction and bounding box predictions. While these models are successful, current limitations include occlusions, varying lighting conditions, and adverse weather that hinder their performance. XGBoost is a machine learning framework that has proven to be a great classification tool and can complement deep learning models. This paper proposes a novel ensemble approach which employs Faster R-CNN to extract features and provide region proposal and XGBoost to classify the regions. It was tested on a comprehensive dataset of 15,000 annotated images with 11 different object classes gathered for the Udacity Self-Driving Car testbed, offering a rich testbed for a wide variety of autonomous driving scenarios. It indicates real improvement in precision, recall, and mean average precision with performance over both standalone Faster R-CNN and XGBoost. This proposed ensemble model is capable of withstanding the possible challenging situations like occlusions and bad weather.