This paper addresses the challenge of developing autonomous control systems for various vehicles and devicesspecifically spacecraft landing procedures. Precise planetary landing is a classical control problem in aerospace engineering, which requires that the spacecraft decelerates/repels, orients and lands safely under uncertain dynamics with limited fuel. Conventional control techniques like PID and model predictive control need good models and substantial tuning. In this paper we explore the ability of recent reinforcement learning (RL) algorithms to learn such control policies directly from interaction with a physics simulator. We compare our approach against four popular algorithms - Deep Q-Network (DQN), Double DQN, Dueling DQN and Proximal Policy Optimization (PPO) on OpenAI Gymnasium and LunarLander-v2 benchmark. We present a unified PyTorch implementation, conduct multi-seed experiments and compare in terms of convergence speed, stability of rewards and computation efforts. We observe that PPO achieves competitive rewards in a quarter of training time compared to value-based methods, while the best peak reward is obtained by Dueling DQN. We discuss for real world autonomous landing and other high risk robotic tasks.