Gesture recognition enables human to communicate with machine and interact naturally without any mechanical devices. The ultimate aim of gesture recognition system is to create a system which understands human gesture and use them to control various other devices. This research focuses on gesture recognition system with a radial basis function network. The radial basis function network is a 3 layer network and trained with a radial basis function algorithm to identify the classes. The complete system is implemented on a Field Programmable Gate Array with image processing unit. The system is design to identify 24 American sign-language hand signs and also real time hand gesture signs. This combination leads to maximum recognition rate. The proposed system is very small due to FPGA implementation which is highly suitable for control of equipments at home, by the handicapped people.