Detection of human emotions by analyzing the speech patterns in the audio signals has various applications including effective human-computer interaction, evaluation or automation of the customer care call quality assurance process, and medical applications such as assisting a doctor in understanding the emotional state of a patient. For instance, speech produced in a state of anger, fear, or happiness has higher and wider pitch and it is louder and faster whereas slow and low-pitched speech is produced when a person is sad. Recognition of emotion, gender and age from speech signals is one of the active research areas in speech processing where there is need for improvement in accuracy. The work proposed in this paper makes use of machine learning neural networks to recognize the gender and emotion of a speaker. The model is designed to classify four different emotions (neutral, happy, sad, angry). The work involves the integration of two modules namely, Gender identification block and Emotion detection block. A simple Feed forward neural network, tested and trained by a large dataset forms the gender identification block, while Convolutional Neural Network (CNN) with adequate datasets were used to train and test the emotion detection block. The proposed model results in detection of gender with 91.46 percent accuracy while the emotion detection module provides 86 percent accuracy.