Using hand gestures as a HCI modality introduces intuitive and easy-to-use interfaces for a wide range of applications. However, the hand is an object with a high number of degrees of freedom and with high similarities derived from the heterogeneities of possible gestures. Moreover, the online detection of a gesture as soon as it happens in a video stream is a very challenging problem. To address these difficulties, we introduce an effective deep learning based approach, which takes advantage of the combined description of the hand shape and its temporal variation. First, we employ a transfer learning strategy to learn coarse and fine hand features from depth image dataset originally created for hand pose estimation. Then, we model the temporal aspect separately of the hand poses and its shape variations over the time using recurrent models, before merging. Our approach achieve significant performance for the task of hand gesture detection and recognition. In online scenario, results show that our approach is able to detect an occurring gesture and to recognize it far before its end, making our system efficient for real-time interactive applications.