Drowsiness detection is still an open issue, especially when detection is based on physiological signals. In this sense, light non-invasive modalities such as electroencephalography (EEG) are usually considered. EEG data provides informations about the physiological brain state, directly linked to the drowsy state. Electrocardigrams (ECG) signals can also be considered to involve informations related to the heart state. In this study, we propose a method for drowsiness detection using joint EEG and ECG data. The proposed method is based on a deep learning architecture involving convolutional neural networks (CNN) and recurrent neural networks (RNN). High efficiency level is obtained with accuracy scores up to 97% on validation set. We also demonstrate that a modification of the proposed architecture by adding autoencoders helps to compensate the performance drop when analysing subjects whose data is not presented during the learning step.