In order to realize lip-to-speech conversion more conveniently, this paper proposed a method to generate speech directly from lip video and studied the related problems. Firstly, it calculated lip motion features and corresponding LPC10 speech coding parameters in video files synchronously, and then carried out an end to end training using the LSTM artificial neural network. Then the trained network model could map the lip motion features into LPC10 coded parameters, and finally converted the coded parameters into speech samples by the speech synthesis algorithm. The advantages of this method were that it was easy to obtain the training samples without time-consuming manual annotation, and also avoided the pronunciation ambiguity existing in the text-speech reconstruction method. The test results show that, in the applications under limited vocabularies, the proposed method can reconstruct clear and understandable speech from lip video.

[2020-06-05] Printed Article /doc/2018.11.0912/v2