Title:
ELECTRONIC APPARATUS FOR COMPRESSING LANGUAGE MODEL, ELECTRONIC APPARATUS FOR PROVIDING RECOMMENDATION WORD AND OPERATION METHODS THEREOF
Document Type and Number:
WIPO Patent Application WO/2018/164378
Kind Code:
A1
Abstract:
An electronic apparatus for compressing a language model is provided, the electronic apparatus including a storage configured to store a language model which includes an embedding matrix and a softmax matrix generated by a recurrent neural network (RNN) training based on basic data including a plurality of sentences, and a processor configured to convert the embedding matrix into a product of a first projection matrix and a shared matrix, the product of the first projection matrix and the shared matrix having a same size as a size of the embedding matrix, and to convert a transposed matrix of the softmax matrix into a product of a second projection matrix and the shared matrix, the product of the second projection matrix and the shared matrix having a same size as a size of the transposed matrix of the softmax matrix, and to update elements of the first projection matrix, the second projection matrix and the shared matrix by performing the RNN training with respect to the first projection matrix, the second projection matrix and the shared matrix based on the basic data.
Inventors:
YU SEUNG-HAK (KR)
KULKARNI NILESH (KR)
SONG HEE-JUN (KR)
LEE HAE-JUN (KR)
KULKARNI NILESH (KR)
SONG HEE-JUN (KR)
LEE HAE-JUN (KR)
Application Number:
PCT/KR2018/001611
Publication Date:
September 13, 2018
Filing Date:
February 06, 2018
Export Citation:
Assignee:
SAMSUNG ELECTRONICS CO LTD (KR)
International Classes:
G06F17/16; G06F17/30; G06F40/20; G06N3/08
Foreign References:
| US20170053646A1 | 2017-02-23 |
Other References:
XIANG LI ET AL.: "LightRNN: Memory and Computation-Efficient Recurrent Neural Networks", 30TH CONFERENCE ON NEURAL INFORMATION PROCESSING SYSTEMS (NIPS 2016, 31 October 2016 (2016-10-31), Barcelona, Spain, pages 1 - 9, XP055552615, Retrieved from the Internet
YUNCHUAN CHEN ET AL.: "Compressing Neural Language Models by Sparse Word Representations", IN: PROCEEDINGS OF THE 54TH ANNUAL MEETING OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS, 7 August 2016 (2016-08-07), Berlin, Germany, pages 226 - 235, XP055552619, Retrieved from the Internet
WENLIN CHEN ET AL.: "Strategies for Training Large Vocabulary Neural Language Models", IN: PROCEEDINGS OF THE 54TH ANNUAL MEETING OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS, 7 August 2016 (2016-08-07), Berlin, Germany, pages 1975 - 1985, XP055552622, Retrieved from the Internet
GRZEGORZ JURDZIDSKI: "Word Embeddings for Morphologically Complex Languages", SCHEDAE INFORMATICAE, vol. 25, 30 December 2016 (2016-12-30), pages 127 - 138, XP055552632, Retrieved from the Internet
HAKAN IRIAN ET AL., TYING WORD VECTORS AND WORD CLASSIFIERS: A LOSS FRAMEWORK FOR LANGUAGE MODELING, 4 November 2016 (2016-11-04)
SAINATH TARA N ET AL.: "ICASSP, IEEE INTERNATIONAL CONFERENCE ON", 26 May 2013, IEEE, article "Low-rank matrix factorization for Deep Neural Network training with high-dimensional output targets", pages: 6655 - 6659
JIAN XUE ET AL.: "Restructuring of Deep Neural Network Acoustic Models with Singular Value Decomposition", PROC. INTERSPEECH, 25 August 2013 (2013-08-25), pages 2365 - 2369, XP055193450
See also references of EP 3577571A4
YUNCHUAN CHEN ET AL.: "Compressing Neural Language Models by Sparse Word Representations", IN: PROCEEDINGS OF THE 54TH ANNUAL MEETING OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS, 7 August 2016 (2016-08-07), Berlin, Germany, pages 226 - 235, XP055552619, Retrieved from the Internet
WENLIN CHEN ET AL.: "Strategies for Training Large Vocabulary Neural Language Models", IN: PROCEEDINGS OF THE 54TH ANNUAL MEETING OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS, 7 August 2016 (2016-08-07), Berlin, Germany, pages 1975 - 1985, XP055552622, Retrieved from the Internet
GRZEGORZ JURDZIDSKI: "Word Embeddings for Morphologically Complex Languages", SCHEDAE INFORMATICAE, vol. 25, 30 December 2016 (2016-12-30), pages 127 - 138, XP055552632, Retrieved from the Internet
HAKAN IRIAN ET AL., TYING WORD VECTORS AND WORD CLASSIFIERS: A LOSS FRAMEWORK FOR LANGUAGE MODELING, 4 November 2016 (2016-11-04)
SAINATH TARA N ET AL.: "ICASSP, IEEE INTERNATIONAL CONFERENCE ON", 26 May 2013, IEEE, article "Low-rank matrix factorization for Deep Neural Network training with high-dimensional output targets", pages: 6655 - 6659
JIAN XUE ET AL.: "Restructuring of Deep Neural Network Acoustic Models with Singular Value Decomposition", PROC. INTERSPEECH, 25 August 2013 (2013-08-25), pages 2365 - 2369, XP055193450
See also references of EP 3577571A4
Attorney, Agent or Firm:
KIM, Tae-hun et al. (KR)
Download PDF:
Previous Patent: ELECTRONIC DEVICE FOR GENERATING MAP DATA AND OPERATING METHOD THEREFOR
Next Patent: SECONDARY BATTERY
Next Patent: SECONDARY BATTERY