Leveraging NLP and Deep Learning for Document Recommendations in the Cloud

Download Slides

Efficient recommender systems are critical for the success of many industries, such as job recommendation, news recommendation, ecommerce, etc. This talk will illustrate how to build an efficient document recommender system by leveraging Natural Language Processing(NLP) and Deep Neural Networks (DNNs). The end-to-end flow of the document recommender system is build on AWS at scale, using Analytics Zoo for Spark and BigDL. The system first processes text rich documents into embeddings by incorporating Global Vectors (GloVe), then trains a K-means model using native Spark APIs to cluster users into several groups.

The system further trains a recommender model for each group, and gives an ensemble prediction for each test record. By adopting the end-to-end pipeline of Analytics Zoo solution, we saw about 10% improvement of mean reciprocal ranking and 6% of precision respectively compared to the search recommendations for a job recommendation study.


Try Databricks
See More Spark + AI Summit in San Francisco 2019 Videos

« back
About Guoqiong Song

Guoqiong Song is a senior deep learning software engineer of the big data technology team at Intel. She has a PhD degree in atmospheric and oceanic sciences from UCLA, with a focus on numerical modling and optimization. Her interest is in developing and optimizing distributed deep learning algorithms on spark