BigDL Movie Recommendation System 🎬

Authors: Akshay Bahadur and Ayush Agarwal

The following blog post is part of Machine Learning in Production (17634) coursework at Carnegie Mellon University. In this discussion, we will consider the scenario of Movie Recommendation with respect to the following two frameworks.


Now that we have mentioned the resources and formed a basic blog premise let’s talk about some standard definitions.

BigDL in the Spark Framework

Problem Statement ✨

Movie recommendations are challenging because we can approach it in multiple ways, especially since we have numerous data points that can be modeled for the recommendation. Let’s try to analyze some of the problems.

Approaches 🎍

Although the problem discussed above is tricky, there have been multiple approaches in the past to address the issue. Let’s briefly go through some of them.

BigDL 🧱

Now that we have discussed the problem and the approaches for the movie recommendation, it is apt to discuss the framework to implement the strategy.

BigDL Movie Recommendation System

BigDL is used in building Large-Scale AI Applications for distributed big data. As part of the BigDL framework, multiple modules handle specific tasks

Distributed Deep Learning using BigDL

In this blog, we will be discussing BigDL in general and specifically about Orca (wherever required).

“More Data = More Compute Resources = More Cost”

Features of BigDL 🦄

Movie Recommendations with BigDL 🍿

BigDL is useful for processing large datasets and doing large-scale distributed training and inference. BigDL can be useful for handling the movie recommendation scenario when we try to scale our movie recommendation system.

For more details on the source code, visit the GitHub Source


Installing java
Installing bigdl-orca
Setting the execution environment


We have used the ‘MovieLens 1 million’ dataset for the assignment.


For the modeling, we have used Keras BigDL abstraction.

Deep Network for Movie Recommendation

Compilation Details


Prediction and Caching

Caching the Recommendations


For more details on the results, visit the BigDL Movie Recomendation Web App

We encourage you to check out the result yourself in our BigDL Movie Recommendation Web App. The web application is built and deployed using Streamlit.

Generating Insights
Generating Recommendations

Advantages 💎

Speed Comparision b/w PyTorch and BigDL

Limitations 🎲

Conclusion 🎮

BigDL is a distributed deep learning framework developed and open-sourced by Intel that can be used easily for developing and deploying an AI system. In this blog, we have given a brief introduction and discussed its strength and limitations. We have also discussed its usage in general and with respect to building a recommendation system.

Made with ❤️ and 🦙 by Akshay and Ayush

Team Ensemblers

References 🔱



Search & ML @ Yahoo!

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store