We propose Youtube video classification at scale, leveraging a dataset of over 8 million videos
with high throughput computing and large data handling system design. The project requires training
on 1.5 terabytes of video and audio files to then predict a video type from several thousand labels. Our
solution uses two bidirectional-LSTM networks, one for audio and one for video, trained on a Spark-based
sitting atop Hadoop YARN and enabled by Elephas: a library that enables training Tensorflow models
in Spark. The infrastructure used consists of a custom cluster of m4.xlarge instances on AWS using Amazon
Machine Images to spin-up nodes with the correct software dependencies. We found this approach produced
reliable systemic speedup and effective classification with our final model.
We have learned many lessons through this project, especially in provisioning our custom Elastic Map Reduce
cluster, as well as in building various software dependencies: Tensorflow-Spark connector for reading
TFRecord files, Amazon-flavored Linux, Apache Maven, and getting all of our various components to be
version-compatible (e.g. Scala, Java versions with PySpark, Tensorflow-Spark and Amazon Linux versions).
We faced many issues in provisioning the cluster as a result of the growing complexity that so many software
dependencies adds to the problem, especially when there are non-backward-compatible components (e.g. PySpark
with Scala).
In the future, we would recommend not using Amazon Machine Images (AMIs) for provisioning, and instead bundling
the Python-based dependencies into jar files (a method which we found was possible only after the fact). This
would certainly save much time as there are many specific "gotchas" that are required for tackling the AMI
method successfully (EBS volume configuration, must be Amazon Linux, etc.). On the positive side, we found that
asynchronous weight updates can be parallelized highly effectively, and we actually achieved superlinear speedup
with this method. Future work may involve training more complex models (e.g. multiple input models), or
implementing alternative parallel Stochastic Gradient Descent schemes (e.g. Baidu's proposed Ring AllReduce).
We hope that you can learn from our work and the particular method we employed to deploy Tensorflow model training
on a massive dataset by utilizing Spark sitting atop a Hadoop-based elastic cluster. Please check out the source
code here.