Google Relies on GPUs for Deep Learning
Were you at the GPU Technology Conference in April? One of the featured presentations was called Large-Scale Deep Learning for Building Intelligent Computer Systems, given by Jeff Dean, a Senior Fellow at Google. Since interest has been steadily increasing in our GPU optimized servers and GPU writ large, we are featuring posts this month about how GPUs are being used. Below is a synopsis of Mr. Dean’s presentation.
What Is Deep Learning in Relation to Neural Networks?
Deep learning is basically the reincarnation of artificial neural networks, which have been around for 30 years – and which are loosely based on how the brain works. The idea behind deep learning is that you want a collection of layered abstractions which form simple trainable mathematical units and those units collaborate to compute some really complicated function.
Google’s Use of Large Scale Deep Learning
Google’s use of large scale deep learning over the past few years initially started focusing on how to build more intelligent computer systems to minimize software engineering effort. Google prefers deep learning algorithms where the algorithms themselves build higher and higher levels of understanding with the goal of the algorithms themselves learning from raw data.
Google’s goal is to solve difficult problems for new and existing products with deep learning technology. In order to do this, the machine needs to perceive and understand the world. Google – maybe more than any other entity – has tons of data including: text in webpages, frames in videos, images, billions of hours of video, thousands of hours from Google speech recognition; user search data patterns, etc. These really all boil down to access to the raw data for:
- basic speech & language capabilities
- language understanding
- user behavior prediction
- ability to interact with environment
Why GPUs Are a Perfect Fit for Deep Learning
Typically, small batches of examples are used in an algorithm; but by running batches through the network simultaneously, Google turns matrix-vector operations for a single example into matrix-matrix operations, which is nicely suited for GPUs. GPUs allow Google to get high performance out of modern computer systems.
GPUs enable dense matrix operations which helps train big neural networks (and deep learning) quickly. In the same way a GPU card is highly parallel inside, Google wants to spread that computation across multiple computational devices via model parallelism. data parallelism, and independent experiments.
Click here to see the full video giving an even more in-depth view of how Google uses deep learning and GPUs.
