PyTorch is an open source machine learning library for Python, based on Torch, used for applications such as natural language processing. It is primarily developed by Facebook’s artificial-intelligence research group, and Uber’s “Pyro” software for probabilistic programming is built on it.
PyTorch provides two high-level features:
- Tensor computation (like NumPy) with strong GPU acceleration
- Deep Neural Networks built on a tape-based autodiff system
PyTorch Tensors
In terms of programming, Tensors can simply be considered multidimensional arrays. Tensors in PyTorch are similar to NumPy arrays, with the addition being that Tensors can also be used on a GPU that supports CUDA. PyTorch supports various types of Tensors.
Modules:
Autograd Module
PyTorch uses a technique called automatic differentiation. A recorder records what operations have performed, and then it replays it backward to compute our gradients. This technique is especially powerful when building neural networks in order to save time on one epoch by calculating differentiation of the parameters at the forward pass itself.
Optim Module
torch.optim is a module that implements various optimization algorithms used for building neural networks. Most of the commonly used methods are already supported, so there is no need to build them from scratch
nn Module
PyTorch autograd makes it easy to define computational graphs and take gradients, but raw autograd can be a bit too low-level for defining complex neural networks. This is where the nn module can help.
6 Neuronale Netze einfach in Python erstellen (In German )
In this course, participants will learn to easily and efficiently program neural networks in Python as well as train them on the graphics card.
This methodology is explained using 6 very different examples covering the most common types of networks:
- Simple feed-forward networks
- Handwriting recognition and image recognition with convolutional networks
- Name recognition with recurrent networks
- Password generation with recurrent networks
- Reinforcement learning for games
Previous knowledge in Python and the theory of neural networks are
It has many useful features, but at a high level:
- It has a first class tensor object similar to Numpy’s array. It allows for optimized storage, access and mathematical operations (dot product, convolution, etc.) on an array-like object. A Torch tensor comes with nice things like GPU support and broadcasting.
- It has an automatic differentiation engine, Autograd. With a simple, initialization flag, requires_grad=True a series of PyTorch operations can be differentiated with the .backward() method.
- It has a gradient-based optimization package with algorithms such as SGD, Adagrad, RMSProp, LBFGS
- It has support for multi-node training. This is important for large datasets where an engineer might want to shard training across many CPUs or GPUs.
- As of 1.0, it has a just-in-time compiler; a powerful tool for compiling python Torch code for faster network evaluation. Compiled models can be read by c++ as well as python for easy sharing of models.
- And the most popular component, Pytorch has a nice, high level neural network (deep learning) API. It supports a software paradigm that your code = your model. This is in contrast to previous deep learning libraries (Theano/Caffe) where your code was effectively written as a config file (e.g., a protobuf like object) and run in a separate VM.
For what it’s worth, TensorFlow has most of these properties as well. While the two libraries were quite different upon initial release, they seem to be converging in features and programming paradigm. Both now have eager mode, TF autograph seems similar to PyTorch’s JIT, both are investing in probabilistic programming libraries, both are available in AWS and GCE standard deep learning VMs, etc.
Tensorflow or PyTorch : The force is strong with which one?
So, since you’re reading this article, I’m going to assume you have started your deep learning journey and have been playing around for a while with artificial neural nets. Or maybe, you’re just thinking of starting. Whichever case it be, you find yourself in a bit of a dilemma. You have read about various deep learning frameworks and libraries and maybe two really stand out. The two most popular deep learning libraries: Tensorflow and PyTorch. And you can’t quite figure out what exactly is the difference. Fret not! I’m here to add one more article to the unending repository of the Internet. And maybe, help you get some clarity. Also, I’m going to make it easier and quicker for you, and give you just five points. Five points of comparison, no more. So, let’s begin!
Point #1:
While both Tensorflow and PyTorch are open-source, they have been created by two different wizards. Tensorflow is based on Theano and has been developed by Google, whereas PyTorch is based on Torch and has been developed by Facebook.
Point #2:
The most important difference between the two is the way these frameworks define the computational graphs. While Tensorflow creates a static graph, PyTorch believes in a dynamic graph. So what does this mean? In Tensorflow, you first have to define the entire computation graph of the model and then run your ML model. But in PyTorch, you can define/manipulate your graph on-the-go. This is particularly helpful while using variable length inputs in RNNs.
Point #3:
Tensorflow has a more steep learning curve than PyTorch. PyTorch is more pythonic and building ML models feels more intuitive. On the other hand, for using Tensorflow, you will have to learn a bit more about it’s working (sessions, placeholders etc.) and so it becomes a bit more difficult to learn Tensorflow than PyTorch.
Point #4:
Tensorflow has a much bigger community behind it than PyTorch. This means that it becomes easier to find resources to learn Tensorflow and also, to find solutions to your problems. Also, many tutorials and MOOCs cover Tensorflow instead of using PyTorch. This is because PyTorch is a relatively new framework as compared to Tensorflow. So, in terms of resources, you will find much more content about Tensorflow than PyTorch.
Point #5:
This comparison would be incomplete without mentioning TensorBoard. TensorBoard is a brilliant tool that enables visualizing your ML models directly in your browser. PyTorch doesn’t have such a tool, although you can always use tools like Matplotlib. Although, there are integrations out there that let you use Tensorboard with PyTorch. But it’s not supported natively.
Finally, Tensorflow is much better for production models and scalability. It was built to be production ready. Whereas, PyTorch is easier to learn and lighter to work with, and hence, is relatively better for passion projects and building rapid prototypes.
Alright enough! Just tell me which one is better?
There is no right answer.(I know, I hate it too when someone says that)
The truth is, some people find it better to use PyTorch while others find it better to use Tensorflow. Both are great frameworks with a huge community behind them and lots of support. They both get the job done. They both are amazing magical wands that will let you do some machine learning magic.
I hope I was able to help you in clearing your confusion(little bit, maybe?). And if you are really confused and haven’t used any of them yet, pick any and just start. You will develop more intuition which will help you decide.
If you are just beginning your deep learning journey, and want to learn how to build deep learning models(like CNNs, RNNs or GANs) in Tensorflow and Keras, try out this Deep Learning Nanodegree by Udacity.
And finally, these are just tools. You can pick any and start learning the science and art of machine learning.