English | MP4 | AVC 1280×720 | AAC 44KHz 2ch | 265 lectures (57h 17m) | 21.9 GB

Master deep learning in PyTorch using an experimental scientific approach, with lots of examples and practice problems.

Deep learning is increasingly dominating technology and has major implications for society.

From self-driving cars to medical diagnoses, from face recognition to deep fakes, and from language translation to music generation, deep learning is spreading like wildfire throughout all areas of modern technology.

But deep learning is not only about super-fancy, cutting-edge, highly sophisticated applications. Deep learning is increasingly becoming a standard tool in machine-learning, data science, and statistics. Deep learning is used by small startups for data mining and dimension reduction, by governments for detecting tax evasion, and by scientists for detecting patterns in their research data.

Deep learning is now used in most areas of technology, business, and entertainment. And it’s becoming more important every year.

How does deep learning work?

Deep learning is built on a really simple principle: Take a super-simple algorithm (weighted sum and nonlinearity), and repeat it many many times until the result is an incredibly complex and sophisticated learned representation of the data.

Is it really that simple? mmm OK, it’s actually a tiny bit more complicated than that but that’s the core idea, and everything else — literally everything else in deep learning — is just clever ways of putting together these fundamental building blocks. That doesn’t mean the deep neural networks are trivial to understand: there are important architectural differences between feedforward networks, convolutional networks, and recurrent networks.

Given the diversity of deep learning model designs, parameters, and applications, you can only learn deep learning — I mean, really learn deep learning, not just have superficial knowledge from a youtube video — by having an experienced teacher guide you through the math, implementations, and reasoning. And of course, you need to have lots of hands-on examples and practice problems to work through. Deep learning is basically just applied math, and, as everyone knows, math is not a spectator sport!

What is this course all about?

Simply put: The purpose of this course is to provide a deep-dive into deep learning. You will gain flexible, fundamental, and lasting expertise on deep learning. You will have a deep understanding of the fundamental concepts in deep learning, so that you will be able to learn new topics and trends that emerge in the future.

Please note: This is not a course for someone who wants a quick overview of deep learning with a few solved examples. Instead, this course is designed for people who really want to understand how and why deep learning works; when and how to select metaparameters like optimizers, normalizations, and learning rates; how to evaluate the performance of deep neural network models; and how to modify and adapt existing models to solve new problems.

You can learn everything about deep learning in this course.

In this course, you will learn

- Theory: Why are deep learning models built the way they are?
- Math: What are the formulas and mechanisms of deep learning?
- Implementation: How are deep learning models actually constructed in Python (using the PyTorch library)?
- Intuition: Why is this or that metaparameter the right choice? How to interpret the effects of regularization? etc.
- Python: If you’re completely new to Python, go through the 8+ hour coding tutorial appendix. If you’re already a knowledgeable coder,
- then you’ll still learn some new tricks and code optimizations.
- Google-colab: Colab is an amazing online tool for running Python code, simulations, and heavy computations using Google’s cloud services. No need to install anything on your computer.

Unique aspects of this course

- Clear and comprehensible explanations of concepts in deep learning.
- Several distinct explanations of the same ideas, which is a proven technique for learning.
- Visualizations using graphs, numbers, and spaces that provide intuition of artificial neural networks.
- LOTS of exercises, projects, code-challenges, suggestions for exploring the code. You learn best by doing it yourself!
- Active Q&A forum where you can ask questions, get feedback, and contribute to the community.
- 8+ hour Python tutorial. That means you don’t need to master Python before enrolling in this course.

What you’ll learn

- The theory and math underlying deep learning
- How to build artificial neural networks
- Architectures of feedforward and convolutional networks
- Building models in PyTorch
- The calculus and code of gradient descent
- Fine-tuning deep network models
- Learn Python from scratch (no prior coding experience necessary)
- How and why autoencoders work
- How to use transfer learning
- Improving model performance using regularization
- Optimizing weight initializations
- Understand image convolution using predefined and learned kernels
- Whether deep learning models are understandable or mysterious black-boxes!
- Using GPUs for deep learning (much faster than CPUs!)

## Table of Contents

**Introduction**

1 How to learn from this course

2 Using Udemy like a pro

**Download all course materials**

3 Downloading and using the code

4 My policy on code-sharing

**Concepts in deep learning**

5 What is an artificial neural network

6 How models learn

7 The role of DL in science and knowledge

8 Running experiments to understand DL

9 Are artificial neurons like biological neurons

**About the Python tutorial**

10 Should you watch the Python tutorial

**Math, numpy, PyTorch**

11 PyTorch or TensorFlow

12 Introduction to this section

13 Spectral theories in mathematics

14 Terms and datatypes in math and computers

15 Converting reality to numbers

16 Vector and matrix transpose

17 OMG it’s the dot product!

18 Matrix multiplication

19 Softmax

20 Logarithms

21 Entropy and cross-entropy

22 Minmax and argminargmax

23 Mean and variance

24 Random sampling and sampling variability

25 Reproducible randomness via seeding

26 The t-test

27 Derivatives intuition and polynomials

28 Derivatives find minima

29 Derivatives product and chain rules

**Gradient descent**

30 Overview of gradient descent

31 What about local minima

32 Gradient descent in 1D

33 CodeChallenge unfortunate starting value

34 Gradient descent in 2D

35 CodeChallenge 2D gradient ascent

36 Parametric experiments on g.d

37 CodeChallenge fixed vs. dynamic learning rate

38 Vanishing and exploding gradients

39 Tangent Notebook revision history

**ANNs (Artificial Neural Networks)**

40 The perceptron and ANN architecture

41 A geometric view of ANNs

42 ANN math part 1 (forward prop)

43 ANN math part 2 (errors, loss, cost)

44 ANN math part 3 (backprop)

45 ANN for regression

46 CodeChallenge manipulate regression slopes

47 ANN for classifying qwerties

48 Learning rates comparison

49 Multilayer ANN

50 Linear solutions to linear problems

51 Why multilayer linear models don’t exist

52 Multi-output ANN (iris dataset)

53 CodeChallenge more qwerties!

54 Comparing the number of hidden units

55 Depth vs. breadth number of parameters

56 Defining models using sequential vs. class

57 Model depth vs. breadth

58 CodeChallenge convert sequential to class

59 Diversity of ANN visual representations

60 Reflection Are DL models understandable yet

**Overfitting and cross-validation**

61 What is overfitting and is it as bad as they say

62 Cross-validation

63 Generalization

64 Cross-validation — manual separation

65 Cross-validation — scikitlearn

66 Cross-validation — DataLoader

67 Splitting data into train, devset, test

68 Cross-validation on regression

**Regularization**

69 Regularization Concept and methods

70 train() and eval() modes

71 Dropout regularization

72 Dropout regularization in practice

73 Dropout example 2

74 Weight regularization (L1L2) math

75 L2 regularization in practice

76 L1 regularization in practice

77 Training in mini-batches

78 Batch training in action

79 The importance of equal batch sizes

80 CodeChallenge Effects of mini-batch size

**Metaparameters (activations, optimizers)**

81 What are metaparameters

82 The wine quality dataset

83 CodeChallenge Minibatch size in the wine dataset

84 Data normalization

85 The importance of data normalization

86 Batch normalization

87 Batch normalization in practice

88 CodeChallenge Batch-normalize the qwerties

89 Activation functions

90 Activation functions in PyTorch

91 Activation functions comparison

92 CodeChallenge Compare relu variants

93 CodeChallenge Predict sugar

94 Loss functions

95 Loss functions in PyTorch

96 More practice with multioutput ANNs

97 Optimizers (minibatch, momentum)

98 SGD with momentum

99 Optimizers (RMSprop, Adam)

100 Optimizers comparison

101 CodeChallenge Optimizers and… something

102 CodeChallenge Adam with L2 regularization

103 Learning rate decay

104 How to pick the right metaparameters

**FFNs (Feed-Forward Networks)**

105 What are fully-connected and feedforward networks

106 The MNIST dataset

107 FFN to classify digits

108 CodeChallenge Binarized MNIST images

109 CodeChallenge Data normalization

110 Distributions of weights pre- and post-learning

111 CodeChallenge MNIST and breadth vs. depth

112 CodeChallenge Optimizers and MNIST

113 Scrambled MNIST

114 Shifted MNIST

115 CodeChallenge The mystery of the missing 7

116 Universal approximation theorem

**More on data**

117 Anatomy of a torch dataset and dataloader

118 Data size and network size

119 CodeChallenge unbalanced data

120 What to do about unbalanced designs

121 Data oversampling in MNIST

122 Data noise augmentation (with devset+test)

123 Data feature augmentation

124 Getting data into colab

125 Save and load trained models

126 Save the best-performing model

127 Where to find online datasets

**Measuring model performance**

128 Two perspectives of the world

129 Accuracy, precision, recall, F1

130 APRF in code

131 APRF example 1 wine quality

132 APRF example 2 MNIST

133 CodeChallenge MNIST with unequal groups

134 Computation time

135 Better performance in test than train

**FFN milestone projects**

136 Project 1 A gratuitously complex adding machine

137 Project 1 My solution

138 Project 2 Predicting heart disease

139 Project 2 My solution

140 Project 3 FFN for missing data interpolation

141 Project 3 My solution

**Weight inits and investigations**

142 Explanation of weight matrix sizes

143 A surprising demo of weight initializations

144 Theory Why and how to initialize weights

145 CodeChallenge Weight variance inits

146 Xavier and Kaiming initializations

147 CodeChallenge Xavier vs. Kaiming

148 CodeChallenge Identically random weights

149 Freezing weights during learning

150 Learning-related changes in weights

151 Use default inits or apply your own

**Autoencoders**

152 What are autoencoders and what do they do

153 Denoising MNIST

154 CodeChallenge How many units

155 AEs for occlusion

156 The latent code of MNIST

157 Autoencoder with tied weights

**Running models on a GPU**

158 What is a GPU and why use it

159 Implementation

160 CodeChallenge Run an experiment on the GPU

**Convolution and transformations**

161 Convolution concepts

162 Feature maps and convolution kernels

163 Convolution in code

164 Convolution parameters (stride, padding)

165 The Conv2 class in PyTorch

166 CodeChallenge Choose the parameters

167 Transpose convolution

168 Maxmean pooling

169 Pooling in PyTorch

170 To pool or to stride

171 Image transforms

172 Creating and using custom DataLoaders

**Understand and design CNNs**

173 The canonical CNN architecture

174 CNN to classify MNIST digits

175 CNN on shifted MNIST

176 Classify Gaussian blurs

177 Examine feature map activations

178 CodeChallenge Softcode internal parameters

179 CodeChallenge How wide the FC

180 Do autoencoders clean Gaussians

181 CodeChallenge AEs and occluded Gaussians

182 CodeChallenge Custom loss functions

183 Discover the Gaussian parameters

184 The EMNIST dataset (letter recognition)

185 Dropout in CNNs

186 CodeChallenge How low can you go

187 CodeChallenge Varying number of channels

188 So many possibilities! How to create a CNN

**CNN milestone projects**

189 Project 1 Import and classify CIFAR10

190 Project 1 My solution

191 Project 2 CIFAR-autoencoder

192 Project 3 FMNIST

193 Project 4 Psychometric functions in CNNs

**Transfer learning**

194 Transfer learning What, why, and when

195 Transfer learning MNIST – FMNIST

196 CodeChallenge letters to numbers

197 Famous CNN architectures

198 Transfer learning with ResNet-18

199 CodeChallenge VGG-16

200 Pretraining with autoencoders

201 CIFAR10 with autoencoder-pretrained model

**Style transfer**

202 What is style transfer and how does it work

203 The Gram matrix (feature activation covariance)

204 The style transfer algorithm

205 Transferring the screaming bathtub

206 CodeChallenge Style transfer with AlexNet

**Generative adversarial networks**

207 GAN What, why, and how

208 Linear GAN with MNIST

209 CodeChallenge Linear GAN with FMNIST

210 CNN GAN with Gaussians

211 CodeChallenge Gaussians with fewer layers

212 CNN GAN with FMNIST

213 CodeChallenge CNN GAN with CIFAR

**RNNs (Recurrent Neural Networks) (and GRULSTM)**

214 Leveraging sequences in deep learning

215 How RNNs work

216 The RNN class in PyTorch

217 Predicting alternating sequences

218 CodeChallenge sine wave extrapolation

219 More on RNNs Hidden states, embeddings

220 GRU and LSTM

221 The LSTM and GRU classes

222 Lorem ipsum

**Ethics of deep learning**

223 Will AI save us or destroy us

224 Example case studies

225 Some other possible ethical scenarios

226 Will deep learning take our jobs

227 Accountability and making ethical AI

**Where to go from here**

228 How to learn topic X in deep learning

229 How to read academic DL papers

**Python intro Data types**

230 How to learn from the Python tutorial

231 Variables

232 Math and printing

233 Lists (1 of 2)

234 Lists (2 of 2)

235 Tuples

236 Booleans

237 Dictionaries

**Python intro Indexing, slicing**

238 Indexing

239 Slicing

**Python intro Functions**

240 Inputs and outputs

241 Python libraries (numpy)

242 Python libraries (pandas)

243 Getting help on functions

244 Creating functions

245 Global and local variable scopes

246 Copies and referents of variables

247 Classes and object-oriented programming

**Python intro Flow control**

248 If-else statements

249 If-else statements, part 2

250 For loops

251 Enumerate and zip

252 Continue

253 Initializing variables

254 Single-line loops (list comprehension)

255 while loops

256 Broadcasting in numpy

257 Function error checking and handling

**Python intro Text and plots**

258 Printing and string interpolation

259 Plotting dots and lines

260 Subplot geometry

261 Making the graphs look nicer

262 Seaborn

263 Images

264 Export plots in low and high resolution

**Bonus section**

265 Bonus content

Resolve the captcha to access the links!