I want to compute the gradient of a loss wrt a large neural network in pytorch. The loss is of form L = g(f(x1), f(x2), f(x3), ..., f(xn)), where there are potentially many xs, f is the neural network, and g is a relatively simple nonlinear function. A naive implementation is:

I want to compute the gradient of a loss wrt a large neural network in pytorch. The loss is of form L = g(f(x1), f(x2), f(x3), ..., f(xn)), where there are potentially many xs, f is the neural network, and g is a relatively simple nonlinear function. A naive implementation is:

I want to compute the gradient of a loss wrt a large neural network in pytorch. The loss is of form L = g(f(x1), f(x2), f(x3), ..., f(xn)), where there are potentially many xs, f is the neural network, and g is a relatively simple nonlinear function. A naive implementation is:

I want to compute the gradient of a loss wrt a large neural network in pytorch. The loss is of form L = g(f(x1), f(x2), f(x3), ..., f(xn)), where there are potentially many xs, f is the neural network, and g is a relatively simple nonlinear function. A naive implementation is:

I want to compute the gradient of a loss wrt a large neural network in pytorch. The loss is of form L = g(f(x1), f(x2), f(x3), ..., f(xn)), where there are potentially many xs, f is the neural network, and g is a relatively simple nonlinear function. A naive implementation is:

I want to compute the gradient of a loss wrt a large neural network in pytorch. The loss is of form L = g(f(x1), f(x2), f(x3), ..., f(xn)), where there are potentially many xs, f is the neural network, and g is a relatively simple nonlinear function. A naive implementation is:

I wanted to understand the purpose of the detach method in PyTorch. Below is an example. If you look into the update_delta method, you see the detach method being used. I don’t understand what the author is trying to achieve by using detach.

I’m attempting to implement a custom single forward pass training algorithm using PyTorch. Since I don’t require backpropagation, I am manually updating the weights of the neural network. However, I can’t seem to get this to work correctly.
After the first pass, I repeatedly get the error that I’m trying to backward through the computational graph for a second time, despite having zeroed out the gradients in the model. Not sure where I’m going wrong.