news.ycombinator.com• Sep 7, 2017• 1 min read
Why is the gradient either +ve or -ve when we use sigmoid activation function?whenever i ask this question people tell me something like ""Gradient of W = X * gradient of Function Now, from the above equation, since all X values are positive, therefore gradient of all W's could either be all positive or negative, depending on the gradient of function."" i get it but i think they are missing a point. consider the multi-class setting - if there are 10 classes, and we use a softmax output. The NN makes a prediction C1, whereas the true class is C2. Now our gradient of the ouput layer(from the loss function) will be a vector which will have one positive term, and another negative term (for C1 and C2). When this vector is multiplied with positive gradients during backprop. Since error gradient has both positive and negative numbers, the gradients of W should also be a mix of positive and negative. NOT JUST THAT... also the gradient over the hidden vectors dhidden when backpropagated will be multiplied by weights which can have positive and negative signs. I am i righ
