Kevin Durant was worried about how his words would be received, so he stopped speaking last year