Durant feels like the Warriors' success created discourse about his time with the team