Granger and James both received technical fouls for the incident, and initially personal fouls, but those were later rescinded. Had James connected with that higher elbow, it could have been bad. You can understand James being upset because of the contact to his upper body and neck, but the league is really sensitive towards elbow swipes to the head. As long as the league considers that he restrained himself to stop it, he won't face any further punishment.
This thing's getting personal.