Well, yeh, getting rid of blackjack would probably bring a ninja back down to earth. Amazingly, getting rid of a skill as effective as this one wouldn't even turn the class from OP to UP. You can't argue that ninjas have a huge set of skills that helps them in mobility, melee, ranged, and survivability. Add on that they still have Kick and Blind and you're left wondering why Ninja has a stun in the first place.
The issue with stuns (aside from piggify) is that when a player is stunned he doesn't move an inch during that time. A ninja can get behind a player and spam left click and he doesn't need to move an inch to 'catch up to him'.
Backstab is a good skill, and a ninja should be rewarded for hitting someone from behind. Without a stun it'll be harder to backstab your opponent, but certainly not impossible. Blind does a good enough job to disorient a player that he's unable to see more than a couple blocks infront of his face. During a blind, a ninja should be able to get off some backstabs. Lets not forget that Blinded players are unable to sprint either. Hell, even if a player is running away from a fight you can catch up to him with your agility/backflip and stab him in the back here and there.
Of course, a Ninja would really shine in a team fight. With so many people running around a Ninja can set himself to focus on backstabbing 1 guy (preferably a squishy class) and make short work of him. If all else fails, run around with a Paladin or Berserker who have stuns, and backstab the poor souls that they stun.
In short, a Ninja is perfectly able to hold his own without blackjack. The catch is, you actually have to be an average PVPer to get a kill. As is, someone who is new to Minecraft can blackjack, get behind a player, and left click to produce ungodly amounts of damage. It's just not right.