Unfortunately that will not work. Each town has less then 5 actives save for Volantis, but only a handful are PVPers.
If you are looking to hold it to townships, instead of groups of players I fear the idea will not turn out to be what you are looking for due to inactivity.
Now if you are wanting to hold even, organized matches of players; then it will open up because all players will be able to participate(even if their towns don't have 5+ players.) Keeping it open to groups of players will allow for even fights even when the numbers on the server are low.
Edit: Of course, I am not saying that anyone should be forced to group with people they don't want to group with, but for the sake of getting even numbered fights, it will need to be open to groups.