There should be other ways in exterminating or raising a port population other than using voodoo cards.
For exterminating, I suggest that bombarding the port to kill some of its population. But, doing so, increases the players' hostility in the nation that's controlling/governing the port. Also this is the reason why players needs to build up the port to gain influence. Bombarding also increases the danger rating of the fleet so that the Marquees of the nation should be able to defend the port or counter the bombardment.
For raising the port population, ships can also transfer citizens from one port to another by the same amount as their cargo. 1:1 ratio I guess.
Opening the link will do you no harm.
http://s2.piratesglory.com/promo.php?id=38872