Have no solid idea on how to implement this, but throwing it out there for discussion, should it spark interest.
The plantation dynamic should play at least a part, if not a big part, in population growth. Healthy plantations equal employment for locals, which means financial means for supporting a family, which means more people, which attracts the means for supporting the population- i.e. other jobs, vendors, businesses, etc. A presence of healthy, growing plantations around a port would attract still others to live in the area. You get the idea.
Unhealthy plantations would discourage immigration to the area, not to mention slow growth in families and the increased worker mortality. It would be in a port governor's best interest to monitor the health of the plantations located around his port. Plantations that are not well taken care of would run the risk of not having their permissions renewed.
Personally, I am in favor of getting rid of TI and BD voodoo cards as a means for altering population at a port and tying population growth to other in-game dynamics.