Immigrants you bring to a port (workers you fire, count as immigrants) increase your influence by 0.1 (1 influence point per 10 immigrants) at the port.
DEV Clarification Notation & Points: As workers do not count as port population, bringing workers to your plantation would not increase influence. As originally noted above, when you fire them, they will then become part of the population and thus will count as immigrants.
Alternatively.....
Workers brought in could gain influence, and workers that die reduce your influence. Workers you fire would gain nothing. This would counter a player buying up workers on the cheap while holding no acres and then firing them for influence.
-Dez