i have had the same guy attack me every port i go to he finds me it is starting to get annoying

He can only see you if your danger rating goes to 3 or more : that's your problem. Every time you hit port, you gain 1 danger, every hour you lose 1 : so you need to time your journeys with that in mind : keep them just over an hour and you wont gain the danger and thus will not be seen.

There is a link on the tavern page called 'port gossip'.. this will show all movements in and out of the port for the previous 50 movements : but you still need to be 3 or more danger to be hit.

Also some pirates have a ship in every port so they can see who is there

...and warehouses in every port so they can always read gossip.