This is to be expected, from the network perspective. If your ping is high, server might not manage to transfer information about you not being close to the vendor yet but on the event itself.
Ping ~300 or more is going to take time to make server itself aware of such a change. If you pressed 'E', server was still thinking you are near the vendor, but teleported (on your client screen) to the event itself.
I'm not sure I can do anything about it. Similar stuff will be happening when you are near items for example or Fast-Travel system etc.
I will look into it, and try. Thanks for reporting.