I've seen some drift that was -3 or -4. Anything to be concerned about?

Stan,

If it is only occasional from maybe one or two stations then it is their problem.
If it is on most spots of your transmission then you are drifting.

A search seems to indicate that some IC-7300s do drift on TX. eg http://www.wsprnet.org/drupal/node/7967

73 Alan G4ZFQ