You didn't specify the switch model. For the answers I assume you have a smart or managed switch.

glasser wrote

I am wondering what is your understanding of the logic behind port QOS: incoming vs. outgoing traffic??
Is port QOS enforced on both Incoming and Outgoing traffic?

Port QOS just defines priority as a property of a port. Ingress traffic can be classified, egress traffic happens as a function of the select scheduling mode for the traffic queues.

Based on testing so far, it seems like the gateway port#1 that carries all traffic should be *lower* than VOIP port#2-3 (Highest). So port#1 Medium, not Highest
So I am not clear on how this QOS works to configure it right.

Port QOS is suitable for prioritizing traffic from a given port, e.g. if in a congestion situation an admin workstation needs higher priority than user's workstations (scheduling mode strictly based on ingress priority) or if in a congestion situation traffic from one department should be weighted more than traffic from another department (scheduling mode based on weighting). In both cases the upstream always needs highest priority to share bandwidth between lower-priority ports.

For VoIP you should use the VLAN-based QOS priority for a dedicated Voice-VLAN. See the user's guide for a smart switch, e.g. the UG for the T-1600G V1 series, page 481 for an example of setting up such a QOS scheme suitable for VoIP.