When a packet comes in, the kernel looks at the existing mappings: how it has mapped this packet in the past, or if this is a reply to a packet it has mapped in the past. If it finds a match, this existing mapping is used.
If no match is found, the rules (created by ipnatctl) are examined. The most specific rule is examined first, down to the least specific rule. If there are multiple rules which are identical (except for the ranges they map onto), their ranges are combined into a cumulative range.
If a rule is found, then a mapping will be created: unless the rule specifies a mapping type (eg. masquerade), the least-used IP address in the range given by the rule is used. If that IP address is full, the next best IP address is used. If the entire range is full, the packet is discarded.
If no rule is found, then a `null mapping' is created; we try leaving it as it is. If there is already a connection mapped onto that address, and the packet is of a protocol we understand (by default this means TCP, UDP or ICMP), then an attempt is made to alter the source port or ICMP id.
The mapping is always created as a packet enters the system, either from a local process, or from a network interface, even if the actual mangling is done later (ie. source manipulation).
If the destination of a locally-generated packet is changed, and that causes the packet to pass out a different interface, then the source address is also changed to that of the interface. For example, changing the destination of a loopback packet to head out eth0 (-b dest -d 127.0.0.1 -t 192.168.1.3) will result in the source also being altered from 127.0.0.1 to the address of eth0; unlike other source mappings this is done immediately. Naturally, both these mappings are reversed on the reply packets which come in.