A few things to consider with respect to the simple case of you and your friend....
(1) are you both running tx and rx 24 hours a day? Natural noise levels tend to peak in the evening and be lowest in the early morning around sunrise. So the best reception conditions will not generally coincide in time with the best propagation.
(2) manmade noise in urban locations can nowadays be much stronger than natural noise sources. The daily variation of manmade noise depends on what equipment is causing the noise and the habits of the people using the equipment. Do you both have similar manmade noise levels or do they differ quite a bit? That's not usually an easy question to answer.
(3) you mentioned various antennas were used. Any very small antenna (or any antenna fed by an excessively long piece of coax) will have poor efficiency and will put out a poor signal compared to a full size antenna fed with low loss feeder cable. On receive on HF, however, a mobile whip (for example) can be just as effective as a full size antenna (not counting directional beams), because the reception is often limited by external noise, which gets attenuated by the losses in the antenna by the same amount as does the desired signal.
With respect to the overall statistics, I am tempted to think that the relative ham populations of the US and UK may have something to do with it. It might be clearer if you could somehow limit your analysis to the same number of transmitting and receiving stations at both ends of the path, and perhaps limit the distribution of US stations to a geographical area the size of the UK.