When I referenced the recording to verify the lyrics, I noticed that "so" is sung with a melisma (D E), also giving a sense of degree.
Later, I was still thinking about the song and realized that the melody to which the line "The difference between us is so small" is sung is entirely conjunct, moving step-wise from A to E. The adjacent intervals here musically illustrate this "small," too.