There does seem to be sort of a duality. Most synthetic chemists I know are bad at math and a little scared of physical/quantum chemistry. Then as you point out there are many physicists / comp scientists who are good at math / physics but find chemistry (i.e., organic) weirdly boring or incomprehensible. I think it's because so much of organic consists of not-completely-consistent 'rules of thumb' and heuristics that generally work; I remember what puzzled me most when I was learning organic was how certain rules seemed to apply in some cases but not in others. All of this is very different from how physics or mathematics is structured. I wonder if there are two different types of thinking involved and only a limited number of ppl who are good at both.
