Eventually, the changes they are differing on may affect something which makes a merge impossible. In the area of sexual selection, a change in an exterior color or small may make the others look or smell foreign, and therefore unsuitable for mating. An otherwise neutral change may cause mating to fail in the part of fertilization where the DNA recombines, and result in non-viable offspring. At this point speciation has essentially become complete.
So, to answer your question, mutations don't create subpopulations; rather they grow up within subpopulations and begin to differentiate them. The passing on of traits is done by the copying of DNA from parent germ cell into the whole of the offspring.
Why should the reproductive process with female be incompatible? The mates are compatible because they are members of the same species; the genetic layout ("genome") of the male matches that of the female, indeed all the females of the species, by definition of the word "species". Anyone in the species who is not compatible with the rest will be childless, and the incompatibility ends with that individual's death.