Biological Reviews 18:32-64
(rec'd 26 June 1942)
Polygenic inheritance and natural selection
K. Mather


(1) The adaptive nature of breeding systems

The control of variability, its storage and release, depends chiefly on the linkage relations of the polygenes within chromosomes. The frequency of recombination determines the rate of exchange between the various states of variability. But segregation cannot occur and recombination is ineffective unless the organism is heterozygous for at least some of its polygenes. With complete homozygosity no potential variability can be released and, apart from some mutation, the free variability must steadily dimish. A homozygous system is static and, though perhaps showing high present fitness, is doomed to ultimate extinction by its inflexibility. Heterozygosity, on the other hand, sacrifices some fitness to the maintenance of flexibility. Genes which control the rate of outcrossing, or breeding system, control the balance of homo- and heterozygosity, and so will have an adaptive value in determining the variability transformations of the population (Mather & de Winton, 1941).

Genetical control of breeding systems is well known. Most animals show sex separation, while in plants there is a wide variety of mechanisms ranging from physiological incompatibility to special pollination structures. Some of them, like sex separation, incompatibility of the Nicotiana type (East, 1929) and heterostyly (Mather & de Winton, 1941), are recognizable as highly efficient means of promoting outbreeding and heterozygosity. These systems depend on genetical heterogeneity, matings between like bypes being prevented. Other outbreeding mechanisms, such as protandry and floral devices for attracting insects, do not depend on heterogeneity since all individuals show the same basic behaviour. They do not usually serve to prevent self-fertilization so completely and hence are of lower efficiency. Inbreeding mechanisms, like premature anthesis in Triticum and intra-uterine copulation in Pediculopsis, also exist and encourage homozygosity. These never depend on genetical heterogeneity.

Morphologically and physiologically these means of controlling breeding are very varied, especially in plants. (Most botanical textbooks, e.g. Kerner & Oliver (1894-5), devote large sections to their description.) The exact nature of the system depends on the morphological and physiological possibilities of the species (Mather, 1940, 1942a). Sex separation, for example, is well suited to mobile organisms and is less efficient in sessile forms (see Lewis, 1942a*), which, however, may be able to develop a highly efficient incompatibility system.

From the point of view of function, breeding systems are classifiable on a two-dimensional basis. Some are more efficient than others in the control they exercise, in that chance plays a less part in determining the mate. Thus incompatibility and sex separation can completely prevent self-fertilization, whereas protandry and protogyny only reduce the chance of this happening in the place where it is most likely to occur, viz. within a flower. The former are more efficient than the latter.

The second dimension is provided by the rate of outbreeding, or, speaking inversely, of inbreeding. Incompatibility and heterostyly lead to outbreeding, homostyly and prepature anthesis to inbreeding. This dimension must be carefully be distinguished from the first one. Inefficient control will permit mixed breeding, but a superficially similar result can follow from a carefully controlled system. The latter, however, will enforce rather than permit mixed breeding, the average degree of outbreeding being thus less subject to chance variation than that resulting from poor control. Balanced controlled systems are achievable by mixed outbreeding and inbreeding systems of kinds dependent on the nature of the organism. Incompatibility systems may have a fertility allelomorph (East, 1929) whose frequency determines the balance of self and cross fertilization. Heterostyled species may include homostyled individuals (Crosby, 1940) giving a measure of inbreeding.

With sex separation the method of balancing outbreeding and inbreeding may be different again. The efficiency of sex separation, even in a mobile organism, depends on the ability of one sex to recognize individuals of the other sex for the purpose of mating. Control of the degree of outbreeding can be established by an extension of this discrimination. If individuals not only recognize those of the other sex, but can also distinguish particular groups within that sex, preferential mating will occur. Where discrimination is against individuals from the same population, the rigour of outbreeding is increased and where mating is preferentially within the population, inbreeding is increased. Evidence of such discrimination, or sexual selection, has been obtained both between and within species from observations on Drosophila miranda (Dobzhansky & Koller, 1938; see also Dobzhansky, 1941). Sometimes the control of breeding may, as with gynodioecy, lead directly to a balanced system (Lewis, 1941).

It is clearly not to be supposed that the balance of inbreeding and outbreeding observed in any given existing organism represents the present optimum, because, obviously, the present optimum depends on the degree to which change in the future will bee advantageous. The existing balance must be one which has been sufficiently close to some past optimum for the ancestor of the present species to survive, and perhaps spread, while competitors, not possessing such a good balance, were extinguished. Such a balance will be of advantage to existing organisms only to the extent that future changes of environment match past changes. The survival of any present day species will depend on the extent to which its breeding balance agrees by chance with, or can be adapted to, an optimum, or series of optima, which only the future can decide. Thus most existing organisms will fail and must perish. Existing breeding systems do, however, show us how future adaptation of the breeding system can occur, for they show us how it has occurred in the past.

(2) The genetical control of breeding

The regular control of breeding can only be achieved genetically, because, otherwise, the controlling mechanism could not be a permanent feature of the species. The rise of breeding control must, however, be gradual, for two reasons. First of all the spontaneous origin of a highly specialized and complex mechanism is extremely unlikely, and, secondly, it is also very unlikely that the conditions determining the advantage of a given rate of outbreeding will either arise suddenly or persist indefinitely. (The rate of environmental change in the glacial period was very different from that of the Pliocene.) The development of control must be by small steps, and any subsequent change in the control must be equally gradual. So we should expect the controlling mechanism to be polygenic. If this be granted, it is easy to see that any organism is capable, under the action of natural selection, of developing and of changing its breeding system, by virtue of the store of polygenic variability with which it is endowed.

The genetical study of breeding control is still in its infancy, but so far nothing has been found to contradict this expectation, and indeed there is a certain amount of evidence in its favour. The homogeneous breeding systerm are uninvestigated, but appear to offer no difficulties. In almost all cases the special arrangement of flowers, or the timing of anthesis relative to receptiveness of stigmata, is obviously a quantative character of the type which we may confidently expect to be polygenic.

On the face of things, heterogeneous systems of control are less amenable to a polygenic interpretation, in that they depend on 'switch' genes determining the classes within which mating is impossible or unlikely. But this is only part of the mechanism, for the strength of the mating reaction, controlled by the switch genes, may itself vary. In the hetestyled Primula obconica, for example, illegitimate matings, between pin and pin or thrum and thrum, never set more than 10% of the seeds obtained from the legitimate matings, pin by thrum and thrum by pin (Lewis, 1942b and unpublished), while in Primula sinensis the ratio may be as high as 70% (Lewis, 1942 b; Mather & de Winton, 1941). Darwin (1877) gives other data of the same kind for a number of plants. The strength of the incompatibility reaction varies similarly in hybrids of the incompatible Petunia integrifolia and the compatible P. axillaris (Mather, unpublished). Sex in Lebistes (Winge, 1934) and Apocheilus (Aida, 1936) is also variable independently of the direct control by X and Y chromosomes. Thus the evidence suggests that this control of reaction strength is polygenic in Primula, Petunia and most cases of sex separation, though in Lymantria Goldschmidt (1933) thinks that it is not so. Winge (1937), however, considers Goldschmidt's results to be amenable to a polygenic interpretation, and Drosophila (Dobzhansky & Schultz, 1934) also appears to agree with the polygenic expectation.

Heterogeneous breeding systems depend on switch genes for their direct working but are probably polygenic for the strength of control and general adaptation. The same conclusion has been reached by Ford (1937,* 1940b) for cases of polymorphism and it would appear to be a general principle, that even where sharp oligogenic variation is observable in an adaptive character, switch genes only provide the necessary trigger action, adaptation being secured by polygenes. In this way the development of such systems is gradual and their adaptation capable of continuous fine adjustment.

The adaptive changes in polygenic constitution may even bring about changes in the switch genes, new ones being substituted for old. This would appear to be the case with the system of sex determination in Lebistes and elsewhere (Darlington, 1934 b). The whole system is capable of change and readaptation.

(3) Breeding control and isolation

The breeding system of a species, like any other phenotypical feature, cannot be immutable. It must be subject to change for the same reasons as are other characters viz. that it represents an adaptation to existing conditions and must alter to meet new conditions if the species is to survive. Most species presumably do fail to survive because they fail to show adequate adaptation; but existing species, deriving from successful ancestors must afford some evidence of what is necessary for success in the breeding system, as in other characters. Furthermore, the evidence suggests that the breeding system changes by exactly the same means as other characters, viz. by selection acting on heritable variability of a polygenic nature.

The general principle that evolutionary changes can never be wholly or even largely reversed (see Muller, 1939) holds true with breeding systems. There is a steady progress towards more exact control, and even where the conditions favouring the mixed system characteristic of the early prototype are re-encountered after a controlled breeding system has developed, the successful organism will be that which shows new adaptation by the rise of a balanced inbreeding-outbreeding mechanism, not by a regression towards loss of control.

Such a change may occur either by adaptation of the existing method of control to the new conditions, or by the superimposition of a new type. Homostyly in heterostyled plants is probably an example of the former (Mather & de Winton, 1941). The outbreeding system is changed, in this case by a reshuffling of the switch gene components, to give an equally rigorously controlled inbreeding. It would seem that balanced systems can then be produced by mixtures of homo- and heterostyled individuals such as have been found in the wild (Crosby, 1940).

Examples of the superimposition of new mechanisms on old are many, and sometimes show evidence of complex series of changes. Wheat provides a simple case. The floral behaviour shows adaptation, and is often described as an example of such adaptation, to cross-breeding by wind pollination; but the anthers burst before extrusion, so giving regular self-pollination. Inbreeding is superimposed on outbreeding.

Some of the Compositae show a more complex situation. Arrangement of the flowers into a capitulum is most easily understood as an advantageous means of promoting outbreeding by attracting insects, and is usually accompanied and strengthened by protandry. But many species also achieve self-pollination by the device of curving the stigma lobes over until they pick up pollen from the anthers of the same flower. This was followed in Taraxacum, for example, by apomixis, which, by sacrificing the normal sexual process, secures even more rigorous 'inbreeding' and the immediate survival of cytologically anomalous types, though it equally determines the ultimate extinction of the species through prohibition of any extensive future change with selection (see Darlington, 1939). In Dahlia, on the other hand, a second cross-breeding mechanism, incompatibility, is in evidence (Lawrence, 1931), so marking a new break towards outbreeding. These complex breeding systems, inexplicable in any other way, are thus seen to represent stratified adaptations towards outbreeding and inbreeding, presumably recording the changing circumstances through which the prototype species passed successfully.

A remarkable feature of these stratified systems is that outbreeding is not often observed to have been superimposed on inbreeding, though the reverse is very common. The reasons for this are to be found in the consequences of inbreeding, which, as we have seen, increases fitness at the expense of flexibility. Inbreeding reduces variability, and an inbreeding species will thus be less likely to develop a new system than will an outbreeding species (Huxley, 1942). Furthermore, a species showing rigorous inbreeding is very likely to be exterminated sooner or later, because of this inability to change with condition, except for the very slight effect of mutation. An outbreeder, on the other hand, is pre-eminently flexible, can change with conditions and so can leave some successful descendants. Inbreeding must frequently lead to extinction while outbreeding need not.

Fig. 9. Breeding control and isolation There is a steady tendency to develop more rigorously controlled outbreeding from uncontrolled breeding. No evidence exists of a similar rise of inbreeding control, but this can be derived from, and may also give rise to, controlled outbreeding. The change from outbreeding to inbreeding is accompanied by the rise of isolation mechanisms (see text).
The change from out- to inbreeding has far-reaching consequences. An outbreeding species will be capable of great variation by virtue of its store of variability, and so will be able to occupy and colonize a wide range of environments. This it must achieve at the expense of a decrease in present fitness, because high local adaptation will not be possible. So long as competition is not so severe as to put an increased premium on high fitness, the outbreeding system, and hence wide adaptability, will persist. But if a higher standard of fitness becomes favoured, by increased competition, a change towards inbreeding will set in. The species or superpopulation will break up into small locally adapted populations which seldom interbreed. The polygenic combinations will become balanced within and not between these populations. Heterosis will occur when individuals from different populations happen to intercross, and the inbreeding system will tend to be strengthened and developed into an isolation mechanism (§V (2)). Isolation and speciation will be the inevitable consequences of change from outbreeding to rigorous inbreeding (Fig. 9), though, if brought about in a different way, isolation may equally cause change of the breeding system.

A further consequence will, of course, be that any of the new species which do not retain some measure of outbreeding, or the ability to revert to such a system, are doomed to extinction. To express the series of changes in terms of the variability states, we can say that outbreeding encourages the replenishment of free variability from the potential store, but that this power is lost when close inbreeding supervenes. The free variability immediately available permits high local adaptation, the potential variability being frozen by the inbreeding mechanism. Since future adaptation depends on release of this store, freezing, unless, as would seem rarely to be the case, reversible, means extinvtion, and supplantation by more flexible competitors. Something of this kind has happened and is probably happening now in such groups of the genus Drosophila as that which includes D. miranda and D. pseudo-obscura (Dobzhansky, 1935). The former species has two partly isolated, and the latter two almost fully isolated, races. This implies developing speciation. It is accompanied bv inbreeding, for Dobzhansky & Wright (1941) have evidence that D. pseudo-obscura consists of a large number of small populations showing free intra- and restricted interbreeding. Sexual selection also seems to occur (§VI (1)). Presumably then, there has been a large freely outbreeding species. Increasing severity of competition favoured the rise of inbreeding which gave rise to speciation. The various species and, more especially, the two races of D. pseudo-obscura seem to be the relicts of a previous break up, having become isolated and so able successfully to populate common localities (§V (2)). The strains within each race are evidence of a new break up now in progress (also see Huxley, 1939). It follows on our argument that many of the species recently developed or now developing are nearing the end of their history. Wright (1940) and Dobzhansky (1941) have developed a different view of the part plaved by small populations. They point out that in such cases the random fixation of genes, or genetic drift, will cause the free variation to exist mainly between, rather than within, the populations. Since populations are not in direct competition this should preserve variability, since it will be less likely to be eliminated by response to selection. This reasoning is quite sound, so far as it goes, but two considerations, dependent on polygenic behaviour, have been overlooked. These considerably lessen the importance of this particular consequence of small populations. In the first place, the greater the number of genes affecting any character, the less is the importance of random fluctuation in the allelomorph frequency of a single gene. With polygenic behaviour each of the populations will settle down to a characteristic phenotypic distribution, broadly similar for them all, though dependent on different genes to an extent determined by population size (§V (2)). The genotypic flux will not be represented fully in the phenotype, on which selection acts. Secondly, Wright & Dobzhansky discuss only the free variability, though, as we have seen, there is reason to believe that polygenic variability is more potential than free. Reduction in population size will protect the free variability from selective diminution, but it will also decrease heterozygosity and so diminish the chance of freeing the potential variability. The quantity of free variability preserved in this way will be more than offset by the loss of new free variability caused by the reduction in transformation of potential to free. Thus the net result will be a loss of, not as Wright and Dobzhansky conclude a gain in, flexibility. The degree of protection of variability from the action of selection conferred by small populations, will be completely overshadowed by the closing of the variability store.

No matter what the mating system may be the major part of the variability must be potential and hence does not have any phenotypic expression. It will not be subject to the action of selection. Thus the value of small populations as a means of protecting and preserving variability will be insignificant, while their action in sacrificing flexibility to fitness, in increasing adaptation at the risk of extinction, will be of great moment. The latter action will detemiine population size.


The occurrence of natural selection demands (1) that there exists genetical heterogeneity, and (2) that unlike genotypes leave different average numbers of progeny. It is now known that both of these conditions are fulfilled, and all the available facts of evolution are in accord with the genetical theory of variation and selection.

Species must, on this view, differ in the same way as, but to a greater extent than, varieties or individuals of the same species.

The application of this criterion leads us to the conclusion that species differences are polygenic, i.e. depend on quantitative characters whose variation is controlled by many genes. These genes have individual effects which are both similar to one another and small when compared with non-heritable fluctuation. Other kinds of heritable difference are ancillary to polygenic variation in speciation.

Each individual polygene is inherited in the same way as the familiar major mutants of the laboratory. As, however, there are many polygenes affecting a given character, the aggregate type of inheritance is distinct from that of the major mutants. Polygenically controlled differences are quantitative rather than qualitative and do not lead to the sharp segregation shown by the more familiar genetical differences. Polygenic characters, such as stature in man, can show any degree of expression between wide limits. Many genotypes may have the same phenotype. Thus polygenic theory relates continuous phenotypical variation to discontinuous genotypical variation, the biometrical to the genetical.

These special properties of polygenic behaviour lead us to a new and clearer understanding of the action of natural selection in producing adaptive and evolutionary changes.

Very fine adaptation of the phenotype to environment is made possible by the existence of such a wide range of phenotypic expression. The frequency distribution of the individual phenotypes found in a population may approximate to a normal curve. It may, however, also be skew, to an extent determined by the dominance and interaction relations of the polygenes and by the scale on which the character is measured. The central, most frequent, phenotype must closely approximate to the optimum for the prevailing environment. Departure from this central type will thus mean poorer adaptation and loss of fitness.

The phenotype is produced by the genotype acting as a whole. Since polygenes have effects similar to one another, a given phenotype may correspond to various genotypes some containing one and some another allelomorph of a given polygene. As a consequences neither allelomorph will have an unconditional advantage over the other, in the way that major mutants do. Rather the advantage of any allelomorph of a polygene will be conditioned by the other polygenes present. Fisher's theory of dominance then leads us to expect that, in wild populations, equal numbers of polygenes will show dominance of the allelomorphs leading to increased and decreased expression of the character. Artificial selection disturbs this equality. The existing evidence is in keeping with these expectations.

The existence of polygenic variation free in the phenotype must lead to some individuals departing from the optimum and so showing reduced fitness. Variation is to this extent disadvantageous, but it is also essential for prospective adaptive and evolutionary change. The polygenic variability necessary for prospective change need not, however, exist as free phenotypic variation which will affect fitness. It may be hidden in the genotype under the cloak of phenotypic constancy, when it will have no effect in lowering fitness. Such hidden, or potential, variability is released, and shown freely by the phenotype, as a result of segregation from heterozygotes. Free variability may pass into the potential state by means of crossing between unlike individuals. Some potential variability will exist as differences between homozygous individuals. Such homozygotic variability can be freed by segregation only after intercrossing has rendered it heterozygotic.

If most of the variability in a population is potential, high current fitness can be combined with the possibility of great, though slow, change under selection. In such cases the response of the organism to selection will largely depend on the fixation of variability as it passes from the undetectable potential to the detectable free state. Thus selection may superficially appear to create its own free directional variability.

The frequency of recombination between polygenes affecting a character will control the rate of variability release. Consequently the effective recombination frequency is itself an adaptive character and will be subject to selective action. The evolution of genetic systems is largely the history of this selective control of effective recombination.

Control of recombination is almost wholly achieved within chromosomes, so that the storage of variability must depend on intrachromosome adjustment. Natural selection will tend to build up balanced combinations of polygenes within each of the chromosomes. These combinations will have the properties of close adaptation to the optimum, great variability storage and slow variability release.

Combinations are characterized by two kinds of balance, that of the individual com bination, as shown in homozygotes (internal balance), and that of pairs of combinations when working together in heterozygotes (relational balance). Dominance permits the adjustment of these balances independently of one another. The theory of polygenic balance shows how polymorphism and clines can be maintained.

Heterosis is due to a particular kind of poor relational balance brought about by artificial selection. The concept of heterosis is now extended to include all types of such unbalance, natural and artificial. Poor relational balance encourages isolation, and so heterosis, in this broad sense, stimulates the rise of isolation mechanisms and hybrid sterility.

The store of polygenic variability, steadily depleted by random fluctuations in allelomorph frequency and by response to selection, is replenished by new mutations. Since all polygenes affecting a given character have much the same effect, the phenotypical properties of a population may be stable or nearly so even though the genotype is fluid. Fixation, mutation, segregation and recombination cause a genotypic flux to exist under the cloak of a phenotypic stability, itself maintained by the action of the same natural selection, which, under new conditions, would lead to new adaptation.

The mechanical relations of unlike combinations, whose constituent polygenes are intermingled along the same chromosome, are sufficient to account for the degeneration of unused organs.

The breeding, or mating, system of a species determines the frequency of heterozygosity, upon which the rate of release of potential variability depends. Inbreeding gives homozygosity and high immediate fitness; but it freezes potential variability in the homozygotic state and so reduces the chance of prospective adaptation. Outbreeding has the reverse effect and sacrifices some fitness to flexibility.

The breeding system is thus an adaptive character. It will be subject to selective change towards more closely controlled inbreeding or outbreeding. A controlled compromise between inbreeding and outbreeding may also occur. The strength and direction of control is probably polygenically determined, though the actual controlling mechanism may depend upon a major switch gene for its direct action.

A change from outbreeding to inbreeding increases local adaptation and so leads to heterosis and isolation. It also freezes potential variability and lowers the chance of prospective adaptation. Thus a species which shows such a change to inbreeding will break up into a swarm of small, locally fit, but inflexible, new species. As a consequence of their inflexibility, most of these must perish when environmental changes set in.