Welcome to evolution in its simplest form.
For higher life forms with longer reproductive cycles, evolution takes a long time. Evolution is not smart. It's a mindless self organizing system. The environment changes. You have x breeding pairs times y breeding cycles for the right characteristics to emerge. Then those characteristics have to spread to enough of the population to make a difference. If a species loses this lottery, it goes extinct. So it not only takes a long time, but it's also basically random chance.
Even if we were able to observe evolution in higher life forms over the timescales we're working with, we probably wouldn't see it in humans. For one thing, we change ourselves and our environment extremely rapidly. We get bigger and stronger through better nutrition. We get smarter through organized education systems. We stay warm by building houses and burning dinosaurs. We address food shortages with agriculture. We don't adapt to our environment. We adapt our environment to our needs.
Our environment is almost entirely artificial. We decide what its characteristics will be, and we decide what traits are desirable or undesirable within it. If a trait arises that we think (we don't always know) makes people less able to thrive within the environment we've created (which may or may not be the best one for us), we don't nurture it and we don't allow it to flourish. We treat it.
A big part of our environment is culture. Societal norms that encourage monogamy, place an inordinate value on traits and skills that have little to do with survival outside the societies we've constructed, and the concept of romantic love all conspire to allow a very high percentage of us to breed. Which makes it much harder for any traits to become dominant and spread across large percentages of the population.