You aren't wrong, and are totally right to be skeptical, but there is logic to it.
Generally, things that are toxic to us are compounds that we either didn't evolve alongside or that are in much different concentrations than what we evolved alongside. There are exceptions of course, but this is strongly true. For example, plutonium is much, much more toxic than Uranium. This is for a couple of reasons but also because we evolved alongside a certain concentration of Uranium in the environment and therefore our biology has ways of dealing with it. Plutonium is recent and our bodies have no way of coping with it.
The argument is that the concentrations of compounds in our modern food supply (modern being relative) are toxic because they are in concentrations our bodies did not evolve to deal with, high sugar/low fiber being the most notable difference. Now, depending on who you talk to there are 2 versions of "modern" diet. There is modern as in the beginning of Agriculture ~10k years ago and modern since the green revolution in the ~1950s. I'd argue that they are both equally significant, and both deleterious to our health overall but that the green revolution combined with modern agriculture policy has had a massively negative impact on public health far more than the advent of agriculture 10k years ago.
A modern apple, although different than an ancient apple, still MUCH more closely resemble each other in what compounds they contain and very importantly the
concentrationof those compounds than say a Hungry Man dinner and its unprocessed constituent parts.
Edited to Add:Lactase persistance, or the ability to digest dairy into adulthood, is a good example of the evolutionary adaptation known as
neotany. This is the retention of juvenile traits into adulthood and is one of those types of adaptations that can happen relatively easily in a population because usually there is a pretty simple switch "turn off" or "not turn on" the adult trait.
Blonde hair and blue eyes is another example of this in humans. These 2 traits are correlated in Northern Europeans because it is suspected that when people living in central Scandinavian took up agriculture the amount of Vitamin D in their diet dropped sharply. Since light skin means you get more UV exposure to make more Vitamin D in the cold, dark north a relatively simple mutation that caused the genes that control melanin pigmentation production to just not turn on or not turn on as much conferring a massive advantage. Note that indigenous people that maintained a high proportion of fish and wildlife rich in Vitamin D as a part of their diet such as the Inuit and Sami kept their darker complexion. Lactase persistence is correlated with this because milk is a very good source of this nutrient and also would have conferred a big advantage.