Finance still existed in the 70s and 80s. The main change in the business has been the increasing scale of productions. Top Gun (1986) had a budget of $15 million, adjusted for inflation that's less than $50 million. ET had a budget of $11 million. Back to the Future $19 million. Star Wars $11 million. Today, $100 million budget is pretty much table stakes to produce an action film with tentpole films doing multiples of that. With more money at stake, it follows that there's a tighter risk management function as well. This is a big part of why we have so many remakes and sequels now - previous box office success is the strongest predictor of future box office success. The Lord of the Rings films that you rightfully laud were not exempt from this sort of Hollywood optimization function. Changes were made to the source material to make the films more marketable to general audiences, such as expanding the role of Arwen and deleting the scourging of the Shire. Even though most reasonable people would agree these changes make sense in the context of making a Hollywood film, there are Tolkien purists (including his children) who reject Jackson's interpretation of the story, just as there are OG Transformers fans who curse the legacy of Michael Bay.
"If you make a good movie, all kinds of people will go see it" is a nice platitude, but it's not reality. It's generally true that "good" (as in generally well received) movies will do better business than ones that aren't, but it's very straightforward to predict the demographics of who will go see your movie based on the content of the film and the cast. For example, we know that Wes Anderson crowds are going to be nearly all white while the audience for the next Madea movie will be nearly all black. We know that a romcom audience will be 55%+ female while action movies will be 60%+ male. UCLA puts out an industry report every year that has some interesting data on audience demographics:
Something I'll point out that isn't directly related to the conversation be relates to audience demographics in general is that whites are underrepresented in movie audiences and minorities are overrepresented. Whites make up ~60% of the US population, but account for only ~45% of movie ticket sales. Table 4 in the above report gives you a glimpse of that. This isn't a new phenomenon coming out of the DEI push either - it's something that's been historically observed. So while Hollywood rightfully gets a lot shit for pushing DEI bullshit on productions and audiences, it's not as crazy or self-destructive as you may think when you understand who is actually paying to see their product.