Colin Baker caused a shit ton of problems, albeit not necessarily his fault. He had immense issues with the costuming (which was bad) and was generally scripted as the asshole doctor during his entire run. He was not well received by fans, who were hoping for a return to the more comedic and dialog driven Tom Baker years, as the whole Peter Davidson era was marred by the "the Doctor is a giant uncaring asshole" and too many fucking companions issue. By the time McCoy took over, the studio was trying to kill the show and it started looking like a community access show as a result, so I don't fault him for anything he said or did.
Mainly, though, the show entered a period where action driven science fiction was starting to get mainstream along with (for the time) good special effects. During the Tom Baker era, you could slog around with shitty rubber suits and matte painting planets because people were willing to accept that if the story and dialog were good and Tom Baker fucking owned the role in a way no one has really done before or since. Once the 80s started, people's standards for visual effects ramped up and BBC failed to change to match it. BBC frankly were coasting on Dalek popularity and the Tom Baker era for a long time without investing any additional effort into the show, just phoning it in to milk the franchise.