Marlo is terrifying because he is emblematic of the hopelessness of the situation. The Avons, Stringers and Prop' Joes of the world, they felt rejected by society and decided to build their own parallel society in which they could thrive. Marlo does not care about any society. He only plays by his own rules, he is ruthlessly anti-social. He has no hope and no dream, only an unquenchable thirst for domination. There are systemic problems that create the Avons, the Stringers and the Prop' Joes and these problems can hopefully be solved. But what can you do about the Marlos?
There is a theme in the show kind of showing the degradation of "Gangster Society", Bunk says something of it in his speech to Omar, about how despite the older generation being bad men, they still had rules. You see it in a few other places, too--like the Sunday truce, and what not.
The theme seems to go along with Plato's republic--too, it's pretty neat.
Proposition Joe=Aristocrat. The philosopher king. Old school, part of the generation that formed things like the Sunday Truce. He runs his own business, helps his neighborhood--as you said, he became a gangster because they needed that society to fill a void left in the system from societies short comings with their people. (Since his age began right after the civil rights movement, and the start of the drug war.) The point of being a Gangster wasn't about power, or honor, but about ruling justly--as much as they could given the landscape they were in. Hence things like respecting civilians (As Bunk said) or the Sunday truce.
Avon Barskdale=Timocrat. The warrior caste. Plato describes these guys as warriors--and that's what Barksdale was. He still played by the "rules" and was concerned about his image. To him, money? Didn't matter. It was more about honor. He was willing to throw his financial empire away, in order to assure his own honor and power. Perfect Timocrat.
Stringer Bell=Oligarch (Capitalist). This theme was played to an extreme. Check out the Wiki's summary of the Oligarch.
, Plato says that he might have been the child of a timocratic man: The son initially emulates the father, and is ambitious and craves for fame and honor. When, however, he witnesses the problems his father faces due to those timocratic tendencies - say, he wastes public goods in a military campaign, and then is brought before the court, losing his properties after trial -, the future oligarch becomes poor. He then rejects the ambitions he had in his soul, which he now sees as harmful, and puts in their place craving for money, instead of honor, and a parsimonious cautiousness
Pretty much Season 3's arc, with Avon as the father and Stringer as the son. When he sees Avon is about to throw away money and perhaps even his life, all for honor--he rejects that "life", and wants them to go legit and use their money to live comfortably, and make more money.
Then you get to Marlo--the Tyrant. That's where we are at with Cartels, and a lot of inner city gangs today. Marlo shows that even gangsters used to have a society, and between the utter hopelessness of that life, the crushing drug laws, and the huge void society has left for all those people (Even the FBI doesn't care about them anymore)--leads to anarchy, and someone like Marlo, the Tyrant, swoops in to take power.