I don't see how "nothing else with a lasting impact" happened. I think this movie set up a few important things that will affect the future films:
- A real re-emergence of SHIELD as an effective force (with the helicarrier and new SHIELD/Avengers base)
- New Avengers roster with Vision, Falcon, and Scarlet Witch
- Upping the animosity between Stark and Captain America
- Popular opinion turning against the Avengers after Hulk's widescale destruction in Wakanda.
- Acknowledgement of the infinity gems, the danger of them coming together, and Thanos setting out to collect them
In fact, rather than saying that nothing with a lasting impact happened, I'd say it spent a little too much time setting up future movies. As others have mentioned in this thread already, it definitely felt like a "middle" movie, meant to build characters and set up the next films in the series moreso than be a standalone, great movie by itself (which it still was, imo).