I was given a new project yesterday. Our director has had this idea for awhile but the previous VPs weren't about it so it never happened. The new VP comes more from an application development background than the previous and he wants it so here we are.
This project is based around the idea of self service for everything. We have a massive amount of Kafka streams for various reasons and more are created everyday. As it is today, whenever one created by the application team it eventually makes its way down to our team to build out something to consume it. Then to expose the Kafka messages in a way the BI people can understand and use in whatever they're doing. This application he is wanting is to do the following:
- Take inputs from the app teams.
- Like a form or something they answer a few questions to identify the Kafka piece and where they want it to go.
- These inputs are used to generate a bunch of DDL and DML for the necessary database pieces.
- Approval step for leadership to yay or nay this stuff.
- Executes all the above DML/DDL.
- Provision access to the users in question.
This is not a big deal but for some reason he is also adamant that everything it generates also be in source control itself. This is where I side eyed him. Generating the above is really no big deal to automate and it is a fairly low effort project. However we have five other code repos where pieces of the overall arch lives and all of that is operated by Snowflake DevOps (cool feature btw), DBT, and now Github Actions.
While yes we can generate files for whatever it would have to creates its own branch, add this to it, commit it, push it, and then generate a PR all on its own. I think this is absolutely insane. Am I just behind the times on something like this? I have never heard of such a thing. It seems. like you can do it just using the github rest api or something.
