Finally, I was able to complete my CI/CD pipeline to streamline flow from development to deployment. I wanted to share my experience related to backend systems and managing your own servers. This is not meant to be copypasta guide but a direction you can look into for managing your services. I will try to cover details in later iterations, this will remain a work in progress🤝
While I understand the lure of managed services for some where only thing you need is the code , rest is a black box , promising security, scaling and most importantly abstraction over all the ugly part. It is usually also a great choice when you are suddenly experiencing traffic with a small team but cost could accrue overtime. There is this huge VPS vs managed services debate on tech twittter these days for small and big teams with many switching to self hosting to save cost, I have been following some of it.
For someone like me who loves building and breaking things, managed services do not make the cut. I don’t have 1000s of users or intensive workloads to manage, even if it were the case I would probably be inclined to a self hosting solution than managed ones, which will eventually save cost and give me more control. I feel this need to own my infra, this might be a wrong take but it is there.
Subhas Chaudhary from Dukaan often talks about how they manage their own infra, you can check out his talk.
Now I would go through how I am doing it, all you need a cheap-ass vps server (1gb memory,1 core cpu) and a domain name and world is your oyster.
I use docker and docker compose extensively to streamline my flow. Put a nginx proxy in front, for DNS you can use cloudflare for added security, create a ssl certificate for your domains and subdomains using something like let’s Encrypt, ( Dive deeper into how let’s Encrypt work and why it is free ).
Now that traffic can safely land to your servers, you have to setup services that will be served. For first time you can build and containerize your angular and backend services push them to dockerHub or any registry of your choice and scp the compose file to your server and simply do a docker compose up in the directory of compose file to run on your servers like you would run them on your own machine, .i.e localhost and proxy requests from nginx.
Now getting to automation part, you are building docker images on your machine, pushing them to a remote registry (which you can host too), then on your server you are manually running a bunch of commands command each time you want to run a services, this is a lot of work especially when you have 5-10 services running and you need granular control over each.
This is where a Continuous integration and continuous deployment comes to rescue. My flow looks somewhat like this:
I have a main branch and a release branch for my services, I develop features in feature branches and once it is ready , it is merged into main. Once I am ready for deploying the changes, I pull those changes into release and do a push. Push triggers a github workflow which builds my image and pushes it to registry.
Now that I have my docker images, the part where I need to build and manage image for each service is automated, I can focus on development now on any machine with git and internet.
Next part is deployment, I still will have to login to my server and manually run bunch of commands to stop containers, delete old images, pull fresh images and then start those containers again. It is a lot of work if you have multiple containers running.
I use a webhook and trigger deployment from github workflow itself, after first stage , deployment step is initiated where I send a protected payload with openssl signature to webhook server that is running on my server ( checkout this fantastic project if you are learning golang), webhook server now handles the deployment.
Bunch of scripts are run by webhook server and Voila🚀 you have your changes on your site.
I would write more about this setup and extend it to be production ready with analytics ,observability and how to manage databases, migrations and security issues once I get some time. Until then Adieu!!🤟