Development Continuous Integration for $5
Before I get to the meat of this post about a cheap continuous integration system, I’ll say that I was caught with my pants down earlier today. My Digital Ocean droplet was ‘hacked’ by a bot and was used as a spamming machine for an hour. I woke up to an email from Digital Ocean telling me my droplet’s network was shut down and that I needed to provide an immediate response.
Seriously?
So, what happened was about 98% my fault, but I want to put the tiniest amount of blame on Digital Ocean for not provisioning their VMs with a quick script to lock down the major problems (especially when logging in with SSH keys).
Essentially, two days ago, I spun up a droplet for what I assumed was going to be about an hour of work, because the machine I run Vagrant on was busy doing something else; so I figured I would pay about a penny to do what I needed to do on a droplet. For some reason, even though I provisioned a droplet with my SSH key, a root password was emailed to me (never happened before).
I logged in with the password and was asked to change my root password… No problem, except, since I was going to be shutting this droplet down in an hour, I set the password as “qwertyuiop” (not even thinking).
What ended up happening is that I logged out of the droplet, and left the city for the night, returned home the next day (yesterday), and completely forgot this droplet existed. And then this morning, I get the email from Digital Ocean telling me that my droplet is causing all kinds of hell.
A bot got in there using a simple dictionary attack (I think it was literally the 20th attempt or something equally absurd), installed a ton of cron jobs that spammed the world, and then DO locked that droplet down (I couldn’t administer it or anything).
Don’t be stupid
I’ve followed this tutorial every time I’ve created a droplet, because those other times, the droplets were intended to last a fair number of weeks, and locking them down was important. So, to save me some time in the future (and anyone else), I’ve written a little script that will provision my new droplet accordingly (small amount of inspiration from here as well)… One big caveat is that you MUST create the droplet with an SSH key attached, or remove the line about ‘PasswordAuthentication no’ (otherwise you’ll be locked out!):
#!/bin/sh
ROOT_PASSWORD="SomeUnbearablyLongButReallySecurePassword!!!!!1{}*_-"
NEW_USER="myusername"
NEW_USER_PASSWORD="mynewusersreallysecurepassword"
PORT_NUMBER=21312 # Security through obscurity
# Change root password
echo "root:$ROOT_PASSWORD" | chpasswd
# Create a new sudo user and set their password
adduser $NEW_USER --gecos "First Last,RoomNumber,WorkPhone,HomePhone" --disabled-password sudo
echo "$NEW_USER:$NEW_USER_PASSWORD" | chpasswd
# Transfer ssh credentials to new user (if any)
cp -R ~/.ssh/ /home/$NEW_USER/
chown -R $NEW_USER:$NEW_USER /home/$NEW_USER/.ssh
# Configure sshd_config to remove root login and allow only SSH keys for the new user
sed -i "s/Port 22/Port $PORT_NUMBER/" /etc/ssh/sshd_config
sed -i "s/LoginGraceTime 120/LoginGraceTime 60/" /etc/ssh/sshd_config
sed -i "s/PermitRootLogin yes/PermitRootLogin no/" /etc/ssh/sshd_config
sed -i "s/#PasswordAuthentication yes/PasswordAuthentication no/" /etc/ssh/sshd_config
echo "AllowUsers $NEW_USER" >> /etc/ssh/sshd_config
# Reload SSH
reload ssh
echo "*************************************"
echo "WARNING!!! Before closing this session, open a new one with your new username to make sure it works!"
echo "*************************************"
Continuous Integration on the cheap
I’ve been taking some time to incorporate more process into my personal project development cycles. To whatever extent I can, I make sure I have a good source control process (e.g. git-flow and FDD) and I have unit tests and do some amount of TDD. That’s pretty standard across the board, but I’ve been slowly trying to incorporate other SW process implementations to my personal development, like continuous integration, code coverage tools, KPIs, static/dynamic code analysis, etc…
As I have projects spanning 6-7 programming languages (not one project, but across different projects), this becomes a bit tricky to implement, but the goal/spirit of what I’m doing is still there. While I have solutions for most of these processes, I keep coming back to continuous integration tools and whether I should self-host, use a hosted solution, pros, cons, etc…
To mention quickly, the major CI options in my mind are:
Travis CI: Free for open-source, links cleanly to GitHub, starts at $129/month for private builds
Bamboo: Atlassian’s offering, great Atlassian integration (BitBucket, Jira, etc…), starts at $50/month for hosted builds (+ compute time on Amazon AWS) or $800 for a self-hosted license (I’m ignoring the $10 license solutions)
Jenkins: Open-source CI system, needs to be self-hosted, no limits
While not everything is about price, for a lot of startups, or even established companies, hosted solutions can add a huge monthly bill. I also don’t recommend buying servers you then need to coddle to do everything in-house (unless you have super sensitive IP).
There is a very legitimate middle-ground…
Enter Digital Ocean
Digital Ocean has those sweet 20GB SSDs with 512MB RAM for $5/month VPSes. You know what that means? A pretty speedy little build machine!
For comparison, Travis CI offers this in terms of hardware, again, starting at $129/month - ranging to $489/month for 10 concurrent builds (not really a lot in an enterprise environment):
We use dedicated hardware to ensure full speed for your tests. You have 3 GB of memory and up to 2 cores available. Your tests are running directly off SSDs with the most common databases optimized to run off RAM disks.
With Digital Ocean, for $20/month, you can get 2GB of memory and 2 cores
- for $40, 4GB RAM. So, for 3x-6x cheaper than Travis, you can get equally powerful hardware.
What about software?
Well, here’s where our old friend Jenkins comes into play…
Setting up Jenkins in Digital Ocean is a 5 minute affair, as illustrated in the build script below (inspired by this page):
wget -q -O - https://jenkins-ci.org/debian/jenkins-ci.org.key | sudo apt-key add -
sudo sh -c 'echo deb https://pkg.jenkins-ci.org/debian binary/ > /etc/apt/sources.list.d/jenkins.list'
sudo apt-get -y update
sudo apt-get -y dist-upgrade
sudo apt-get -y install build-essential autoconf libtool git jenkins
sudo service jenkins start
I included a few tools that are nice to have on a build machine as well (e.g. Git).
From here, navigate to your IP address and port 8080, and you’ll see Jenkins is ready to rock.
Hop over to “Manage Jenkins” and follow the security setup, and then install whatever plugins you’ll need for your build environment (e.g. Git Plugin). From here, I highly recommend following the Jenkins docs to get setup and building.
Incredibly minimalistic example
Just to show how fast you can get started, here is what you need in a new project to build Google’s Protocol Buffers (for example).
Create a new project
Setup the project to use Git to pull down the code (https://github.com/google/protobuf.git)
As your build process, use a Linux shell script with the following:
sh ./autogen.sh ./configure make make check
Save and run the build. In 5 minutes, you’ll have built (and tested) Google’s Protobufs! This skips out on about 50 really cool things you can do with Jenkins (split up building and testing, running code coverage, running lint tools, getting reports on passing/failing builds, etc…), but still…
Is it worth it?
Well, as with everything, it entirely depends on your company’s setup. Do you have dedicated build masters? Do you need them? Do you want them? Is the cost of hosted/licensed build tools prohibitive?
In my opinion, continuous integration systems are a must have for every project, as they really inject accountability/pride into your team (no one wants to be the guy who is associated with failing the communal build).
Whether you go with Travis, Bamboo, or Jenkins completely depends on how concerned you are about hand-holding your builds. In my experience, a fragile build process will crap out regardless of the tool used to host it, so there’s something to be said about robustness (or lack thereof) of the builds themselves (hardcoded links, environment variables, library versions, etc…).
I think Jenkins is fantastic and have never had issues with it. Some anecdotal evidence.
At my old company, we had a build system that literally needed 1 or 2 dedicated people to hold its hand every single day. Builds failed and then magically started working again. Sometimes there would be incremental builds, sometimes full re-builds. Who knows!
Then we switched to Jenkins and while there was a learning curve, once you knew it, you were gold. We had a half-decently powered server which had about 40-50 projects on it from my team alone, and got clean nightly or continuous builds, error reports, testing reports, commit info, etc… And no one needed to babysit it for the year and a half I used it.
In both of these instances, we still used the same cobbled together ant build scripts, but Jenkins managed to power through!