McVeigh's Blog buzz     About     Archive     Feed

Simple app, simple deployment

Not really.

Background

So I was trying to learn a bit of scala, and tinkering with neo4j. I had built a simple web app dashboard for fitbit that displayed some data. The app itself compares two user’s step counts for the previous day and basically displays who “won” that day.

It led to a bit of good natured slagging when configured for my main rival (the girlfriend) and myself. I hadn’t really intended for it to exist anywhere else other than on my own machine, but when asked if we could see it from anywhere, well it just would have been rude not to go and deploy the thing.

The plan

I had a criminally under used ubuntu server at home, which was basically acting as a glorified NAS device, so this became the target home for the dashboard app.

I do fancy myself as not being completely clueless when it comes to server administration - at least in the linux world - I’m reasonably well adversed with setting up services, cron jobs, file permissions, network interfaces, port bindings and the like.

It is a simple app, only a couple of components. I already had scripts to run most of the ‘bits’ locally. I knew I had been lazy leaving configuration in the source code, but other than that, it was in decent shape. I’ll copy up the jars in a couple of minutes and be ready to go! Feck it, I’ll even copy up the code and the build files so I can update the configuration class and re-build it on the fly. And I am a fairly experienced developer, surely at this stage I would naturally be developing in a way that makes my life easier when it comes deployment time. A couple of hours at most to get this simple app running on another server.

The problems

Yeah it didn’t take a couple hours. Here’s just some of the things I ran into outside of the nice, safe, easy local development bubble.

  • Use of a private key for access to external server api - I can’t keep that in source code and publish it to github as well!
  • Use of relative paths in scripts - when running as a cron job it doesn’t get executed from where the script lives.
  • Server bound to ports on the localhost network interface - these connections are refused from outside, i.e. other computers on the network
  • I had to change the code for starting the embedded Jetty server to get around the localhost bindings (eventually used “0.0.0.0” for server host to work both on local dev and on the server - this binds it to all network interfaces)
  • This last change was one of about 5 source code changes I had to make (e.g including updating paths to files that I forgot were in use).
  • I also forgot I had disabled authentication to the neo4j db rest service - needed to do this on the target server as well (a common problem with doing things manually).
  • the jetty server could not write to the specified log file - chmod permissions, surprise surprise.

There is nothing particulary difficult in this list, this is common stuff that I’ve had to deal with in some form previously. All in all, it took a full days work to get all the problems sorted and the application running for real.

Now imagine this…

Take this simple example, warp it a bit, scale it up more traditional “real-life” deliverables. Imagine there are different teams of developers working on different services that form a larger platform. Imagine at the end all the outputs from the development teams get handed over to some poor guys who are then told to deploy it (probably over the weekend because some reason). Never mind that all the developers who had written the actual code have forgotten how much of it works, as they wrote it 3 months previously. Imagine that there are some of these services that the guys expcetd to deploy them know nothing about, they only have a vague idea about what they are susposed to do, never mind how they actually work. Imagine that the guys deploying also have to deal with a bunch of different suppliers who are actually the ones in control of the target platform - they don’t have root access to change the things they need once they have troubleshooted for hours to find out why it isn’t working. And thats just 1 of the 5 new services they need to deploy. They have to fill out forms to open a port on a managed and accredited environment like this, and then wait days for a turnaround.

Some of what I have written about here is kinda contrived, I’ve been through the ringer a few times so that I am not naive enough to believe I wouldn’t run into things I hadn’t considered before the actual deployment. I had more problems than I listed above, these are just the ones I remember. But I think there are some good lessons here that is good to keep in mind while you are developing.

Even if I went to my simple example project now after the fact and tried to make the building and packaging automated, there are lots of things I would still miss.

In my case I was in total control of all the variables here, the source code, the environment, the firewall etc. And yet it still took me ~5 times longer to deploy the thing than I said it would.

DevOps, Continuous Deployment

In the industry now, we’ve all heard the arguments for changing attitudes here. We’ve watched the seminars, talks, and conferences. The information is all around us now. Can any one can really argue for continuing to work the way we used to in the past. Look at the difficulty I had when deploying this simple app!

But yet it’s still not great out there. We are not getting this quite right, especially with large projects. This sort of stuff does go on. Even with people that do know better, and have the best of intentions.

We should start with ourselves and try to make small changes to get a little better next time. When we are working on delivering software, before starting to build anything, if you are not the person that is going to deploy it (and even if you are), talk to the guys that know about deploying to production environments. Get their input. Don’t assume someone else will do this and everything will be ok.

You’ll not be able to cover absolutely everything, especially for a first deployment to a new environment or platform, but the skill and experience of these guys will help you get there. Get your walking skelton app deployed to where it will live as soon as it is safe to do so. Make the time to learn the tools to get stuff deployed. Get involved with it!

Do not think it is purely up to “ops” to sort out the parts of the development you don’t understand, or you haven’t thought about, or you don’t know about, because it is their job, not yours. That’s not really fair, is it? Come on, it is all our jobs.