Outside-in — To the max! (BDD and Microservices part 2)

This is part 2 in a series about my 20% project BDD across team bordersPart 1 can be found here.  

When writing any piece of software, it’s very easy to start working on the details before you have enough knowledge about the targeted context. I think most developers recognize this pattern:
You start with some innocent (often unintentional) guessing about one or a few unknowns. Pretty soon the guesses turn into assumptions and before you know it they have morphed into ‘truths’. You happily code away with these truths as your guiding light. Later, when you try to integrate your code into a bigger context, the assumptions and truths fall apart and you need to rethink your work.

– Oh, is that how the incoming data looks like!
– Why didn’t anybody tell me I have to ask the database for that value?


Personally, I have made these kinds of mistakes countless of times, and I will probably make many similar ones in the future. I think this is due to human nature; We really like making assumptions for some reason. To minimize the impact of this phenomenon, we have to try really hard to be aware of it at all times. We also have to find strategies and techniques that help us avoid making these false assumptions.

spiralTest Driven Development (TDD) is one excellent tool for this purpose. It forces us to step outside of the current context and look at the problem from the outside, or from “one level above” if you wish. More importantly, the TDD approach fits on all levels of software development! No matter if your context is a oneliner method in a class or the public API of a complete service, you always benefit from taking one symbolic step out of your current context.

In my 20% project BDD across team borders, I wanted to take this line of thinking to the extreme, and work on the highest level possible, thus making sure I was taking the “outside-in” approach to the max, maximizing our helicopter view in the process.
I set as a constraint on my work going forward that I was only allowed to write the BDD tests against our publicly available APIs and web pages. No matter how tempting, I would not “cheat” by calling internal APIs or access anything not available to a user in production. If I did, I would risk getting my assumptions wrong, as described above.

Working against public APIs also provides another great bonus;  It forces you to write user-centric stories that are sliced vertically. A good rule is that “Every story must have a noticable effect on the public API level”. If you write a story that has no noticable effect for any user, you have not succeeded in writing a vertical story.

My claim to test on the “highest possible level” is not entirely true, of course. There will always be third party dependencies and other integration points that we cannot incorporate in the system under test. We will have to fake or mock things like partner APIs, for instance. As long as we don’t mock or fake our own code or apps, we should be good to go!

I want my helicopter view back! (BDD and Microservices, part 1)

502377266

Here at SpeedLedger Engineering, we took the plunge into the world of microservices some time ago. We have gradually phased out our old monolithic style applications in favor of many small and independent micro services/apps.

Overall, the micro services paradigm aligns very well with our organization (small independent teams) and we are getting pretty good at it, I must say! We are constantly improving our infrastructure and deployment tools, enabling us to develop and deploy quicker and more often. As a result, the general time-to-market has decreased and our customers are getting more new features, sooner.

However, one big drawback of working in isolated islands of people and code is that you risk losing the big picture, the “helicopter view”. Unsurprisingly, this is what happened to me and my teammates. We often found ourselves asking questions like:

  • What’s supposed to happen in this business flow?
  • What are the possible inputs to this API?
  • Which apps will call this endpoint?

The answers to these and similar questions are not impossible to find but it always takes valuable time to track them down and to get them clarified.  

I felt something had to be done about this situation. For a long time, I have been a fan of BDD (Behavior Driven Development) but I never really got my feet wet in a real world situation. I suspected our current situation could benefit a lot from a successful BDD effort, and that it provided a perfect opportunity for me to experiment with and learn more about BDD.

For someone who doesn’t know what BDD  is, it can be described as “executable documentation” or “executable examples” that a non-programmer can understand and edit. More about this later…

After some initial discussion, we decided that I would do this in the context of a “personal 20 percent project”, famously invented and used by Google.  The main reason being that part of the work had to be done outside the team boundary, touching all the other teams in some way. In line with this, we actually named the project “BDD across team borders”. 

This was in February, and since then I have spent most of my Fridays exploring the possibilities of introducing BDD at SpeedLedger, and – more generally – automated testing on the highest possible level in a microservices environment.  In a series of coming posts, I will blog about BDD and the progress of the project. Hopefully someone will find it interesting. Also, I would love to get some feedback from you blog readers along the way!

 

QCon London venue

QCon London 2016

 

I’ll try to summarize three great days at QCon in London, not going into detail about all sessions that I attended but high-lighting some of the ones I really liked.
Most sessions were recorded and will be available for the public audience over time, make sure to check them out!

 

QCon offers a wide variety of tracks ranging from low level “Close to the metal” to more “soft” skills like “Optimizing You”.  There were also a track for the main sponsors of the conference. The wide variety of content and speakers made choosing what session to attend somewhat of a problem; I had made a schedule before traveling to the conference, and it broke down during the first presentation of the tracks before the first keynote…
Kicking of QCon was a great keynote “Unevenly Distributed” by Adrian Colyer. Adrian reads a paper a day, summarises it and publishes it on his blog “The Morning Paper”. This was a very inspiring and well presented keynote that raised my interest in reading papers and as Adrian said “5 reasons to love papers”:

  • Great thinking tools
  • Raise your expectations
  • Applied lessons
  •  Great conversation
  • Unevenly distributed

The only problem with reading more papers and learning more is that:

“The more I learn, the more I realize how much I don’t know.” – Einstein

After the keynote we listened to Gavin Stevenson, Engineering Lead at WilliamHill, who talked about WilliamHills betting engine and how they are transitioning from a large database centric solution to a micro service based architecture (this was a common theme during the conference). They were building a “production ready” betting engine in 2 week sprints, testing it with real production data. The most interesting take away was how important it is to really try to break your system. When the system fails, that’s when you learn. The old saying “If it ain’t broken, don’t fix it” just doesn’t apply anymore.

“If it ain’t broken, try harder!” – Gavin on testing

One of the few really non-software related talks was held by long distance runner Simon Wheatcroft.
When losing his sight the age of 17 due to a genetic degenerative eye condition, he began a journey of adapting tech to achive the impossible.
Through the aid of the Runkeeper application, he started running solo outdoors. Simon will be running his first solo race in May 2016; the Four Deserts Series Sahara Race in Namibia. There, he’ll use GPS coordinates and a mobile app to navigate across the 250 kilometer distance.
A very inspirational and humbling talk!

“You just have to tell yourself that pain doesn’t last forever”- Simon on running 250k

From non-technical to real in-depth low level Netty implementation details. Norman Maurer from Apple described how Apple is using Netty as a web service delivery platform for most of Apple services. Apple are running 550000+ Netty based services, handling 10s of Petabytes of data every day and millions of request per second.
Norman guided us in different aspects of the Netty framework and how the default JDK implemtations just aren’t good enough for this kind of load and how they’ve commited several improvements to the Netty open source code. Very interesting and down to the metal of how to hack the JDK and using JNI to get better performance with; for example memory allocation and SSL.

Martin Kleppmann held one of the best presentations of the whole conference where he talked about keeping data sources in sync, moving away from (distributed) transactions to streams. The content was not very in-depth, but Martin had deep knowledge of the subject, excellent slide and a lot of energy when presenting. This is a talk everyone should watch and learn from.

“Stupidly simple solutions are the best” – Martin Kleppman

Day two started with Linda Northrops keynote “Reflections on Software Architecture”. Linda looked at software architecture and how it’s importance and acceptance have changed over the last 20 years. To summarize: architecture is important, and it’s a way to manage technical debt.

Josh Evans from Netflix presented how Netflix have expanded there streaming services to almost the entire globe (Netflix#Everywhere).
Netflix have had some major outages and failures, both in their own software and the underlying cloud AWS-platform. Josh concludes that “Failure is inevitable” and that one really have to embrace the failure and not fail in the same way twice. This had lead Netflix to embrace a “Failure-driven architecture” approach when building their platform.
Netflix’s architecture is really impressive, although not applicable for most comapanies/services, so it’s always interesting to hear what they are doing to actually run the platform at that scale.
Josh presented Netflix four architecture pillars; data, caching, traffic and micro services, and how they use (among other techs) EVCache, Cassandra and DNS to keep their services up and running in case of total failures of an AWS datacenter/region. He also showed how they test failure in different regions and route trafic to another region to minimize customer impact.
If infrastructure and architecture at scale is of any interest, watch this talk when it comes online!

“Never fail in the same way twice” – Josh Evans

Mitchell Hashimoto (founder of Hashicorp) gave his talk “Observe, Enhance, Control: From VMs to Containers”. In his talk he takes us back to 2006 and the age of VMs and how the datacenters and the problems to solve are driving the architecture of the software for Monitoring, Configuration and Deployment. Jumping to 2016 and the age of containers, Mitchell argues that the “state-of-the-art” tools from the age of VMs are not really suited to handle the tasks anymore. Even though the tools are extremely good, they do solve a completely different problem. The content of the talk was nothing new, but it is really inspiring to listen to Mitchell talk.

Gil Tene talked about Hardware Transactional Memory. Really low-level stuff about CPU pipeline and cache optimization. HTM in the JVM is not new, Azul has been delivering both hardware and a customized JVM with JVM for 10 years. What’s interesting is that it will become mainstream now when Intel is shipping CPUs with support for HTM. Gil succeeded in a very educational way describe the complexity of HTM and how it can be implemented in for example the standard JVM. In the end Gil talked about how the developers must reason about locking and synchronization to make the most of HTM in their code.

One of my most anticipated talks during the week was Dan North‘s “Making a sandwich”. I hade very high expectations for this talk, and Dan managed to exceeded them (as usual). Dan talked about giving feedback, how feedback in itself is a system and why we should do it. Giving and receiving feedback (which is really just to say ‘Thank you’) is, in my opinion, one of the hardest skills to master and we should really practice a lot! Dan presented some useful techniques and tricks, but you should really watch this yourself!

Last day started with a very entertaining and inspiring keynote delivered by Kolton Andrus (Netflix) and Peter Alvaro (University of California). Peter and Kolton shared their expereience of a very successful collaboration between industry and academia. Peter had a “big idea”, Lineage-driven fault injection, and together with Kolton this evolved from a theoretical model into an automated failure testing system that leverages Netflix’s state-of-the-art fault injection and tracing infrastructures.

“My code is now actually running live on Netflix…” “…well, minus all of the println statements”

Vikki Read and Alex Wilson from Unruly described how they are using the extreme programming (XP) ideas to deliver high quality software and how it can be made to work in a very agile environment. Since agile and XP focus a lot on collaboration and knowledge sharing, they shared some problems that they’ve had with getting new employees up to speed and how growing teams make for example (too long) stand-ups being a problem.

The final session I attended before heading to the airport was Tammer Saleh at Pivotal talking about the mistakes people make when building a microservices architecture, i.e. Microservice anti-patterns. When are microservices appropriate as an alternative to the monolithic app? The problem with monolithic apps is not about the code, it’s about the teams! Large teams (or multiple teams) can’t work effectively in the same codebase.

Its not about code, its about teams

Its not about code, its about teams

Tammer stressed that “the most common mistake is to start with microservices”. Start monolithic and extract as needed, because microservices are complex and impose a constant tax to your development.
Tammer explores how to draw the lines between services. dealing with performance issues, testing and debugging techniques, managing a polyglot landscape and the explosion of platforms, managing failure and graceful degradation.

“Boring is beautiful” – Tammer Saleh

Most of the interesting sessions I attended during the week were about failure, and how to handle failures. Quotes like “Failure is inevitable”, “Failure is an opportunity to learn” and the importance of building an architecture that can manage failures were common topics. Migrating from a monolithic application to a more micro service oriented architecture were also popular.
Overall QCon London was a great conference, I will most likely try to get back next year! All tracks had great speakers, which is problematic since you have to choose between sessions – on the other hand most of the sessions are recorded so I know what I will be doing the coming weeks.

 

Do we need the database at all? – Event Sourcing!

In my previous blog post I reasoned about how the database could be hidden behind an ORM tool. And logic could now be written in lambdas (java or scala) instead of SQL. The ORM tool will transform the data model (residing in the database) to an object model (consisting of  java/scala objects).
I pinpointed  some big advantages of this, but I also mentioned that there was an even bigger advantage with the object model that I will address in this post.
Event Sourcing, let me explain what it is with a simple example.

Example: The dry cleaner

The dry cleaner is a place where you can leave your clothes to get them cleaned. In return for your clothes you receive a receipt. When the clothes are cleaned, you can get them back by showing the receipt.
A simple object model for this could be:

(more about how the DryCleaner trait could be implemented later)
The dry cleaner trait has methods for leaving and retrieving clothes as well as a method for looking at the dry cleaners current inventory of clothes.
A simple example of using this model:

The implementation of the DryCleaner could be backed by a database using an ORM tool, but let’s consider another alternative right now. What if all modifying methods of the DryCleaner implementation wrote their arguments to a log file? (the leave() and retrieve() methods modify the object model while the inventory method is a read only method).

The Event Log

What if all (modifying) operations on the object model resulted in a log entry. If all modifications was stored as “events” in a log.
A log created from the example above could yield the following log:

Note: the receipt number need not be present on “leave” rows, but increases the readability

This log contains all the information contained in the method calls of the example plus a serial number of the call and a time stamp. (The timestamps in this example has been modified to improve readability, the code example above would of course execute instantly).
Could you recreate the object model from the information contained in this log?
Yes!
How could the object model be recreated?
By using the information on each log row to call the right method on the DryCleaner object.
Does this work for complex object models?
Yes, the object model can be as complex as you need, below I will describe how to automate the process of reading/writing this log.
In that case, can you do without a database?
Yes, you can choose to store this log in a text file if you would like.

This is called Event Sourcing, and here are some benefits of Event Sourcing:

  • The object model will be portable using a simple text file
  • Human readable, if your events have good names, the resulting log is easy to understand for a human.
  • The log itself is a very elaborate audit log.
  • No database data model needed. Even if you store this log in a database, you do not need to model the entities in a data model.
  • A time machine, you know everything that has happened and when (unlike a data model in a relational database).

The time machine

What if I need to know the inventory of the dry cleaner at a specific point in time?
If I want to know the inventory of the dry cleaner at 2015-08-28 at 09:00, I simply load all events that has a timestamp less than this time. Thus we will have recreated the exact state of the dry cleaner at this moment!
In fact, we can recreate the state of all involved objects for any time we choose!
This is an extremely useful feature of event sourcing. You can return to any point in history to see exactly what happened.

Additional benefits of event sourcing

  • External systems can listen to the events (collect statistics or react to events)
  • Extreme performance. All operations on the object model will be in memory (no need to go to a database). This will be extremely fast.
  • Performance for persistence. All events are immutable and are always added, never removed. You can use an append-only operation for storing them.
  • Simplification. The events are simple. No complex domain object or graphs of objects need to be saved. The object model can of course be as complex as needed, the complexity is not transferred to the persistence mechanism.
  • Testing. It is easy to set up test data, a pre recorded event log can be used to set up the test data. No messy mocking needed.
  • Flexibility of design. The events does not describe the domain objects. So the object model can be redesigned without any need to migrate existing data to the new design.

And some Cons

  • You might need to support ‘old’ events for a long time. You can easily add new types of events, but removing old events is harder. Because you still have to be able to read old event logs. There is ways to remedy this (by making snapshots for example). But still, it could be a pain.
  • Event Sourcing is a less known technology compared to RDBMS etc. It will pose a learning curve to most software developers. And it might be harder to sell to an organization.

Automate the reading and writing of the event log

How do you implement this? Do you have to implement logging to the event log in every (modifying) method? And to parse and apply the states again, do you need to implement this for every class?
No, the implementation could be done through the reflection mechanism. Once and for all for all entities. You could use a naming convention or annotations to mark which methods that modify the object model.
In our bookkeeping application we implement the logging to the event log through a single class of 100 lines of code. All our 120 entities use this single class for all event writing and reading. This is an extremely big win, only 100 lines of code to handle all persistence for a very big application. All the slick code (see my last blog post) could also be removed! We can now concentrate on writing application logic!

Application of Event Sourcing at Speedledger

Our main product at Speedledger is our bookkeeping application. And bookkeeping seems to be the perfect match for event sourcing as described above.

  • We can restore any customers bookkeeping to any point in time. And thus we can let our customers do this in some cases (however, there are some legal aspects what you can do with your bookkeeping).
  • We can easily transfer data for one customer from one system to another. For example, we can transfer customer data from our production environment to the system used by our support personnel simply by exporting the event log as a text file. The support person can then view an exact copy of the customers data in their system while helping the customer.
  • Our support personnel can use the event log itself when helping customers, they see exactly what the customers has done in their bookkeeping. The support personnel has no specific technical training but can still read and use the event log since it only describes bookkeeping events (and not in a very technical way).

You need a database!

Just because it is possible to store the events in a text file without the database entirely, I do not recommend this. A database has many other advantages (transactions for instance). The point is that you do not need a relational database or SQL. You could use something much more lightweight, like a document oriented database. And the data model can be extremely simple, you just need a place to store the event log.

100% consistent object model and no bugs (next blog post)

What about database constraints? The database helps you perserve data integrity by using constraints. If you do not use a real data model (or a database at all), how can you preserve data integrity?

In my next blog post I will describe how to preserve data integrety in an object model with Event Sourcing. And in a much more complete way than you can hope to acheive by database constraints. And this will in turn lead to a manner of coding that makes bugs much less likely.

I want to write lambdas, not SQL

When lambda was introduced in Java, I was over the moon. I thought it was the best feature ever… soo useful! Though lambdas is no longer a new feature in Java (it has been in production for ~1.5 years), I have not used it nearly as much as I wanted. Why? Because the application I write most of my code for use plain old JDBC. Why is that a problem, why can I not use lambdas anyway?

Look at the figure below, these two examples do the same thing, one in SQL and one with Lambdas:

SQL

private static final String GET_CALCULATED_RESULT =
    "SELECT SUM(vertrans_amount) AS verifSum " +
    "FROM verif_trans vt, verifs v, periods p, accounts a, financial_years fy " +
    "WHERE vt.acc_id = a.acc_id AND a.acc_num >= 3000 AND a.acc_num < 9000 " +
    "AND vt.ver_id = v.ver_id AND v.period_id = p.period_id " +
    "AND p.finyear_id = fy.finyear_id AND fy.finyear_id = ? ";

Lambdas / Object model

SLMoney totSum = allVerifications.stream()
    .flatMap(verification -> verification.getVerifTransactions())
    .filter(verifTrans -> verifTrans.getAccount().isProfitAndLossAccount())
    .map(verifTrans -> verifTrans.getAmount())
    .collect(SLMoney.sum());

The two examples above do the same thing, why is the lambda version better?

Here are some reasons, the really good reasons require some more explaining (more about that later):

  • The lambda version can be parsed by the IDE and compiler (that will tell me if I am making some syntactic error or misspelled something)
  • The lambda version can be easily tested in a unit test. The JDBC version require testing in a real database that first must be populated with test data.
  • The lambda version require no extra code around. The JDBC code is best written in a separate class with a DAO interface in front to keep the java code readable.
  • The lambda version is written in Java, the other is not (only wrapped in java).

Object model – the key to using lambdas

To efficiently use lambdas on our domain, we need a real object model. An object model where our entities have real relations to each other. Not relations through database ids but real java references. Like this:

organization.getFirstFinancialYear().getAccouns().get(0);

And not like this.

FinancialYear financialYear = financialYearDao.getFinancialYear(organization.getFirstFinancialYearId());
List<Accounts> accounts = accountsDao.getAccounts(financialYear.getId());
Account account = accounts.get(0);

One way of getting what I want

It would be tempting to throw away the application and start all over again with the object model approach. But for several reasons, a big rewrite of our application is not the best option . Instead, we’ll have to “dig where we stand”. When we introduce an object model, the “old” code must still work in parallel. The new object model can be used when implementing new features.

The obvious way to introduce the object model is an ORM tool! This way we can keep the database while still exposing it as an object model.

An ORM tool would have to have the following characteristics:

  • It must be uncached, because it will have to work in parallel with old code written for JDBC.
  • I would like the entities to be immutable (why? more about this in my next blog post : Event Sourcing)
  • I would like the object model NOT to mirror the data model entirely. A naive ORM tool would map database tables to objects that exactly mimic the table with all the attributes. I do not want this because I would like to hide some attributes that are not needed in an object model (like database Ids and unused columns). And in some cases, I want to remodel the entities and relations between them in a way that does not directly map to the database.

My choice fell on Slick, a Scala ORM framework.. And as an extra bonus, no SQL whatsoever is needed since Slick lets me write all database queries/updates/inserts in pure Scala (with lambdas!).

20% Project

I told my boss Pål about my ideas, how we could simplify things, reduce the number of lines-of-code (LOC) and make the work more fun. Being a visionary guy (and an ex developer himself), Pål immediately saw the potential. We discussed the idea and decided that I could dedicate some time to work on this. I would dedicate one day a week to this project, a so called “20% project”.

Results

I have now spent 60 hours implementing the object model in our flagship product (the accounting application). And it really looks promising! We can now write new functionality using lambdas (in Java or Scala) like figure 1. The amount of code needed to write a new service in the application is reduced to a fraction, while the readability is improved.

Exporting the object model in JSON

A really cool side effect of this is that we can now export customer data from the database as JSON! Since the object model is entirely made of Java/Scala objects, I can use reflection to export all the data in the model as JSON (and then importing it again). We have attempted this before (extracting customer data for one customer in JSON or XML) but doing this directly from the database has has been too complex (there is 120 database tables involved). Now it is trivial The class that serializes the entire object model as JSON is 58 LOC long and the deserializer is 73 LOC.

Number of code lines left (a thought experiment)

Code is a liability. This is the view of myself and my colleagues. If some LOC can be removed, they should! (if readability is maintained). Less code means less number of bugs and less code to maintain. Therefore it is interesting to see if and how much code we can save if we would rewrite the entire application using the new object model and lambdas.

Of course, all the old JDBC code could be removed. All other Java code could be rewritten using lambdas (I assume this code would be ~1/3 of the original size, I think this is a conservative estimate). Some code will be added to the new Scala slick database classes and entities.

Total accounting code 161857
JDBC classes (SQL + Java) -17606
DAO interfaces -5181
JDBC test classes -16869
Old Model objects -27419
All other code -2/3 * 94782
Scala slick DB classes +4572
New entities +6508
Total LOC left 42474

LOC left: 26% of the original

That’s a real big win by itself! But, wait till you see whats comes next…

Next blog post

In my next blog post, I will explain the really big advantages of the object model. Namely: a way to introduce Event Sourcing and removing the entire database.

Ship your Scala code often and easy with Docker

At SpeedLedger we have during the last year been working with gradually containerizing our services using Docker. At the moment we have 20 different services containerized that are running in production. During this period we have learned a lot and made a lot of mistakes. We are using a number of tools and developed some of our own. For example our deployment tool is built in-house and is responsible for starting and configuring our Docker containers in a standardized way to handle logging and service registration for our applications.

At the end of April it’s time for this years edition of the flatMap conference in Oslo. I (Marcus Lönnberg) will be hosting a talk there about how to deploy Scala applications using Docker. Some of the talk will be general about Docker and various tips from my journey with it at SpeedLedger. I will also talk a bit about how one can use sbt-docker, which I recently released version 1.0.0 of, to easily build Docker images using sbt. There will also be a few other tools that we will look into, for example the deployment tool we have developed at SpeedLedger.

SpeedLedger at the DataTjej conference

Earlier, in February, I participated in the DataTjej Conference held in Gothenburg. “DataTjej” roughly translates to “ComputerGirl” and DataTjej is a non profit organization aiming to increase women’s interest in computer science and IT, disregarding age or experience. Their long term goal is to increase the number of women in computer science and software development. Each year peaks in a conference held at different universities in Sweden. As of this year it was held at Chalmers University of Technology in cooperation with Gothenburg University here in Gothenburg. The conference aims to enhance the student-business relationship within computer science, IT and information systems. You can find out more about the organization and the conference here.

I have attended the conference several times as a student, but this was the first time representing my company. The very first time I came in contact with DataTjej was during my first year at Chalmers University of Technology. Some of my seniors organized the conference in Gothenburg 2009 and they encouraged me to go.

This year my co-worker, Marcus Lönnberg, and I attended the conference in order to promote SpeedLedger as an awesome employer and to recruit more women to SpeedLedger’s engineering department. We took part in three different activities during the conference; a presentation, a dinner and a fair.

So I had a few ideas when I put together the presentation. I knew we were going to have a booth at the fair, so I didn’t feel the need to promote SpeedLedger during the presentation, instead I wanted to give the audience something useful. Something code related. I also remembered my first time at DataTjej. We had just finished the second course in object oriented programming, which basically was Java with Swing. I hate Swing. During this period my mind was set on surviving the programming courses and then graduate to work with development processes and not touch code for the rest of my life.

But over time though, I came to realize that there are a different types of programming. Just because you don’t like one specific type, doesn’t necessarily mean that you dislike programming. You need to find the type of programming you like. As I see it, it’s always easier to learn something that you like and feel joy doing. For me it was datastructures and algorithms that changed everything.

So I let my presentation revolve around algorithms and how it awoke my interest for programming. Algorithms can be used to solve everyday problems and I gave the audience a demonstration of how two algorithms find the fastest path between two points, plus the theory behind it. To do this I hacked together a small program showing how the algorithm thinks in every step of the problem solving process.

A* solving a maze

A* solving a maze

I really wanted the presentation to have a technical aspect, rather than the typical business-connecting-with-students type. I hope the audience appreciated the technical connection and if anyone of them felt insecure I hope they were given a motivation to why coding can be fun. They have already chosen to study computer science, now we need to convince them to stay and enjoy the ride.

Back at the office I got the question: Why is this so important and why do we need girls in our teams?
This is a very relevant question. We actually don’t need women in our programming teams because they are just that, women that is. We need women in our team to create a heterogeneous group. Just as we need male nurses. There are numerous studies showing that a heterogeneous group is more efficient and drives each other forward at a much higher pace than a homogeneous group.

DataTjej’s most important task is to create an environment free from prejudice and to be a place where women, who otherwise might feel alone in a classroom full of dudes, can meet and share experiences.

So, to summarize: More diversity in software development! That is what we need to be stronger, better and faster as a team!

DockerCon Europe 2014

I just came back from DockerCon Amsterdam 2014 and here is a summary of the conference with some comments.
 
The theme of the conference was of course Docker. Just for fun I wrote down the most used terms on the conference just to give you a hint of what the sessions were about…
Orchestration, micro-services, cloud-native, distributed apps, scale and continuous delivery.
 
The common thread through almost all sessions were orchestration. Seems to me like the word orchestration means a whole lot of stuff while listening to the sessions. It involves creating Docker hosts, containers, clusters and monitoring just from the top of my head.
 

Docker and especially Solomon Hykes (original creator) and Ben Golub (CEO) spent some time explaining how the project is maintained, governed and how they will try to scale the project to keep up with all Pull Requests coming in. Some stats were presented and the numbers are quite astonishing.

Docker PR stats
It is neat how they handle organizational changes by doing PRs. The Open Design principle is also applied on organization changes by maintaining a file called MAINTAINERS in the repo. Check it out on Github.

Henk Kolk from ING presented how they have reorganized in order to increase speed of development. Developers are first class citizens.
IMG presentation

 Announcements

Docker Compose

An apparent theme of the conference was distributed apps. Meaning that you will run several containers cooperating to fulfill an applications goal. The way Docker has solved it with Docker Compose is very similar to how Fig works. A file describing what containers to bring up and links between them etc. I’m really looking forward to start utilizing the container composition capabilities on our systems.

containers:
web
:
build
: .
command
: python app.py
ports
:
- "5000:5000"
volumes
:
- .:/code
links
:
- redis
environment
:
- PYTHONUNBUFFERED=1
redis
:
image
: redis:latest
command
: redis-server --appendonly yes
And to have the containers up and running
% docker up

Docker Machine

In order to address some of the problems involved by getting Docker to run on a new host they announced Docker Machine. It comes with a new command ´machine´ that effectively creates a new Docker enabled host. There are a number of providers that you can use such as AWS, DigitalOcean, VirtualBox etc. When a new host has been created you can run Docker commands on your local machine that actually run on the newly created host. How cool is that?
machine create -d [infrastructure provider] [provider options] [machine name]
I can directly see the use of it in development. However, for production I don’t know yet.

Docker Swarm

Another announcement made on DockerCon Amsterdam 2014 was Dockers new cluster system. It enables us to control a cluster of Docker hosts that automatically puts a new container on a suitable host when running the usual ´docker run …´ command. You can set constraints on the run command for all properties that ´docker info´ gives you. So for example, you can state that OS must be Ubuntu.
% docker run -d -P -m 1g -e "constraint:operatingsystem=ubuntu" redis
 

Docker Hub

The team from Docker Hub announced a couple of new official images, Tomcat was one among them. They also announced Docker Hub Enterprise for on-premise use cases.

About CoreOS Rocket

The announcement of the CoreOS Rocket project was not commented and discussed officially on the sessions by the Docker team. Rocket is a new container engine project with a similar goal as Docker. It consists of two parts if I am correct; a container specification and an implementation. They spent quite some time talking about some of the things that CoreOS mentioned as flaws in Docker. I had a chat with a guy from the CoreOS team attending the conference. He made some good points and presented why they have started Rocket. The main reasons he mentioned:
  • The all-in-one binary for doing everything. Rocket is split up into multiple binaries more in the way the initial standard container manifesto Docker had. For example. There is probably no need for having the ability to run ´docker push´ or ´docker build´ on a production Docker host.
  • The design choice of having a daemon process for Docker. The design of Rocket is the reverse. Running containers should not depend on a running daemon. A start of a new container is a one-off job for Rocket which terminates after the container is up and running.
He also mentioned that Rocket probably will be somewhat interoperable with Docker.

Fluffy but important takeaways from DockerCon

DocerCon panel discussion

The conferences ended with a really good panel discussion. Here are some quotes I found inspiring.

Speed, especially in development process, is everything! – Adrian Cockcroft


Go deep, not broad. – Adrian Cockcroft


Do yourself a favor and present a new idea/technique in a representative way. A good idea deserves a nice presentation and good argumentation. Try to work with people upstream and downstream. – Solomon Hykes


If you know you have an excellent idea, believe in yourself. Don’t let negative opinions get in your way. – Solomon Hykes


All in all. DockerCon Amsterdam 2014 was a good conference. A lot more stuff can be found on Twitter.

Docker meetup startup resumé

The day after creating the Docker Göteborg Meetup group, around 30 members had joined. Now the are 65 Dockers in the group. The interest and buzz about Docker is intense for sure.
We started the group 21th of October this year and we decided to start off with a introduction meeting. It felt right to have a small group of people (20) and focusing on discussions instead of presentations. Here follows a brief summary of the meeting.

Short introduction of all participants

Everybody introduced themselves. The DRY principle was heavily applied since some people worked at the same company =)
 

Group introduction

Me and Marcus Lönnberg (the group organizers) are working at SpeedLedger and we started using Docker at SpeedLedger half a year ago. We started to use Docker for some of our internal services and single purpose production services. When we were in  the starting blocks of putting Docker in to use on our flagship product we felt the need to discuss Docker topics in a bigger forum. Hence the meetup group. We also have a co-organizer Katya from Docker who offered her help.
 

Leading presentation and discussion

We started by defining Docker together by writing down properties and characteristics. Most of them were positive but also some negatives were addressed. We moved on and started talking about how we are using Docker at SpeedLedger. We draw our production environment and a lot of question came up. These were some of the questions:
“How are you coping with docker host updates?”
“Where do you keep configuration? Inside or outside container image?”
“How do you communicating between containers?”
“How is the new production environment working out? Problems?”
“How do you deploy new images?”
etc.
 
A lot of good questions and interesting ideas. To summarize, the majority of the participants are not using Docker in production yet. Some are using it for other purposes such as test environments though.
 

What now?

We talked about practicalities around the next meetup. We will surely try to gather many more participants for the next meetup in order to involve and attract as many people as possible to the group. Seems like the next meetup would gain popularity by doing having more in-depth presentations. Lets see how that turns out!
 
Right now I am in Amsterdam and looking forward to attend the DockerCon Europe 2014 conference. If you want to see what comes out of it, follow me on Twitter and continue to follow this blog…
 
Take care folks!

Updating your Java server in runtime

It is sometimes convenient to have the ability to update an application in runtime. You might want to change the log level to track down some bug or maybe change the setting of a feature toggle. But it’s difficult to redeploy the application without disturbing your users. What to do? At SpeedLedger we decided to put the properties we want to change in runtime in a file on disk outside of the application. We then let our application watch the file for changes using Java’s java.nio.file.WatchService. So whenever a property in the watched file is changed the application is automatically called to update its state. We currently use it to add new endpoints in our traffic director.

Doing this first appeared to be really simple, the functionality is built in to Java since version 7. But doing it in a stable and controlled way required some thought, you cannot have your file watch service crash your application in production. So we created a small helper class to hide the complexity and decided to open source it. The code is available on GitHub.

To use it, simply add this dependency to your pom (or similar):

<dependency>
   <groupId>com.speedledger</groupId>
   <artifactId>filechangemonitor</artifactId>
   <version>1.1</version>
</dependency>

To create a new file watch you can do something like this when your application starts:

public class MyFileWatcher {

   final static String CONFIG_HOME = "/home/speedledger";
   final static String CONFIG_FILE = "file.json";

   public void init() {
      FileChangeMonitor monitor = new FileChangeMonitor(CONFIG_HOME, CONFIG_FILE, this::updateSettingsFromFile, 30);
      new Thread(monitor).start();
   }

   void updateSettingsFromFile(String directory, String file) throws IOException {
      ObjectMapper mapper = new ObjectMapper();
      String json = new String(Files.readAllBytes(Paths.get(CONFIG_HOME + File.separator + CONFIG_FILE)), "UTF-8");
      List<Endpoint> endpoints = mapper.readValue(json, mapper.getTypeFactory().constructCollectionType(List.class, Endpoint.class));

      // Update the endpoints
   }
}

In this example we watch a JSON file containing “endpoint” objects. Whenever someone writes to the file updateSettingsFromFile is called and the endpoints are read from the file and updated. If something goes really wrong, like the disk becomes unavailable or someone deletes the watched directory, the monitor waits for 30 seconds and then tries to restart the watch service.

It is a good idea to validate the data from the file before updating, if someone makes a mistake when editing the file we want to keep the current state and log an error message.

Note that if you run this on OS X you will notice a substantial (a few seconds) delay in the watch. This happens because Java on OS X doesn’t have a native implementation of WatchService. This shouldn’t be a problem as long as your don’t use OS X in production.

The file change monitor is available on GitHub and Maven Central, we hope you will find it useful!