Automated Exploratory Test Fixtures

This is part 3 in my series about  BDD and MicroServices. You may want to read part 1, and part 2 first.  

As I have blogged about earlier, I have been spending some time exploring BDD/executable examples in a MicroServices environment, and how we can benefit from executable examples (SBE, Specification By Example) here at SpeedLedger. It has been a really good learning experience, and fun too.
Thanks to the initial progress, I have managed to persuade my boss to let me work on this full time! We are doing it as an time-limited experiment, but hopefully this will be extended.

In this blog post, I would like to share one of our greatest takeaway so far: That End-to-End Exploratory Testing is more valuable than End-to-End Regression Testing.
Both categories are valuable of course, but if I had to choose to have a really good setup for exploratory tests OR only a great suite of regression tests, I would definitely choose the former. Having automated fixtures (setup procedures) for interesting “moments” in the lives of our customers when using our product is super valuable, and I will list the reasons below.

 

When all this started, we named the experiment “BDD across team borders”. This was a good name at the time, but we later realised that the benefits of BDD/SBE can be appreciated by all co-workers, even those outside development teams. We have identified three main areas where we – as a company – can benefit from automated environments and automated test data:

Learn

If you are teaching yourself a new app/system/service, wouldn’t it be great if you could experiment and explore all aspects of it without being afraid to destroy anything? And in a simple way set up situations that the typical customers face every day?
Similarly, if you are training a new hire about your system; Wouldn’t the training be more efficient and fun if you could go through these scenarios interactively, setting them up with the touch of a button?

Discuss

When discussing existing or new features with you peers, what would it be like if you could throw up a the system on a wide-screen monitor, demoing the current feature in question and the proposed solution respectively. While being able to repeat all setup step automatically and indefinitely?
Compare this to discussing features with the help of only stale sketches or obscure user stories.

Verify

This is the obvious one, of course. Most people think that verification of current features is the only benefit of automated test. But as we have seen above, there are several nice bonus effects.
In addition, there are other – maybe more subtle – positive side effects that come from using the test-first mindset. But that is a different story…

How about you? Can you automatically setup the preconditions for your customers’ most interesting interactions with your product?

 

 

 

Speaking at FlatMap Oslo about Property Based Testing

In the beginning of May I’m heading to FlatMap in Oslo to give a 90 minute workshop about Better testing with less code.  I will go through the basics of property based testing with ScalaCheck. About half the time will be hands on exercises to get a feel for the framework a get comfortable enough to start using it in your own projects. The exercises and presentation can be found in my github repo.

I find it very rewarding to speak at conferences and agree with everyone that says that the best way to get a deeper understanding of a subject is to start teaching it to others.

Outside-in — To the max! (BDD and Microservices part 2)

This is part 2 in a series about my 20% project BDD across team bordersPart 1 can be found here.  

When writing any piece of software, it’s very easy to start working on the details before you have enough knowledge about the targeted context. I think most developers recognize this pattern:
You start with some innocent (often unintentional) guessing about one or a few unknowns. Pretty soon the guesses turn into assumptions and before you know it they have morphed into ‘truths’. You happily code away with these truths as your guiding light. Later, when you try to integrate your code into a bigger context, the assumptions and truths fall apart and you need to rethink your work.

– Oh, is that how the incoming data looks like!
– Why didn’t anybody tell me I have to ask the database for that value?


Personally, I have made these kinds of mistakes countless of times, and I will probably make many similar ones in the future. I think this is due to human nature; We really like making assumptions for some reason. To minimize the impact of this phenomenon, we have to try really hard to be aware of it at all times. We also have to find strategies and techniques that help us avoid making these false assumptions.

spiralTest Driven Development (TDD) is one excellent tool for this purpose. It forces us to step outside of the current context and look at the problem from the outside, or from “one level above” if you wish. More importantly, the TDD approach fits on all levels of software development! No matter if your context is a oneliner method in a class or the public API of a complete service, you always benefit from taking one symbolic step out of your current context.

In my 20% project BDD across team borders, I wanted to take this line of thinking to the extreme, and work on the highest level possible, thus making sure I was taking the “outside-in” approach to the max, maximizing our helicopter view in the process.
I set as a constraint on my work going forward that I was only allowed to write the BDD tests against our publicly available APIs and web pages. No matter how tempting, I would not “cheat” by calling internal APIs or access anything not available to a user in production. If I did, I would risk getting my assumptions wrong, as described above.

Working against public APIs also provides another great bonus;  It forces you to write user-centric stories that are sliced vertically. A good rule is that “Every story must have a noticable effect on the public API level”. If you write a story that has no noticable effect for any user, you have not succeeded in writing a vertical story.

My claim to test on the “highest possible level” is not entirely true, of course. There will always be third party dependencies and other integration points that we cannot incorporate in the system under test. We will have to fake or mock things like partner APIs, for instance. As long as we don’t mock or fake our own code or apps, we should be good to go!

I want my helicopter view back! (BDD and Microservices, part 1)

502377266

Here at SpeedLedger Engineering, we took the plunge into the world of microservices some time ago. We have gradually phased out our old monolithic style applications in favor of many small and independent micro services/apps.

Overall, the micro services paradigm aligns very well with our organization (small independent teams) and we are getting pretty good at it, I must say! We are constantly improving our infrastructure and deployment tools, enabling us to develop and deploy quicker and more often. As a result, the general time-to-market has decreased and our customers are getting more new features, sooner.

However, one big drawback of working in isolated islands of people and code is that you risk losing the big picture, the “helicopter view”. Unsurprisingly, this is what happened to me and my teammates. We often found ourselves asking questions like:

  • What’s supposed to happen in this business flow?
  • What are the possible inputs to this API?
  • Which apps will call this endpoint?

The answers to these and similar questions are not impossible to find but it always takes valuable time to track them down and to get them clarified.  

I felt something had to be done about this situation. For a long time, I have been a fan of BDD (Behavior Driven Development) but I never really got my feet wet in a real world situation. I suspected our current situation could benefit a lot from a successful BDD effort, and that it provided a perfect opportunity for me to experiment with and learn more about BDD.

For someone who doesn’t know what BDD  is, it can be described as “executable documentation” or “executable examples” that a non-programmer can understand and edit. More about this later…

After some initial discussion, we decided that I would do this in the context of a “personal 20 percent project”, famously invented and used by Google.  The main reason being that part of the work had to be done outside the team boundary, touching all the other teams in some way. In line with this, we actually named the project “BDD across team borders”. 

This was in February, and since then I have spent most of my Fridays exploring the possibilities of introducing BDD at SpeedLedger, and – more generally – automated testing on the highest possible level in a microservices environment.  In a series of coming posts, I will blog about BDD and the progress of the project. Hopefully someone will find it interesting. Also, I would love to get some feedback from you blog readers along the way!

 

Maker Days 2016 Q1

Wait, what?

Maker Days is what you get when you take a team of 20 or so developers, feed them sugar and a pseudo-scientifically regulated amount of beer and sit them down at a computer for an extended period of time: An in-house hackathon arranged a couple of times a year here at SpeedLedger.

We form small teams around interesting or novel challenges and go at it. Some times the tasks are technically advanced dares on the outer rims of known code-space; other times, more banging rocks together in the hopes of having them throw beautiful sparks.

The execution

Anton and Lis looking dapper in their MakerDays tees

Heroes.

To kick things off we met for lunch at John Scott’s stable: A local pub with free access to a projector for presentations. As we ate we shared our ideas and formed loosely knit teams to take them on.
The actual event started Wednesday, two days after our lunch meet. With an array of snacks and caffeinated beverages at our disposal, we donned our mandatory MakerDays 2016-shirts and set to work. We were heroes.

Two days later we emerged from our makeshift cave to present the result of our labour. Each team got 5 minutes to give a brief recap of the original idea and the end-result. Encouragingly, all participants had something to demo and presentations proceeded with only the briefest technical hiccup.

The projects

Das blinken lights lets us know that while some of us are at work, others aren’t.

Das blinken lights lets us know that while some of us are at work, others aren’t.

So what wondrous creations did we come up with this time?
Thanks for asking!

  • Das blinken lights
    This mysterious LED-adorned box houses a Raspberry Pi programmed to detect mobile devices on the local network. It doubles as nerdy christmas decoration.
  • The SpeedLedger Button™
    Ah, the button. So succinct, so tantalisingly clickable.
    “Why don’t we make a few for our partners’ admin websites, so that they may synchronise stuff like their member registry and what not?” mused Per, and so he did.
  • Frikkin’ Flags
    Dynamic and highly configurable feature toggles to let us try things out for a select group of users — like A/B testing — or simply activate and deactivate a service-feature without having to redeploy. Implemented using LaunchDarkly. (Note: No flags were harmed in the process of development.)
  • Listen2Speedledger
    Based on the theory that everyone loves geometric shapes and random intermittent noise, one brave team set out to visualise user interaction events in almost-real time. The events were dirty polled from Elasticsearch and linearized so that they played one after another, rather than all at once much like a grand piano hitting concrete after being dropped from a tall building.
  • Async-duh!
    A lot of our intra-service communication is done synchronously even when it would be simpler to use an asynchronous data flow. This project experimented with exposing streams of logging events to consumers, using Kafka so that interested parties wouldn’t have to resort to dirty polling.
  • Hipstereport
    Finding Clear Reports’ PDF generation capabilities “slightly cumbersome” (“… holy crap, just look at all those boxes crammed on top of each other!”), another Per came up with the idea to replace it all with a custom-built report tool using HTML5 and CSS3 for layout.

Lastly a team of talented designers managed to freshen up the careers section of our website.

Awesome work everybody!

The wrap-up

Presentation of The SpeedLedger button™.

Per, demonstrating the undeniable affability of buttons.

Last year we had some issues with screen sharing between teams; each team were responsible to share their presentation over the network so that remote location participants could partake. This year we opted for a simpler solution by having the teams connect directly to the projector and offering a video stream to remotes. Much simpler.

All in all, the event was a triumph (I’m making a note here, huge success). Most of the projects initiated were completed and the overall experience was rewarding.

Two days of hacking feels like a reasonable timeframe for hackathon projects; any less and we would be rushed to produce a viable candidate. It would be interesting to see what we could do with three days.

From Business to Buttons - Al Gore

From Business to Buttons

Last thursday we traveled to Stockholm for “From Business To Buttons” conference on friday, hosted by the lovely people at inUse. A full day conference around User Experience, Service Design and Sustainability.

Thursday night we separated to meet friends for dinner. Johan went to have a bite at Yuc and I went to Jamies Italian. Myself finished the night with a good night beer at the hotel bar before a good night sleep on the boat and hotel m/s rygerfjord.

Main experience and impressions after the day

  • Meeting great people who want to build great human experiences
  • Great design starts with culture: Work should be a safe place to be
  • We should challenge ourselves to focus on improving human lives, and don’t forget about the disabled ones
  • Problems can be solved through technology without yet another screen
  • We spend way too much staring at a screen during a day, what takes us away from our social life
  • In fact, we don’t need a screen interaction for everything
  • We need to gain more real human empathy for those who we design for
  • Challenge ourselves and colleagues to dig deeper and ask for the root cause of their beliefs through using the five WHYs
  • Al Gore is vegan
  • Scientists breed goats that produce spider silk in larger quantities
  • We can not use as much resources as we do today in order to keep climate in a shape that allows human and animal beings to live


ROBOTS, VIRTUAL REALITY, AND ARTIFICIAL INTELLIGENCE: THE FUTURE OF HUMAN INTERACTION WITH TECHNOLOGY

Dr. Susan Weinschenk – CEO of The Team W Inc., Author and Behavioral Scientist

Susan’s Slide Deck

Notes from Susan’s talk:

  • Transequence (temporary content) will increase in value – eg. Snapchat
  • People (will) use technology to be social
  • Having mobile phones on the table, even if it not ours, makes us feel socially less connected towards the ones sitting across the table
  • Unconscious data processing will increase – Technology allows us to influence decision making through signals send to our brains (What If You Could Have A Direct Feed From The Internet Into Your Brain?)
  • Technology recognizes our decisions 0.7 seconds faster from our brains before we actually take actions
  • Technology will increase neuroplastic to help people see, feel and move
  • Objects are expected to manage themselves
  • People trust machines more than human to make better decisions
  • If a car has a voice, and the voice has a name, it is more likely that we can trust it
  • The simplest robot made out of cardboard with a voice of an ten year old boy, made people talk right out of their hearts
  • Experiments with robots show how human can build relationship with digital mediums (BlabDroid)
  • We tend to have more emotions when robots are animal like, rather than human like (Watch robot dog ‘Spot’ run, walk…and get kicked – YouTube)
  • Robot pets are used today in nursery home to minimize loneliness

So what….

  • Human can in fact be emotionally attached with technology
  • Digital should feel more human, as if there is no technology involved


THE BEST INTERFACE IS NO INTERFACE

Golden Krishna – Design Strategist at Google

Goldens’s Slide Deck

Notes from Golden’s talk:

  • Sometimes we make tasks more complicated with interfaces when in fact no interface is needed
  • So called ‘backpocket applications’ allow us to interact without picking up the mobile to achieve specific tasks – the signal of a mobile simply does it for us
  • Screens now have taken over our lives
  • Screens everywhere, in cars, in your pocket, refrigerator, and even on your nose
  • An average person spends 150 times looking at their mobile during a waken day, and more than eight hours staring at a screen
  • There are apps for everything. Are you sick? There’s an app for that! Need to pray? There’s an app for that! Dead? Well, there’s an app for that, too!
  • Most apps are intentionally addictive distractions that end up taking our attention away from things like family, friends, sleep, and oncoming traffic
  • We are eager to use new technology, like screens etc. – But we forget to ask ourselves what the bigger problem is we are trying to solve for people

So what…

  • We can build a technologically in an advanced world without digital interfaces


HOW TO MAKE SENSE OF ANY MESS

Abby Covert – Independent Information Architect

Abbys’s Slide Deck

Notes from Abby’s talk:

  • Many people get overwhelmed when encountered a mess
  • The majority of mess is made of information (and people)
  • Information is not like content or data
  • Data are facts, observations, and questions about something
  • Content is whatever a user interacts with
  • Information is whatever a user interprets from the arrangement or sequence of things they encounter
  • Information architecture is how we arrange the parts to be understandable as a whole
  • Language matters
  • The goal is not simplify an arrangement or sequence of things – the goal is to be clear with you mean when you say what you say
  • There is no right way
  • There are only five ways to organize anything: 1. Location, 2. Alphabetical, 3. Time, 4. Category, 5. Hierarchy
  • Organizing things isn’t the hard part – Agreeing is the hard part
  • We need pictures – Pictures give us something in common to point to
  • Visualizing something when it is hard to explain in words
  • Show process, not just results

So what…

  • Involve real users to solve the mess, understand how they would organize the content
  • Group navigation rather so it makes sense to users, not how it reflects internal structure or your own common sense


FROM DAY ONE TO DAY NONE: THE LIFESPAN DESIGN CHALLENGE

Patricia Moore – President at MooreDesign Associates

Notes from Patricia’s talk:

  • Don´t make things for disabled people, make them for everyone. If someone disabled could use them, everyone can
  • Get empathy for your user, put yourself in their position – feel how they would feel

So what…

  • Being a user for a day helps to bridge empathy
  • Talking, listening and understanding users allows us to dive deep into their context, needs and goals


DESIGNING CULTURE

Jeff Veen – Design Partner at True Ventures / Former CEO of Typekit

Jeff’s Slide Deck

Notes from Jeff’s talk:

  • Equanimity (Latin: æquanimitas having an even mind; aequus even animus mind/soul) is a state of psychological stability and composure which is undisturbed by experience of our exposure to emotions, pain, or other phenomena that may cause others to lose the balance of their mind
  • Don´t find someone to blame, leaders need to build a safe workplace and spark a mental calmness
  • The five whys model helps to identify the root cause of a problem or critique
  • Share why something went wrong, how can we solve it and how could we fix so it won’t happen again?
  • Great teamwork with no disturbance could solve almost every problem
  • Focus to solve the task. Things that Jeff’s team at Typekit planned to take months was solved on a weekend
  • Culture should allow us to share our true thoughts
  • Chatt allow us to collaborate and act quicker
  • Chatt allows employees to respond more open on an opinion
  • We should feel safe in a work environment in order to spark innovation and creativity
  • Grow a team that trusts and respect each other leading to better products and service

So what…

  • Be open for critiques
  • Ask why five times in order to dig for the cause root of an opinion


THE MINDFUL MANAGER

Simon Bennett – Managing Principal for LASTing Benefits (AU/UK)

Simon’s Slide Deck

Notes from Simon’s talk:

  • Usually it doesn’t feel good when you screwed up
  • Being wrong is entirely different from realising that you are wrong
  • We are expected to be professional
  • Being professional should mean being complete human
  • Usually, the higher in an organisation you are, the less you have to defend yourself
  • Distinguish between being emotionally intelligent and being emotional
  • Ask yourself, do you hold beliefs or do you have knowledge?
  • Human beings are social animals, our communities can define, coerce and corrupt us or blame us
  • Share the burden
  • Safety is everyone’s responsibility
  • We experience others from the outside – but ourselves from the inside
  • From the outside the irrational use of power looks irresponsible and ugly
  • Leaders should use their power to shape new realities instead of distorting our view of the existing one

So what….

  • It is ok to screw up, as long as we realise we did
  • Screwing up helps us to learn
  • Encourage assumptions before knowledge


ELEGANT TOOLS: THE FOUR PRINCIPLES OF BUSINESS DESIGN

Margaret Gould Stewart – VP, Product Design at Facebook

Margaret’s Slide Deck

Notes from Margaret’s talk:

  • Advertising industry need to adapt to reality
  • We need to earn the right to be in people’s pocket
  • If Facebook wants users to continue trust, they need to continue deliver values
  • Design and act for people where they are
  • Understand device and internet connection people have access to
  • Giving advertising industry possibility to reach people in a relevant context

Four principles for designing quality business products

  1. Help people grow
  2. Balance efficiency and effectiveness
  3. Bring clarity to complexity
  4. Be accurate and predictable


TRANSFORMING A CITY: RE-DESIGNING DENVER

Kjell Persson – CEO at inUse

Kjell’s Slide Deck

Notes on Kjell’s talk:

  • inUse to help Denver to be a better city
  • Denver wants to be a more smarter and more sustainable city
  • Learning about citizens through data, observation and interviews
  • Bringing highest managers from different unity together once a month
  • Building shared understanding across different denver departments
  • Building measurable short term, mid term and long term goals


THE DEMOCRATIZATION OF TECHNOLOGY

Al Gore – Nobel Peace Prize winner, former US Vice President and Environmentalist

Notes from All Gore’s talk:

  • It was less technological advancement and user involvement we payed attention to, Al Gore impressed us mostly with his passion and engagement for environmental change
  • Guest in restaurant: If you dye your hair black, you’d look exactly like Al Gore, and you sound like him too
  • Science allows us to combine a spider with a sheep. (Spider sheep) – Interesting for biologist, more scary for normal human beings
  • We need to react on keeping smaller families in order to keep the population in a manageable amount
  • Al Gore is vegan
  • We all must make a change to environment now
  • Al Gore is positive towards winning the global warming issue
  • Do we have to change? – Yes!
  • Can we change? – Yes!
  • Will we change? – It is up to all of us
  • There is no plan B if we fail to prevent the climate change
  • We couldn’t rescue people from hurricane Katrina in New Orleans, we definitely cannot rescue world’s population from failing to prevent the climate change
Stockholm
Stockholm (Photo credit: Johan Magnusson)
From Business to Buttons - Golden Krishna
Golden Krishna (Photo credit: Johan Magnusson)
From Business to Buttons - Al Gore
Al Gore (Photo credit: Johan Magnusson)
Florian UX Design
Florian Fiechter, User Epxerience Designer (Photo credit: Johan Magnusson)
johan-magnusson-visual-designer
Johan Magnusson, Visual Designer (Photo: Florian)
QCon London venue

QCon London 2016

 

I’ll try to summarize three great days at QCon in London, not going into detail about all sessions that I attended but high-lighting some of the ones I really liked.
Most sessions were recorded and will be available for the public audience over time, make sure to check them out!

 

QCon offers a wide variety of tracks ranging from low level “Close to the metal” to more “soft” skills like “Optimizing You”.  There were also a track for the main sponsors of the conference. The wide variety of content and speakers made choosing what session to attend somewhat of a problem; I had made a schedule before traveling to the conference, and it broke down during the first presentation of the tracks before the first keynote…
Kicking of QCon was a great keynote “Unevenly Distributed” by Adrian Colyer. Adrian reads a paper a day, summarises it and publishes it on his blog “The Morning Paper”. This was a very inspiring and well presented keynote that raised my interest in reading papers and as Adrian said “5 reasons to love papers”:

  • Great thinking tools
  • Raise your expectations
  • Applied lessons
  •  Great conversation
  • Unevenly distributed

The only problem with reading more papers and learning more is that:

“The more I learn, the more I realize how much I don’t know.” – Einstein

After the keynote we listened to Gavin Stevenson, Engineering Lead at WilliamHill, who talked about WilliamHills betting engine and how they are transitioning from a large database centric solution to a micro service based architecture (this was a common theme during the conference). They were building a “production ready” betting engine in 2 week sprints, testing it with real production data. The most interesting take away was how important it is to really try to break your system. When the system fails, that’s when you learn. The old saying “If it ain’t broken, don’t fix it” just doesn’t apply anymore.

“If it ain’t broken, try harder!” – Gavin on testing

One of the few really non-software related talks was held by long distance runner Simon Wheatcroft.
When losing his sight the age of 17 due to a genetic degenerative eye condition, he began a journey of adapting tech to achive the impossible.
Through the aid of the Runkeeper application, he started running solo outdoors. Simon will be running his first solo race in May 2016; the Four Deserts Series Sahara Race in Namibia. There, he’ll use GPS coordinates and a mobile app to navigate across the 250 kilometer distance.
A very inspirational and humbling talk!

“You just have to tell yourself that pain doesn’t last forever”- Simon on running 250k

From non-technical to real in-depth low level Netty implementation details. Norman Maurer from Apple described how Apple is using Netty as a web service delivery platform for most of Apple services. Apple are running 550000+ Netty based services, handling 10s of Petabytes of data every day and millions of request per second.
Norman guided us in different aspects of the Netty framework and how the default JDK implemtations just aren’t good enough for this kind of load and how they’ve commited several improvements to the Netty open source code. Very interesting and down to the metal of how to hack the JDK and using JNI to get better performance with; for example memory allocation and SSL.

Martin Kleppmann held one of the best presentations of the whole conference where he talked about keeping data sources in sync, moving away from (distributed) transactions to streams. The content was not very in-depth, but Martin had deep knowledge of the subject, excellent slide and a lot of energy when presenting. This is a talk everyone should watch and learn from.

“Stupidly simple solutions are the best” – Martin Kleppman

Day two started with Linda Northrops keynote “Reflections on Software Architecture”. Linda looked at software architecture and how it’s importance and acceptance have changed over the last 20 years. To summarize: architecture is important, and it’s a way to manage technical debt.

Josh Evans from Netflix presented how Netflix have expanded there streaming services to almost the entire globe (Netflix#Everywhere).
Netflix have had some major outages and failures, both in their own software and the underlying cloud AWS-platform. Josh concludes that “Failure is inevitable” and that one really have to embrace the failure and not fail in the same way twice. This had lead Netflix to embrace a “Failure-driven architecture” approach when building their platform.
Netflix’s architecture is really impressive, although not applicable for most comapanies/services, so it’s always interesting to hear what they are doing to actually run the platform at that scale.
Josh presented Netflix four architecture pillars; data, caching, traffic and micro services, and how they use (among other techs) EVCache, Cassandra and DNS to keep their services up and running in case of total failures of an AWS datacenter/region. He also showed how they test failure in different regions and route trafic to another region to minimize customer impact.
If infrastructure and architecture at scale is of any interest, watch this talk when it comes online!

“Never fail in the same way twice” – Josh Evans

Mitchell Hashimoto (founder of Hashicorp) gave his talk “Observe, Enhance, Control: From VMs to Containers”. In his talk he takes us back to 2006 and the age of VMs and how the datacenters and the problems to solve are driving the architecture of the software for Monitoring, Configuration and Deployment. Jumping to 2016 and the age of containers, Mitchell argues that the “state-of-the-art” tools from the age of VMs are not really suited to handle the tasks anymore. Even though the tools are extremely good, they do solve a completely different problem. The content of the talk was nothing new, but it is really inspiring to listen to Mitchell talk.

Gil Tene talked about Hardware Transactional Memory. Really low-level stuff about CPU pipeline and cache optimization. HTM in the JVM is not new, Azul has been delivering both hardware and a customized JVM with JVM for 10 years. What’s interesting is that it will become mainstream now when Intel is shipping CPUs with support for HTM. Gil succeeded in a very educational way describe the complexity of HTM and how it can be implemented in for example the standard JVM. In the end Gil talked about how the developers must reason about locking and synchronization to make the most of HTM in their code.

One of my most anticipated talks during the week was Dan North‘s “Making a sandwich”. I hade very high expectations for this talk, and Dan managed to exceeded them (as usual). Dan talked about giving feedback, how feedback in itself is a system and why we should do it. Giving and receiving feedback (which is really just to say ‘Thank you’) is, in my opinion, one of the hardest skills to master and we should really practice a lot! Dan presented some useful techniques and tricks, but you should really watch this yourself!

Last day started with a very entertaining and inspiring keynote delivered by Kolton Andrus (Netflix) and Peter Alvaro (University of California). Peter and Kolton shared their expereience of a very successful collaboration between industry and academia. Peter had a “big idea”, Lineage-driven fault injection, and together with Kolton this evolved from a theoretical model into an automated failure testing system that leverages Netflix’s state-of-the-art fault injection and tracing infrastructures.

“My code is now actually running live on Netflix…” “…well, minus all of the println statements”

Vikki Read and Alex Wilson from Unruly described how they are using the extreme programming (XP) ideas to deliver high quality software and how it can be made to work in a very agile environment. Since agile and XP focus a lot on collaboration and knowledge sharing, they shared some problems that they’ve had with getting new employees up to speed and how growing teams make for example (too long) stand-ups being a problem.

The final session I attended before heading to the airport was Tammer Saleh at Pivotal talking about the mistakes people make when building a microservices architecture, i.e. Microservice anti-patterns. When are microservices appropriate as an alternative to the monolithic app? The problem with monolithic apps is not about the code, it’s about the teams! Large teams (or multiple teams) can’t work effectively in the same codebase.

Its not about code, its about teams

Its not about code, its about teams

Tammer stressed that “the most common mistake is to start with microservices”. Start monolithic and extract as needed, because microservices are complex and impose a constant tax to your development.
Tammer explores how to draw the lines between services. dealing with performance issues, testing and debugging techniques, managing a polyglot landscape and the explosion of platforms, managing failure and graceful degradation.

“Boring is beautiful” – Tammer Saleh

Most of the interesting sessions I attended during the week were about failure, and how to handle failures. Quotes like “Failure is inevitable”, “Failure is an opportunity to learn” and the importance of building an architecture that can manage failures were common topics. Migrating from a monolithic application to a more micro service oriented architecture were also popular.
Overall QCon London was a great conference, I will most likely try to get back next year! All tracks had great speakers, which is problematic since you have to choose between sessions – on the other hand most of the sessions are recorded so I know what I will be doing the coming weeks.

 

DESIGN FOR DELIGHT WITH THE KANO MODEL

Delightful, excellent, fantastic, awesome, great, incredible, satisfaction.

Which of the above differs from the others? — Why aiming for satisfaction, when we can build for delight?

Developed by Japanese economist, the Kano Model shows how to map a customer’s journey from frustration to delight – and keep them there. These are the main principles in the Kano model:

  1. Eliminate unused features to avoid experiences rotting.
  2. Look out for missed and failed expectations.
  3. Use joy, flow and meaning to define what pleases users and customers.
  4. Remember that today’s “OMG” is tomorrow’s “same old, same old.”
  5. Innovating our way to aspirational experiences, one small step at a time.

When following these 5 steps, we are on our way to a winning a strategy.

Source: “Understanding the Kano Model – A Tool for Sophisticated Designers,” by Jared Spool

Design as a value for growth

Remember your last bad experience you had with a service or product, and how that made you feel?

According to Nobel Prize-winning scientist Daniel Kahneman, we experience approximately 20,000 moments in a waking day. Each moment lasts a few seconds in which our brain records an experience. The quality of our days is determined by how our brains recognise and categorise our moments; either as positive, negative, or just neutral. Rarely do we remember neutral moments.

This fact is interesting for design. We need to ask ourselves how big of a human problem are we solving? How can we intentionally design for positive moments? Moments that is remembered as genuinely delightful will drastically affect how people feel about a brand, service or product.

People talk to others about moments as they remember it. Think about it as all delighted customers become a part of the marketing team. That’s why we strongly believe delighted customers can help us truly grow for the long-term.

That doesn’t mean that we should ask ourselves – can we build it? Because the answer is yes. Today’s technology allows us to build almost anything.

Therefore we should truly ask ourselves why, and what is the future that we want to build together.

Inspired and adopted from “HOW DESIGN BECAME THE NEW LANGUAGE OF BUSINESS,” – A documentary by Invision

Do we need the database at all? – Event Sourcing!

In my previous blog post I reasoned about how the database could be hidden behind an ORM tool. And logic could now be written in lambdas (java or scala) instead of SQL. The ORM tool will transform the data model (residing in the database) to an object model (consisting of  java/scala objects).
I pinpointed  some big advantages of this, but I also mentioned that there was an even bigger advantage with the object model that I will address in this post.
Event Sourcing, let me explain what it is with a simple example.

Example: The dry cleaner

The dry cleaner is a place where you can leave your clothes to get them cleaned. In return for your clothes you receive a receipt. When the clothes are cleaned, you can get them back by showing the receipt.
A simple object model for this could be:

(more about how the DryCleaner trait could be implemented later)
The dry cleaner trait has methods for leaving and retrieving clothes as well as a method for looking at the dry cleaners current inventory of clothes.
A simple example of using this model:

The implementation of the DryCleaner could be backed by a database using an ORM tool, but let’s consider another alternative right now. What if all modifying methods of the DryCleaner implementation wrote their arguments to a log file? (the leave() and retrieve() methods modify the object model while the inventory method is a read only method).

The Event Log

What if all (modifying) operations on the object model resulted in a log entry. If all modifications was stored as “events” in a log.
A log created from the example above could yield the following log:

Note: the receipt number need not be present on “leave” rows, but increases the readability

This log contains all the information contained in the method calls of the example plus a serial number of the call and a time stamp. (The timestamps in this example has been modified to improve readability, the code example above would of course execute instantly).
Could you recreate the object model from the information contained in this log?
Yes!
How could the object model be recreated?
By using the information on each log row to call the right method on the DryCleaner object.
Does this work for complex object models?
Yes, the object model can be as complex as you need, below I will describe how to automate the process of reading/writing this log.
In that case, can you do without a database?
Yes, you can choose to store this log in a text file if you would like.

This is called Event Sourcing, and here are some benefits of Event Sourcing:

  • The object model will be portable using a simple text file
  • Human readable, if your events have good names, the resulting log is easy to understand for a human.
  • The log itself is a very elaborate audit log.
  • No database data model needed. Even if you store this log in a database, you do not need to model the entities in a data model.
  • A time machine, you know everything that has happened and when (unlike a data model in a relational database).

The time machine

What if I need to know the inventory of the dry cleaner at a specific point in time?
If I want to know the inventory of the dry cleaner at 2015-08-28 at 09:00, I simply load all events that has a timestamp less than this time. Thus we will have recreated the exact state of the dry cleaner at this moment!
In fact, we can recreate the state of all involved objects for any time we choose!
This is an extremely useful feature of event sourcing. You can return to any point in history to see exactly what happened.

Additional benefits of event sourcing

  • External systems can listen to the events (collect statistics or react to events)
  • Extreme performance. All operations on the object model will be in memory (no need to go to a database). This will be extremely fast.
  • Performance for persistence. All events are immutable and are always added, never removed. You can use an append-only operation for storing them.
  • Simplification. The events are simple. No complex domain object or graphs of objects need to be saved. The object model can of course be as complex as needed, the complexity is not transferred to the persistence mechanism.
  • Testing. It is easy to set up test data, a pre recorded event log can be used to set up the test data. No messy mocking needed.
  • Flexibility of design. The events does not describe the domain objects. So the object model can be redesigned without any need to migrate existing data to the new design.

And some Cons

  • You might need to support ‘old’ events for a long time. You can easily add new types of events, but removing old events is harder. Because you still have to be able to read old event logs. There is ways to remedy this (by making snapshots for example). But still, it could be a pain.
  • Event Sourcing is a less known technology compared to RDBMS etc. It will pose a learning curve to most software developers. And it might be harder to sell to an organization.

Automate the reading and writing of the event log

How do you implement this? Do you have to implement logging to the event log in every (modifying) method? And to parse and apply the states again, do you need to implement this for every class?
No, the implementation could be done through the reflection mechanism. Once and for all for all entities. You could use a naming convention or annotations to mark which methods that modify the object model.
In our bookkeeping application we implement the logging to the event log through a single class of 100 lines of code. All our 120 entities use this single class for all event writing and reading. This is an extremely big win, only 100 lines of code to handle all persistence for a very big application. All the slick code (see my last blog post) could also be removed! We can now concentrate on writing application logic!

Application of Event Sourcing at Speedledger

Our main product at Speedledger is our bookkeeping application. And bookkeeping seems to be the perfect match for event sourcing as described above.

  • We can restore any customers bookkeeping to any point in time. And thus we can let our customers do this in some cases (however, there are some legal aspects what you can do with your bookkeeping).
  • We can easily transfer data for one customer from one system to another. For example, we can transfer customer data from our production environment to the system used by our support personnel simply by exporting the event log as a text file. The support person can then view an exact copy of the customers data in their system while helping the customer.
  • Our support personnel can use the event log itself when helping customers, they see exactly what the customers has done in their bookkeeping. The support personnel has no specific technical training but can still read and use the event log since it only describes bookkeeping events (and not in a very technical way).

You need a database!

Just because it is possible to store the events in a text file without the database entirely, I do not recommend this. A database has many other advantages (transactions for instance). The point is that you do not need a relational database or SQL. You could use something much more lightweight, like a document oriented database. And the data model can be extremely simple, you just need a place to store the event log.

100% consistent object model and no bugs (next blog post)

What about database constraints? The database helps you perserve data integrity by using constraints. If you do not use a real data model (or a database at all), how can you preserve data integrity?

In my next blog post I will describe how to preserve data integrety in an object model with Event Sourcing. And in a much more complete way than you can hope to acheive by database constraints. And this will in turn lead to a manner of coding that makes bugs much less likely.