The challenge of change

My previous topics - about making things continuous, automating things and renewing processes - require all major changes to existing way of working. Deploying a change is never easy. Most of us are happily living in our comfort zone which we fear a change may disrupt.

However, all people are not the same. We may find individuals that are eager to follow us, especially if we are properly prepared and can justify why the change is good. Getting some people to support the change could help us to get rest of the people to follow us, too. But the change will get harder towards the end, the last laggards are the most difficult to convince, even after we already have transformed the great majority.

The heaviest resistance is provided by narcissists. Those are people who think they know everything, are absolutely right about everything, and will never give up. Often these people represent some specific area of work, and are being impacted by the change in a negative way, i.e. increasing their workload. They are pushing the change to follow their own interests without taking any responsibility for the consequences. They ignore the benefit of the company while their own comfort is threatened.

If these people are not crucial to our change, we are lucky. We may then just try to ignore them. In any case, we need to be well prepared so that the narcissists don't convert other people to support them. We need to have facts right, information shared and know that the thing we are trying to accomplish is the right one.

If the narcissists are blocking the change, we need to focus on showing what is the value of the change for the company, and what does it cost for the company that some individuals are blocking it. Then, it will be finally the task of higher management to decide and give support, or postpone or reject the change.

But we mustn't forget that there is always also valuable resistance, which should be used as feedback to tune the change into perfection. Thus, even in the presence of narcissists, we need to keep our posture and openness towards the community, so that we will be able accomplish the change and do it right.

For a quick handbook on making a change try reading How to Change the World, pdf available just for a couple of bucks!

SW development of the industrial era

Think about a situation where you would create a specification of a new system and with a push of a button you would get the implementation and a set of tests to validate it. Utopia? Not exactly, even though not yet reality.

The problem with SW development is that the code is something that typically only the developer(s) understand. SW engineering has three major problems:
1. Software is written in a language not commonly understood.
2. Engineer is usually expert of some domain. SW engineer is an exception, known more for expertise of a programming language more than a domain.
3. SW development is still handicraft. It lacks the performance improvements many other areas of engineering have reached via industrialization.

To tackle the first problem, we would need a language that is understood by all parties involved in the development of a product. For second, SW engineer would be transformed to an interpreter for that language and the implementation. For third, development would be done as configurable components, which could mostly still be utilized even if the description done in the common language changes.

Starting with the language problem, solution would be modeling, because that could provide an opportunity to make high-level descriptions of the components and their interactions. Typically, when talking about modeling of SW people think about UML (Unified Modeling Language). But that's not the right option. Problem with UML is that it's supposed to be an all-purpose modeling language and it's thus very complex. I've seen attempts to make specific UML limit-and-extend solutions, but it just won't work out. In addition, UML doesn't raise the level of abstraction, it's basically just another programming language.

A better option would be to create an own domain-specific model (DSM) and a language for that (DSL). Creation of a DSL is often expected to be complex, but it probably isn't - it really depends on your domain and how do you want to show it. There are tools to help you in the creation. And one of the problems to overcome is that we will most likely try the creation of DSL with SW engineers that are experts in some programming language and thus will be out of their comfort zone when working with DSL. So growing the SW engineers to DSL world would be necessary to tackle problem two. That might be the biggest change project when starting DSM.

For problem three, we need to create a code and test generator for our DSL. That isn't as difficult as it sounds first, but finally it depends on what you are working with. If the application is mostly about interactions (e.g. UI or some protocol), DSL could work out. For mathematical algorithms, DSL might not provide that much benefit, but might still help to understand and manage the entity.

Working with DSMs require specific tooling. There are several tools available, commercial and free, for doing modeling for code or tests. When starting modeling, it's important to select the tools carefully, because we might well be tied to the use of the tools for the lifetime of the application. That's probably the biggest hurdle in taking DSL approach in use.

As I mentioned in the beginning, it's not yet reality to have a high level model from which both code and tests are generated (AFAIK, but if you know a case please share it to us). If you wish to learn more, check for example what MetaCase has done in DSM tooling and Conformiq in the area of model-based testing.

Automate... everything?

In the previous post, I encouraged to make everything continuous. In order to do so, we need to automate a lot of things... but we can't automate everything. Automation suites to activities that are manually demonstrated to be monotonic, and it's beneficial in things that are done repeatedly.

Considering what I suggested to make continuous, planning is always heuristic science, not monotonic, neither repeatable. Development is mostly heuristic but partly it's about common routines that are duplicated here and there. A basic solution for the repetitive parts is to form libraries of functions or macros.

Integration should be mostly repetitive and monotonic, thus it should be mostly automated. However, integration tasks are often lengthy and therefore it should be automated as much as possible with attention of skillful people. It will pay off, because the alternative is that developers will manually do those long tasks, leading to errors and waste of time. We shouldn't use dedicated integrators, because they will get bored with the work, and it's also waste of resources. Problem with dedicated integrators is also that they know much less about the change to be integrated than the people who have written the code, thus giving an opportunity for extra bugs to be inserted. Separate integrators would also add a handover to the development process, which is waste in lean principles. And when dedicated integrators are present, developers take much less responsibility of the changes they submit.

Testing is something we should automate a lot, because running tests is very repetitive and highly monotonic. However, not all tests are suitable for automation. Tests that require complex procedures (and therefore are also seldom executed) might not be worth automating. Delivery and deployment are such that should be at minimum very well documented, but preferably automated.

When automating things, it would be beneficial to pick a framework for automation. Thereby, we can reduce a lot the need for maintenance later on. This is often forgotten when we eagerly start developing new tools and systems, that those need to be also maintained. However, the framework should be carefully selected, seeking for a versatile, cost-effective alternative that is not overwhelmingly complex and heavy for the purpose. Prospects for further development of the system, possible vendor lock-in and available support should also be checked when making a selection.

Then, we could think about automating the whole development process. More about that next...

Continuous... everything!

The buzzword in modern SW development is continuous. Disruptive, big bang work is out of fashion. Or at least it should be.

Avoiding big bangs starts with continuous planning. We shouldn't have content for our next big release planned a year before, but instead the planning should happen in smaller pieces, giving new content timely for development to be used in their short-term planning.

Traditionally (in embedded systems), SW development is done in big chunks. Typically, every chunk is a project with own branch in version control, and a project manager. Development is done in silos, fixes are copied by developers (or worse - integrators) to every project branch - just because every project manager cares about his/her project. This kind of approach is waste of resources and possesses a major risk in quality when changes are manually copied between branches.

A better way would be to focus efforts on one branch (called usually trunk or master/main code line), branching only for verifying that the code base possesses the level of quality that customers expects to get. This typically requires a short period of time for fixing the remaining bugs, expected that quality has been constantly kept on a good level already before branching.

This I would call continuous development. It should be usable in most cases. If there's need to make several consecutive releases with remarkably different content partially from the same code base, this approach might not work any better than the traditional one.

Often it's tempting for developers to take their own or team branches and continue developing SW there for long periods, ending up in a big bang integration after the development is "ready". Integration will take a lot of time, and will face complications with other developers wishing to integrate their code. Development in own branch is often done to avoid the "bad quality" code written by other developers.

Continuous integration (CI) is an approach where developers commit their code frequently to trunk, at least once a day. The submissions don't need to provide new functionality, the only requirement is that one doesn't break any existing functionality. To make CI work, there needs to be an engine that relentlessly and swiftly checks, builds and tests every submission, and naturally the system needs to be very fluent to use for the developers. Finally, there needs to be fluent process for making daily releases of the latest changes for the developers to be used as a base for their submissions next day. It's also useful to get a release for more extensive testing overnight. We need to have continuous testing instead of leaving it to be done after the development.

So the checks, builds and tests need to be fast, and there needs to be enough capacity to verify all submissions. If capacity is limited, verifications may be done for only every nth submission, but in that case finding a breaking submission from a set of n will lead to blocking of further submissions until the bad submission is found and removed.  If blocking is not done, removal of the bad SW might get very difficult and may lead to even to discarding of the whole stack of later submissions above it.

In many embedded systems, deployments can't be continuous, SW updates require typically a service break, often also extensive testing to guarantee that the existing quality of service will be preserved - there is a fear that the new release breaks some of the features already in use (as testing in the lab never uncovers all the bugs). And that fear is just common sense, because uptime of the systems needs to be often practically 100%, unlike many web services were a break for a few seconds every now and then might be acceptable. Updating remotely a huge number of devices around the world is also a case where the quality of the update must be guaranteed, because reputation may be lost for a long time just by one bad delivery. Deployment to production of the devices could be attempted more often, in order to get the latest and greatest SW to new customers, but the concerns are the same as above.

But it's great if you are able to please your customers with continuous delivery of SW updates. Thereby, they don't need to wait until the quarterly or yearly release project has got in all (or most) of the planned features, but they'll have an opportunity to update when it suites them best.

For continuous delivery and deployment, we need to consider what is our support capacity and scheme. Distributing numerous SW releases requires resources from the support organization, as it would be expected to support all of them. We may try to deploy a scheme were the default action is to offer latest SW as the solution, but if the deployments can't be done very often...

However, if we have a process in place for making good-quality deliveries continuously, we will have the freedom to choose the moment when we think the content for our next major release is ready, instead of following a plan we might have made long before, where everything relies on a bug fixing period to be done after development is "ready".

If you wish to learn more about making things continuous, you could read book Continuous Delivery, which gives very deep insight of many of the aspects I mentioned here.

Foreword

I started this blog to share what I've learned and experienced in SW development, just to document those and improve my own understanding. Many of the things have been said before, but I try to present the ideas here concisely to give a quick insight, and give references for further reading on the topic. I hope you are already familiar with basic concepts in modern SW development, like agile and lean.

Feel free to leave comments or questions to any of the topics, I'll try to give meaningful responses promptly. You may also request additional postings on deeper insight of the topic.