Quantifying the Encapsulation of Implemented Software Architectures

TL;DR: We studied which static architecture metrics are correlated with a high ratio of local changes (i.e. changes made to only a single component). An analyses of 10 open-source systems shows a positive relationship between the percentage of code only used within a component and local change. We conclude that having small, clearly defined interfaces for your components lead to more local changes, which are easier to implement and test. 
 

---

This week I had the pleasure to present our paper Quantifying the Encapsulation of Implemented Software Architectures at the 30th International Conference on Software Maintenance and Evolution. What follows is the high-level story I have presented (using these slides), if you want all the details you can find the complete paper here.

Inspecting the title of our paper we see that it is about quantifying the encapsulation of implemented architectures. To understand what this paper talks about let's start by examining these concepts more closely.

Implemented software architectures

As a whole, software architecture is defined as:

the organisational structure of a software system including components, connections, constraints, and rationale. 

If we focus on the implementation within the code, we can only observe the components and the connections, stuff like constraints and rationals are normally defined in the documentation. 

As an example, consider the figure on the right, which depicts a hypothetical system. We can clearly see the high-level components which (hopefully) implement a distinct functionality, and the connections that exist between these components. To get such a high-level overview of the system you can normally open up the source-code repository or look for it in the documentation. Should that fail you can always fall back to a whiteboard, a marker, and a software engineer working on the system, I still have to meet the software engineer who cannot draw such a picture of their system.

Quantifying encapsulation

Encapsulation revolves around localizing the design decisions which are likely to change (a process also known as information hiding). If done correctly, we would see that the changes to a system are done to source-code modules which are located near each other, preferably in the same component. This makes it easier to implement the change (since we do not have to jump between components) and easier to test the change (since we have less components to test).

Given a system, a definition of its components, and all changes made in the past years we can easily determine whether the process of encapsulation has been successful by using the concepts of local change and non-local change as introduced by Yu et al. 

As a first step we classify each change-set in the history of a system (e.g. all commits or pull-requests) as either local or non-local. When a change-set contains source-code files from only a single component it is considered to be local, if more than one component is touched the change-set is considered to be non-local. The figure on the left shows an example of each type of change-set, blue for local and brown for non-local.

After this classification we can quantify the success of the encapsulation by simply dividing the number of local change-sets by the number of total change-sets. For example, the figure on the left shows a change-set series containing ten change-sets of which seven are local, leading to a quantification of 0.7 for encapsulation.

As explained above we would like to see as many local change-sets as possible, so we want this number to be as high as possible. However, since we also expect to see some non-local changes for cross-cutting concerns such as logging we would not expect to see a ratio of 1 that often. To get a feel for which numbers are good we can calculate this metric for many systems, thus creating a benchmark which can tell us whether this 0.7 is relatively good or bad compared to other systems.

Up until now we have only seen concepts introduced by others, which makes a rather sub-standard research paper. So what is the problem here?

The timing problem(s)

The main issue with calculating the encapsulation of an implemented architecture using the concepts above is that it can only be done after a project has been finished. Although nice to know at that point in time, it would be nicer if we could calculate a metric on the project which provides some sort of indication of the encapsulation of the system now. Given that the current literature lists over 40 software architecture level metrics (an overview can be found here) we should be able to find something right?

So we designed an experiment to see which software architecture metric we can calculate on a single snapshot of the code (i.e. snapshot-based encapsulation) is correlated with the encapsulation calculated over time (i.e. the historical encapsulation). 

The first set-up was straight-forward, select some systems, calculate the snapshot-based metrics, calculate the historical encapsulation, run the statistics, and bob is your uncle. The figure on the right shows a sketch of the outline of this set-up, using the number of components as an example of a snapshot-based metric. At a glance this set-up seems correct, but after a while we figured out there it contains a (rather serious) flaw, any thoughts?

Notice that we calculated the number of snapshots based on the situation after the last change-set. But this change-set makes a change to the system, and can also change the number of components! 

More graphically, consider the chart on the left which shows the number of components on the x-axis and the change-sets on the y-axis. We see that there is a period where we have 2 components, then a period where there are 5 components, only to drop to 4 components in the last change-set. Trying to correlate the historical encapsulation of 0.7 with a number of four components is clearly incorrect, since most of the time the number of components was either 3 or 5. 

To remedy this problem we adjusted the design of our experiment. Instead of using all of the history to calculate the historical encapsulation based on all of the change-sets we instead calculate the historical encapsulation based on the periods in which the snapshot-based metric is stable. 

In the example above this gives us two pairs of numbers, (2, 0.6) and (5, 0.75), to indicate the number of components and the historical encapsulation for that period. Note that we do not calculate a pair for when there are four components, since we do not consider a single change-set a 'period'.

 

The results

Now that we know how to do the experiment we can execute it. First we selected 10 open-source software systems to investigate, giving us over 60 years of historical data. Secondly, we filtered the snapshot based software architecture metrics available down to a list of twelve metrics. This list includes simple metrics such as the number of cyclic dependencies or the number of binary dependencies, but also more involved software metrics such as the metrics which form the basis for our dependency profiles. 

These last metrics are (unfortunately) not yet widely known, so let me explain them quickly. In a dependency profile we divide the source-code modules within a component into four distinct categories based on the dependencies from and to other components. We can calculate the profile by calculating the percentage of code of the system in each category, thus a profile (50, 20, 25, 5) indicates that 50% of the code is internal to components, while 20% is depended upon from other components, 25% of the code depends on code from other components, leaving 5% in the last category of code which is depended upon and depends upon code from other components. 

After crunching all the numbers the result is that there is a positive correlation between the historical encapsulation and the percentage of internal code. In other words, we observed that systems which contain a higher percentage of internal code also exhibit periods with a higher ratio of local changes.

So what can I do with this result?

Given that more internal code is related with a higher ratio of local changes I would argue that you should strive towards an implementation with as much internal code as possible. 

One way to achieve this is to define clear, small, and specific interfaces for your components. While this is often done correctly for the incoming interface of a component, the outgoing interface is often overlooked, leading to a large outgoing interface with a higher risk of needing change when other components are touched. 

More details ...

Interested in reading more about the design of the experiment? Or do you want to know how well other metrics correlate? (spoiler: they don't) Want to know more about our ideas about how the inspected software architecture metrics can be improved? Download the full paper here!

One step towards a software metrics catalog

A little over a year ago my proposal was accepted in the Tiny Transaction on Computer Science. If you have not read it, the body of the publication is:

To enable the effective application of software metrics, a pattern catalog based on real-world usage scenarios must be developed.
— TinyToCS (@TinyToCS) May 3, 2013

Unfortunately, such a catalog has not materialized instantly :(

However, last week we did take a small step forward during the Workshop on Emerging Trends in Software Metrics. During this workshop, Arie van Deursen presented our proposal for a Software Metrics Catalog Format. This format is specifically designed to provide a concise, yet meaningful overview of a software metric, while also showing the relationships a software metric has with other metrics.

You can read the complete description of the Software Metrics Catalog Format in our publication, but it is probably more appealing (and fun!) to visit our demo implementation using a semantic wiki hosted at referata.com.

Naturally, all comments, questions, remarks and contributions to the catalog are more than welcome!

Update:If you want to use the format in a scientific publication, this LaTeX template might be helpful (thank you Joël Cox)!

Defending the propositions

As explained before, a thesis in Holland is accompanied by a set of propositions. These propositions are considered to be a part of the thesis, which means that the members of the doctoral committee are allowed challenge them if they desire.

This is why the regulations state that the propositions:

'...shall be academically sound, shall lend themselves to opposition and be defendable by the PhD candidate, and shall be approved by the promotor.'

Furthermore, at least six of the propositions should not concern the topic(s) of the thesis, and at most two propositions can be playful in nature. 

So let's see whether we succeeded, there is the list of my propositions (each one linking to this blog-post which contains an explanation for the proposition):

• To enable the effective application of software metrics, a pattern catalog based on real-world usage scenarios must be developed.
• The software architect should take the responsibility for the implementation of the system
• If software engineering PhD students spend 20% of their time `in the field', their research will be based on more realistic assumptions.
• Making the names of reviewers public will make reviewers more inclined to write better reviews, which increases the quality of the overall review process.
• Hiring a skilled typist is an often overlooked option during the design of an automated process.
• Understanding your goal makes it easier to deal with unpleasant chores.
• The replication of experiments becomes easier when all PhD students must replicate an existing study during their research.
• Using only metrics as acceptation criteria leads to undesired optimization.
• The most important objective in [Boy Scout] training is to educate, not instruct. (cf. Lord Baden-Powell)
• The fact that the McChicken tastes the same everywhere, proves that it is possible to have distinct teams produce the same results.

 So what do you think, did I succeed? And do you agree with them all?

The propositions explained

This post explains all of the propositions listed here.

To enable the effective application of software metrics, a pattern catalog based on real-world usage scenarios must be developed.

This proposition is actually a complete publication, it was published in the Tiny Transactions on Computer Science, Volume 2. As explained in that paper, there has been a vast amount in research in the area of software metrics. Many software metrics have been designed and validated over the past decades, but only a few software metrics are used by project teams to identify and solve problems in a timely manner.

One reason for this lack of adaption is that it is currently hard to decide which software metric should be used in which situation. Documenting the benefits and limitations of metrics makes this decision easier, which ultimately leads to more successful software projects.

 

 

The software architect should take the responsibility for the implementation of the system.

According to the global IT Architect Association:

"The software architect has mastered the value, use, development and delivery of software intensive systems. They have developed skills in software development lifecycles, software engineering and software design. They are also responsible for communicating software concepts to all levels of management and for ensuring that expected quality attribute levels are achieved. "

On different occasions I have encountered a software architect which only concerns him/herself with the design of the system, not with the actual implementation. In other words, the architect does not communicate with the development team. This way of working is based on the assumption that the design is such that all of the quality attributes are achieved. Thus, when the implementation follows the design, the implemented system will also achieve the desired quality attributes.

Unfortunately, the implementation only follows the design in very rare cases. During the implementation developers will run into nasty problems with the used technologies, unexpected events and border-cases, or simply with errors in the design.  It is crucial that the development team can rely on the software architect reflect upon these situations and make the decisions that are necessary. In other words, the software architect should be an integral part of the development team, and the one person that makes all final decisions.

 

If software engineering PhD students spend 20% of their time 'in the field', their research will be based on more realistic assumptions.

The field of software engineering research should reflect upon the way in which professionals design, construct, test and maintain (e.g. 'engineer') software systems. In my opinion, the best way to do this is to observe professionals to identify some of the problems that they are facing. The researcher then develops solutions for these problems and verifies whether these solutions indeed solve the identified problem.

For this last step it is crucial that the research has been based on realistic assumptions about the data available, the effort people want to invest or the processes that can be changed. If any of these assumptions are incorrect it is going to be hard to a) get professionals to apply the solution for the verification, and b) to get wider acceptance for the solution after the initial validation.

By spending time together with professionals in industry, a PhD student gets an idea of what constraints are put upon these professionals in terms of time, resources and data. This knowledge can immediately be used to test the assumptions for potential solutions, avoiding the development of unnecessary or unrealistic ones. 

Making the names of reviewers public will make reviewers more inclined to write better reviews, which increases the quality of the overall review process.

In the majority of cases the review process for conferences and journals is a closed process. The input is a submitted paper and the output is a decision and (possibly) a set of reviews. Most of the time the paper has been read by two or three reviewers who wrote a review, and in some cases these reviews are discussed by the program committee (either online or in person).

Within the current process the names of the reviewers are known to the rest of the committee and the chairs, but the authors normally do not know who wrote the reviews. In theory this makes sure that the reviewers can be honest in their reviews without being afraid that their feedback gets back to them in undesired ways. Unfortunately, this also enables reviewers to reject papers on loose claims or false beliefs. In addition, this cloak of anonymity provides the reviewers an opportunity to be less civilized than they could be. Lastly, being anonymous decreases the reward for writing a very good and detailed review, since only a few people witness and appreciate it if you do.

Especially this last problem can be solved by making the names of reviewers public, since the authors know who to thank for reviewing their paper. In addition, just as a paper should not contain claims without evidence, a reviewer should be less inclined to make a loose claim if his name will be known. Of course, an author still needs to accept a 'reject'-decision, but this should be possible if the feedback in the review is honest, civil and supported by facts.

 

Hiring a skilled typist is an often overlooked option during the design of an automated process.

When I just learned to program during my university training I wanted to automate everything, all repetitive behavior was to be captured in scripts, macro's or programming tools. Using this strategy you quickly run into a situation in which the effort to automate some steps is way bigger than the eventual costs savings. 

This XKCD-comic actually summarizes pretty clearly how much time you can spend on the automation of a certain tasks before you spend more time on the automation instead of the task itself. For some of the systems I have seen this table would have saved quit some time, money and irritation of the peoples involved.

 

Understanding your goal makes it easier to deal with unpleasant chores.

In every job, study or other day-time passing activity there are chores that are not fun to do. For example, in this PhD project my initial reaction to a complete restructuring of a paper is always annoyance. Yes, it might be a good idea. Yes, it will make the paper easier to read. Yes, it does make the story better. But most of all, it requires me to work another three evenings and it requires me to throw away two days of work! Did I mention that the deadline is just three days away?

At these points in time I always tried to take a step back and look at the overall goal, which is to write a nice PhD thesis. The restructuring is gonna cost me now, but the realization that the paper becomes better (okay), which gives a higher chance of acceptation (good), which means that I can finish the overall project on time (awesome!) usually makes me want to do the restructuring anyway.

So, whenever an unpleasant chore comes along I try to ask myself: 'what goal am I getting closer to by completing this task?' In most cases, this helps me to do the task anyway. And in those cases that I cannot figure out why the task helps in reaching a goal, this helps me to not feel bad about not doing the task at all.

 

The replication of experiments becomes easier when all PhD students must replicate an existing study during their research.

Replicating an experiment is performing the same experiment with slight variations in terms of data or set-up. This is in contrast with reproducing an experiment, which is geared towards reproducing the exact same results. Within academia, replication of experiments is needed to confirm earlier results and to broaden the common body of knowledge. Replication of an experiment can be viewed as performing a double-check of the work, making it less likely that an error has been made earlier.

By replicating an experiment, a PhD student learns about the choices that need to be made during an experiment, and the possible effect(s) of these choices on the outcome of the experiment. In addition, the PhD student probably finds out that an experiment cannot be (easily) replicated because of missing data, too few details in the description of a procedure, or the absence of running source-code. Because of this first-hand experience (and frustration) about missing details the documentation produced by the student about his own experiments will be of higher quality, thus making the experiments easier to replicate.

 

Using only metrics as acceptation criteria leads to undesired optimization

 

This proposition is based on chapter 5 of the thesis, which in turn is based on the article called 'Getting What You Measure'. This article describes four pitfalls I have seen over and over again when metrics are being used in a project management setting. The most widespread one of these is probably 'treating the metric', e.g. making changes to a system just to improve the value of a metric. 

In some cases this is not problematic because improving the value of the metric also helps in reaching a goal. For example, if you want people to write more code you could require them to check-in 2000 Lines of Code each day. However, you probably want them to write useful code, something which is hard to capture in a metric. And even if you could, there would a long list of other characteristics that are desirable, but you didn't think of specifying at the start of a project.

Therefore, I think that using metrics as a formal acceptation criteria is fine, but it should always be clear why a specific value of the metric is desirable. In other words, always communicate the overall  goal along with the formal metrics, and focus the acceptation on the goal itself instead of the metrics.

 

The most important objective in [Boy Scout] training is to educate, not instruct. (cf. Lord Baden-Powell)

I spend a large part of my life being a boy-scout, only the last few years I have been a bit busy with a different project. By joining the scouts movement I have had many great times, I have traveled to many different places and have meet a wide range of interesting people. Because of all this, I figured that a quote from the founding father of the scouts movement should be in my list of propositions.

My interpretation of this quote is that you should not strive to tell somebody on what to do next, but that you should help the person to understand the current situation such that he can derive useful actions himself. I think the best way to summarize the benefits of this approach is to refer to the old saying: 'give a man a fish and he can eat for a day, teach a man to fish and he can eat for a lifetime'.

 

The fact that the McChicken tastes the same everywhere, proves that it is possible to have distinct teams produce the same results.

 

I have personally sampled the McChicken in many places in the Netherlands, and this PhD project allowed me to sample them in Spain, Portugal, Canada, the USA, Germany, Belgium, Italy and Switzerland. I am always amazed that this piece of fast-food does not only look similar, but also has the same taste (or lack according to some, 'lack of taste') in all those locations. 

Unfortunately, I do not have any insight into how this distribution/production process works. And although I probably do not want to know all of the details, I would really like to understand which conditions have to be met in order to replicate this achievement in other fields.    

Time to defend the dissertation

Since October 2008 I have been introducing myself as 'a technical consultant and a PhD student'. On the 28th of this month, around 16:15 hours I hope to drop the second part of this sentence!

Because June 28th (at 15:00 hours) I will start defending my dissertation against the eight members of my doctorate committee. The ceremony lasts for a little over an hour and is carried out according to a strict set of rules, which prescribe everything from the way in which everybody needs to be addressed up until the clothes that will be worn by the committee, my paranymphs and me.
 
The subject of the defense is my dissertation, titled 'Metric-Based Evaluation of Implemented Software Architectures'. If you have followed this blog you know most of the content by now since it is basically a compilation of my previous publications. If not, the easiest thing to do is to read through the summary enclosed in the PDF version of the dissertation. 

At first I was a bit skeptical about the usefulness of bundling all of the papers together since they are already published. However, writing down the overall story felt pretty good, and I must say that it was quite a joy to unpack the printed copies of the resulting book! (BTW, there are still a few copies left, so you can probably still get one of them at the defense).

Apart from asking questions about the dissertation itself, the members of the doctoral committee are also allowed to ask questions about one of the ten propositions that accompany the dissertation. Although I am not sure whether I can pull off a 'Project #tweetprop' (e.g. a short blog-post per proposition), but I'll definitely discuss the propositions in a later entry. So stay tuned!

When do you consider a software metric useful?

Do you use software metrics in your project? Which ones? Why do you use those software metrics?

The answer to question one is probably 'Yes'. The answer to question two may vary, but hopefully the answer to question three is: "because I find them useful".

For me, the usefulness of a software metric is determined by two properties. On the one hand the software metric should be a correct quantification of what I want to measure, while on the other hand the value of the metric should provide enough information to make a decision.

To verify whether a metric measures what you want it to measure you can examine the value of a metric for a small number of cases, or you can conduct a more quantitative experiment to understand the statistical behavior of a the metric on a large group of systems/components/units. The nice thing about such an experiment is that you can conduct it in a relatively safe lab-environment using open-source systems.

Because of its relative easiness this type of evaluation has been done extensively over the past years.  Virtually every scientific paper on software metrics includes at least one or two case studies, but often researchers also examines the statistical relationship between the value of the (newly proposed) metric and other desirable attributes. For example, we did this for our Component Balance and Dependency Profiles metrics.

To understand whether a metric can be effectively used in a decision making process is more complicated. First, you need to ensure that the metric is available for a large number of projects for an extended period of time. Secondly, you need to observe the people involved in the projects and record discussions/decisions involving the metric. Lastly, the gathered data needs to be analyzed to extract usage patterns and identify areas for improvement.

This second type of evaluation requires quite some time, patience, access to a wide range of software projects in various stages of development, and you need to be able to communicate with the people involved in these projects. Basically, you need to find a company which allows you to conduct this type of research, which might be the reason why I did not find any study which evaluates software metrics in this way.

You can probably guess which company allowed me to conduct this research. Indeed, within the environment of the Software Improvement Group me and my co-authors were allowed to study the usefulness of our architectural metrics. The full details of the evaluation design and the results are available in our ICSE 2013 SEIP paper:

Evaluating Usefulness of Software Metrics - an Industrial Experience Report.

which is going to be has been presented at the ICSE conference in San Fransisco! The slides of this presentation can be found by clicking this link.

Naturally, I am very proud of this paper. In particular because it takes the evaluation of the software metrics one step beyond the usual statistical validation. What do you think, should all metrics be validated like this or should we look at other aspects as well?

Detecting Cross-language Dependencies Generically

Most systems I see are not written in just a single technology. Instead, dedicated technologies  are used for the front-end (think JavaScript, HTML, JSP), the server-side business logic (think Java, C#, C, insert-your-favorite-programming-language-here) and the data-storage (think any flavor of SQL or NoSQL). 

To get an initial understanding on how all of these technologies work together in the system it is needed to determine what the dependencies between these technologies are. Where does the JavaScript connect to the server? And how is the stuff we see on the front-end connected to the stuff stored in the database?

For the most common cases there is tool-support available to detect these dependencies (semi)-automatically. However, extending these tools with support for a new language is not trivial because the techniques used requires you to deeply understand the full language of the new technology.  

Until now...

Last year I had the pleasure of co-supervising Theodoros Polychniatis for his internship at the Software Improvement Group on the subject of Detecting dependencies across programming languages. After diving into the available literature and soliciting requirements from consultants he managed to create a prototype tool which implements a relatively simple algorithm to detect dependencies across source-code modules written in different technologies. 

The first evaluations show that the algorithm is capable of producing a relatively good recall (e.g. finding the dependencies that you want) and a reasonable precision (e.g. finding only actual dependencies). As always, the behavior of the algorithm greatly depends on the parameters used, but the initial results are positive enough to continue exploring this idea.

Apart from producing the thesis Theodoros was also the driving force behind the paper:

 'Detecting cross-language dependencies generically'

by  Theodoros Polychniatis, Eric Bouwers, Joost Visser, Jurriaan Hage and Slinger Jansen.

This paper will be presented (and published) at the 17th European Conference on Software Maintenance and Reengineering (CSMR 2013). If you are considering attending CSMR make sure you drop by at the Software Quality and Maintainability (SQM) workshop for a chat and a very interesting key-note!

As a closer, here is the abstract of the paper:

In order to evaluate large, heterogeneous information systems (i.e., comprising modules developed in diverse programming languages) a method to detect dependencies among these modules is needed. Although there is a variety of methods that can detect dependencies within a single programming language, the available cross-language detection methods use extensive language specific information to parse and analyze modules written in different languages.
In this paper, a new method for detecting cross-language dependencies is proposed. This method is generic, yet accurate and can support new languages with minimal effort. To evaluate the method, a tool was created and a series of experiments was conducted on a small case study for which dependencies had been extracted manually. The evaluation shows that the method is effective, extensible and easily explainable.

Tutorial: Software metrics - Pitfalls & Best Practices

The International Conference on Software Engineering (ICSE) is one of the (if not the) largest conference on software engineering in the world. This year, the conference will take place in the Hyatt Regency, San Francisco, U.S.A. 

At this conference, Arie van Deursen, Joost Visser and I will be organizing a three-hour tutorial. To quote our proposal: 

Using software metrics to keep track of the progress and quality of products and processes is a common practice in industry. Additionally, designing, validating and improving metrics is an important research area. Although using software metrics can help in reaching goals, the effects of using metrics incorrectly can be devastating. 

In this tutorial we leverage 10 years of metrics-based risk assessment experience to illustrate the benefits of software metrics, discuss different types of metrics and explain typical usage scenario’s. Additionally, we explore various ways in which metrics can be interpreted using examples solicited from participants and practical assignments based on industry cases. During this process we will discuss four common pitfalls of using software metrics.

In particular, we explain why metrics should be placed in a context in order to maximize their benefits. A methodology based on benchmarking to provide such a context is discussed and illustrated by a model designed to quantify the technical quality of a software system. Examples of applying this model in industry are given and challenges involved in interpreting such a model are discussed. 

This tutorial provides an in-depth overview of the benefits and challenges involved in applying software metrics. At the end you will have all the information you need to use, develop and evaluate metrics constructively.  

It is yet unclear when the tutorial takes place (either before or after the main conference), I will update this post as soon as this information is available. The tutorial has been given on May 21, see the ICSE program for more details. The slides are now available by clicking here.

Meanwhile, please feel free to share your thoughts, remarks or questions on this topic via the comments or any other means of communication!

A last post?

It took some time to actually post it here, but I am very pleased to say that the paper "Dependency Profiles for Software Architecture Evaluations" by Bouwers, van Deursen and Visser has been accepted at the Early Research Achievements-track of the 27th IEEE International Conference on Software Maintenance.

Before dumping the abstract I want to confess that I have given in and created a twitter-account. As you might have noticed, my updates have been infrequently at best. This is mainly because it takes me a long time before I start writing. Let's see whether this twitter-thing makes this easier!

Abstract:
In this paper we introduce the concept of a “dependency profile”, a system level metric aimed at quantifying the level of encapsulation and independence within a system. We verify that these profiles are suitable to be used in an evaluation context by inspecting the dependency profiles for a repository of almost 100 systems. Furthermore we outline the steps we are taking to validate the usefulness and applicability of the proposed profiles.

WICSA 2011

And yet another publication to announce! I am very happy to tell you all that the paper "Quantifying the Analyzability of Software Architectures" by Bouwers, Correia, van Deursen and Visser has been accepted at the 9th Working IEEE/IFIP Conference on Software Architecture!

Abstract:
The decomposition of a software system into components is a major decision in any software architecture, having a strong influence on many of its quality aspects. A system’s analyzability, in particular, is influenced by its decomposition into components. But into how many components should a system be decomposed to achieve optimal analyzability? And how should the elements of the system be distributed over those components?
In this paper, we set out to find answers to these questions with the support of a large repository of industrial and opensource software systems. Based on our findings, we designed a metric which we call Component Balance. In a case study we show that the metric provides pertinent results in various evaluation scenarios. In addition, we report on an empirical study that demonstrates that the metric is strongly correlated with ratings for analyzability as given by experts.

(and yes, maybe twitter is not such a bad idea if I keep on writing these short posts :)

SQM 2011

Yes I know, it is a bit late. However, I am still very pleased that the paper Preparing for a Literature Survey of Software Architecture using Formal Concept Analysis by L. Couto, J.N. Oliveira, M.A. Ferreira and E. Bouwers has been accepted (and presented) at the Fifth International Workshop on Software Quality and Maintainability!

The abstract:

The scientific literature on Software Architecture(SA) is extensive and dense. With no preparation, surveying this literature can be a daunting task for novices in the field. This paper resorts to the technique of Formal Concept Analysis (FCA) in organizing and structuring such a body of knowledge. We start by surveying a set of 38 papers bearing in mind the following questions: “What are the most supported definitions of software architecture?”, “What are the most popular research topics in software architecture?”, “What are the most relevant quality attributes of a software architecture?” and “What are the topics that researchers point out as being more interesting to explore in the future?”. To answer these questions we classify each paper with appropriate keywords and apply FCA to such a classification. FCA allows us to structure our survey in the form of lattices of concepts which give evidence of main relationships involved. We believe our results will help in guiding a more comprehensive, in-depth study of the field, to be carried out in the future.

A new year, a new set of chances

At the end of the old year (or the start of the new year) there is usually some time to clean up the old stuff and get around to the things you keep postponing.

In my case, I finally got around to put the slides of my Landelijk Architectuur Congres (National Architecture Congress) presentation on-line. Currently the slides are in Dutch, but fortunately I got an invitation to present the work to an English audience. A nice chance (and 'stick-behind-the-door') for me to translate the slides.

And although it might be a bit late (the deadline is just around the corner), the new year still offers you a chance to participate in the Second Workshop on Managing Technical Debt 2011, to be held on the 23th of May.

It might be fun to try to explain what 'Technical Debt' is and why you should care about it, but I believe others already did this better than I could. So let me just point you to this post of Philippe Kruchten, which contains all the information and links you need to get started. Looking forward to your contributions!

Attending LASER 2010

Today I managed to go through my notes of the 2010 edition of the LASER summerschool series. While doing this I realized that, apart from the fancy hotel, the good food, the nice people and the great weather, the trip also had a high educational value!

• Walter Tichy explained several important aspects, which I need to take into account while designing my next empirical study.
  
• In addition, Natalia Juristo pointed out several things I will keep in mind during this design in order to make sure others can repeat it.
• Tim Menzies gave a very good introduction into the ins and outs of data-mining (in such a way I actually want to try it out)
  
• Bertrand Meyer (and his students) made me aware of some testing techniques I forgot about.
• The presentation of Joshua Bloch was, for me, less relevant from a PhD point-of-view, but his thoughts about good API design are extremely interesting from a developers point of view.
• It was really fun (and interesting) to hear Barry Boehm explain his research on estimation models and how empirical SE can be integrated into education.
• Last but not least, the presentations of Victor Basili contained a lot of stuff which I believe is relevant for what we are doing at SIG.
  

Now, all we have to do is to figure out how we can free up some time to apply all that we learned :)

A magazine article

This week I received some hard-copies of the July/August issue of the IEEE Software magazine. This issue contains an article called
A Lightweight Sanity Check for Implemented Architectures
which is written by Arie van Deursen and yours truly.

The article looks really great, it has multiple colours, pull-quotes, a side-bar and pictures! And although good looks are sometimes deceiving, this article also contains some pretty interesting content (if I do say so myself).

As expected, it describes a check-list which can be used to get a quick overview of the state of the implemented architecture of a system. Such an overview helps in determining which parts of the architecture can be improved, or confirms that everything is in good shape. I believe that both outcomes are interesting, so I recommend everybody to give LiSCIA a try!

A 5 star system!

My presentation on JFall 2009 went very well, there was a big audience (they barely fit into the room) and there was lot's of interaction going on during the talk. People where asking critical questions and provided good comments.

The topic seemed to make people pretty enthusiastic. Especially the examples of a really complex method, the biggest method and a method with a lot of parameters.

In the presentation I answered the following two questions: 1) what is a 5-star system, and 2) how can I get one? The answer to the first question is: "a system that receives the highest rating in the TUVit Trusted Product Maintainability Certification". I explained what this certificate entails and how the complete procedure works (something which is also summarized over here).

The second question is answered by the following mantra: "make sure that your code is small, simple and specific".

If you cannot make your code small, simple and specific, take a step back and look at the abstract picture (the overall design) of your project. Why does this design disallow you to make your code small, simple and specific?

Can you model stuff differently? Is there a way to abstract over details that clutter your code? How can this type of functionality be grouped in a better way? Try to determine why the design (or your framework) is restricting you and solve that problem. The result will be a more maintainable system.

Naturally, these answers are just the core of the talk, a more complete overview of the presentation is provided by the Atos Application and Development blog (Dutch). For those interested, the slides can be found here.

All and all a nice experience, on to JSpring 2010!?

A 5 star system?

This week I received notice that my proposal for a presentation at the J-Fall 2009 was accepted. So on the 11th of November I will be discussing what a 5-star system is and how to get it. For those of you who cannot wait to find out what it means to get stars for your system can take a quick look over here.

The full abstract of the presentation can be found here (in Dutch). If you want to attend you should register yourself over here.

Hope to see you in Bussum!

I present, you present ....

Last week I was in Canada for the ICSM 2009 (combined with a very relaxing vacation btw) where I got to present my first paper of my PhD. The room was a filled with about 50 attendees and, even though I was a bit nervous, the presentation went pretty well. There where some interesting questions and comments, and even a small discussion that had to be taken off-line (which is now again on-line).

During the rest of the ICSM I talked to various people about the presentation and the paper and it seems that the topic certainly appeals to several people. Which is good to know since I think I will be spending some more time in researching it.

For those of you that are interested in the slides, please take a look at the slides page of the ICSM, or just follow the direct link.

To be honest, these slides are not exactly the ones I used during the presentation. I cheated a little bit by splitting a slide 22 in two slides, just so that you can actual see the contents of a table. In the original version there was an animation that removes the numbers before showing the crosses, but this behavior is not transferred to the pdf-version of the slides. I can tell you from personal experience that it will increase your blood pressure if you notice this when you are giving your presentation.

Anyhow, at least I learned now to check the pdf-version of my slides before I give I a presentation, just in case the laptop on which I made the presentation does not play nicely with the available beamer :)

A first chapter

Yesterday morning at about 5.48 am I received an email with the following message:

We are pleased to inform you that your paper, entitled
    Criteria for the Evaluation of Implemented Architectures
has been accepted for publication as a FULL RESEARCH PAPER in the conference proceedings.

The e-mail was send by the ICSM 2009 Research Program Co-Chairs. It also explained that out of the 162 research papers submitted, 35 were accepted as full research papers (21.6% acceptance ratio). A nice beginning of my PhD thesis :)

Abstract:

Software architecture evaluation methods aim at identifying potential
maintainability problems for a given architecture. Several of these
methods exist, which typically prescribe the structure of the
evaluation process. Often left implicit, however, are the concrete
system attributes that need to be studied in order to assess the
maintainability of implemented architectures.

To determine this set of attributes, we have performed an empirical
study on over 40 commercial architectural evaluations conducted during
the past two years as part of a systematic ``Software Risk Assessment''.
We present this study and we explain how the identified attributes can be
projected on various architectural system properties, which provides an
overview of criteria for the evaluation of the maintainability of implemented software architectures.

Solving bad smells

Last week I took some time to checkout the JDeodorant Eclipse plugin. The plugin was introduced to me at CSMR 2009 by Nikolaos Tsantalis. The demo he gave me already showed some potential, and playing around with the plugin confirmed this first impression.

Installing the plugin was easy, I just followed the installation instructions on the website. At first I used the tool on one of our smaller projects to find refactorings for long methods. After some minutes the results showed a long list of refactor-candidates. Unfortunately, this lists also contained many propositions to move, for example, all statements of a method to a new method. Even though it is a theoretically correct proposition it does not really make sense in general.

Since I did not feel like analyzing the complete list I tried a different approach. I first used checkstyle to find potential candidate methods and let JDeodorant analyze these files one at a time. This does not only speed up the process of finding the candidates, it also provides a list that is way easier to handle. Using this approach I found some new ideas for a several smaller refactorings in just a few minutes.

In the end I think that, even though it still has some rough edges, this plugin can be very useful. Anyway, the plugin is definitely put onto my watch list.

SQM and CSMR 2009

Last week, I was in Kaiserslautern to attend the 2009 version of SQM and CSMR. Both the workshop and the main conference were hosted at Fraunhofer IESE, which turned out to be an excellent host.

All of the conferences I have attended up until now lasted only for a day, so I did not really know what to expect of four days of presentations. I was a little bit afraid that I would not be able to pick up any new information on the last day, but everything turned out to be fine. Every day brought at least two interesting presentations, and definitely one interesting conversation about the current research somebody is doing. I have met several interesting people (who might be interested in doing some research together), and picked up quite a few new ideas. Unfortunately, it is too much to write down all at once, but I will try to post updates on this on a more regular basis.

What I found particularly interesting was the fact that several people mentioned the same two things on various occasions. The first one is that PhD students should not start from scratch when they are implementing their ideas. Although some students are using more generic development frameworks (like for example Eclipse), there are still people out there that start everything from scratch. This does tend to take up a lot of time and usually results in tools that "should be publicly available before I finished my thesis, I just want to clean it up a bit first". I am not really sure how this problem can be solved (and neither did the people at the Industry Panel Discussion), but I do hope that my future tools will not suffer the same fate.

The second thing that was repeated a few times is the call for more empirical evidence for the usefulness of ideas and tools. However, the lack of publicly available industrial systems and cost-data makes it hard for the average researcher to provide this evidence. Fortunately, there are ways to overcome this issue. One could, for example, work together with companies that have access to this type of data. Just a matter of sending and e-mail and explain your ideas and needs to the right person I guess... ;)