Making Good Software

They are evil!!

It always comes the time in almost every project when some difficult task needs to be performed and so the team will ask themselves: “How are we going to implement it?” is usually at this stage, after several meetings, brainstorming and debates, that an approach is decided, and a proof of concept is arranged… And that’s when the team fails in realizing that they are about to waste their time.

Proofs of concept look ideal on paper; they are supposed to be an excellent mechanism to validate the approach that is about to be taken in the project, but the reality is that they are not, and that’s what I would like to talk about in this article.

Why proofs of concept are bad.

1. They don’t tell you if you are about to do the right thing, they only tell you if that specific approach is feasible. Most of the times, is not about if what we are proving is valid or not, is about how much overhead there is in maintain or integrate what we are proving, but proofs of concept are usually very limited telling us what these boundaries are.
2. People who sponsor a proof of concept get emotionally attached and loose their objectivity. After a proof of concept, a decision needs to be taken, but the people who sponsored the proof of concept, because their ego, will have their objectivity affected and will push to have the proof of concept accepted no matter its output.
3. Proofs of concepts tend to be very specific and focused; they leave outside of the picture many critical areas that need to be proved. For instance, if proving a reporting tool: The proof of concept may be focused in generating some particular type of report, but it may leave outside of the picture some other areas, as the integration of the reporting tool with the application being developed or how customizable the generated reports are.
4. Designing a good proof of concept requires a full understanding of the domain. To design a good proof of concept, is important to identify upfront what are the different risk areas, so they can be proved, but for most situations that’s impossible, so is very likely that after adopting some approach the team is going to hit some unexpected limitations on the approach.
5. Proofs of concept distract the team from the main goal and make them switch to an R&D mentality. Teams doing proofs of concept usually switch to a R&D mentality, with a R&D mentality is very easy to focus in gold platting the application.

All this disadvantages may translate in the following four real major issues in your project.

1. Waste of time
2. Technical debt
3. Fail to deliver all the requirements.
4. Integration issues.

How to avoid Proofs of concept

These are my 5 key practices for avoiding proof of concepts

1. Find the next baby step. Usually when a proof of concept is kicked off is because the team fails to split the biggest problem into smaller issues. Tackling the smaller issues one at a time is easier and is likely to produce a more integrated solution.
2. Keep it simple. Is usual to find proofs of concept to validate at the start of the project how the latest technologies, frameworks… would fit in the application, but I wouldn’t even consider them, I like to start simple, and if necessary add them as we go. I think adding them upfront can become a huge risk factor to the team as they introduce new learning, complexity and integration factors into the project.
3. Just do it. When a team is stuck, I find that the best remedy is just find what the next baby step is and implement it.
4. Work several options in parallel. Sometimes is just impossible to determine in what direction the next baby step should be, for these occasions just work in parallel all the different solutions. The overhead of working in parallel is paid back early in the project because it will be very clear what solution is the most adequate.
5. Delay decision for as late as possible. The general rule of thumb for the whole article can be resumed with the lean principle “Delay your decisions for as late as possible”, find baby steps, work on parallel… but don’t commit to an approach that may cause your project to fail.

Micro optimizations are one of many different bad practices that can be found in a project.

Micro optimizations are bad responses to potential or real project issues, they are created when the team fails to identify the root causes and instead they address the consequences, this usually happens because:

1. Urgency. Not enough time and resources are used to analyze the issue.
2. Experience. The patch strategy is always been followed.
3. Fear. Sometimes people know what the root cause is, but they are afraid to mention that.

Micro optimizations are bad because:

1. They switch the focus of the team from the big picture, they provoke the “Can’t see the forest for the trees” effect.
2. They provide partial feedback.

Micro optimizations are usually implemented as check lists, metrics, reports, documentation or some sort of paper work that needs to be approved in order to move forward.

Some common scenarios of micro optimizations that I have myself observed are:

• Because of the high rate of defects, the team decided they had to freeze the development from time to time and wait for QA to pass complete regression tests. This turned out to be a micro optimization and didn’t help at all. The root cause for the high rate of defects was that the source code quality was very poor, which in turn, was caused by a complete lack of experience and knowledge on the team on the technologies used for that particular project.
• Because of low productivity, management decided to hire more people for the project. Again, this turned out to be useless, the root cause for this issue was that development and product owner were not collaborating as they should, and as consequence, development never understood what was really required by the business.
• Because of poor performance, the team discovered that the bottleneck was on the messaging layer, so they decided to expend lots of money in a commercial solution that delivered better performance, this turned out to be a micro optimization and the performance issues came back. The root cause was that the overall architecture was poorly designed.

To avoid micro optimizations, focus always in the big picture, not in the details; use an out-of-the-box mindset. One helpful, very well known and very simple technique to avoid micro optimization is “the 5 whys technique”, this technique is better demonstrated with a sample.

The team only deliver half of what it was expected for the deadline. (The issue)

1. Why? They didn’t have enough time
2. Why? They had too much work
3. Why? They failed to estimate how much work they could take.
4. Why? They thought that the required activities to complete were simpler.
5. Why? The requirements were not detailed enough.

Avoiding micro optimizations doesn’t mean that the aspects they would usually be focused on, as: performance, punctuality, code coverage… are not important; it means that they serve to a higher purpose.

The biggest effect on moving away from micro optimizations is that you are likely to find that your main issues in your project are not technical, so probably you may want to take a complete new approach on them.

In my opinion coupling is a very dangerous metric, is something desirable, but if it drives the architecture, it will make your project over complicated, to try to prove my point I am going to start from the basics.

In object oriented programming, coupling refers on how strongly linked are two objects that need to talk to each other.

The degree of coupling is important because, given a change in one of the components, the chances of having to change the other linked object, is proportional to the degree of coupling. When talking about coupling, objects are considered loosely coupled when they are designed so that a change in one object doesn’t require the other object to change.

Loose coupling is one of the most desired qualities in modern software development, but because its very subjective nature, and because the lack of analysis on the different types of coupling, some wrong decisions may be taken in the architecture just for the sake of making the application more loosely coupled.

Considering “the looser the architecture, the better”, is wrong, for each two objects that need to be coupled, their circumstances should be analyzed and based on these circumstances, the most appropriate type of coupling should be used.

Types of coupling.

Direct coupling.

Direct coupling happens when the two objects that need to be linked, talk to each other directly.

From a hardware point of view, this coupling always happens between two objects from the same application which are sitting in the same shared memory area.

Direct coupling is the simplest coupling to implement, but is the one with the highest level of coupling. A change on any of the linked objects is very likely to cause a change in the other object.

There are two styles of direct coupling. Black box coupling and White box coupling.

• Black box coupling is a style of direct coupling where the link between the two objects is created without revealing any implementation detail. The most common form of black box coupling is when object A calls directly a public method in object B.
• White box coupling is a style of direct coupling where the link between the objects carries some implementation information. The most common form of white box coupling is when object A access directly a member variable of the object B. White box coupling is discouraged because it creates even higher coupling than black box coupling and the complexity and effort remains the same.

Indirect coupling.

Indirect coupling uses a third, or more elements, to couple the two objects that need to talk to each other. The purpose of adding additional layers between the two components is so that even if one of the linked objects change, the intermediary doesn’t, or if it does, it changes internally so from the other linked object perspective everything remains the same.

Indirect coupling adds complexity to your application but it offers the loosest coupling.

Indirect coupling can be classified depending on

1. Location of the components.
• In memory. The two objects that need to be linked are part of the same application and are loaded at the same time in memory.
• Remote. The two objects are part of different applications, so they don’t share the memory space. Remote coupling adds complexity but is considered to have a lower coupling than in memory coupling
2. Synchronicity.
   • Synchronous coupling. The standard indirect coupling.
   • Asynchronous coupling. Also known as fire and forget, if asynchronous coupling is required, the complexity of the application would be higher, but it will be likely to have a lower coupling.

Conclusion.

The main conclusion is: The lowest is the coupling, the highest is the complexity. The “Oh yeah! Let’s just take a loose coupling approach”, doesn’t work. What is really important in your design, is how simple it is, so it should be prioritized the simplest approach as possible, which, funny enough, will probably carry a higher coupling.

The degree of coupling should be dictated by the requirements, and the simplest approach should be preferred.

Not dealing with uncertainty efficiently is one of the main causes for software development projects to fail. Traditional approaches assert that uncertainty can be defeated by designing and planning ahead, but that’s wrong. Even in a small development, uncertainty is so high that to discover all of it up front is impossible. That’s why classic approaches, as waterfall, fail in dealing with uncertainty, and that’s why, in my opinion, they also fail in dealing with change.

Change and uncertainty are at the core of any software development, as Heraclitus said: “Change is the only constant”, and software development is not exception. That’s why dealing with uncertainty is so important. Some of the most common consequences of not dealing properly with uncertainty are: false expectations and bad estimations.

False expectations. Uncertainty is going to cause change, and change needs to get fed back to all the stake holders, but is very easy to leave information and people outside the loop, if this happens then the expectations are going to be different across the parties involved in the project causing that some of them will have false expectations.

Bad estimations. As I’ve already said before in this blog, I strongly believe that big planning up front is a waste of time, and that’s mainly because uncertainty. The cone of uncertainty is a very well known diagram which graphically shows this.

Source: http://www.codinghorror.com/blog/archives/000623.html

These are my 3 advices in order to deal with uncertainty.

Small steps

It is better to take many small steps in the right direction than to make a great leap forward only to stumble backward.

Source: http://atriskliving.blogspot.com/2008/09/goodbye-baby-step-1.html

It is impossible to get rid of all the uncertainty, so the best way to deal with it is to take as less uncertainty as possible at a time, your estimation then is going to be more accurate and the expectations across all the stakeholders in the project are going to be aligned. Small steps will also provide with quick feed back, so you can correct the direction in your project as soon as is necessary, and it will help you to reduce the total remainder amount of uncertainty.

As a rule of thumb, I don’t like to have tasks longer than 2-3 days. These tasks should cover a whole end to end scenario in your application and should have a clear acceptance criteria, an example of task for a online book store could be: “Add the option of payment with credit card to the checkout page”.

Iterations

Can’t See the Forest for the Trees

Small steps need to have some sort of higher purpose, if not it would be like trying to climb a mountain by never looking further than 5 meters, just always taking the steepest path, but that is usually not the best path to climb a mountain.

Iterations are short time boxed periods that wraps small steps, their purpose is to serve as control points to demo functionality to the product manager and ensure that the direction of the project is correct.

Communication

Good communication plays a primary role when dealing with uncertainty, it is key, that all the parties involved in software development are aware on how uncertainty develops.

What follows is a summary of my own high level approach for testing software, this strategy is based in facts and principles.

Facts

1. It is impossible to detect all the bugs from an application.

The closer you get to 100% coverage, the harder it is to find the remaining bugs.

2. The more important bugs are in the core layer and in the integration layer (backend).

That’s where the testing needs to be focus on. Core layer bugs and integration layer bugs are the most important because they create a cascade effect causing several parts of the application to fail.

3. Using UI automated tests makes harder the detection of bugs.

Even though they are still very popular, UI automated tests are not very effective finding bugs because they test the core layer and the integration layer indirectly. Testing indirectly the backend makes difficult to exercise it, and makes hard to tell where an error is coming from. It is also important to notice that UI automated tests are also slow and expensive to maintain.

4. Manual testing is still necessary.

There are some important bugs that can only be detected through manual testing, that’s the case of the bugs that can be found doing usability testing and exploratory testing.

5. Testing is worthless if is not executed in a continuous basis.

What’s been proven correct through testing now is going to change very soon so it will have to be proven right again. If this feedback is not fast enough, new changes won’t get proven and eventually new bugs will be entered into the system.

Principles

1. Prioritize what’s going to be tested.

Never have a test strategy that expects to cover 100% of the application.

2. Have as much automated tests as possible.

From your previous prioritization, automate as much as you can.

3. Schedule time for the necessary manual testing to be performed on the project.

Two of the main manual testing activities that are necessary to perform are usability testing and exploratory testing.

4. Use preferably backend automated tests instead of UI automated tests.

I am privileged to work on something which I actually enjoy. It is like being a professional sportsman; I get paid to do stuff that I love.

Actually, based on the majority of programmers I know, I would say that most of us feel the same. We are basically a bunch of geeks trying to prove to each other who is a better programmer, we somehow see the day to day in the office as a grown up version of…

<geekstuff>
     <choices>
          <rts>Civilization</rts>
          <mmorpg>World of warcraft</mmrpg>
          <role game>Lord of the rings</ role game>
          <movie>Star wars</movie>
     </choices>
</ geekstuff >

Going to the office is then like a game, and as in any game, if you are not having fun, what´s the point? So, are you still having fun? Having fun at work is, in my opinion, one of the differences that can make a great software developer.

If you are not having fun, you won´t probably be motivated, and if you are not motivated, you are going to do a poor job, so to try to help you, let me present my four golden rules to keep it fun at work!

See your colleagues, and show yourself as a friend, not as a competitor.

Try to avoid getting too emotional when you have arguments with your colleagues as Dale Carnegie said “The only way to get the best of an argument is to avoid it.”

Change your mind set from “what can I do to show they are wrong” to “what can I do to help my colleagues”.

Look for challenges.

Doing easy and repetitive stuff is simply boring.

Don´t take it too seriously, it is only a job.

At the end is only a job, don’t get too stressed if you don’t want to eventually see yourself with an anxiety attack in the office.

If still is not fun, just find another job.

To me not having fun is critical, we expend a huge amount of time at work, so don’t waste it, if you are not having fun just find another job, even if the money is not as good!!!

Continuous integration is in my opinion the most important practice to successfully deliver good software. Continuous integration should go beyond a simple set of practices and tools to build your source code every now and then; doing so is continuous compilation, not continuous integration.

Continuous integration should drive the way the software gets build and how the different actors from the project interact with each other.

The five main characteristics of continuous integration

In my opinion these following five characteristics are the main characteristics of an effective continuous integration.

1.- QA and development work together.

Having QA and development working together mainly means two things:

1. There are no handovers between development and testing. Coding and testing are not two separate processes, they are performed in parallel, and one can’t be completed without the other one and vice versa.
2. Developers and Testers have a common goal, they don’t perform different activities, developers are not just focused on developing and testers in testing. They share a common goal, produce quality software in coordination with the product owner expectations.

2.- QA act as a business representative for the developers.

The integration with the business requirements is critical to successfully develop quality software, failing to do so can have catastrophic consequences. QA should take special care special of the integration with the business requirements, they should make sure at every time that all the tests and developments are aligned with the business requirements.

3.- The build is the most important asset in a software development.

The build is the image of all the previous activities; it contains the software which is already been developed.

1. The build should never be broken
2. If the build breaks, fixing the build is everyone’s top priority

4.- Continuous feedback and inspection.

Knowing at every time what is the status of the project is also critical, there are a few things to be considered:

1. Automatically check the build every time someone checks code in.
2. Run full regression tests every few hours.
3. Every time a build runs, automatically generate reports including, code coverage, code quality metrics, new lines of code…

5.- Communication is king

Everything is about communication, developers talking to testers and business, and testers doing the same. An important rule of thumb for communication is called “power of three”, in every single important discussion there should be at least, a developer, a tester and a representative from the business, if all the rest works, but communication doesn’t, the project is doomed to fail.

Top recommended actions to improve your continuous integration.

1. For every story, first, do a high level design of what needs to be tested. Prepare this design with QA, development and the product owner. Drive your development with that test design.

2. Only consider two states for your stories: “in process” or “completed”. Don’t use intermediary states as “in QA” or “60% completed”. If you do so you will never have QA and development working together, but against each other.

3. Have testers pairing with developers every now and them. This will enhance the communication between them, and will help to understand what needs to be tested.

4. Make QA people work very close with the product owner. Make sure they keep him well informed of what is going on in the sprint and show him if possible any progress in the stories.

5. Make developers responsible for breaking the build. Developers should perform two steps before committing any code:

1. Update all source code.
2. Build locally the whole application. (If the complete build takes too much time, split it and make the developers run all the unit tests before committing any code).

6. Make your CI continuous integration server to run the build every time someone checks in any code into your CSM. This will make sure that no one checks in broken code.

7. Run full regression tests as many times as possible during the day. All your tests should be automated, hook them if with the build which gets trigger when anyone commits code, if they take too much time, schedule them to run as many times during the day as possible.

8. Add a few visual devices in the office to tell when the build is broken. There are many visual devices that can be configured with almost any continuous integration server, place them in the office strategically, so whenever they turn on, everyone will realize that the build is broken, some examples of this devices are: traffic lights, a red light, a lava lamp…

9. Include an inspection report for every build. In modern continuous integration servers is easy to add different plug-ins so for each build you can obtain additional information, as test code coverage, compliance with coding standards, lines of code duplicated…

Lots of people confuse “test first methodologies” with TDD, it is very common to listen comments like “TDD is just about writing your tests first”, which are completely wrong, these kind of affirmations are not describing TDD at all, they are talking about test first development.

The main reason for confusing TDD and test first development is its own name: “test driven development”. If someone that doesn’t know about TDD would had to guess based on the name what TDD is, would probably guess that is just a test first methodology. But is not!

TDD as invented by Kent Beck, (who also invented Xtreme programming and Junit), goes beyond that. In the core of TDD there is a process to follow, which makes it already different from a simple test first approach.

Image source: Wikipedia

This is also known as red (make your test fail), green (make it pass) and refactor . Where this may be seen as a small difference from a test first approach, if combined with some other agile engineering practices and development philosophies, makes TDD quite different from any other test first approach, it actually switches the focus from testing to design.

TDD is a design practice, is more related with emergent design than with testing. TDD looses much of its potential if it is not combined with other agile engineering practices, or agile philosophies as YAGNI and KISS, in TDD having a large set of tests is a nice side effect, not just its purpose.

To know if you are doing TDD the right way, look at your code, TDD code should look simple and lean, TDD also usually generates more testing code than production code, and you should feel that it helps you to design your code.

So, next time you say you are doing TDD make sure you don’t mean that you are doing test first development.

Creating services is quite an abstract subject, everyone has his own ideas and preconceptions, so is worth clarifying what I mean by services in this article. A service is a component that holds some business logic which can be easily reused anywhere else in the application no matter where the service is located, or what the communication mechanism is used between the client and the service. The following code samples detail my own personal approach on how to implement services in Java.

First thing is to design the service itself through its interface

public interface ServiceA {
	public void doThis();
	public void doThat();
}

Next step is writing the implementation of the service. Is also important to remind that the implementation should just read business logic, it shouldn’t be aware of if is going to be called through a web service, jms…

public class ServiceAImp implements ServiceA{

	protected ServiceAImp (){};

	@Override
	public void doThat() {
		System.out.println("do that");
	}

	@Override
	public void doThis() {
		System.out.println("do this");
	}

}

Now it comes the tricky part, I like to make the services look as if they were local to whoever is using them, it makes them easier to use and because the communication code is hidden and doesn’t need to be written every time a service is invoked they are also less prone to bugs. For this purpose I like to use the proxy pattern.

public class ServiceAProxy implements ServiceA {
	ServiceA implementation;

	protected ServiceAProxy(ServiceA imp) {
		this.implementation = imp;
	}

	public void doThat() {
		implementation.doThat();
	}

	public void doThis() {
		implementation.doThis();
	}
}

Next thing is to deal with the communication issue, now we have a proxy that makes the service look as local to the client, but there isn’t an implementation of a communication mechanism between the client and the service, to do so, we create as many different “stubs” as communication mechanisms we want to implement. For this example, we are going to support a remote WebService call, and a remote JMS call.

The WebService stub.

public class ServiceAWebServiceClient implements ServiceA{
	ServiceA webService;
	public ServiceAWebServiceClient() {
		//webService = code to get the reference to Service A from the webservice call
	}

	@Override
	public void doThat() {
		webService.doThat();
	}

	@Override
	public void doThis() {
		webService.doThis();
	}
}

The WebService implementation could be something like:

@WebService
public class ServiceAWebService implements ServiceA{
	ServiceA imp = new ServiceAImp();

	@Override
	public void doThat() {
		imp.doThat();
	}

	@Override
	public void doThis() {
		imp.doThis();
	}
}

The JMS stub

public class ServiceAJmsClient implements ServiceA{
	ServiceA jmsObject;

	public ServiceAJmsClient() {
		//jmsObject = code to get the reference to Service A from the jms call
	}

	@Override
	public void doThat() {
		jmsObject.doThat();
	}

	@Override
	public void doThis() {
		jmsObject.doThis();
	}
}

Now the service is almost ready, everything is kept separated so changes to the business logic are only done in the ServiceAImp, and changes to the actual transportation mechanism are implemented in their own stub, also we provide with a proxy so that whoever is using the service will use it transparently, the only problem left we have, which is a big one, is that this is quite a collection of classes we are asking our clients to understand in order to use our service, to fix this, I like to use a factory.

public class ServiceAFactory {
	private ServiceAFactory() {}

	public static ServiceA getLocalService(){
		return new ServiceAImp ();
	}

	public static ServiceA getRemoteServiceUsingJms(){
		return new ServiceAProxy (new ServiceAJmsClient());
	}

	public static ServiceA getRemoteServiceUsingWebService(){
		return new ServiceAProxy (new ServiceAWebServiceClient());
	}
}

The previous factory is the only thing our customers need to know about us! To put all the pieces together, let’s look at how a client code using our service will look like.

public class MainClass {

	/**
	 * @param args
	 */
	public static void main(String[] args) {
		//Calling doThis with a local instance of the ServiceA, very useful for testing!!!!
		ServiceA serviceA1 = ServiceAFactory.getLocalService();
		serviceA1.doThis();
		//Calling doThis with a remote instance of the ServiceA using Web services.
		ServiceA serviceA2 = ServiceAFactory.getRemoteServiceUsingWebService();
		serviceA2.doThis();
		//Calling doThis with a remote instance of the ServiceA using Web services.
		ServiceA serviceA3 = ServiceAFactory.getRemoteServiceUsingJms();
		serviceA3.doThis();
	}

}

[EDIT] Now you can make the client even less aware of the service location, you can use a dependency injection framework as Spring to set the implementation of your service.

`
public class MainClassUsingSpring {
	public static void main(String[] args) {
		ServiceA serviceA = loadFromSpring();
		serviceA.doThis();
	}

	private static ServiceA loadFromSpring() {
		ApplicationContext context = new ClassPathXmlApplicationContext ("beans.xml");
		return (ServiceA) context.getBean("serviceAProxy");
	}

}

Then you can have a spring file configuration similar to this.

By changing your spring configuration file now you are able to change the implementation of your service and its location without even changing your source code!

The research to become a better software developer is core to any good software developer. Being a better software developer reflects directly in the code you produce, the better software developer you are, the less bugs, easier to read and easier to maintain code you will produce.

What follow are the four rules I use on my day to day work to help me become a better software developer.

Rule number 1:  My code is crap.

All code is crap, and that includes yours, (and mine, of course). No matter how brilliant a piece of code you think it is, there will be always better ways to implement it, and several programmers who will hate it.

Even if you are sure your code happens to be good, force yourself into “my code is crap” mode, and ask around about improvements, be open minded and you will see soon how you will find issues in your code that you never have thought about.

Rule number 2: Even if it’s crap, I care about my code.

It is not about creating the best code upfront, is about making it better every time you visit it. When you finish some functionality, you are not done, I haven’t met any programmer that produces acceptable code on their first attempt, you need to polish it. Refactoring is then one of the main activities you should be performing.

Rule number 3: My opinion about my own code is wrong.

We are so egocentric that we end up believing that what we have produced, no matter how bad it is, is the best solution as possible. Never trust your own opinion, show your code to as much developers as possible and listen to them.

Rule number 4: My manager doesn’t care about my code, and he pays me.

This rule regulates the other three, it is easy to forget that you actually get paid to get things done, your manager should also care about the code, but he is more focused in completing things on time with an acceptable quality, so be careful not to expend too much time just making sure your code is not that crap. The key is being able to balance getting things done and their quality.

Bonus: A few tips to help you with these rules.

• Code reviews.
• Keep it simple.
• Test.
• Refactor.
• Non technical factors.