Category Archives: testing

Teaching TDD from the ground up

One of the first steps in any embedded software project is to implement an assert macro.  This is true for a few reasons:

  • you have certain assumptions about how the hardware should behave and you want to know when those assumptions are broken,
  • you need a way to signal outside of the custom hardware when something has gone wrong,
  • there is no standard function that can deal with these problems.

This might seem like a wasteful thing for every project to undertake, but actually it is a very constructive place to start.  Firstly, it frames the development around verification.  Secondly, when you find a bug you don’t start printf debugging, instead focusing on adding assertions.  And finally, from the very outset of the project you demystify the black box of your embedded system.

In the same way, I think that the first step in learning test driven development should be to write a testing framework.  And it needn’t be difficult – a simple test framework can be written in a few lines of code.  This is the test framework for testing the fake function framework.

/* Test Framework :-) */
void setup();
#define TEST_F(SUITE, NAME) void NAME()
#define RUN_TEST(SUITE, TESTNAME) printf(" Running %s.%s: \n", #SUITE, #TESTNAME); setup(); TESTNAME(); printf(" SUCCESS\n");
#define ASSERT_EQ(A, B) assert((A) == (B))
#define ASSERT_TRUE(A) assert((A))

https://github.com/meekrosoft/fff/blob/master/test/c_test_framework.h

It could be even more concise, but I wanted to make it compatible with the googletest framework so I can re-use the test cases in both C and C++ without modification.  Anyway, you get the point.

I am a big fan of testing frameworks, and for sure there is no need to re-write junit every time we start a new java program.  But for learning TDD, I think a lot of the mystery would disappear when participants make a simple test framework and use that to make their first few tests.

Advertisements

3 Comments

Filed under fff, software, testing

Fake Function Framework – Request For Comments!

I have a little micro-framework called fff.h for generating fake functions (mocks) in C.  I have blogged about it in the past, and there have been some exciting changes over the last few weeks that I’d like to share.

Background

The basic premise is that testing a C source file is difficult in idiomatic C because of all the external function calls that are hardwired into the production code.  The way fff.h helps is to make it a one-liner to create fake implementations of these for the purposes of testing.

Now, the basic formula on my last project for testing a C module is like this:

Typical approach to testing legacy C code

This was crude but effective.  What really grated with me though was the amount of repetitive code that had to be written to answer very basic questions.  Was my fake function called? How many times? What parameters were passed in on each occasion? Was the unlock function called before the write function? To answer these questions we ended up writing endless amount of (not very pretty) code like this:

Manual Fake Functions

It seemed to me that it should be possible to write a macro to generate this code.  This would tidy up the test files in terms of readability and would make it easier for my team to make tests.  And that was the genesis of the Fake Function Framework.

The best introduction to the Fake Function Framework is on the fff github site so I won’t rehash that here.

New Developments

I have since moved on to a new project and haven’t thought about fff.h in a wee while.  Through a variety of coincidences it happened that Tore, the architect on my previous project, met James Grenning during a training course and introduced him to fff.h. James played around with fff.h and sent me some great suggestions for cleanup, and how to improve fff.h to produce globally linkable fakes.  At first I thought that this was an unneeded complication for an otherwise simple framework, but James convinced me that reusable fakes had a lot of value.

I set to work on making the generator able to generate the new style fakes.  I don’t know I would have attempted this without having a full regression test suite for the framework.

The way it works is instead of using the usual syntax to create a fake using the FAKE_xxx_FUNCn macros, you create a header file to hold the declarations:

Global Fake example header file

Global Fake example header file

And then create an implementation file for the function definitions:

Global Fake implementation file example

Global Fake implementation file example

Then you can link many test suites against this single object without any problems. Simple!

Breaking Changes

There have been some breaking changes to fff.h to enable these new features, and I have also taken that opportunity to clean up some weaknesses in the previous design.  But since fff.h is just a header file, both will happily exist in your codebase. All you have to do is name the new version of fff.h fff2.h and start using fff2.h in your new tests.

So theses are the basic changes you should know about if you are already familiar with the fake function framework.  Somewhere in the test executable you must define the globals for fff.  Why not put it beside the main function?

Define FFF globals

In the old version of fff.h there was a shortcut for resetting all the fakes in one step.  This is now gone. The reason is that it only worked in C++ using static initializers and the introduction of global fakes were incompatible with this.

Reset Fakes

There has also been some cleanup.  All the fake function framework variables have been moved into a struct to avoid polluting the global namespace.

fff globals now in struct

fff globals now in struct

And the variables for individual fakes are now defined in a struct.

Fake function data now in struct

Acknowledgements

The fake function framework would not exist as it does today without the support of key folks.  Tore Martin Hagen (and his whiteboard), my partner-in-crime in Oslo, was instrumental during the genesis of fff.  Jon Jagger, who during ACCU 2011 helped me teach the preprocessor to count.  And James Grenning, who convinced me the value of global fakes, sent me a prototype Implementation, and showed me how expressive a DSL can be.  Thanks to you all!

Request for comments

I hope you like the new changes, and if you have any feedback or suggestions for further improvements don’t hesitate to leave a comment or contact me via twitter.

Further reading

40 Comments

Filed under fff, quality, software, testing

Retrospective – Global Day of Coderetreat, Beijing!

On the 3rd of December a small collection of developers gathered in a basement conference room in Beijing to practice the craft of software development. The participants came from diverse backgrounds: architects, developers, students, and managers. There were people comfortable with C++, C#, Java, Python and others hadn’t programmed in a couple of years. Here are my notes from the day:

Introduction

Coderetreat is a day-long, intensive practice event, focusing on the fundamentals of software development and design. By providing developers the opportunity to take part in focused practice, away from the pressures of ‘getting things done’, the coderetreat format has proven itself to be a highly effective means of skill improvement. Practicing the basic principles of modular and object-oriented design, developers can improve their ability to write code that minimizes the cost of change over time.

A coderetreat is a language-agnostic event. In each session, the pair chooses what language they want to work in. The day focuses on practicing the fundamentals of software: TDD and the 4 rules of simple design; these are applicable regardless of language.

Alex and Zilong pairing

Chocs Away!

After a brief introduction, we went into the first session to familiarize ourselves with the task: Conway’s Game of Life. Participants struggled with deleting the code at the end of the session; this was one of the biggest challenges that they faced throughout the day. The second session was to swap pairs and have a second go at the problem with a clean slate. The third session introduced the concept of ping-pong TDD.

Gospers Glider Gun

For lunch we went out to a local restaurant to get a chance to stretch our legs and have a fresh perspective.

Lunchtime!

Lunchtime!

In the afternoon we tried a couple of CyberDojos. The first session we didn’t change pairs so we could have a chance to get familiar with the CyberDojo software. After a few technical issues we were on our way. In the second session we changed the pairs every five minutes, really challenging ourselves to write code in very small increments.

CyberDojo

CyberDojo

The final session I gave the participants a choice: we could agree as a group to either try to create the absolute best code we could for the solution, or we could try to create the absolute worst code possible that implements the solution. It was a tough call but the dirty code challenge won out. In this session there were many creative approaches and a notable visual implementation that looked great in the UI and was a hornets nest in the implementation – you know who you are! ☺

At the end of the day we held the usual Closing Circle, where we each share with the group our feelings on what went well, what was surprising, and what we can take away from the event. There was a general consensus that the day was fun and that it highlighted the importance of communication, both between people and through the code. Also surprising was how many different approaches there were to the same problem.

Globalization

Hello from Japan!

Hello from Japan!

The event is called a “Global Day” for a reason; the same event was happening in over 90 cities across the world on the same day. There were many ways the events connected, lots of activity on twitter with the hashtag #gdcr11, and several events talked to each other via google hangouts or skype. We had a quick chat with the coderetreat in Tokyo in the morning, and before lunch we managed a chat with the folks in Perth, Australia. This was a lot of fun and helped to energise the group.

Hello from Perth!

Hello from Perth!

Thanks to Corey Haines and Jim Hurne for organizing the global day, and also thanks everyone that helped me organize in Beijing. Thanks to Tokyo and thanks to Perth. And special thanks to everyone in Beijing who came along on a blue sky day to spend their Saturday in a basement coding with other software craftspeople.

p.s. I am starting a local software craftsmanship meetup group here in Beijing, sign up to hear about future events and meetings!

Related Posts:
Interview with Corey Haines
CyberDojo

Tags: software testing craftsmanship coderetreat coreyhaines tdd

Leave a comment

Filed under coderetreat, software, testing

C++ for Marine Streamer Positioning and Navigation

Here are the Slides from my ACCU 2011 Conference presentation.  The presentation was about how to use C++ in resource constrained devices, and building a C++ cross compile toolchain for the LM3S8962 Cortex-M3 processor.

Topics include:

  • Short introduction to the domain
  • Building a gcc cross compile toolchain
  • Programming and On-chip debugging
  • Stacks & Heaps: Linker Scripts
  • The time before “int main()”
  • C++ on a diet (living without dynamic memory allocation, the standard library, exceptions, and rtti)
  • Testable Embedded C++

Leave a comment

Filed under software, testing

Book Review: Growing Object-Oriented Software, Guided by Tests

In 2010 I read 37 books of all different flavours but my favorite software book of the year was definitely Growing Object-Oriented Software, Guided by Tests by Steve Freeman and Nat Pryce.  I have written about an idea presented in the book previously , but I thought I’d repost my amazon review here for posterity:

Growing Object-Oriented Software, Guided by Tests is an excellent book by Steve Freeman and Nat Pryce. In it, as the title suggests, they describe an agile approach for growing software incrementally over time, using tests as a tool to provoke progress and to maintain focus on quality. The authors are the creators of the jMock mocking framework, and as such they approach the design of OO software with a heavy bias towards interface-based design.

The book is concise and very well written, it feels like it has been edited to within an inch of its life. It is full of tips and patterns for common testing challenges that can be applied to any project. The book is split into five main parts:

  1. Introduction, explaining the motivation behind test-driven development and a basic high-level introduction to TDD. They explain the reasoning behind the Object-Oriented design principles that they follow in their process.
  2. The Process of Test-Driven Development, describing the test-driven process they follow in-depth. The key differentiator of their method is that they follow an “outside-in” development approach where they begin first by writing an integration test to define the feature to be implemented, then use a unit testing approach to tease out the implementation. For them, design for testability should be the highest design goal for software.
  3. A Worked Example, where they describe the development of a complete system following their development approach. They start from the very beginning, getting a “walking skeleton” of the application up and running right through to adding functionality feature-by-feature. There are lots of code examples and you really should pay attention to get the most of this section. As the system is developed they describe the decisions and trade-offs they make.
  4. Sustainable Test-Driven Development, techniques to make TDD projects work in the long-term. Topics like test flexibility, test diagnostics and test readability are covered.
  5. Advanced Topics, focuses on the tough topics in automated testing: persistence, multi-threaded, and asynchronous code.

This is a book which investigates what Michael Feathers calls the “deep synergy between testability and good design” in-depth. It will teach you how to create cleaner, more expressive, more sustainable code. It might even change they way you think about software development. This book is definitely is my software book of the year.

What was your favorite book of 2010?

Leave a comment

Filed under book-review, goos, software, testing

Behaviour Focused Tests with Given/When/Then

When writing unit tests it is important to keep in mind that the purpose of any unit test is to validate behaviour of the system under test.  A good unit test is focused on one specific aspect of the behaviour of the system.  That way, when a test fails it is clear what particular functionality is out of specification.

A unit test can give you some clues when your tests are unfocused.  For instance, having more than one or two asserts in the test is a clue that you are testing more than aspect at a time.  Just like production software should only have one reason to change, unit test code should have only one reason to fail.

Another hint that there is room for improvement is where a test has complicated setup code or code duplication.  If there is a lot of work involved creating a suitable test environment for the unit under test it is usually a sign that a class or module has too many responsibilities.

One helpful way to keep your tests focused on behaviour is to follow a given/when/then template.  This technique started in the Behaviour Driven Development (BDD) approach proposed by Dan North and Joe Waines.  As with most testing styles this began life in the Java and Ruby communities, sprouting a plethora of frameworks, but it can equally be applied to testing C and C++ code.

Here is an example of a unit test that is testing an embedded C module.

TEST_F(PublishMfgInfoTestSuite, when_allocate_cmdbuf_fails_no_eeprom_read_performed)
{
    // given
    allocateShouldSucceed = false;
    // when
    MFGI_sendMfgInfoToBox((DispArg_t)pCmdBuffer);
    // then
    ASSERT_EQ((uint32_t)0, VEEP_getSectionCrcAndLocation_callCount);
    ASSERT_EQ((uint32_t)0, VEEP_readPart_callCount);
}

There are a number of interesting points to note here.  Firstly, the name of the test clearly describes the functionality that is expected.  If you were using a Ruby DSL to describe the tests it might read like “given an allocation failure when sending MFG info to BOX then no EEPROM reading will be performed”.  If this test fails you can tell immediately how the software under test failed to meet requirements.

Secondly, the body of the test contains some comments.  These are entirely optional, but I find it useful to keep them around after I have finished writing the test as a reminder for the future me.  The first, given, is used describe the initial environment for executing the test.  The when clause is where the execution of the functionality under test is performed.  Finally, the then state is where the expectations or end state of the test are asserted.

By following the given/when/then template and being conscientious in our test naming we have arrived at a test absolutely focused on one particular behaviour of the system.

An important part of any technique is understanding when to not use it.  Some people advocate using a given/when/then structure to your test names.  In my opinion (and many others) this leads to verbose naming and unnecessary noise.  As with any recipe, context is king and you should use your own judgement to decide what will give the most readable and focused tests.

Further Reading:

http://blog.dannorth.net/introducing-bdd/

http://en.wikipedia.org/wiki/Behavior_Driven_Development

http://blog.objectmentor.com/articles/2009/12/19/the-polyglot-tester

Leave a comment

Filed under software, testing

Internal vs. External Software Quality

A topic of interest to me is the different ways it is possible to view software quality, so I was delighted to find a new way to categorize software quality while reading Growing Object-Oriented Software, Guided by Tests[1], by Steve Freeman and Nat Pryce. In it, they propose viewing a software codebase in terms of its internal and external quality. Fascinated, I decided to do a bit of research into the approach to see what I could learn.

Origins

The earliest example I have found of the internal/external categorization was from Steve McConnell’s Code Complete [2], where he describes the characteristics of internal and external quality:

External Quality Characteristics: Correctness, Usability, Efficiency, Reliability, Integrity, Adaptability, Accuracy, and Robustness.

Internal Quality Characteristics: Maintainability, Flexibility, Portability, Re-usability, Readability, Testability, and Understandability.

These characteristics are reasonable enough, but as Steve goes on to say the “difference between internal and external characteristics isn’t completely clear-cut because at some level internal characteristics affect external ones”.

Onwards

For me though, the breakdown of software quality into Internal and External Quality is pretty simple:

Internal Quality determines your ability to move forward on a project

External Quality determines the fulfillment of stakeholder requirements

Software with a high internal quality is easy to change, easy to add new features, and easy to test. Software with a low internal quality is hard to understand, difficult to change, and troublesome to extend. Measures like McCabe’s Cyclomatic Complexity, Cohesion, Coupling and Function Points can all be used to understand internal quality.

External software quality is a measure of how the system as a whole meets the requirements of stakeholders. Does the system provide the functionality required? Is the interface clear and consistent? Does the software provide the expected business value?

Benefits

Why have the distinction? It is clear that a software project needs both External and Internal quality in order to succeed, but what value does the categorization provide?

  • The business case for unit/integration/system tests is much clearer
  • The motivation for a particular type of test is easier to understand
  • Understanding the motivation for a particular test avoids mixing test abstraction levels

Feedback from tests

In their book, Growing Object-Oriented Software, Guided by Tests, Steve Freeman and Nat Pryce have a wonderful illustration relating the level of internal/external quality feedback certain test-types can give you.

GOOSGBT Chart

What is interesting about this definition is that using different testing approaches you can garner quality feedback of different types. It shows that end-to-end system tests provide to most amount of feedback on the external quality of the system and unit tests provide the most amount of feedback on internal quality.

It also underlines the importance of multiple testing approaches. If you want to make a system that both meets the stakeholder requirements and is easy to understand and change (who doesn’t?) then it makes sense to develop with both unit and system level tests.

How to Automate Internal and External Quality Feedback

There are many well established methods for monitoring and ensuring internal software quality. In fact, this has been a major part of the agile software movement. Practices like unit testing, TDD, code reviews, pair programming, static code analysis, code coverage and the like all can form a good process to ensure internal quality.

The picture isn’t quite as clear-cut for External Quality. The most popular techniques are continuous integration, end-to-end systems tests, and iteration demos with stakeholder feedback. In addition, there are two relatively new techniques that can be employed: Behavior Driven Development and Mock Objects.

A full explanation of BDD is outside the scope of this article, but the gist of it is in creating executable specifications, usually in a domain specific language, driven by business value. These specifications are to be created in concert with stakeholders to support automation of executable requirements.

A New Approach to Test Coverage

What this shows is that code coverage alone is not a good enough metric for ensuring internal and external quality. A part of the system might have 100% coverage from integration tests but be excruciatingly difficult to change. Alternatively a part of the system might have great unit tests but have a terrible interface, or worse, provide no business value at all.

I have found that the division of quality into external and internal has been a useful tool to help appreciate the value of different testing types and strengths and weaknesses of approaches in different contexts. This is why I am now leaning towards measuring code coverage separately in two execution environments: firstly from the integration test suite and secondly from the unit test suite. This will provide a much richer picture of the internal and external quality of your system.

—————————————————–

[1] Growing Object-Oriented Software, Guided by Tests Steve Freeman and Nat Pryce. ISBN: 0321503627.

[2] According to Wikipedia the definition comes from p558 in the first edition, but I found it in the second edition on page 463.

Further Reading:

Comparing internal and external software quality measurements:

http://portal.acm.org/citation.cfm?id=1566295

Page on the Portland Pattern Repository Wiki:

http://c2.com/cgi/wiki?InternalAndExternalQuality

4 Comments

Filed under goos, quality, software, testing