Triptych: Red, green, refactor!
Inspired by James Grenning.
Retrospective – Global Day of Coderetreat, Beijing!
On the 3rd of December a small collection of developers gathered in a basement conference room in Beijing to practice the craft of software development. The participants came from diverse backgrounds: architects, developers, students, and managers. There were people comfortable with C++, C#, Java, Python and others hadn’t programmed in a couple of years. Here are my notes from the day:
Introduction
Coderetreat is a day-long, intensive practice event, focusing on the fundamentals of software development and design. By providing developers the opportunity to take part in focused practice, away from the pressures of ‘getting things done’, the coderetreat format has proven itself to be a highly effective means of skill improvement. Practicing the basic principles of modular and object-oriented design, developers can improve their ability to write code that minimizes the cost of change over time.
A coderetreat is a language-agnostic event. In each session, the pair chooses what language they want to work in. The day focuses on practicing the fundamentals of software: TDD and the 4 rules of simple design; these are applicable regardless of language.
Alex and Zilong pairing
Chocs Away!
After a brief introduction, we went into the first session to familiarize ourselves with the task: Conway’s Game of Life. Participants struggled with deleting the code at the end of the session; this was one of the biggest challenges that they faced throughout the day. The second session was to swap pairs and have a second go at the problem with a clean slate. The third session introduced the concept of ping-pong TDD.
For lunch we went out to a local restaurant to get a chance to stretch our legs and have a fresh perspective.
Lunchtime!
In the afternoon we tried a couple of CyberDojos. The first session we didn’t change pairs so we could have a chance to get familiar with the CyberDojo software. After a few technical issues we were on our way. In the second session we changed the pairs every five minutes, really challenging ourselves to write code in very small increments.
CyberDojo
The final session I gave the participants a choice: we could agree as a group to either try to create the absolute best code we could for the solution, or we could try to create the absolute worst code possible that implements the solution. It was a tough call but the dirty code challenge won out. In this session there were many creative approaches and a notable visual implementation that looked great in the UI and was a hornets nest in the implementation – you know who you are! ☺
At the end of the day we held the usual Closing Circle, where we each share with the group our feelings on what went well, what was surprising, and what we can take away from the event. There was a general consensus that the day was fun and that it highlighted the importance of communication, both between people and through the code. Also surprising was how many different approaches there were to the same problem.
Globalization
Hello from Japan!
The event is called a “Global Day” for a reason; the same event was happening in over 90 cities across the world on the same day. There were many ways the events connected, lots of activity on twitter with the hashtag #gdcr11, and several events talked to each other via google hangouts or skype. We had a quick chat with the coderetreat in Tokyo in the morning, and before lunch we managed a chat with the folks in Perth, Australia. This was a lot of fun and helped to energise the group.
Hello from Perth!
Thanks to Corey Haines and Jim Hurne for organizing the global day, and also thanks everyone that helped me organize in Beijing. Thanks to Tokyo and thanks to Perth. And special thanks to everyone in Beijing who came along on a blue sky day to spend their Saturday in a basement coding with other software craftspeople.
p.s. I am starting a local software craftsmanship meetup group here in Beijing, sign up to hear about future events and meetings!
Related Posts:
Interview with Corey Haines
CyberDojo
Tags: software testing craftsmanship coderetreat coreyhaines tdd
Filed under coderetreat, software, testing
Less Architecture, More Microtecture
I’m going to jump straight onto my soapbox: the world needs fewer architects. Am I suggesting that engineers of the world unite and overthrow these unjust overlords and reclaim software by the people, for the people? Well, yes, actually. Let’s round up all the architects and send them to a dark place with no whiteboards.
Is there room for reform in my plan? Maybe. I’m not hopeful though. It is a long and tortuous penance that must be exacted, and it will take all the concentration and commitment that an architect can muster. It may also include humiliation and a few slices of humble pie.
You might ask yourself, why has this bozo got a chip on his shoulder? Well let me tell you. The world is full of huge systems built on the watch of a network of highly talented architects performing due diligence and risk control. Yet, 5 years later all that remains is a few MLOCs of hell that not even consultants can be tempted to work on.
How does this happen? Was there a lack of boxes and arrows written down in some document? Was poor requirements management to blame? Hardly. Codebases become unmanageable not because of the high-level design decisions, but because of the low-level details. No UML drawing ever caused a maintainence clusterfuck (apologies to my mother for dropping the f-bomb). What destroys software is engineers. We do this. It’s us.
We write 100-line functions. We inherit for code reuse. We write 10-line comments instead of taking 15 minutes to discuss naming a variable. We don’t write tests. We rush to deadlines. We put third-party framework calls in the middle of our production code. We think multi-hour-long build times are acceptable. We copy/paste follow patterns.
No more. I’m out. Done. Finished. I no longer wish to be associated with this crap. But I have a cunning plan. Just like the architects, I want to solve this problem with job title. From now on, you can call me a microtect. You are welcome to join me on this crusade, of course. Viva la revolution!
Of course no revolution is complete without a manifesto, so without apology I offer my Microtecture Manifesto:
- Sweat the small stuff
- Small, loosely coupled over large, monolithic
- Prefer composition over inheritance
- Code is king
- Optimise for change
- Tight feedback loops
- Value people, not technology stacks
- Value reducing lines of code over adding more code
- Name stuff good [sic]
Remember kids, take care of the pennies and the pounds take care of themselves.
Filed under Uncategorized
Global Day of Coderetreat, Beijing!
Beijing is participating in the Global Day of Coderetreat 2011!
Coderetreat is a day-long, intensive practice event focused on the fundamentals of software development and design.The unique coderetreat format (which eliminates the pressure of ‘getting things done’ and focuses on practicing basic principles of good design) has proven to be a highly effective (and fun) means of skill improvement. Check out to Corey Haines’s coderetreat site if you want further details on what to expect.
On December 3rd, Coderetreats will be held at cities all over the world. To celebrate this significant event, we’ll be participating in some global activities in addition to our normal Coderetreat activities. Afterwards, stick around and we’ll all go out to dinner or for a few drinks. It’s going to be a lot of fun!
As with other Global Day of Coderetreat sites, registration will open on Thursday, November 3rd, 2011. You can register at:
http://beijingcoderetreat.eventbrite.com/
You only need to bring a laptop with the development tools you require to write code using your chosen programming languages. A continental breakfast and lunch will be provided.
Attached is a flyer for the event. Feel free to print it out, post it in approved locations at work, share it via email, etc.
This event is hosted and sponsored by Schlumberger Beijing Geoscience Center. See the Google Map for directions and Parking suggestions. If you have any problems registering, or have any other questions, feel free to contact Mike (mikelong2005@gmail.com).
Filed under Uncategorized
Ye Gong Hao Long (Lord Ye Loves Dragons)
An executive software manager recently told me a tale of Agile adoption within an organization. There was a management team that has discussed, promoted, and advocated agile techniques to the engineering population for years. When the tide finally turned in favour of agile in the company the same management team then became the focal point of resistance. The reality of the consequences self-organization and “Individuals and interactions over processes and tools” were too alarming to be accepted.
He then went on to tell me the tale of Ye Gon Hao Long:
The reality of Agile is that it is vastly easier to talk about than to implement, and to fully commit means to change the organization. So beware the manager who loves dragons…
Filed under Uncategorized
Code Retreat: An Interview with Corey Haines
In June this year, we were lucky enough to have Corey Haines come by the Westerngeco offices in Oslo to host a code retreat with the seismic embedded software engineers. We had a great time and learned a lot. Corey kindly agreed to give a short interview for our software newsletter, and here it is:
Q. Could you tell us a little about who you are and what you do?
I’m an independent software developer trainer, specializing in working
with teams to improve their fundamental software design skills. In
2008, I lost my job and decided to go on a pair-programming tour,
traveling around writing code with people in exchange for room and
board. Over the past few years, I’ve been working on developing and
augmenting a training format called code retreat. When I’m not
traveling, I live in Chicago with my girlfriend and a cat named Zak. I
spend my time at home working on my application, MercuryApp, as well
as advising non-technical founders of startups.
Q. What is a code retreat? Where did the idea of code retreats come
from? What can you expect when you go to your first code retreat?
A code retreat is a day-long, practice-oriented training event. It
stems from the idea that, as software developers, we spend most of our
performing on live code. Since we are always trying to ‘get things
done’, we necessarily have to cut some corners in our code. We often
neglect some of the basics of our craft, such as good abstractions.
Using a focused practice approach, we can make ourselves a bit more
effective at designing and developing software. So, the format is
designed to give the opportunity to really work on writing better
code.
During a person’s first code retreat, the morning is usually spent
getting intimately acquainted with the problem. Then, in the
afternoon, we really push hard on practicing some variants of design,
focusing on aspects of the 4 Rules of Simple Design. Depending on the
group, we might then move into pairing games, design constraints (no
if statements, no loops, ultra-small methods) or a
TDD-as-if-you-meant-it exercise. And, lots and lots of intense coding.
Q. You have been facilitating code retreats all over the world. What
are some of the surprising things you’ve experienced during these
sessions?
A major thing that has surprised me over the past couple years is just
how much progress a person can make in their understanding when
working under the code retreat constraints. Having the opportunity to
delete your code every 45 minutes allows you to really try new things
out and not have to live with the mistakes of the past. In our
day-to-day coding, we often write production code before we really
have a great grasp on the problem. By giving ourselves the freedom to
explore, the solution we come up with at the end of the day is so much
better than at the beginning.
Another surprising thing has been the number of people who get hooked
on working on Conway’s Game of Life as a regular practice problem. I
often hear from people later that they can’t stop working on the
problem, approaching it from different directions, trying out new
abstractions, etc.
I’ve also had more than one company tell me that they’ve begun
integrating the idea of ‘do it once and throw it away’ in their
regular routine. This has a surprising effect on the codebase, as it
decreases the amount of technical debt in a codebase, as spiking the
solution first gives a better understanding of the domain.
Q. First there was OO, then there came patterns, followed closely by
agile. It seems that the movement du jour is software craftsmanship.
What does software craftsmanship mean to you? Is it important? Is
there anything new in it?
Software Craftsmanship is a natural partner to all those things. It is
an outgrowth and evolution of our understanding of how to develop
software. Just like OO and the agile methodologies were culminations
of previous work, Software Craftsmanship brings together a lot of the
lessons that have been learned over the past decades around building
and delivering software products.
For me, Software Craftsmanship is about working to build a programming
industry that can be proud of its professionalism, delivering the
right product for the right situation at the right time. This involves
the whole range of activities, from effectively mentoring new
developers to working closely with our client businesses to understand
the values that a modern software development process can provide.
Q. What excites you about software development now? What technologies
do you have on your radar?
I’m quite excited about the push to get kids interested in
programming. I’ve been spending a lot of time in Scratch
(scratch.mit.edu) lately, as I’m helping teach a class for teenagers.
There is a dearth of developers, especially experienced ones, so it
would be good to start working with the next generation.
I think it is cool to see some of the more fundamental languages
coming back into vogue, such as lisp and smalltalk, emphasizing
functional and pure object-oriented paradigms. The emphasis on being
adept in different language paradigms is an encouraging trend.
Q. How do you see the future of software development?
First, I’d like to see a continued renewed energy and joy in the
younger generation. It is exciting to see the push in the past couple
years to focus on showing kids just how exciting our craft can be. We
wake up in the morning, start with nothing, and have created something
new by the end of the day. It is an amazingly heady feeling to be a
maker.
Second, I’d like to see the industry focus back on fundamentals of how
to deliver quality software for the businesses we are part of. Rather
than always looking at the latest shiny toy, let’s also take some time
to really work on improving our core skills as software developers:
abstractions, modular design, naming, etc. It is important to keep
innovating, but the past has seen a push to focus too much on the next
great thing. For me, code retreat is one of the core activities in the
quest to improve our fundamental skills.
—-
Corey is organizing a global day of Code Retreat, find out more at the Code Retreat Blog.
Filed under Uncategorized
Hardware is a global variable.
There, I’ve said it. No going back now. But when you think about memory mapped IO, a device register is just a globally available address that any Joe can read or write, right? It is only convention that keeps us from accessing the hardware willy-nilly.
Why are global variables considered harmful?
Reasoning about the behaviour of software with global variables is hard. The potential scope for state change is unbounded. The reason is poor locality. Software that is hard to reason about is also hard to test.
But a memory mapped IO register differs from a standard global variable in these ways:
- It is necessary (there are no formal methods to localize the scope of hardware in software)
- By definition it is volatile. Even if the software doesn’t write to the address the value that is read can change unpredictably.
- It is not a memory location. What you write might not be what you read.
Enter the Device Driver
So given that memory-mapped IO is just a fancy type of global variable, how do minimise the impact of this necessary evil? The standard approach is the device driver.
A device driver is just a software module that is used to create an abstraction of the hardware to the rest of the software system. Some drivers can be simple translations of function calls into hardware access, but there is usually a lot more going on behind the scenes in a typical device driver. Devices such as flash memory might require complex communication protocols and timing considerations, and interrupt handling routines might have to delegate to program mode functions with queue worker threads. With this much complexity in the software, it is a good idea to make an automated test suite for drivers.
The challenge then becomes: how do I test software that is so intimately tied to memory mapped IO?
How to unit test device drivers
There are many approaches to testing driver software, but I want to share my favorite. I like it because it has zero runtime overhead in production, but gives full flexibility during testing.
In production code use a compile-time macro to access hardware registers. In test builds replace this with a fake function. This is quite simple to do with a header file:
#ifndef HARDWARE_ABSTRACTION #define HARDWARE_ABSTRACTION #include <stdint.h> #ifndef TESTING #define IO_MEM_RD8(ADDR) (*((volatile uint8_t *)(ADDR))) #define IO_MEM_WR8(ADDR, VAL_8) (*((volatile uint8_t *)(ADDR)) = (VAL_8)) #else /* In testing use fake functions to record calls to IO memory */ uint8_t IO_MEM_RD8(uint32_t reg); void IO_MEM_WR8(uint32_t reg, uint8_t val); #endif #endif /* Include guard */
Then, it’s simply a matter of implementing the driver in a standard way:
#include "HardwareAbstraction.h"
#define DRIVER_OUTPUT_REGISTER 0xFFAA
#define DRIVER_INPUT_REGISTER 0XFFAB
void write_to_driver(uint8_t val)
{
IO_MEM_WR8(DRIVER_OUTPUT_REGISTER, val);
}
uint8_t read_from_driver()
{
return IO_MEM_RD8(DRIVER_INPUT_REGISTER);
}
Now I know this is a contrived example, but it illustrates the point. In a production compilation the preprocessor substitutes this with direct memory access. However, in the tests there is a link seam which can be overriden as you please.
Capture and Assert
So what would a test of this code look like? Well, there has to be fake implementations for the IO_MEM_RD8 and IO_MEM_WR8 functions. They should capture the arguments passed to them so we can verify the driver has the correct behaviour. So an initial stab at the problem might look like this:
extern "C"
{
#include "driver.h"
#include "registers.h"
}
#include <igloo/igloo.h>
#include <map>
#include <iostream>
using namespace igloo;
extern "C"
{
static uint8_t readVal;
static int readCalled;
static uint32_t readRegister;
uint8_t IO_MEM_RD8(uint32_t reg)
{
readRegister = reg;
readCalled++;
return readVal;
}
static uint32_t writeRegister;
static uint8_t writeVal;
static int writeCalled;
void IO_MEM_WR8(uint32_t reg, uint8_t val)
{
writeRegister = reg;
writeVal = val;
writeCalled++;
}
}
Context(AHardwareDriver)
{
Spec(WritesDataToCorrectRegister)
{
driver_write(0x34);
Assert::That(writeCalled, Equals(1));
Assert::That(writeVal, Equals(0x34));
Assert::That(writeRegister, Equals(DRIVER_OUTPUT_REGISTER));
}
Spec(ReadsDataFromCorrectRegister)
{
readVal = 0x55;
uint8_t returnedValue = driver_read();
Assert::That(readCalled, Equals(1));
Assert::That(returnedValue, Equals(readVal));
Assert::That(readRegister, Equals(DRIVER_INPUT_REGISTER));
}
};
These tests are written using the Igloo C++ testing framework, but of course any testing framework will do. These tests are clear and verify that the driver performs the reads and writes that are expected of it. But there are a couple of things missing here. The capture variables and return variables maintain state across tests so there needs to be additional reset code run before every test. If a function of the driver performs several reads and writes it is impossible to capture them without making very complex implementation of the fake function. And with the current implementation it is not possible to track the order of function calls, so we can’t tell if (for example) a read came before a write. To make this work for more interesting scenarios we need to write a lot more code.
Fake Functions
Thankfully there is an alternative to handwriting these complex fake functions by using the Fake Function Framework. All the faking code in the previous example:
extern "C"
{
static uint8_t readVal;
static int readCalled;
static uint32_t readRegister;
uint8_t IO_MEM_RD8(uint32_t reg)
{
readRegister = reg;
readCalled++;
return readVal;
}
static uint32_t writeRegister;
static uint8_t writeVal;
static int writeCalled;
void IO_MEM_WR8(uint32_t reg, uint8_t val)
{
writeRegister = reg;
writeVal = val;
writeCalled++;
}
}
Can be replaced with this:
FAKE_VOID_FUNC(IO_MEM_WR8, uint32_t, uint8_t); FAKE_VALUE_FUNC(uint8_t, IO_MEM_RD8, uint32_t);
Of course that is much less typing. But there are additional benefits with using the framework. All the tricky testing scenarios involving multiple call sequences, return values, and argument histories are taken care of by the framework.
A less contrived example
Imagine that there a particular revision of the hardware requires a peripheral to be enabled before it can be initialized (this is quite common in low power hardware), how can I write a test for this?
I need to be able to simulate the hardware revision, verify that the enable register is written to before the initialize register, and verify that the correct values have been written It turns out that by using the fake function framework this is a quite simple operation. The frameworks captures all this information for us, so all we need to do is assert our expectations in the tests:
Context(DuringSetupOfRevisionBHardware)
{
Spec(EnablesPeripheralBeforeInitializingIt)
{
IO_MEM_RD8_return_val = HARDWARE_REV_B;
driver_init_device();
// Gets the hardware revision
Assert::That(call_history[0], Equals((void*) IO_MEM_RD8));
Assert::That(IO_MEM_RD8_arg0_history[0], Equals(HARDWARE_VERSION_REGISTER));
// Enables Peripheral
Assert::That(call_history[1], Equals((void*) IO_MEM_WR8));
Assert::That(IO_MEM_WR8_arg0_history[0], Equals(DRIVER_PERIPHERAL_ENABLE_REG));
Assert::That(IO_MEM_WR8_arg1_history[0], Equals(1));
// Initializes Peripheral
Assert::That(call_history[2], Equals((void*) IO_MEM_WR8));
Assert::That(IO_MEM_WR8_arg0_history[1], Equals(DRIVER_PERIPHERAL_INITIALIZE_REG));
Assert::That(IO_MEM_WR8_arg1_history[1], Equals(1));
}
};
How does this work? The Fake Function Framework is contained in a single header file which needs to be included into the test code. When a fake is defined, all the boilerplate code is created by the framework. That’s all there is to it.
Summary
- Hardware is a global variable.
- Device drivers are a means of localizing their influence.
- Device drivers can be complex, and therefore should be tested.
- Device drivers produce side effects not seen in software.
- We can test these side effects with zero runtime overhead using the right techniques.
- The Fake Function Framework can significantly reduce the amount of boilerplate code required to test C code.
The code examples for this article can be found on GitHub:
https://github.com/meekrosoft/driver_testing
The Fake Function Framework and documentations can be found here:
https://github.com/meekrosoft/fff
Filed under Uncategorized
Improve your codebase with a Code Dugnad
In an ideal world the codebase you work on is a pristine oasis of pure logic, a magnificent expression of the problem domain, a testament to power of rationality. Unfortunately we don’t always live in an ideal world and even the best codebases have some dust in the corners. In our projects we have used a technique borrowed from the Norwegian dugnad to stop the rot before it becomes an issue.
What is a dugnad?
School children cleaning the roadside
From wikipedia: Noun Singular: dugnad Plural: dugnads
dugnad (plural dugnads)
- Unpaid voluntary, orchestrated community work.
A dugnad is a norwegian tradition of working together to clean something up, usually a communal area. The idea is to get everyone together with a common goal for the common good.
How to run a code dugnad
It’s not hard to have a code dugnad, just follow these simple steps:
- Have everyone on the team in the same room.
- Try to focus on one area to cleanup.
- Have a collection of small cleanup tasks prepared to get the ball rolling, preferably on a whiteboard so that the list can be updated and items ticked off.
- Focus on communication and teamwork.
- End with pizza. :-)
It might be useful to experiment with different durations, although from experience I’m pretty sure the optimum time is between 2 hours – 1 day.
Parting Shots
Keep in mind that a dugnad is not a license to forgo essential development practices like refactoring and it is still important to follow the boy scout rule. The dugnad creates a forum to focus on areas of a large codebase that have been neglected or unchanged for some time, including documentation.
Unit Testing Legacy C
Here are my slides from the lightning talk I gave at the ACCU 2011 Conference.
Related links:
Filed under Uncategorized
