Five years ago, I was sitting in a conference front row feeling puzzled. I was listening to Woody Zuill on Ensemble (Mob) Programming and the idea he presented felt too extreme to be good:

Whole team working on the same thing using one computer.

I had to try it and it transformed me from a non-programming tester to polyglot programmer I had always been since age of 12. To help other people get started, I started Ensemble Programming Guidebook. But the most significant transformation came from what I started calling Ensemble Testing.

Ensemble Testing

Ensemble Testing is to testing activities what Ensemble Programming is to all activities. While I can easily recommend Ensemble Programming 40 hours a week, if you would do Ensemble Testing 40 hours a week, you would soon find yourself grown into Ensemble Programming.

Ensemble Testing is ensembling on testing activities:

• Cleaning up test automation code
• Creating test automation code on any or many layers
• Exploring an application while writing code
• Exploring an application without writing code

Before when testers got together to test and learn about testing, we would work in same space, everyone with our own computers or a minimum of a computer per pair. Ensemble Testing gave us a vocabulary to say that one computer or set of computers with control together as a group would be an option. And that while we might do less, we might learn more. Usually in ensemble testing we don’t uncover so many issues, but we learn a lot. And we learn to work better together over time!

A Facilitating Teacher’s Golden Egg

As I discovered Ensemble Testing, I started using it for all my hands-on exploratory testing courses. As a teacher trying to enable movement in habits and skills for a group of 10–16 students, ensembling took my ability to teach to next level. I would always see what the students could do (not just what they said they could do), I could have the students teach each other providing great leveling of next steps to learn, and I could take control myself to teach hands-on how to apply a particular idea.

I would find myself saying “Let me navigate for a while” and showing how I would test it. I would ask questions on the intent, asking people to give words to what they were trying to do. I would have them write it down on a whiteboard so that they could anchor their learning. And I would watch people move to the keyboard, and know how to do a thing they did not know to the same level on a previous round.

In particular, I learned that when there are things you do not know you do not know (unknown unknowns), ensembling is the way to reveal them.

I moved my courses from exploratory testing without automation to very specific courses on automation, and regardless of what would come up, I could rely on the group’s collective power intertwined with my knowledge on solving the problems. I have no fear.

Skills and Habits

Skills are about actionable knowledge: being able to do something, in a smart way — the right tool for the right job. Habits are about consistency: being intentional not accidental about our results.

I learned people are often at their best behavior in groups. We want to please our peers, be it kindness or innovation.

Best results from Ensemble Testing come over time. Not just one training, just one session but doing your testing work in a group setting every now and then. Mixing up unexpected people. Building bridges. Transforming practices and organizations.

If you have not yet tried, please do. If you have, tell me what your experience has been like with #EnsembleTesting.

Developer-Centric Ways of Working

As much as people like to draw ownership of products into a product management organization, I see the true ownership lies within the developers. If they don’t, successfully, create a pull request that implements a change, the users will not see change. The core of the process is turning ideas into code.

No matter what else we agree that goes on, it all flows through here. Smart ideas, into smart code by people able to do that transformation.

There’s three clear points of failure we often create processes around:

• What if the ideas are not smart? What if the ideas are not worth spending time on?
• What if the people doing the transformation don’t have ideas of high quality, in context and are missing relevant perspectives?
• What if people turning ideas into code could do things faster having help?

Clearly, smart people can learn to have smarter ideas. Looking at what they do and what is the impact of it, they can change. Externalizing the looking often has adverse effects to motivation and ability to hit the mark on delivering, eventually, something of value.

Heart for the Customer

A smart developer would not be very smart if they did not care why they are building the software. There’s someone, somewhere willing to trade money for having something they’re building. We can recognize that the relationship is complex and requiring more attention, especially in scale than a developer can put there, but behind all software is a need it is supposed to fullfil.

Some people talk about being customer obsessed, but obsession sounds like a negative thing. I believe we need to have our hearts set out to hear, to understand and to care. From this relationship of us caring, we also find inherent motivation. Addressing a real need and a right need is motivating. Filters and proxies dampen the motivation, while listeners help manage the relationship.

Being Smart Takes a Village

Looking at the way of working as developer centric, we also soon come to realize that a developer, as smart as they may be, don’t need and don’t want to be alone. Making changes through pull requests that end up delivered to millions of computers out there is a heavy burden being left alone. So we have a principle of always having at least two pairs of eyes on every change.

A lot of times the process focuses on how the others are intended to contribute in the process of turning ideas into code — improving the ideas about to be turned into code, or already turned to code needing adjustment. The product owners, the designers, the testers, all work particular perspectives of improving the ideas.

A Transformative Way of Thinking

Looking at software product creation this way, every developer welcomes help in understanding what is the right thing to build and how we together could learn about it more effectively.

Lets face it: only through making a change through means of coding and delivering it all the way to the hands of the users things change. We can analyze, plan and prepare all we want, but unless that helps a smart developer have smarter ideas, we are probably not improving the impact we make with software development.

Let’s address the weak points: ideas become smart by working together and learning; people are smarter in diverse groups; the work we do can be shared in many ways.

How am I supposed to describe this in process model?

Living Life on Defaults

Teaching programming to a group of children, we look at the little program we’ve created following a recipe. The little turtle we have learned to refer to as tortoise and give commands to draws a square, just as we’ve told it to. We’re ready to look for something more interesting, to unleash the power of programming, allowing the computer to do things we would not do. The children have no concept of this, yet.

We identify attributes with the square. We find things that could be different, things we later will call variables: number of sides, length of a side, color of the line, and eventually, we look at the line to characterize its width. We realize it is not very thick, but we also realize that there is nothing in our code telling how thick the line should be. Jumping to the conclusion of a hidden default value usually takes only a moment.

We learn that the first step to changing the default is to reveal the default.

A lot of our life, just as a lot of software building we do, makes us move on defaults. We let life happen on defaults, when we could take control and optimize for what we find important. This is an insight that I recently found through proxy, Scott Hanselman talking about wisdom his wife shares on one of the Hanselminutes podcast episodes.

Even in life, and in our work, we move with defaults. And the first step to changing the default is to reveal the default.

What does this have to do with Exploratory Testing?

The Bridge Between What Comes Easy and The Real World

At work, we were building a new major functionality to replace the old, and I had a history with the old. The old had a developer known to frustrate the likes of me, testers, for breaking things left and right with every change. They would commit their code and pass it on without testing it. They would then start testing it, and often by the time I had tried if it works (and it didn’t), all they had to say is that they know. The functionality was also complex, so I could have lived through this groundhog day more that I care to admit.

I was particularly frustrated. The other tester assigned to the work approached the work with automation, and had a habit of explaining in meetings how they had 100% of the functionality automated. Reading their code, I knew they had 3 scenarios automated, and it was far from sufficient. The bar for 100% was low. The bar for 100% was their defaults: what they were aware that the functionality needed to do.

Armed with their defaults, they looked at the functionality like a bystander. Whatever was coming their way, they would take it, and turn it into automation. But where does the work come from? The first three cases came from the programmer dealing with the bulk of the new functionality. There were two main options for recognizing more:

• Exploratory testing: Finding new perspectives beyond what was already in the table that could show things were not what they should be
• The Real World: Finding out what users would say

The programmer was heavily opting for the Real World, after all, they had already seen that tester assigned to work was 100% done and everything they shared as knowledge was automated.

The missing piece in the puzzle was someone, with the Explorer mindset that could find out more of the real world, before and while production.

Adding perspectives that were not addressed, charters to explore, was not complicated. Some of them would result in changes without testing anything, as the questions revealed problems. Others we would insist could not cause problems, and yet when spending time exploring with the functionality, would have unexpected side effects.

In hindsight, without exploratory testing in between the bystander perspective and the real world, the real world would not have received what we generally expect to deliver. The work here in between, modeling and actively learning about the real world, is what I think of as exploratory testing.

Exploratory Testing, Exploratory Development

We often come to realize all testing is exploratory to a degree. Similarly, we realize all programming is exploratory to a degree. Why then we have an approach, exploratory testing, when we don’t have the named practice on the programming side?

What we seem to be missing is that testing and debugging is the name we’ve given to the simplest exploratory loop in programming. We test to learn about the program. Exploratory testing says we test to learn about the program, and about the testing we should be doing. It says we move from the bystander role to an active contributor.

Both on building and breaking (illusions) perspectives, we want to move smart people from moving with defaults, being bystanders in their projects, to active people making choices. Exploratory testing is, and has been a way of moving testing away from defaults.

In a pair, we have a driver and a navigator. Think of these roles as they are in driving rally. The driver is hands on the wheel, and in control of paying attention to the immediate task at hand. They are busy looking at the road ahead, optimizing the drive and not checking the map. Checking the map would be a distraction. That is where the navigator jumps in. Keeping track of goals on a larger scale, paying attention to other level of details they can help significantly in getting to better results. Navigator takes care of directions and instructions, and feeding them to the driver as needed.

This same distribution of tasks is what is the core of strong-style pairing. Instead of making the navigator decipher what is going on in the drivers head, the control is placed with the navigator. For strong-style pairing, we say:

“I have an idea, you take the keyboard”.

For traditional pairing, we’d say: “I have an idea, give me the keyboard”.

Getting Set Up For Pairing

For the pair to work well, both parties in the pair need to work well in their roles, and there’s specific skills and techniques you can learn to work better from get-go with a new pair in either one of the roles. Techniques, however, are just ideas until you apply them in practice. Only through doing you can learn to pick up the small hints on what would be right thing to do, as you both are equally responsible for your mutual success.

We may think of the driver as an intelligent input device. Intelligence means that while the driver takes orders on what to do and where to go, the driver can also guide to get on the right abstraction level or improvise within the box the navigator gives them.

The navigator is responsible for mining the to-do list and passing the next things to do with instructions on the highest possible abstraction level the driver can consume.

This all builds on trust. The navigator may be just one step ahead of you, and thus unable to give you a clear overview of what you’re about to do. The direction the navigator is going to is where the driver goes. When the driver has an idea, the roles can be switched keeping a basic rule in mind: for an idea to get on the computer, it must go through someone else’s hands.

Who Should Drive and Who Should Navigate?

You have a pair of people. You are different people, with different ideas and experiences. The two of you are stronger together because you build on each other, and also because each of you have unique qualities. Sometimes the qualities are more of a foundational or long-lasting kind, like experience that does not build up over night or transfer in a few days of pairing. Sometimes the qualities are more of temporary, like one having a rough day just today.

One of you will take the role of the Driver — the intelligent input device. One of you will be the Navigator, working out the direction you’re about to head to together. Which of the two of you should be which?

The first rule on being the navigator would be that whoever has the idea to get on the computer, should navigate. Remember: for an idea to get on the computer, it must go through someone else’s hands in strong-style pairing. This brings quickly the idea that perhaps the more experienced one should navigate. At least first. At least until it’s time to switch. You learn very different things in being a driver and being a navigator.

The dynamic of newcomer and a seasoned professional is interesting. Having the newcomer just drive is already a relevant service for the navigator, freeing you from typing to think on a bit higher level. The information for the newcomer sticks in a completely different way from doing it, hands on the keyboard, than watching what someone else is doing.

Whoever had the idea of what to do will navigate. When I started pairing on exploratory testing with a developer, I was the Navigator. When I started pairing on unit tests with a developer, I was the Driver. Over time with both tasks, I will take both roles. And I will grow towards navigating so that the first idea that I have that I could navigate us through, I volunteer to do.

There’s three trigger options on change of roles:

• On time: you can just change roles on a regular interval. Start with shorter times and grow the time as you grow to work as a pair.
• On task: you can change roles when some task is completed. Like a typical ping-pong style pairing, you change so that one creates a unit test and other implements, and both get to do both types of tasks.
• On idea: you just change when the driver feels there’s a direction he’d like to navigate to. It’s like saying “hey, I have an idea, you take the keyboard now”.

Regardless of the role you’re in, the pair of you should remember to take breaks. Pairing can be very intensive work and outside getting the work done in a pair, you’re also responsible for building forward your pairing experience. And that usually works best by inviting for feedback, like taking a small walk around and retrospecting your most recent pairing experience together.

Driver: Things to Do

You are the intelligent input device. What is expected of you except for typing as you’re told?

• Push back. Express that you need to modify the way the navigator is navigating you. You may want to change the box in which you work, make it bigger to give you more freedom or make it smaller to be clearer on what to do. You may realise you need to try something else, and express that in questions. Key is to be active even when driving.
• Improvise. The navigator gives you a box within which you operate. You have choices on how you can do things. You choose what you believe to be best for context at hand, and driver gives you more detail if they disagree with your choices. Improvising is about adding your intelligence to the collaboration.
• Switch level of abstraction. If you feel the abstraction level of navigating could be higher or lower, talk back to the navigator. If the navigator is using too low level, you can give them the higher level. “You can just tell me to run it”, when navigator is telling you shortcut in keystrokes. If the navigator is using too high level, you can ask “Tell me what to type?” or “Where’s that located?”. Change the level of abstraction and enable a common experience of learning to work together better.
• Ask questions. If you feel something isn’t right, ask about it. You can simply go for “Are you sure?” to stop the navigator to think about what is going on. Keep your questions specific so that the answers can be short. You can try validating questions to clarify what is going on right now, “It seems we’re using zip-add-object to regulate temperatures, is that correct?”. You can check if the thing you did was correct against your understanding: “I thought the database only accepted one connection at a time. Why did we do two?”. You can suggest where to go, without deciding for the navigator: “Shouldn’t we do this first?”. Avoid general why-questions and work to prevent long explanations.
• Initiate role switch on an idea. When you realise what should be done and think you could navigate this task better, initiate a role switch. You can switch roles on idea, without waiting for the timer. Or you could never use a timer and switch on tasks.

It’s good to remember that you as the driver are helping the navigator to navigate. Sometimes navigators will try a general avoidance technique of declaring tasks mundane, and great driver will volunteer to be around even for that task. Sometimes you are just literally trying to go against the excuse. Sometimes the two of you just need to get through the bad stuff together to get to the fun stuff together, to build the long-term relationship.

Underneath what we actually do as driver, there’s a bunch of attitudes to consider:

• Trust your navigator. Stay in the moment. Be ready to work with partial knowledge. Your navigator is probably only one step ahead of you.
• Just try it. You can always do it both ways and end up with a third that builds on top of both. Learn by doing. Learning while doing is just as important as getting the end result out of the process.
• Constant self-reflection. Investigate what is going through you. Focus on learning. When following your navigator, you’re reverse-engineering. Navigator keeps telling you stuff. You get a very thin, narrow view into the system. You can start putting things together a lot and this gives you a chance to reflect on what you’re learning individually as driver with your pair.
• Patience. Give the relationship time. It’s good on both sides, but it’s extra useful on the driver side who works with incomplete knowledge.

There’s two main pitfalls to being a driver.

• Thinking. You think within the box or change the box. Thinking too far as the driver takes the navigator’s focus on bringing you back to the task at hand.
• Silence. It’s not an open box to do anything you think of. If you want to take the lead, switch roles, but try not to run your own way with the silence.

Navigator: Things to Do

As navigator, you are in responsible for caring for your driver. If the driver is the intelligent input device, for them to operate properly, you need to care for the conditions of work. Being the navigator, you need to pay attention to your driver, to constantly know where they are going. And you need to enable them to go as fast as possible.

As a navigator, you have three main tasks you’re paying attention to:

• Feed driver the next thing to do. You’re creating the driver the box they work in on as high abstraction level as your driver can handle.
• Mine the to-do-list. Create an idea on where to go next. You might be just one step ahead, but being that one step ahead is important. Where to go to get your thing done?
• Observe your driver. See where they are and where they are going, and correct if the direction does not match what you had in mind. Pay attention on how they are doing, and help them whenever they need it.

Three main tasks might sound simple, but there’s a bundle of advice to do them slightly better.

Programming style matters

The style of programming matters. Pair programming would seem to work a lot more fluently if the programming style is consume-first. With this the idea is that you start with an end result at first and then one by one build the things you wish you had in order to have the end result. Consume-first enables the navigator to go immediately, coming up with things to do while figuring things out themselves. And the end result and it’s division keep a visible checklist of what there is to do to get to the end result.

Some navigators prefer to work with bottom-up programming style. They build the image of the end result in their head, and feed the smallest possible pieces to the driver, one at a time. In this style, the navigator has a lot more of the information about what we’re trying to achieve, leaving the driver to interpret more of why the navigator is having them do things.

The third style is hardest, when there’s no ability to work in small pieces. If the navigator has to figure out the whole thing before feeding driver work, most of the time the driver is paused or participating in design discussions over taking the implementation forward. As a navigator, you’re supposed to care for your driver, not just having them type for you. Keeping driver waiting while navigator figures things out by themselves isn’t exactly the optimal way of caring. Working with partial information is essential.

The abstraction level dilemma

As the navigator feeding the driver the next thing to do, finding the right abstraction level to communicate with them is relevant, even key to making progress. For driver to go forward, you need to find the right level on which to talk. If you are using an abstraction level too high, you will see nothing happening at the keyboard. If you are using an abstraction level too low, you’re not harnessing the powers of your driver by keeping them on too short a leash.

The core of going as fast as possible is navigating on the highest level of abstraction. Talking in keystrokes when talking in concepts would suffice can feel insulting. So pay attention to raising the level of abstration.

• Intent. What is the thing we want to accomplish now?
• Location. Where does doing that start? Is the cursor in the right place or moving to the right place so that what we want done could be done?
• Details. What exactly to do or type so that the intent is fulfilled, allowing for the drivers contribution while avoiding mistakes?

You notice if the level of abstraction is too high with the puzzled look on their face and the lack of action you expect to see. Drill in if needed.

I was particularly puzzled with the idea of using highest abstraction level possible, until I realised finding the level is really a listening and observation exercise. If you give instructions and following them is hard, you’re probably working on a too high abstraction level. Drilling down can be instant, just be more specific. If you’re not noticing the need to change level, you’re perhaps risking an experience of failing with tasks that lowers motivation in pairing, so it’s good to keep your eyes and ears open. If you start from a very low abstraction level, the driver can always correct you by telling you the level they are comfortable with. But if they have not yet learned how to, they might get a feeling of being talked down. And I found I am particularly sensitive to that, being the only woman and paying attention to being treated differently.

Mining the to-do-list

There needs to be an idea of where we’re heading. But the road ahead might be only clear for the next few steps, instead of all the way. An important thing for a driver is keeping track of the work ahead. There’s three main things to consider on this:

• Timing: Find the right time to use an item on the list. What should be done first, what would make a coherent shared story in your pairing. What choices would enable you as the navigator to care for your driver in the best possible way?
• Backlog: what is there on the list of things to do? Your backlog is best if it can somehow be part of the code and become a shared view — with consume-first style. But you can also make notes by scribbling on a piece of paper of a whiteboard. Whatever you need to keep track of things. Notes are disposable, code (including test code) is what remains when you’re done.
• Prioritisation: Deciding what comes next and in what order to do things. It’s not just about the right time in long term, but the right thing right now. And it keeps changing as you learn.

Express a in-a-nutshell idea of what you’re doing

As a navigator in strong-style, you are not supposed to have to justify all your chosen actions to your driver. Sometimes the driver has little clue on where you’re about to be heading, and a great practice in these cases seems is to invite temporary trust saying you’d like to just go to this direction for like 10 minutes, and if they are still unsure about doing this, you can change then. The argument of what is right thing to do takes easily more time than that. And nothing stops the pair of you implementing both of your ideas and then deciding that a third, completely different option is what you should go for. With programming in particular, there’s a lot of uncertainty that unfolds only through implementation and experimentation.

If you need to express what you’re doing, you should learn to express that in a very concise format. Instead of all the rationale, pick only the part of the message that is absolutely relevant to care for your driver.

Immediate feedback

When you navigate, keep an eye on your driver. Double-check with questions what the driver does. And if the driver does something you consider a problem, help them correct it right away. Feedback belongs with the action the feedback relates to.

Closure Activities

Recognizing time to switch

Either one in the pair can suggest a switch of roles. But as a navigator, you are usually in a controlling position, so your position gives you a bit of extra responsibility to pay attention to when the driver would have ideas that she should navigate on. You want, in the long term, to build a pair where both can contribute. Switching pairs gives a chance to try out the other role, and only practice helps you get better.

Step back and think about what you’re doing

Sometimes you may feel you get bogged down into the details of implementation, and you might want to take a step back and think if you are spending your efforts in the right area. You may not notice without intentional step back to look at the bigger picture, because you’re going fast with support of your pair (and usually a nice bunch of unit tests too).

Retrospectives

Looking at your collaboration and how the two of you feel about it on regular intervals is also necessary. Not just looking at what you implemented and stepping back on that, but talking about what you learned and what you like and dislike about your collaboration, in a constructive tone.

Ending this with a word of warning

I’ve started to really enjoy looking into the dynamics of this style of pairing. But there’s a story I learned, that I also find very specific to this style.

Two programmers were pairing in strong-style and had the time of their life. There was a lot of talk, as all ideas must go verbally for two people to share. Sometimes the discussions were very enthusiastic and to an outsider they might have appeared even heated.

While the pair was enjoying, the environment that did not recognise the different pairing style was betting on which one of the two in the pair would quit first.

Don’t care just for your pair, care also for your environment. Verbalising your thoughts makes them available for more people than just the two of you. Take it as an opportunity to build relationships outside your immediate pair.

And it is not always that the pair is enjoying. Your pair not opting to work with you again is feedback that the way you’re connecting isn’t appropriate. A pair is two people, where both need to feel comfortable.

Individual Work

Let’s first think of testing as individual work. Core to the work is the a skilled tester, with ideas of what to do to uncover information. They use the keyboard to drive the computer, and as ideas emerge and evolve, write down notes in a tool of their choice. There’s just one person for all the activities around the task, and no need for changing roles other than within the tester’s head.

Traditional Pairing

In traditional pairing, we introduce a second person and split the work to do. We introduce roles and rotation, either on task or on timer. The work is split so that whoever is on the keyboard is in control of the testing session. The one on the keyboard has the ideas, and focuses on tactical implementation of those. The other one thinks more strategically, and makes notes. We often suggest that the one on the keyboard must think out loud to allow the pair into the insights of what goes on in the testers head. After all, very small portion of testing happens at the keyboard and can be visually followed.

The problem with this style of pairing is that the one without the keyboard is continuously decoding what they are observing from the other’s testing, building assumptions. It is hard to see what exactly is going on and this easily creates a bit of a disconnect in the pair. At worst, the disconnect shows as the watcher zoning out and not paying attention. Regularly, it shows as the notes not matching the testing that was done.

Strong-Style Pairing

In strong-style pairing, we split the work differently. The one who is not on the keyboard is now the one with the idea. They must vocalize the idea for the one on the keyboard. The one on the keyboard asks for clarifications and can challenge the direction, suggest ideas and ask questions. However, decision power is on the one without the keyboard. Notes are also taken on the same computer, representing ideas and initiated by the one not on the keyboard. As they are typed in and reviewed, they become a shared representation of the idea.

The benefit of this style of pairing is that the one without the keyboard must vocalize all their ideas. This style creates a stronger connection in the pair. Speaking about what you want to happen on the keyboard is a skill of its own, and it takes a bit of practice to get used to. The person on the keyboard sets the pace in which things happen within the pair.

Different Skillsets, Unequal at a Task

Strong-style pairing becomes particularly useful and necessary when pairing amongst different skillset so that your level of skills and knowledge is very unequal at a task. The difference could come from having product and domain expertise, while the other one would usually test another product. Or it could come from pairing a tester with a programmer. With traditional pairing, the one with more knowledge on the task at hand would be on the keyboard. The skills difference makes following what goes on harder in general.

With strong-style pairing the one with more knowledge is hands off keyboard, and the less knowledgable sets the pace of how fast they can absorb the pieces of the task.

Ensemble Testing — Pairing on Turned Up

Strong-style pairing is also the foundation of ensemble testing (and ensemble programming). In this style of working, you have group of people working on one task, taking turns on the computer. Ideas come from the crowd not on the computer. This is about getting the best out of everyone into the work we are doing. Everyone takes turns at the keyboard, and if you have someone unequal at task, the ensemble helps out navigating at an appropriate level of abstraction.

There’s a lot more on the dynamics of pairing and ensembling. Check out the Ensemble Programming Guidebook I’m writing, it is available on LeanPub.

For me adjusting to strong-style is what made pairing turn from frustration to learning fun and something I’d volunteer for. Similarly, ensembling was the gateway to pairing making me comfortable in a group before being alone with each of my team mates. I encourage you to experiment with what works with you.

Introducing Gilded Rose

Gilded Rose is a refactoring Kata (practice) created by Emily Bache. The idea with Gilded Rose is simple. There’s an inn somewhere that has an inventory system implemented. They would want it extended for new requirements but that won’t be all straightforward. Don’t break anything that used to work!

I’m a tester, so I don’t naturally come to the problem as it has been given, but I come with the idea that after someone changes it I may have to test it. How would I do that?

Priming With Information Sources and Tools

At work, things don’t come to you with the full range of sources and tools readily handed in. For doing the exercise, I no longer drop people in cold to “just figure it out” but I give them a few starting points.

• Requirements. Gilded Rose comes with a requirements specification. Would that be of use?
• Code. It “works in production” now and you can look at the implementation. If Java isn’t your cup of tea even if I use it while I teach this, Emily has been nice to provide it with tons of other languages.
• Unit test. The one unit test gives a starting point of how to execute the code so that you don’t need to figure that out.
• IDE with visual code coverage. Code and unit test in an environment where you can start working on it, with a code coverage tool. I use Eclipse with Code Coverage as I just like the visual nature of it showing green, yellow and red for the branches.
• Ideas of the domain. You have past experiences of what makes sense for a shop and inventory management system. You have ideas of what inputs are meaningful, and what could be confusing. Everything you’ve learned about what makes software worthwhile is with you.

To make the exercise slightly more tester friendly and approachable in coaching people who never work with code, I extracted a method out of the original unit test.

If you want to try things out before spoilers, pause here and go do the exercise. Figure out what your test cases for it look like and why they are the way they are.

Getting to Work — How Would I Test this?

We’re approaching the exercise with exploratory testing, and all of our options are open. What makes this exercise particularly exploratory is that I will rule out the option of going away to your cubicle to write test cases based on the specification without running a single test. I expect you to design your tests as you go and allow you to learn rather sooner than later.

Unlike for me right now writing this article after having done the exercise, you are now at a time you know the least. You know there’s two information sources and either specification or code could be your starting point. If you are a tester by trade, you are likely to lean towards the specification and if you are a developer by trade, you are likely to lean towards the code and coverage tools.

As the exploratory tester, you are on the driver seat. You get to choose which way you go. And any mix is possible. There is no recipe. But there is options.

Just Try It

You could just forget about the specification for now, as well as not read the code that implements this, and start playing with values you can enter into the CheckItem-method. It takes three inputs:

• a name (String item)
• a number of days we sell the item (Integer selling)
• a number indicating how valuable it is (Integer quality)

If we didn’t look at the specification, deducting this much info out of the interface is unlikely. We would just see it takes text and two numbers. But that is enough to play with it! This brings you to the problem with least structure, highest chance of getting confused and highest chance of learning something outside the spec and the code.

You could look at the problem with the simple ideas around the inputs. What if the input is really really long? What if the number is really really big? Oo, negative numbers? Special characters? All the things forbidden? Hey, there’s numbers that are special: what if your input is 00100, is that 100 or something different?

Read the Specification

Exploratory testing does not mean you have to jump in without considering any of the sources. It means you are intentional about what you choose, and you combine things in ways that keep you engaged as well as ensure you do a good job tracking coverage and meaningfullness of your work. Reading the specification gives you one way tracking coverage.

The specification is full of claims. Some of them are clear. Some will turn out not so clear when you’re testing and seeing what the application does. Some lines are single claims, some are have multiple claims in them. Some lines stand on their own, others depend on other lines of text. Not all lines are meaningful at all.

Which one do you choose to start from? How do you know? Actually, you don’t know. So you sample something and hope to have made a selection that leads you to understanding rather than confusion. And that if it leads you to confusion, you get out of there with later samples.

Read the Code

You could also choose to read the code. You could choose to introduce some tests that enable you to step the logic through in a debugger so that maybe you could see some patterns in how it is implemented. Maybe you just read it without executing so much. Read line by line, or read paying attention to some aspects like variable names or values the code checks against.

Code is the ultimate truth, what is not in the code that now “works in production” isn’t working in production. The spec and the code can be in sync or not, but if they disagree, the code wins until it gets changed.

Think about the environment

The program you’re supposed to test is probably intended for some use by some people somewhere. That somewhere most likely isn’t the test machine you’re using now, and the end user interface most likely isn’t going to be the method you have your hands on now.

What would change in the way you look at the application of the environment was different? Would someone try running it concurrently? Would it need to speak to an external system? Could it be limited to run with miniscule amount of memory, or in an environment that does not allow Java Virtual Machine to be running?

The First Test

There is no absolute choice for the first test, and having tested this with a good crowd of people both individually and in mobs, some people still make a different choice for the first test in the setup we test in.

For me, the first test is to see we can actually test. Running the test that has been given to us as an example. The test that reveals our ability to run any consequent tests.

There has been days of going into doing the exercise where this test fails, because I accidentally cleaned up more than I should after the previous run through the exercise.

Generally, this test results in a green bar.

As we are exploratory testing, we learn that the interface provided is good for us to go further. We learn that the tool reports us with green when a test is given that passes. We might even read the test to figure out that it says that when we start with item named “Any”, no days to sell it and quality of zero, the quality isn’t changing anywhere.

The Second Test

With the second test, we arrive at the significant divergence of choices.

• the freeform value exploration
• the specification
• the code through measurable coverage
• the context of use

The real choice is actually to realize they are all different dimensions and for properly testing this, some work on all might need to happen. Some tracking of coverage on all of them would need to happen. Some discipline in not abstracting results of one to fully cover the results of another would need to happen.

What I often see with the choice of the second test is that some people choose to just pay attention to the code, and end up missing out on all the problems the specification leads you to uncover. Some people choose to pay attention only to the specification and report problems that aren’t really problems and fail to cover the code. And some people just don’t feel like they are empowered to add anything beoynd the given artifacts, which is detrimental to their ability to uncover yet another category of problems.

Reading the Spec While Aiming for 100% Branch Coverage

Let’s assume we intentionally, not accidentally, chose to approach this code and code coverage first, with the help of the specification.

Here’s tests from one group to get to 100% branch coverage. 18 tests. Took two hours in a mob.

At first they tried giving good names, but knowing the least to begin with, the names were not really good. Halfway through they got tired of trying to think of names and just gave up thinking and trying to understand other things and focused on getting through the coverage of specification and code.

In every single test their assert was on quality, and they never see a requirement around how sellin-number would behave in case of legendary items.

Power of the group lead them to pay attention to the specification and they found the discrepancies there:

• The lack of input validation implied by many requirements
• The shorthand of naming items in the specification in comparison to full names used in the code
• The fuzziness around limits the rules defined that behavior would change at

By the time they were done, they had worked significantly. Yet they called done before it was time. No cleaning up the names, no looking for rules they might have missed.

Tools Ease The Exploration

To contract to the 18 handpicked tests over 2 hour intensive work, I’ll show you a few minute example of just covering the code with ApprovalTests-library.

This tool includes a possibility to pass a group of values and automatically generate combinations of those values. The generated tests are pushed into a text file where we can visually verify and approve them. Within the minutes version, I would use the principle of them all being correct because this is legacy code that Works in Production. Within minutes, I generated 41616 tests to get to 100% branch coverage, and to run them again to make sure nothing breaks it takes 1.028 seconds to run. The code to do that is four lines and I go with the ultimate lazy of not even hand-picking relevant integer values but using all between -1 and 100.

If you miss the copy-pasteable example of this, I put in a gist. You may notice I needed to do one change to the original method so that the returning object would have a toString() to write to file. What was object type GildedRose in the first example, is no object type Items in the latter as it already had the necessary toString() defined.

With the 41616 tests, I have now a list of things I can verify with specification. Obviously I would sample heavily over checking them all. But with then visible, I can run into problems serendipitously. I can notice that for

[Sulfuras, Hand of Ragnaros, 27, 23] => Sulfuras, Hand of Ragnaros, 27, 23

the second value, sellin, isn’t in fact changing. I could visually and conceptually compare it to another item

[Foo, 1, 10] => Foo, 0, 9

Seeing that for all others but the legendary item Sulfuras, as speficied, “never has to be sold” in requirements means the sellin date won’t be changing.

Summing up to some fun pitfalls

For a little exercise like this, it has surprising dimensions. To end this article, I wanted to share lessons learned with one newbie tester who found out there was a lot to learn.

They did not get bit by missing the first test, because they were lucky enough to have it in a safe and good green starting point.

They started creating their tests from the end of the specification, where many claims depend on each other. They assumed they spotted an easy claim to verify which I find a fascinating judgement call of what is easy and what is difficult. Defining a test around that claim lead them to learn things that were not true, and double the time to finish the exercise to include unlearning things they had started to firmly believe in.

The lessons were:

• First test was a poor choice leading to long-term confusion, we could either choose differently or pay attention to what we really know more actively.
• Any coverage will do, some here some there and testing is done! That is true, but tracking how much done are you is a big part of testing. Even the code coverage can fool you because the number shows line coverage and the colors show branch coverage. Calling it done on line coverage left out lines leading to new branches.
• Naming coverage dimensions isn’t a thing everyone does, they didn’t. Conceptualizing code, specification, environment and risk coverage is a necessary thinking model.
• Code coverage can be achieved without proper asserts and gives a very false sense of security. You don’t even need to assert values of quality for the 16 tests above to have them still run to 100% branch coverage.
• Problems against specification were all assumed distinct problems, no grouping around the fact that there was full areas of “I expect input validation” that were not implemented anywhere.
• Specification having shorthand like saying “Sulfuras” instead of “Sulfuras, the Hand of Ragnarok” isn’t a major problem with the specification even if it bit them in the exercise. Best of specifications are helpful, yet incomplete. Best of testers using specifications don’t need them to be complete to do complete testing.
• When results did not match the specification, quickly jumping to conclusion that we are testing a buggy software which wasn’t the case. The models to pinpoint whether problem could be in my tests that I have control over were not in place. We practiced many rounds of “is there another test we could do that would give us a second data point to verify we understood the requirement”.
• One test one requirement isn’t a thing. They are a messy network of dependencies.

There is no easy recipe for testing any application. Stop to think. Approach from different angles. Check and double-check. And don’t hide in your corner, maximise your chances of getting to the right information by working with the others.

What Testing Gives You?

As a programmer, you know what you’re implementing. When you don’t know, you’ll learn. You explore the problem to figure out a solution.

Back in the days some bad organizations told you that they’ve hired a group of testers. They might even say that since they pay other people for testing, you shouldn’t be bothering yourself with that.

But you know you want to test. Because testing has direct value to you as a programmer. It gives you four things:

• Specification
• Feedback
• Regression
• Granularity

Specification means that tests you write can be concrete examples of what the program you’re about to write is supposed to do. No more fancy words around high level concepts — give me an example and what is supposed to happen with it. And when moving on, you have the specification of what was agreed. This is what we made the software do, change is fine but this was the specification you were aware of at time of implementation.

Feedback means that as the tests are around and we run the tests — they are automated, of course — the tests give us feedback of what is working and what not. The feedback, especially when we work with modern agile technical practices like test-driven development, gives us a concrete goal of what we need to make work and if it is working. They help us anchor our intent so that given the interruptions, we still can stay on the right path. And we can figure out if the path was wrong. This is the test that passes, yet you say it isn’t right. What are we not seeing?

Regression means that tests don’t only help us when we’re building something for the first time, but they also help us in changes. And software that does not change is dead, because users using and loving it will come back with loads of new ideas and requirements. We want to make sure that when we change something, we make only changes we intended. And regression is the perspective of doing more than we intended, without tests without us knowing.

Granularity comes to play when our test fail for a reason. Granularity is about knowing exactly what is wrong and not having to spend a lot of time figuring it out. We know that small tests pinpoint problems better. We know that automation pinpoints problems better than people. And we know that we can ask people to be very precise on their feedback when they complain something doesn’t work. Not having to waste time on figuring out what is wrong is valuable.

This is not exploratory testing. Exploratory testing often guides this type of testing, but this is testing as artifact creation. Exploratory testing focuses on testing as performance — like improvisational theatre — in a multidisciplinary way beyond computer science.

Exploratory Testing, eh?

I get all of this good stuff from testing as I know it, unit tests and test automation. What is this exploratory testing stuff then?

Exploratory testing is basically saying that spending time thinking with your application, your APIs, your environments as an external imagination, you will find things you did not realize you were missing. And time thinking with something you’ve implemented is real, empirical. Not just something you wish was true like most of our designs.

Exploratory testing is multidisciplinary, basically saying that it is not enough for you to take orders from whoever gives you requirements, but you have to think critically on why they are asking what they are asking and its practical implications. If someone asks you to do something that is illegal, you need to point it out. If you don’t know if it is illegal, you need to spend time figuring out what is and isn’t illegal, and how that would show up in the application you are creating. GDPR is a great example of this: not caring for privacy can cost your organization a lot. And it is not just law you need to care for. You need to care for business, users, environments your software runs in, ethics, psychology — and many more. You want to do no harm, and be conscious about what you are doing. Intentional, not accidental.

With the way you look at testing as programmer, what more is there that exploratory testing promises to give you? It’s an approach that focuses on learning while evaluating what you know and don’t know, giving yourself chances of learning the things you don’t know you don’t know but can see while using the application like your users would.

It gives you four things:

• Guidance
• Understanding
• Models
• Serendipity

Guidance is not just about the specification, but general directions of what is better and what is not. Some of it you don’t know to place in yes/no boxes, but have to clarify with your stakeholders to turn them into something that can be a specification.

Understanding means that we know more about the place of our application in the overall world of things, why people would find it valuable, how other’s design decisions can cause us trouble and how what should be true if things were right always are not. It helps us put the details we’re asked to code into a bigger picture — one that is sociotechnical and extending beyond our own organization and powers.

Models are ways of encoding knowledge, so that we can make informed decisions, understand things deeper and learn faster next time or with next people joining our teams.

Serendipity is the lucky accidents, running into information you did not expect to find. The lucky accidents of new information about how things could and do go wrong when using your application emerge given enough time and variety to use of your application. And knowing helps you not get the escalations waking you up to critical maintenance tasks because who else would fix all of this than the programmers?

An Approach To Testing

Exploratory testing is a approach to testing. It says whoever tests needs to be learning. Learning needs to change what you are doing. You can’t separate designing of tests and executing them without losing learning that influences your next tests. It is an approach that frames how we do testing in a skilled way.

We don’t do just what is asked, but we carefully consider perspectives we may miss.

We don’t look at just what we are building, but the dependencies too, intentional and accidental.

Unsurprisingly, great programmer teams are doing exploratory testing too. Creating great automation relies on exploratory testing to figure out all the things you want to go check. While with exploratory testing we believe that premature writing of instructions hinders intellectual processes, we also know that writing that stuff as code that can be changed as our understanding grows, this frees our mental capacity to think of other things. The executable test artifacts give us that peace of mind.

Programmer teams are also doing what I would consider low quality exploratory testing with limited ideas of what might make a difference in new information being revealed. And that is where testers often come in — mindspace free of some of the programmer burdens, they focus their energies elsewhere, raising the overall quality of work coming out of teams.

Finally, I want to leave you with this idea — bad testing is still testing. It just does not give much of any of the benefits you could get with any testing. Exploratory testing and learning actively transforms bad testing to better.

More where this came from

This story is published in Noteworthy, where thousands come every day to learn about the people & ideas shaping the products we love.

Follow our publication to see more product & design stories featured by the Journal team.

An Approach to Testing

Exploratory Testing is an approach to testing that centers the person doing testing by emphasizing intertwined test design and execution with continuous learning where next test is influenced by lessons on previous tests. As an approach, it gives a frame on how we do testing in a skilled way.

We use and grow multidisciplinary knowledge for fuller picture of empirical information testing can provide. With the product as our external imagination, we are grounded on what is there but inspired to seek beyond it.

We learn with every test about the application under test, ourselves as the tool doing the testing, other tools helpful in extending our capabilities and the helpful ways we can view the world the application lives in.

We keep track of testing that has been done, needs doing and how this all integrates with the rest of the people working on similar or interconnected themes.

What makes our activity exploratory testing over other exploratory approaches founded in curiosity is the intent to evaluate. We evaluate, seek for information we are missing, making sure what we know is real with empirical evidence.

Specifying It By Contrast

It’s not a surprise that some folks would like to call exploratory testing just testing. In many ways it is the only way of testing that makes sense — incorporating active learning is central to our success with software these days.

To contrast exploratory testing with what people often refer to as manual testing, exploratory testing as a skilled approach encompasses use of programming for testing purposes. We use our brains, our hands as well as programs to dig in deep while testing. Sometimes the way of including test automation happens through means of collaboration, where ideas from exploratory testing drive implementation of automation that makes sense.

To contrast exploratory testing with people refer to as scripted testing, exploratory testing isn’t driven by scripts. If we create scripts from exploratory testing, we know to use them in exploratory fashion remembering that active thinking should always be present even when the script supports us in remembering a basic flow. We’ve talked a lot about scripted testing as an approach where we separate design — deciding what to test, and execution — making the test happen, and thus lowering our chances of active learning targeting the most recent understanding of the risk profile.

Another contrast to programming centric views to testing comes with embracing the multidisciplinary view of of testing where asking questions like “is my application breaking the law today after recent changes?” is something routinely encoded into exploratory testing, but often out of scope for a group of automators.

The Three Scopes of Exploratory Testing

As an approach, we can time and integrate this into our development efforts in many ways.

I find there are three ways of scoping:

• exploratory testing as a way of extending existing test cases
• exploratory testing as a limited timebox
• exploratory testing as frame of all thinking

When exploratory testing is scoped as a way of extending existing test cases, the way of working does not look very exploratory. In places like this, you find yourself wondering why Tina is so successful with the same tests Toni is missing problems with. The secret is that one of them actively extends how they understand the test cases, refuses to follow exact instructions or only instructions. They learn and find new perspectives.

When exploratory testing is scoped as a limited time box, you find people feeling they can only be free on Friday afternoons. These are moments in project life that are structured separately, with different expectations of how things flow and where focus is. This is a great way of scoping exploratory testing into a process where the natural inclination is to believe we know where the tasks begin and end, and need explicit encouragement to see if what holds true.

When exploratory testing is the frame of all thinking, it encompasses all considerations about testing. We start with exploratory testing to identify something that needs documenting, and take time to document it — perhaps with automation that is the modern way of documenting as executable specifications.

Listen to Language

If you hear: “My boss asked me to test search so I searched something that was found and something that wasn’t and reported I was done” you may be witnessing very low quality exploratory testing. It relies on following high-level orders to an extent the tester can imagine based on their knowledge.

If you hear: “My colleague wrote me 50 cases of what to try out with search and I tried them and reported I was done” you may be witnessing testing that isn’t exploratory. There is no hint of learning, and owning responsibility of quality of the testing that happens.

If you hear: “My boss asked me to test search so I came back with 50 quick ideas of what could be relevant and we figured out I’d just do 10 of them before we decided if going further was worthwhile“, you are likely to be witnessing exploratory testing.

Similarly, in the automation first space, if you hear: “I got a Jira ticket saying I should automate this. I did, and found some problems while at it, and extended existing automation because of the stuff I learned.”, you may be seeing someone who is exploratory testing.

If you hear: “The Jira ticket said to automate A, B, and C, and I automated A and B, C could not be automated.”, you may be witnessing testing that isn’t exploratory.

Look at who is in the center: is the tester doing the work and learning actively applying a better way of doing the overall testing? If yes, that is exploratory testing.

As skilled approach, it is only as good as the skill of the person applying it. With focus on learning through, skill may be a problem of today, but improved upon every day as exploratory testing is being done. If you find yourself not learning, you most likely are not exploring. With the product as your external imagination, you should find yourself imagining new routes through the functionalities, new users with new perspectives, and relevant information your project teams would be happy to make justified decisions on their take for the risk. With and without automation, in a good balance.