Thoughts on devops, and random sysadmin related rambles.

After hearing about Kanban as an Agile method for managing workflow at the excellent DevOpsDays ‘09 conference, I was immediately struck by how well it fits the workflow of tasks that we face in our team.

After a brief discussion with my MD (’we need to manage our workload better, and Kanban seems a good fit. It means we will be limiting how many tasks can be on the go at any given time though – but we’ll get more done overall, and you’ll see what we are doing on a big board’) to get his buy-in on the Work-In-Progress (WIP) limit, my team and I got together to plan what the board should look like – and get it in action!

Firstly, a little background on $WORK and in particular my team within it. The core business is web development for our clients, who are mostly large academic publishers, though we also offer hosting and systems management solutions to provide a full managed solution for them. I head up the hosting and support side of the business.

My team comprises of three people, including myself, and we have four main functions: 2nd/3rd line support for our software; systems management of the customer systems and supporting infrastructure; architecture work for new customer projects (and re-engineering of existing projects), and finally supporting all the internal staff.

Dan primarily handles the support side of things. Giles mostly does systems management/operations type work, but helps Dan out when he’s busy, and does some project work. My time is about 25-50% project work, 25-50% operations, and 25% management. The support load is light in number of tickets – about 40 per week, but they can range from simple 15 minute responses to heavily involved long running tasks taking several weeks to resolve.

So, a couple of days ago, Dan, Giles and I went into a meeting room, with a blank magnetic white board, 60 magnets and a whole load of 4×3″ cards, and came up with the following:

The board is divided into three areas: Support Bucket (Blue); Emergency (Red); and normal workflow (Green).

We decided to keep things as simple as possible for our first board, so simply divided our workflow into our team members. We did wonder whether splitting up based on task type would have been better, but considering we all have a primary role in the team, it seemed more sensible to start with our own workflow.

The Emergency lane is there so that we can have any ‘massive priority task’ given to us without requiring that we free slots available in our WIP board. WIP stops for at least one member of the team whilst the emergency box has cards in it.

The Support area is, I admit, not very Kanban (as I understand it!) We felt it necessary to have this on the board, to prevent support issues that are blocked on 3rd parties from clogging up our WIP. If an issue is awaiting more information from the customer, it comes out of the WIP, freeing a slot for another card. If a ticket requires assistance from a Dev team member, then we pull it out of the WIP in a similar manner. Honestly, I’m not sure this is a good idea, but it does seem to fit with our support workflow – it can take an age for a customer to get back to us about a problem.

Off the board, we also have the ‘Backlog’s Backlog Box’ – a card box that contains new (unsorted) tasks, and tasks that did not make the cut to get on the board in our scheduling meeting (which we will hold weekly). It is divided into three sections: ‘New’, for tasks that have not been prioritised; ‘Awaiting $DATACENTRE’ – for tasks that can only be completed next time we schedule a visit to our DC; and ‘Awaiting Other’ for any other task that cannot be started until something else has been completed.

The weekly meeting (of which we’ve only had one so far!), has the following format:

• Get the cards out of the backlog’s backlog box
• Sort them into a 2×2 grid, based on how much benefit they have to the company and our customers (High, Low), and how much effort/time is involved (Quick, Slow).
• Colour the corner of the card with a highlighter: Quick+High – green, Quick+Low – blue, Slow+High – yellow.
• Slow and Low gets no colour, in case it becomes easier to do, or more important, later.
• Put all cards that are Quick+High onto the backlog on the board, writing the date that it went onto the board on the front of the card.
• Work out what in the boards backlog is ready for action in the coming week, and move those cards into the ‘Ready’ column.
• Of the tasks in the ‘Ready’ column, we each moved the most important tasks into our WIP, limiting as per the numbers on the board.
• We have a ‘once it’s on the board, it doesn’t come off’ policy. Not sure how practical this is yet.
• We have a ’special’ card, labelled ‘General Support’. This takes up one WIP entry for Dan or Giles, depending on who is focussing on the smaller support tasks this week. It gets assigned in the weekly meeting, though can move mid week if need be.

We’re also having a 5 minute daily stand-up, which is in front of the board in our main office. The format is:

• Quick run through our current WIP, with an estimate of work remaining to complete a card, are there any blockers?
• Discussion of any tickets in the support ‘awaiting boxes’ – can they be moved back into the WIP/Ready flow?
• Is there anything in the emergency WIP?
• What do we expect to be working on next in the Ready column?

Update: We’ve now been working with the board for over a week. It’s been going reasonably well, and has definitely improved our task-focus and visibility within the team. It’s pretty obvious that we need to tweak how we work on/around the board, but we’re still learning.

My main concern at the moment is that the 5-minute stand-ups are a little too micro-management-like – but I think as time goes on they will feel more natural.

Any suggestions are greatly appreciated in the comments, and I’ll try to get another post up soon with how we’re progressing!