ComPart Evaluation - Three key roles to support collaboration and learning

We are still thinking through the implications of the recent ComPart evaluation, first announced on this blog, We know it wasn't the perfect time, given the changes the ICCO Alliance has been going through lately, but if you have been able to contribute to the evaluation in one way or another, we really want to thank you for your time and effort.

One of the suggestions of the evaluation team was to develop and work with three key roles (though not necessarily three distinct people) to support this way of working. If you're somewhat familiar with ComPart, you probably know the ComPart admin team (based in Europe) relies heavily on the continuous efforts of the ComPart ambassadors (personnel of the regional offices) and the ComPart enablers (web 2.0 and collaboration experts in the regions).

The evaluation suggests working with three 'roles':
converer - technology steward - facilitator

The evaluation team suggest a somewhat different constitution of the roles: conveners, facilitators and technology stewards. The conveners are the ones that bring people together and they make sure that there is trust among the group. The facilitators focus on the processes and the activities that come with it. The technology stewards, in their turn, will assist the group on technological issues, but without loosing the community perspective.

We believe that these three key roles can have a big impact on the adoption and implementation of new ways of working where collaboration and learning are vital. When looking back at our own experiences with ComPart, we have seen several examples of how this can work in practice.

One such case was when ComPart went global for the first time. During the Conference on Informal Education and Literacy in Ouagadougou (Burkina Faso) several sessions were to be held in which the participants would be introduced to ComPart with exercises to get acquainted with this new way of working. The very first day of the conference, however, the people who had organised the workshop came to us with a problem. During the presentations, the three countries had so many interesting questions about the others' programmes that they needed to spend at least an additional two hours to this session, which, in turn, would seriously mess up the schedule. So we were asked if ComPart couldn't solve this urgent problem. And it could...

We quickly created wiki-pages for the questions per country and instead of starting a 'ComPart session', as we had planned, we briefly introduced the event wiki as a way of getting the work at hand (i.e. answering all the questions) done as quickly as possible. So instead of having a seperate ComPart session about wikis, we integrated it in the session Presentations – part 2.

The wiki was used to let the participants of the
different countries asked and answer each others questions

There were several reasons why this worked so well. First of all, we were lucky to have conveners and facilitators like Machteld Ooijens (ICCO) and Tine Veldkamp (IC Consult) present. They knew the group very well, spotted the need for a ComPart intervention and had the authority to suggest and introduce this 'new way of working'. Local enablers Christophe Hien (Burkina Faso) and Oudou Bengaly (Mali) had good knowledge of the ComPart tools but also had a perfect understanding of the local context, the technical difficulties people might encounter during their daily work and they were acquainted with the theme of the conference. The results of this experiment (it was improvised after all) were remarkable. All questions were answered in less than an hour, where it would have taken more than 2 hours to answer them in the 'traditional way'. Everybody had used and contributed to the wiki without it feeling as a burden or something 'extra'. Instead, it was seen as a real time saver and something that helped them get on with the job at hand.

We think this example not only shows the validity of working with conveners, facilitators and technology stewards. It is also linked with other recommendations of the evaluation team, like blending learning with getting work done, create time-delimited experiments or change one thing at a time.

The evaluation has a lot of valuable suggestions for the ICCO way of working. Luckily, we don't have to start from scratch and we can build on previous experiences (successes but also failures) ComPart and ICCO have had. And that's what learning's all about...

Danny Aerts

Evaluation of ComPart – The real question

As if there wasn’t enough going on in ICCO in the last quarter of 2010, the ComPart evaluation came to a head in December. Marc Coenders and Nancy White, the lead evaluators, held a validation workshop on 9 December 2010 to discuss their findings and the final report was presented to management on 12 December 2010. It was a deep investigation and the findings described in the report are rich, balanced between positive and negative, subtle and far-reaching so it is hard to summarise the main points in a blog post. This is therefore a personal take on the findings, from my own point of view, as both a ComPart admin team member for three years and an independent consultant with experience in KS and the use of digital technology to support the work of development organisations.

A storm tossed cork

In terms of context the report acknowledges that ICCO has been through a period of turbulence unrivalled in recent times, one that will unfortunately continue for at least the early part of 2011, and not simply because of the current financial pressures. The longer term change illustrated opposite is about

continuing to move to a more networked organisation at a time when public money is reducing sharply. In that context staff and partners are much less likely to be able to – or want to – prioritise learning about a ‘change project’ such as ComPart, especially one that is associated with a ‘way of working’ that is different to the status quo. ComPart, by definition, is focused on ICCO’s partners and alliance members, which, the report notes, has meant that the ComPart way of working has become associated with the wider changes taking place across ICCO during this period, which in many cases produces negative connotations. This provides a context for the finding that internationally based people generally had more positive views of ComPart and its role than Utrecht based staff. This is despite the fact that, as the report notes, international offices in their set-up phase were generally unable to engage with ComPart as they would have liked to and so the ambition to use ComPart with partners has not yet been achieved.

The ComPart Lab

The report notes early on that, "ComPart turned out to be a laboratory with a series of experiments rather than a project with defined, measurable goals." In terms of evaluating the results and impact of ComPart, the report acknowledges and speaks positively about the range and scale of the work that has been done; the quality of support provided; the continual experimentation and the consequent well documented lessons learnt; and the resultant growth in awareness across ICCO of how digital tools can support business and learning processes. However, the number of active users is still small. This, suggest the evaluators, is a product of the complexity of the ComPart toolset, the difficulty of choosing between or integrating ComPart with corporate IT and Communications tools and the fact that its usage is not seen as an essential part day to day working but as an extra. The one occasion where ComPart was integrated into a core Business Process – the development of the Business Plan – did raise ComPart’s profile and usage of the wiki but also generated a lot of extra work and some confusion as it was introduced rapidly and with inadequate preparation. That example also highlighted one of the key weaknesses raised in the report, that wikis are often difficult to introduce into organisations because a lot of people do not like their structure – or lack of it – and are not comfortable with the way of working they encourage. The ComPart wiki has become a rich source of information and learning but many people are less aware of the other tools in the ComPart toolkit.

Questions to do with ways of working as well as attitudes to technology and information storage or retrieval are central to the report, mainly because the evaluators note that there isn’t a consensus about these issues within ICCO. For example, to some people a wiki is a place to store information and should become as reliable, searchable and up-to-date as other online resources. Other people see a wiki as a dynamic and flexible tool to support collaborative processes and are not rigorous in how information is organised and stored. Wikis, of course, can be used in both ways but the result is an inconsistent collection of material not always accessible to new staff or others not so engaged in the ComPart experiments. At the same time, to many people ComPart was simply the toolkit and the ‘way of working’ which it promoted – and indeed depended on for its success – was not conceptually well understood, perhaps too open and flexible for the current ICCO way of working. This leads to one of the most fundamental findings which is that the questions which are raised in the review are not essentially to do with ComPart but to do with ICCO itself, how the organisation wants to go forward in terms of learning and Knowledge Management. The evaluators suggest the most important questions are about what ICCO wants to do with its learning and knowledge sharing, about what is the ICCO way of working - rather than the ComPart way of working; about how Departments work with each other and how ICCO is going to adapt to its evolution to a networked organisation. They suggest that ICCO should concentrate on thinking about those issues, and then look at what can be learnt from the ComPart programme to support those new ways of working.

A rose by any other name…

Names are important. Marc has some interesting things to say about learning and naming/theory (see blip below). The recognition that ‘ComPart’ has become associated with problematic issues for ICCO staff suggested to the evaluators that, “the ComPart brand carries enough misunderstanding that it could become a liability. We suggest this is a turning point where ComPart becomes the historical name of a period of experimentation and learning”. We need to talk about the “ICCO-Alliance way of working” rather than the “ComPart way of working”.

From pioneering to deploying

In the new decentralised (and down sized) ICCO social media and other digital tools definitely have a role to play. So the pioneering phase is over. The time has come to really deploy and integrate working with digital tools in the organisation its processes as it regionalises. Therefore it is time to launch a new phase, taking into account the experience gained and all lessons learnt from the “ComPart phase”. As well as the ongoing regional office development programmes there are other new initiatives, such as the learning communities programme, which could benefit from that experience. However, the report also suggests that two other key areas of attention. Firstly, ensure that the next phase is integrated within the whole organisational way of working. Secondly, as part of that, there must be smoother and more cooperative relations between directly involved departments (P&D, ICT, Communications) and thereby with all staff within the Alliance.

Pete Cranston

(If you are interested you can ask for a copy of the full report at compartadmin@gmail.com)

Evaluation of ComPart kicks off

Even if the name ComPart - Communication with and amongst Partners - came in only at a later stage, at ICCO we have been busy with this project since 2007.

The main idea behind this project was that it would contribute to ICCO Alliance capacity building, learning networks, and knowledge in two main ways: first, by supporting the information, knowledge and communication capacities of partner organisations in the south; second, by supporting these same activities within the current alliance members.

During the implementation of the project ICCO was undergoing a profound organizational transition process. One of the major elements of this process is the decentralisation, which means that much of the daily work is being transferred to Southern based regional offices, with the transition of ICCO from a Dutch-based and located organisation to an international decentralised network organisation. This meant that new (internal and external) communication issues arose. Already in its early stage it was foreseen that ComPart could play a role in helping to address these communication issues.

The ComPart project has been designed and implemented with the strong support from Euforic, a cooperative of European Development organizations mainly supporting their members with relevant information and the use of information management tools. This cooperative closed down in the end of 2009 and fortunately part of the involved staff could continue the support to the ComPart project by way of the newly created consultancy firm Euforic Services.

During the 4 years of the projects, as documented here in this blog and in the ComPart timeline, numerous achievements have been reached, and different challenges emerged as well. More in general, a great deal of learning has been done as far as introducing and applying social media and collaborative tools in an changing organisational context is concerned. It seems now a good moment to look back at what has been done so far, reflect on the project sore assumptions and implementation patterns and learn from our achievements and, more important, form our mistakes. Therefore, since a couple of week, the ComPart project is undergoing its evaluation phase.

The purpose of this evaluation is twofold. In the first place to get an appreciation of the ComPart approach itself, the way it has been introduced and the results of the project until now. In the second place it is expected to get practical recommendations on how to further improve and implement the “ComPart way of working” within ICCO, the ICCO Alliance and among their partners. This implies that it is expected to get an overall validation of the project towards its objectives and based on this to get recommendations on how to better use and introduce internet based tools for an open, collaborative and learning culture within the decentralized organization/alliance and also possible (technical) improvements on the tool-set itself.

Nancy White and Marc Coenders are leading this evaluation, and we're glad we can count on their knowledge and expertise to help us learning and reflecting on ComPart.

The methodolgy follwed for the evaluation comprises the review of the documentation, plans and the ComPart tool-set; the preparation of a learning history for this project; a web survey; individual and collective interviews with champions and active users, new users, users who aren't very active, non users, the ComPart admin team and ICCO management; a 'findings validation' workshop with users, the admin team and others involved to share findings and gather feedback.

Most of the above is already happening these days and the evaluators are busy reviewing the documentation available, as well as interviewing people in ICCO Global Office in Utrecht and in the Regional Offices. The validation workshop will be held in Utrecht on 9 December 2010.

As usual, we'll use this blog to document on the progress of the evaluation and report on its finding and the validation workshop.

PADEV - Participatory Assessment for Development

On 15 September 2010 ICCO hosted a public seminar titled 'Insight in Complexity - Possibilities for scaling –up a bottom-up evaluation approach'. The main purpose of the event was to present, review and discuss the findings of the research project on participative impact assessment from a beneficiary perspective, carried out since 2007 by Tamale University of Development Studies, Ghana and UvA, Amsterdam and supported by the ICCO Alliance.

After the welcome and introduction to the event by Jack van Ham (Chairman ICCO Board of Directors), Professor Ton Dietz from the University of Amsterdam who coordinated the research projects, presented the PADEV manual and its main findings. Roger Bymolt (master student International Development studies UvA) offered some critical remarks from a field perspective. Further, Agnieszka Kazimierczuk (former master student UvA) focussed on her findings in the use of the methodology with ‘the (very) poor’. Lastly, Robert Chambers (IDS) and Irene Guijt (Learning by Design) presented their critical comments on the research project and the PADEV methodology.

A lively plenary discussion moderated by Kees Biekart (ISS) followed this round of presentations and allowed participants to debate around issues related to the methodology proposed in PADEV; the results that emerged in the research and what their follow up will be; the issue of relevance of PADEV and who will benefit form it.

The PADEV manual, as well as a set of multimedia outputs from the seminar are available on the ICCO Policy and Development workspace.