INEGMA-E2 took part in BALANCE FSX to conduct a workshop with a team of 22 dedicated evaluators. In the workshop, the following topics were discussed:
- What are the skillsets that the exercise evaluator should have
- What tools are they using
The evaluation team had a strong pracademic structure as there was a strong presence from the CERIDES research institute from Cyprus and Urban Search and Rescue experts from Greece, Bulgaria, and Croatia plus the independent evaluator who was with extensive exercise experience and UNDAC member. This fact contributed to a lively and open discussion in the room. Some of the key inputs received concerning the required skillset of the evaluator:
- Empathy and humility towards the organizers of the exercise but also towards the participating teams. The evaluator must show appreciation for the tremendous efforts put in place to organize a simulation exercise especially FSX and for the teams that are participating in such a controlled experiment.
- Communication skills. The evaluator needs to be an excellent communicator to take out the best of the training audience. Established rules of engagement in advance, acceptance of those rules by all participants, and presenting the evaluation team to the audience are crucial for the learning element to happen.
- Language. The importance of understanding the working language of the exercise (English) was emphasized not only for the role of evaluators but for other segments of Exercise control – team leaders, liaison officers, incident commanders, safety and security officers, and trainers. Understanding the local language is considered an asset.
- Experience. If it is a USAR exercise then the evaluators need to know the topic in order to better position themselves with the training audience and to understand the challenges that they are facing. However, the evaluation team even in a USAR exercise could benefit from the presence of non-USAR experts within the team. A combination of people with academic and practical experience as part of the evaluation team is considered as an added value to the process.
- Understanding of the process of evaluation. Evaluators should know the basic theory behind evaluation methodologies and tools that are used. This is especially important for the evaluators who are with a purely operational background – firefighters, rescue staff, etc. Understanding what the data collection process looks like, how it is structured, and how it proceeds toward the analysis and preparation of a meaningful report is needed. If an evaluator sticks only to his/her specialty like in this example USAR they might easily miss other segments that are important for the exercise and they might also enter into the shoes of the trainer.
- Not knowing the results of the evaluation. Although not a skill set for the evaluator, the group mentioned this element as well which is a contributing factor to a successful process of exercise evaluation. The comments were > I was many times in an exercises setting and evaluated but never had the chance to see the results of that evaluation.
As BALANCE was unique in terms of the number of evaluators, it was assessed as helpful to have more evaluators in the team.
Conclusion
INEGMA E2 was introduced for the 3rd time to BALANCE project and now during the FSX in Kotor, Montenegro. The colleagues from CERIDES Research Institute from Cyprus remained very open, supportive, and cooperative on the topic of exercise evaluation. From the preparation process of the evaluation itself and the effort invested in compiling the Evaluation Plan it is evident that the knowledge on the topic within CERIDES Research Institute is on a high level and should be explored more. Finally, all 22 evaluators as part of the team expressed interest to join the pool of evaluators that will be designed with WP4 and contribute to the topic further.