Dear reader,
Wishing to learn from good practices and to share the experiences gained in the Holocaust, Police and Human Rights program, started in 2014, Kazerne Dossin: Memorial, Museum and Documentation Centre on the Holocaust and Human Rights, together with the Belgian Federal Police set out in 2016 on an ambitious two-year program entitled “Connecting Law and Memory” (CLM). The central objective of this program has been to encourage reflection on the integrity of civil servant practices in extreme situations in the past and to link these reflections to the current work of the civil servant institutions.
The Kazerne Dossin was inaugurated in November 2012, with a team composed of experienced researchers as well as colleagues with new profiles. Its enthusiasm for the Connecting Law and Memory program has been, was shared and supported by the International Holocaust Remembrance Alliance (IHRA), the Council of Europe and the Government of Flanders. Other organizations have also cooperated in the program, including the Mechelen City Festival “Op.Recht. Mechelen” (“Up.Right. Mechelen”), the Belgian Interfederal Centre for Equal Opportunities (UNIA) and, especially notably, the Anne Frank Foundation.
We wish to take this opportunity to thank the supporting organizations and institutions, their experts – in particular the keynote speakers and panelists – the participants and especially the colleagues from Kazerne Dossin and the Belgian Federal Police who have spared neither time nor effort to make the program a great success.
From its inception, one of the essential objectives of the CLM program was to set up an open-ended coordinating forum (platform) to permanently connect, share and benchmark existing outreach programs of memorial museums. A key objective here was to learn from good practices and experiences between interested institutions and to foster innovative approaches for dealing with collective violence.
Obviously, new partners would be encouraged and supported in developing their programs. Over the three years of the CLM program, networking became one of the most rewarding experiences.
An equally important objective was the drafting of a manual to support the implementation of programs for practitioners, policymakers and civil servants who are seeking to connect Law and Memory.
The foundations for these major objectives were laid at the opening session and workshops in Mechelen in September 2016, mapping experiences and good practices in outreach programs and training courses on human rights, with special focus on those designed for law enforcement, including police officers, judicial authorities and other civil servants. These programs have sought to strengthen civil service resilience. Over the following two years exchanges between the first group of participants and newcomers and the establishment of other working groups progressed rapidly towards the goals.
In November 2017, at the invitation of the Anne Frank Foundation, workshops were held in Amsterdam. Although the previously noted objectives remained a central focus, additional attention was given to the possibility to better engage museum visitors and to the need to find ways and means to create the conditions and incentives to move a person from empathy to compassion to action. The Inzovu Curve was presented as a potential model for use in memorial museums. There was also general agreement among the participants that, in order to reach sustainability, extended connections should be established between museums and visitors and between the groups of civil servants.
The launch of this manual, “Structural quality standards to connect Law and Memory,” which is tailored to practitioners, policymakers and civil servant program implementers, offers a unique opportunity to share with many interested experts and groups the myriad insights gathered over the last two years.
In our view, the publication of the manual and the establishment of the platform should not be considered the conclusion or the ending of Connecting Law and Memory.
This manual, as well as the platform, marks the beginning of new exchanges and will help towards generating innovative solutions and experiences that CLM adepts wish to share with one another, with the expanding CLM community and with us.
We look forward to continued exchanges and cooperation.
Christophe Busch Jan Deboutte
General Director Kazerne Dossin CLM Program Chair
Historical research has uncovered the crucial role that civil services throughout occupied Europe played in the discrimination and murder of Jews and Roma. These insights are of invaluable importance in Holocaust and human rights education for civil servants. Civil servants, especially those in law enforcement, are endowed with far-reaching powers and authority over their fellow citizens. Thus, their work requires great responsibility and critical reflection.
In focusing on the causes of the Holocaust and reflecting on its consequences in this light, it becomes clear that the subject of civil servants during the Holocaust is a topic that offers much potential for gaining knowledge and insights about the role of civil servants. This historical and scientific point of view demands research in order to render it applicable to more universal mechanisms that remain active in modern societies today.
It also important to connect research and education. The Holocaust is still the general comparative framework for assessing mechanisms of collective violence in other situations. There is a need for specific training programs on these issues; they are crucial for raising awareness and knowledge about the Holocaust and remembrance education.
Civil servants often consider human rights not as the foundation of their work but rather as an obstacle to it. This can become especially relevant for law enforcement officers, who, unlike most people, face human rights dilemmas on a daily basis. In emphasizing the importance of these officers’ dual role to protect and respect human rights, these training programs have the potential to change such attitudes.
The overall aim of Connecting Law and Memory was to establish a coordinating forum and network for sharing and benchmarking expertise, so as to strengthen the civil service resilience through discussion, reflection and dilemma training. This project will offer tools and attitudes to prevent and resist racist, xenophobic or homophobic behaviors and other forms of intolerance – which are frequently aggravated by peer pressure and/or group loyalty – within law enforcement and other civil service institutions.
We wished to connect, link, share and benchmark existing outreach programs, good practices and experiences. We did so by organizing a series of conferences over three years. The participants were a mix of decision makers and educators from museums, memorials and civil service bodies involved with establishing training programs on human rights (or showing strong interest in doing so), based on Holocaust and genocide studies. There was also an emphasis on bringing together civil servants (especially law enforcement) and personnel from museums and memorials.
An important objective during the Connecting Law and Memory project was to reflect on the circumstances to initiate and develop these programs and to take existing programs to a next level, in particular the level of concrete action. The key question in this regard was to what extent training programs can be optimized in order to have more impact.
There is a need to evaluate the impact and quality of existing educational programs and the experiences of trainers. This must be done at two levels: individual (participant) and collective (legislation, structures, whistleblowing).
During the Connecting Law and Memory conferences, we were able to look into research about the impact and effect of training programs for law enforcement groups. This specific target group is strategically important not only because of its role in, and for, society but also because of the larger social perception of this role.
This project also allows us to present the impact of existing training programs to countries who need this kind of knowledge and insights so as to be able to better reflect upon their own history. We are convinced that the Connecting Law and Memory project can also contribute to the process of transitional justice in certain countries and provide them the proper elements to do so.
The structural standards outlined here are designed to aid the efficiency and effectiveness of human rights training programs for civil servants, in particular law enforcement, and to facilitate proper evaluation. As an initial foundation for further process and impact evaluations, this manual serves policymakers and public bodies when evaluating program implementers’ structural quality, and offers opportunity for the implementers themselves to establish the structures of their work and develop them further.
The main body of the manual covers eleven practices and approaches, analyzed on four topic areas: running and developing a program; training of the trainers; participant classification; and quality assurance and quality control.
Based on the good practices presented during the first Connecting Law and Memory conference, in 2016, in Mechelen (www.connectinglawandmemory.eu), we asked all contributors to join us in Amsterdam in 2017 to find together answers to the following questions:
– How can we improve the museum experience and better engage law enforcement civil servants during their visits to museums dedicated to remembrance and human rights?
– How can we channel civil servants’ experiences of visiting or receiving training at a memorial museum into action? How can we incorporate the feelings of sympathy, sadness and outrage into an experience that activates visitors towards working for a better humanity?
– How can we turn institutions memorializing genocide into institutions helping to prevent it?
The results of the Amsterdam working sessions were harvested and resulted in a questionnaire by which to more deeply analyze the good practices.
This manual is particularly addressed to policymakers, civil servant program implementers and practitioners who are working on Connecting Law and Memory.
Specifically, this handbook aims to help:
Running and developing a program
Training of the trainers
Participant classification
Quality assurance and quality control
Based on the experiences of the existing training programs for civil servants, initiating and developing human rights training programs are based on two mean reasons:
When these two ideas encounter each other, a fruitful and successful collaboration is possible. Whatever the reason may be for initiating these kinds of programs, the ultimate goal is to encourage reflection about the integrity of civil servant practices during extreme situations in the past and to link this to their daily work today.
Ensuring support of the Senior Management within the civil servant organization is crucial when developing these programs. Buy-in from the policy and decision makers will likely be needed in order to ensure implementation of the programs. Moreover, such top-level backing can convey a system-wide message of endorsement that validates the professional value of the training approach.
For this reason, the program development process is highly important. Training programs must be established on a robust theoretical basis, so as to apply a systematic method to the program approach. Besides this, it is also crucial to know what the target group is experiencing in their daily professions. These experiences are an important, if not the most important, insight to take in consideration when designing a training program.
What else should be verified?
In the development phase of a training program, it is essential to look for possible partnerships. A partnership can be between different museums/memorials, different civil servant organizations and other organizations. A key benefit of these kinds of partnerships is that expertise is shared on different levels, which affords potential for continued developments in connecting law and memory. A list of inclusion or exclusion criteria can be helpful in determining partnerships; such criteria can include long-term engagement in the project, contributing without financial compensation, expertise in a specific domain, embracing the mission statement of the particular museum/memorial or the values of the civil servant organization, the size of the civil servant organization and its influence in the field, the mandate of the civil servant organization, and maintaining political neutrality as an organization/museum/memorial.
It must be borne in mind that years of activity do not automatically equate to a high degree of competence or quality. The vast majority of these programs in this field have not yet been comprehensively evaluated as to their impact on civil servant organizations.
The concrete aims of the program must be clearly defined. A clear aim is essential for the success of the program.
Both the target group and the staff themselves must understand as clearly as possible the actual aims of the program. Effective evaluation is only possible when formulated aims can be measured against ultimate outcomes.
The practices used in writing this manual have each established clear objectives on which the different training programs are based.
Depending on the practice, the training program is given by museum/memorial educators, or by trainers linked to the civil servant organization. Sometimes a third party, e.g., anti-discrimination trainers, is involved in the training. In some cases, a mixed team is formed, based on the various trainers’ expertise.
Some good practices include exclusion criteria for becoming a trainer. Such criteria can include the mandate of the chief, integrity issues, etc.
Staff training should include ethical guidelines for trainers as well as ensure that the staff will have opportunity to provide feedback about the course content. This ensures that trainers know, share and support the goals of the project. Ongoing further training is also a central component of quality assurance.
The materials used to implement the training strategy must be made available for the evaluation, because this strategy forms an integral part of the overall project, and should be in line with its focus and aims.
For all the good practices, the trainers receive an intensive training before giving the programs. All trainers are supervised on a regular base. In some programs, regular team meetings are organized to discuss the difficulties that trainers may encounter during the training or to provide them with new material. These team meetings contribute to the quality of the training programs.
The quality and effectiveness of human rights training programs depend chiefly and crucially on the trainers. Thus, the importance of the trainer should not be underestimated. It is essential that the trainers ensure intensive and close support and supervision for the newly recruited trainers as well as maintain short and effective channels of communication between trainers, steering committees, etc.
The target group for the training program must be clearly defined and appropriate to the program format. Different types of programs are suitable for different kinds of target groups; it is therefore important that, during the program development stage, the social and political context is carefully evaluated.
The defined target group, with its own specific characteristics, directly influences the core offerings and services of the program. These factors also determine the skills which trainers should possess.
In a few practices, the choice of a specific target group within the civil servant organization is connected to the influence it can have in policies and practices to prevent genocides and other atrocities. In such cases, exclusion criteria are used to select the participants, such as position of influence, seniority, mandate of the chief, etc.
Other practices are developed at the behest of a civil servant organization. In these cases the target group is determined by the organization itself. It is essential that the developers of the training have a good understanding not just of the civil servant organization and their expectations but also of the target group. It is crucial to understand what they are dealing with in their daily profession and how this can relate to the training program.
In some good practices there were no exclusion or inclusion criteria determined during the development phase of the training. However, during the course of the trainings and based on feedback from participants, the initial target group was expanded, or a more specific, additional training for a clearly delimited group was developed. This expansion was often needed to comply with specific needs of the (new) target group, such as senior officers, cadets, etc.
Other organizations organize thematic events, in house or on location, in order to address specific needs of target groups. The thematic events can treat topics such as polarization, radicalization, etc.
For some organizations, expansion is not (yet) possible, due to capacity problems or limited financial resources.
A steering committee helps to direct a project from start to completion. Sometimes it is comprised exclusively of staff from the organization developing and implementing the project; most often it is made up of representatives of the different partners in the project, each of whom have a particular expertise to lend to the project, and/or whose “clients” are the intended users of the output of the project. It is very important (and useful) to include at least one client of the service, or potential user of the project that is being developed, as their viewpoint is crucial in ensuring that the project is targeted accurately and effectively.
A steering committee should be helpful, and not distracting, to the project manager. Thus, membership should be considered carefully.
The steering committee’s role is to provide advice, ensure delivery of the project outputs and confirm achievement of project outcomes. It provides support, guidance and oversight of progress. Members do not usually work on the project themselves.
This may include such tasks as:
The development of the training programs by museums and memorials does not necessary mean that the training program is integrated into the curriculum of civil servant training (such as police training) or that it depends on the policies of the civil servant organization.
In some cases, a distinction must be made between cadets and police officers already in the field.
Depending on the particular police organization’s structure, the trainings can be part of the curriculum for cadet training. For police officers in the field, participation can be made mandatory for obtaining a senior position. In other organizations, public servants participate voluntarily in this kind of training.
The trainings can be integrated into the lifelong learning (ongoing, voluntary and self-motivated pursuit of knowledge for either personal or professional reasons) of the civil servant organization. This is done in only a minority of the gathered practices.
In some organizations, the staff is required to follow an “ethics” training each year.
One good practice mentioned continuous engagement not only with the program alumni but also with their offices and colleagues. This engagement is intended to ensure an effective, durable change within the civil servant organization.
As part of a structural quality assurance it is essential – insofar as possible – to gather feedback from participants about the program. This can be done by offering participants an opportunity to provide anonymous feedback, on a voluntary basis, at the end of the training. This provides a key resource for both internal and external impact and process evaluation. Other possibilities include asking for feedback via e-mail, online forms or after-program surveys.
In the practices we collected for this manual, requests for feedback are done mostly through feedback forms at the end of the training. In some practices the evaluation is done independently by a third-party consultant. This independent evaluation gives a robust report with quantitative and qualitative measures related to a variety of substance-based and logistic-based considerations. In other cases, feedback from participants is analyzed by a pedagogical team of the police academy.
Feedback is gathered not just from participants but also from the trainers. This can be done by having the trainers write a report about their experiences during the training or by filling in a questionnaire.
In one practice, the training program is the subject of an optimization study, conducted by a university (see chapter 5).
Some organizations do not organize feedback; instead, they call upon the feedback procedures of the civil servant organization they work for. Still others gather feedback on their work only in an informal way.
Some practices emphasize the importance of having close contact between the trainers and the program coordination. Trainers are regularly invited to reflection days organized by the program’s coordination and steering committee. Central topics addressed during these reflection days include the quality and the evaluation of the program.
The gathered feedback is, for all practices, the most important input for adapting the program.
A quality assurance assessment is performed during the project, so as to ensure the product meets quality standards. This differs from quality control that is effectuated after the product has been created.
There are various tools by which to assess quality assurance, including benchmarking, cause and effect diagrams, Pareto charts, inspection, flow charting, cost-benefit analysis, etc.
It is clear that a quality assurance mechanism should be built into the program’s structure. This mechanism should be divided into an internal and external part.
The internal quality assurance checks for actual compliance with the formulated aims and standards and envisaged processes. This includes verifying that trainers’ knowledge is regularly tested and/or updated, organizing regular learning moments, checking (on an unscheduled basis) on individual trainers’ performances and maintaining data on the financial parameters of the project.
These internal quality assurance mechanisms ultimately help in making refinements and optimizing the processes with regard to the actual aims and tasks of the program.
A further aspect of internal quality assurance is determining participant satisfaction, as far as possible within the parameters of the program.
External quality assurance is based on comparisons using specifically defined criteria and benchmarks. It is of fundamental importance to the structural integrity of these training programs that the program undertakes regular, third-party external quality assurance. The professional examination and evaluation of various aspects of the program should always be performed by people and bodies with either relevant practical experience in the field or the necessary academic expertise.
In conclusions, the internal and external quality assurance must entail a critical and transparent dialogue between trainers, the coordination and the steering committee about mistakes made during the training and about possible structural problems. As part of the structural integrity evaluation, there must be critical discussion concerning the knowledge about elements that have not run smoothly. Questions to address include whether or to what extent such incidents have been addressed and what has been done to minimize any risk of repetition.
In some practices there is no follow-up concerning the trainers. Other practices organize their follow-up process through a team of coaches who use an observation grid and debrief the trainer at the end of his/her performance. The observation grid is kept in the trainer’s file, so as to chart evolution in his/her training skills.
Other practices organize capacity-building activities based on need assessment and feedback with the aim to enhance training skills and knowledge.
As mentioned, the vast majority of the programs in this field have not yet been comprehensively evaluated as to their impact within civil servant organizations.
One practice noted that their evaluation is based on different inputs, from participants and trainers, and is written in an annual report. This report describes quantitative and qualitative data and compares them to previous years of the training program. The evaluation form includes criteria to gauge the usability of the program in the daily working environment.
Other inputs can include anecdotal evidence, such as thank-you letters and emails from participants.
Only one practice conducts a scientific research on the impact of the training program on their target group. The results were not yet available during the writing of this manual.
Memorials play an important role in teaching and compelling us so that we will never feel detached from and complicit in tragic events that injure humanity. Most of the time, however, when we visit such places we remain shocked yet unable to act beyond that immediate experience. Virtually every visitor to a genocide memorial or a Holocaust museum can attest to overwhelming feelings of sympathy, sadness and outrage, but most visitors can also attest that they did nothing substantively differently as a result.
How can we redesign the experience of those memorials to activate visitors and civil servants for a better humanity?
In our work with organizations around the world, including the Kigali Genocide Memorial in Rwanda, we believe we have come to see the outline of a new institution needed.
We believe the next generation of genocide museums and memorials will not be like conventional memorials. They will be a new kind of institution – an institution that proactively addresses global challenges of intolerance and violence, leverages new technologies to enable global impact, connects the next generation to the stories of past atrocities without the help of generations that witnessed them and a more determined, humble view of how outsiders should intervene to stop violence.
What does this new institution look like and how can their experiences activate visitors and civil servants for a better humanity? And, how can you begin to understand the changes required in your own institution and its programs and services to further engagement and shift towards a stronger agent for change?
The Kigali Genocide Memorial is built on the site of a mass grave, housing the remains of 250,000 Rwandans who were killed over three months in the 1994 Genocide against the Tutsi. Like all such memorials, it is intended as an antidote to genocide itself – teaching us and moving us to ensure we will never again be detached and complicit. But, for the most part, we remain unchanged.
During the UX for Good (www.uxforgood.org) challenge in Rwanda (June 2014) we partnered with Aegis Trust, the Kigali Genocide Memorial and the Rwandan people to develop a model that focuses on turning institutions that memorialize genocide into institutions that end it.
A team of researchers, designers and social innovators assembled in Kigali to work in partnership with Aegis Trust, the founding partner of the Memorial, and the local Rwandan team managing the Memorial itself. The design team made use of an array of resources, from experts on museum design to their own personal observations at the memorial site.
Collectively, the design team completed over 500 hours of research, prior to and during the event, to understand, assess and evaluate the visitor experience of the Kigali Genocide Memorial. Research activities included:
But the team was most inspired by the young people who visited and worked at the Memorial. In workshops and curricula, portable posters and personal stories, the next generation of Rwandans are figuring out how to convert the story of one of history’s worst genocides into hopeful action in their own lives.
Carefully observing these young people, the design team developed a model the Kigali museum — and all museums — can use to convert profound emotional experiences into action. They nicknamed it “the Inzovu Curve” after the Kinyarwanda word for “elephant,” because the arc users travel resembles an elephant’s trunk. Visitors to a memorial or museum first descend into a state of (often painful) empathy with the victims of violence whose stories they encounter. Many institutions simply abandon them there; the Inzovu Curve instead advises them to provide additional experiences that lift visitors into a state of compassionate action. The model also identifies specific moments of reflection and transformation that will help equip all visitors to make a difference in the world.
The Inzovu Curve

At its core, the Inzovu Curve is a model that maps specific designed activities to the emotional response of the individual experiencing them. The model contains three elements:
Aegis Trust’s leaders in Africa and Europe endorsed the Inzovu Curve model as a way of inspiring action against genocide. But we believe it has implications for all museums and memorials, not just those that commemorate atrocities. Therefore, while we have continued to partner with Aegis Trust and the Kigali Genocide Memorial to further integrate the model into the visitor experience, we have also continued to refine the model, based on new insights. For the first time, the Memorial is placing systems and metrics in place to begin measuring the visitor experience, and we have already seen some practical application of the model and its supporting recommendations, including:
Additional information can be obtained at www.inzovucurve.org.
Starting from that first application a few years ago, the potential of the Inzovu Curve model continues to be scaled up to any other similar museums that aim at reaching the ambitious goal of motivating visitors towards activism.
In fact, work continues on examining, refining and applying the Inzovu Curve model in various settings – from working with world-renowned memorials and museums to international conference workshops to continuing its extension and application at the Kigali Genocide Memorial. The model is now recognized as a useful tool in helping frame the visitor experiences at institutions looking to transition to even greater agents of social change.
Most recently, we shared the Inzovu Curve model and thinking as applied to civil servant engagement at a workshop during the Connecting Law and Memory conference in Amsterdam in November 2017. Our question for participants was:
How can we redesign the experience of your organization/program to activate civil servants for a better humanity?
Our goals and objectives for the workshop included:
The workshop and associated discussions provided a wide range of insights. Participants also generated a rich and diverse set of ideas in response to applying the Inzovu Curve to their own organizations and programs. Here are some of the ideas that were captured:
Workshop participants agreed on the need to further strengthen museum visitors engagement and to create pathways between empathy and compassion to action. Additionally, participants agreed that in order to become more sustainable, connections between museum visitors and civil servants should be further established.
Overall, the Inzovu Curve model was considered a useful model for helping to assess and guide thinking for the visitor and civil servant experiences.
While museums and memorials may be successful in telling the significant stories of the past and in sharing reflections of such history confronting the future, there’s growing opportunity and need for the institutions to become exemplar “civic connectors” within the local and global community.
But that is not an easy task at hand, as it impacts the visitor and civil servant experience, the people, programs and processes supporting that experience and, perhaps, even reshaping the operating model of the institution itself. This opportunity is both complex and critically important towards ensuring these institutions are not limited to tourism and education but rather are seen as agents of social change.
How might we continue making the effort to keep the institution’s experience relevant for all audience types?
While the Inzovu Curve model can provide reflection and help facilitate ideation and discussion, institutions will need to address many additional questions in the longer term to achieve success, whether with visitors or with civil servants. Here are a few to continue your journey:
Create a learning-centered culture:
Focus on the end-to-end human experience:
Perhaps a conference attendee best stated our shared efforts:
“Society is a partnership of the past, the present and the future. We can’t control the future today, but we can influence the partnership, and therefore influence the future.”
In its most general form, effectiveness research (sometimes called Impact Research) is driven by the questions of whether a particular type of education has an impact on the participants, what effect it has and why it is effective. Research into the effectiveness of educational initiatives is relatively young, having first emerged in the late 1970s. Initially, such research was limited to purely cognitive academic performance in schools, especially in the areas of mathematics and language proficiency. Today, it constitutes a broad research stream that assesses the impact of education on beliefs, attitudes and preferences. Effectiveness research is also no longer limited to education offered in schools; increasingly, it now focusses on myriad kinds of educational initiatives, including programs for remembrance education such as the Holocaust, Police and Human Rights program which is taught at Kazerne Dossin.
In this chapter we reflect on this evolution. Conducting effectiveness research for programs for remembrance and peace education poses a number of specific challenges. As effectiveness research on remembrance education is a newly developing field, the number of “good practices” is rather small. Moreover, effectiveness research has been criticized on more fundamental grounds. It is accused, for example, of paying too little attention to the restrictions and limitations confronting education, of being neo-liberal (i.e., equating educational quality with a narrow set of quantitative outcomes), etc. Many of these accusations rest on misunderstandings and persistent myths. As such, they disregard the contribution effectiveness research can deliver to the broader area of remembrance and peace education. It is this potential contribution this chapter aims to draw attention to.
This chapter has two general objectives. First, we discuss some general observations concerning the existing effectiveness research on remembrance education and the broader field of peace education. In this way we aim to make clear what effectiveness research is and should be and to signal some of its most important pitfalls. Second, throughout this discussion we seek to raise more specific questions that can be used by any specific program, regardless of whether an effectiveness research is conducted. In this way we aim to illustrate how the core questions of effectiveness research can be used both for further optimization of existing programs and for development of new ones.
In assessing the current state of literature on effectiveness research on remembrance education and the broader field of peace education, four general observations can be made. These observations highlight several critical preconditions for conducting successful effectiveness research.
Booming business and the right motivation to do effectiveness research
Recent years have seen a sharp increase in the number of studies claiming to assess the impact of remembrance and peace education. This tendency is part and parcel of the growth of an accountability culture with respect to all types of education. In societies that are highly focused on the pursuit of efficiency through scientific research, there is great pressure on organizers of remembrance and peace education to “prove” their effectiveness and, in this way, to justify why their programs exist.
In such circumstances two problems arise. First, in some cases effectiveness research becomes an end in itself (backward looking), rather than what it should be, namely, a means towards improving education (future oriented). Second, some effectiveness studies are started even if the essential means to conduct the study are lacking. People who are pushed to justify their work use the means they have at their disposal. The overall result is that the effectiveness research of remembrance and peace education is characterized by large variation in methodology and quality. This disparity contrasts sharply with the more general effectiveness research, where there is fairly broad consensus concerning the key methodological elements of good effectiveness research.
In our view, it is not reasonable to ask every program to prove its effectiveness, especially when available means do not allow for setting up a full-fledged study. Overall, the added value of an effectiveness research or impact study will be limited when (1) a program has just started, and/or (2) a self-reflective attitude and a culture of evaluation among the organizers are lacking. One reason we thought that an optimization study for the Holocaust, Police and Human Rights educational program was worthwhile was that evaluation and self-evaluation were from the very beginning an integral part of this project. From the very beginning, in 2014, there were reflection groups with the coaches, perception research among participants, structured feedback between the organizers of the program and the police department, etc. A large-scale optimization study was the logical next step to take. In many cases this is not so, and efforts and means can better be invested in the further development of an existing program. This does not mean, however, that the effectiveness literature is irrelevant for such programs. Indeed, we argue that, even when a full-fledged effectiveness study is not feasible, the particular effectiveness research literature can be used by organizers to improve their existing programs without doing an effectiveness study themselves. Simply by applying the key questions that drive the effectiveness literature (see further), one can already detect the most important blind spots or weak points in one’s program.
Effectiveness research differs from perception research
In many of the existing studies on the effectiveness of remembrance and peace education, the assessment of a specific educational program is based on the self-assessment of the participants or of their teachers. In such cases, people are asked whether they had the impression that the program had certain effects on them. This type of research, known as perception research, can be considered part of effectiveness research but it is certainly not identical.
The crucial point here is that perception research documents only experiences which the participants are explicitly aware of. There is no good reason to limit “the effect” of an educational program to these experiences. Just as people may sometimes have the feeling that they were influenced when in fact they were not, much of the learning – perhaps even the major part – is an unconscious process. Indeed, many of the effects that are documented, particularly in the literature concerning the broader area of citizenship education, are ones that the individuals involved are likely entirely unaware of. The analogy with advertising applies here. People are strongly influenced by advertising, but when they are explicitly asked about this, they tend to deny it. Consequently, no one would maintain that the effectiveness of advertising can be assessed only by asking customers whether they have been influenced by advertisements. For the very same reason, perception research cannot demonstrate the effectiveness of an educational program.
The objective of an educational program is to create a change in abilities (knowledge, skills, attitudes) and behavior. The assessment of effectiveness must therefore test whether a change in ability and behavior has occurred, not whether the participants have the impression or are willing to say that such change has occurred. In short, assessment must focus on learning gain, which in turn can only be established by observing, measuring and comparing the relevant knowledge, skills, attitudes and behavior. On this point, effectiveness research into remembrance and peace education needs to correspond to the existing body of effectiveness research. Within this research, effectiveness is expressed in terms of learning gain. In order to discuss the practical implications of this, we must take into account the next point.
Remembrance education, like any other type of education, may not be equally, or in the same way, successful for everyone.
One of the remarkable characteristics of the existing effectiveness research on remembrance or peace education is that the analysis often assesses the impact of a program without making any distinctions between subgroups of participants (i.e., by analyzing the group as a whole). Such practice (implicitly) assumes that an educational program has a general effect on all its participants. This is a strong assumption. If a program does not hold in practice, there is high risk that one will mistakenly conclude that a program has no effect or that the effect is underestimated. Effectiveness research should assess which groups gain the most benefit from remembrance education. If the effect of education is more pronounced in one subgroup in comparison with another, we refer to an interaction effect.
The existing literature signals a wide range of possible interaction effects. It is known, for example, that the impact of an educational program tends to be stronger when the people involved had positive expectations of the project as compared to people with less sympathetic views. It is therefore important to include people’s openness to change. People who really do not believe that a program can affect their thinking are unlikely to be affected by the program. This is not so much because the program is ineffective, but rather because it takes much more time to create a change in these people. We also know that programs tend to have a stronger impact on people when there is more space for improvement. People who are already very tolerant, who already adopt a self-reflective attitude and who feel responsible for themselves and their environment will mainly find confirmation for the views they held before they started the program. Finding no improvement among these people should not be interpreted as indication that the program lacks effectiveness.
The foregoing has two clear implications. First, a good effectiveness study starts with an extensive pre-test. This test should not only provide a clear picture of the knowledge, attitudes and beliefs to which the project seeks to contribute – a precondition to assess learning gain. It should also draw a clear profile of the participants – a precondition to distinguish meaningful subgroups among participants. As concerns the latter, the following questions can be used as guiding principles:
As applied to the Holocaust, Police and Human Rights program of Kazerne Dossin, police officers strongly differ in terms of their working conditions, contact with violence, experience, openness to change, etc. It is likely that these differences will generate differences in how the program is received and experienced. In studying the effectiveness of a certain program, it is also best to keep in mind the context – i.e., relevant characteristics of the school or work environment of the people who attend the program. It is indeed possible that the impact of a program that is effective in itself could be counteracted by the prevailing school or work environment of the participants. The general point we try to make is that it is of crucial importance to establish a good understanding of all relevant characteristics of the program’s initial context. Ideally, these questions are not only raised when one aims to set up an effectiveness study but are also part of the design and continuous monitoring of the program; thus they should be asked even if one does not intend to set up an effectiveness study.
Second, the question of whether or not the target population characteristics should be taken into account when developing the program should also be addressed. This is an extremely difficult question, because tension may arise between, on one hand, the idea that remembrance and peace education is primarily orientated towards evoking a change of culture (which presumes the inclusion of all people and groups involved and as such leads to heterogeneous groups) and, on the other hand, the more pragmatic consideration that a program has only limited time/resources available to affect the participants (which can encourage a focus on those participants who are most likely to benefit from the program).
Good effectiveness research always commences from the learning processes that the education is expected to trigger. An attempt is then made to measure the results of these processes and to assess whether process characteristics can explain the observed changes in outcomes. In short, in current effectiveness research there is strong emphasis on the relevance of process variables. This contrasts somewhat with the strong and sometimes exclusive focus on final outcomes in effectiveness research on remembrance and peace education.
A focus on process variables is informed by the simple observation that education never has a monopoly of influence on people. The key question, then, concerns whether an observed change in beliefs, knowledge and behavior between the start and the end of a program is actually an effect of the program. Many things may have happened during the course of a program that have nothing to do with the educational program in question, yet nonetheless may be responsible for the observed shift. Thus, when assessing the impact of a program, it is not only important to observe a positive change in the outcome variable after the program. It is also crucial to try to “explain” this change by characteristics that refer to the process of the program.
It is at this point that perceptions of participants may be relevant. When this is the case, the perceptions act as explanatory variables rather than as the outcomes. For the Holocaust, Police and Human Rights program, for example, we not only used the conventional perception measures (e.g., satisfaction with different aspects of the program, whether people felt that they had learnt something, etc.). We also included a focus on the emotions that people felt during the session. In this way we have attempted to assess whether a possible change in the outcome variables (attitudes, prospective behavior) can be attributed to what participants experienced and felt during the session. If such attribution is in fact the case, this would strengthen the claim that possible observed changes after the program are also due to the program.
The durability of effects and identifying what exactly the long-term effects of remembrance education should be.
As noted earlier, education is embedded within society and cannot claim a monopoly of influence over young people. This is precisely why it is so interesting to determine how long the impact of remembrance education lasts. By the “durability” of effects, we mean both their extent in time, and how far the newly acquired attitudes and skills can be generalized when facing new situations. A number of important challenges arise at this point, however.
Education always endeavors to have long-lasting effects but not every type of education has the same means to achieve this. As compared to more general education, remembrance educators have even less a monopoly of influence over people. In general, people simply spend much less time in remembrance and peace education as compared to general education; this implies that the overall chances of influencing people are lower in remembrance and peace education. The Holocaust, Police and Human Rights program of Kazerne Dossin, for example, has a single day to influence people’s thinking and behavior. Secondly, the outcomes typically studied in effectiveness research are greatly influenced by factors that come into play outside the educational setting. Moreover, such factors have much more influence as compared to more general outcomes like reading, mathematics, etc. These elements raise questions concerning reasonable expectations of the durability of effects.
When looking at the existing programs of remembrance and peace education, the precise long-term objectives are often far from clear. They frequently sound grand and ambitious, yet at the same time are rather impractical and vague. This causes a practical problem, since effectiveness research assesses the effect of education on pre-specified educational targets. Educational targets are objectives that are deemed feasible in the given circumstances. Anything that it is merely hoped to be achieved, or to which a contribution is attempted, is not part of the educational objectives and also falls beyond the scope of effectiveness research. This consideration is especially important for the long-term effects of remembrance education.
It is tempting to disregard long-term effects of remembrance education by not assessing them in effectiveness research. Such a position could be supported by various considerations, and is above all a manifestation of modesty. The time available for a program is limited, and many conflicts are rooted in structural inequalities that cannot be changed through education. We live in a culture where violence is sometimes even glorified. Literature tells us that great generals receive far more attention in history books than those who build bridges and that conflict is mentioned far more than peace, even though, paradoxically, the great majority of people are confronted more with peace than by violence. Therefore, the effectiveness of remembrance education and peace education in general cannot be measured by the degree of attention afforded to racism, violence and conflict in a society. All of the above is true, but what is the ultimate implication?
Some would argue that because people are exposed to so many stimuli it is impossible to determine the long-term impact of remembrance education. In our opinion, this seems like an unproductive argument that is out of line with the premise of effectiveness research, since it implies that remembrance education is, at most, no more than a drop in the ocean. But if it is not possible to determine the long-term effects, then what is the value of the short-term effects? Again, some would argue that, as long as we do not know for sure that it has no effect, it remains meaningful to keep trying. This opinion is open to debate. All too often, the “there’s no harm in trying it” argument leads to vague and unclear phrasing of the goals of projects under implementation. In our view, it is impossible to disregard the more fundamental discussion about the long-term effects of a program. Only by adequately addressing this question can we formulate reasonable goals and develop effective strategies to achieve them.
At this point, sometimes inadequate distinction is made between two elements, namely, the immediate, practical value of effectiveness studies and the more research-based value. While the former is used by teachers and educational staff to move ahead with the immediate job, the second may provide significant insights for the development and comparison of programs. This is an additional reason why it seems to make little sense to limit research on the effectiveness of remembrance education a priori to its short-term impact.
The bottom line of the argument is that reflecting on effectiveness research in the case of remembrance education not only can provide insight into the question of “what works?” It will also help to define more precisely the long-term objectives that are promoted – or not promoted – by remembrance education itself.
If one is investing in remembrance and peace education, it makes sense to check whether the education is effective and whether the investment is achieving its goal. Effectiveness and efficiency may be important criteria, but they are certainly not the only ones against which the usefulness and value of knowledge should be measured. Moreover, establishing a full-fledged effectiveness research requires substantial means. If it is used solely as a tool to justify one’s own project’s existence, the added value of effectiveness research will be limited. To put it bluntly: the effectiveness of effectiveness research depends on its integration into a culture of self-assessment, self-reflection and continuous innovation.
As such, we think it is not reasonable to ask that every program “prove” that it works. At the same time, we do believe that every program can derive pertinent questions from effectiveness research literature. The following questions should be used to critically engage with the current ways of working:
The answers to these questions constitute what the literature on education calls a formative evaluation, which is distinguished from a summative evaluation (effectiveness research). It should not be conducted only after a project. It is much better to consider and respond to these questions in advance. Indeed, only those programs which are able to provide clear answers to these questions can benefit from more extensive and thorough effectiveness research.
The increased demand for effectiveness research on remembrance and peace education has led to an ever growing number of such studies. All things considered, however, this type of research is still in its infancy. Thus, for precisely that reason, replicating research is of crucial importance. This presupposes a lively dialogue between effectiveness research, empirical research into the nature of planned programs and projects, and theoretical work. The purpose of effectiveness research is to contribute to the improvement of current education. It can do so by improving specific programs; however, precisely because it is not feasible for every program to design an effectiveness study, there is great need in collective knowledge building. Currently, three different bodies of literature are very well developed. First, there is a very elaborated and extensive social and political psychological literature that studies the origins and consequences of socio-political opinions and behaviors. Lab experiments have empirically proved the causal mechanisms at work. The main limitation of this body of literature is that the ecological validity of certain mechanisms has seldom been studied. We know that it works in the lab, but what about real-life situations? Second, there is extensive historical literature that has designed remembrance education programs. Such programs can be further developed and improved only if we can garner insight into the impact they have on participants. And third, there is extensive literature on educational effectiveness research; however, this has been primarily developed for traditional education and so cannot simply and without adjustments be applied to existing remembrance education initiatives. What is lacking is a body of literature that connects these three fields of research.
To conclude, the question of whether remembrance education really works is beginning to be raised with increasing emphasis. The increased demand for effectiveness research comes not only from outsiders, such as policymakers or program designers. It is also increasingly present in the literature concerning remembrance and peace education itself. The latter tendency is based on the idea that the insights that effectiveness studies provide can be helpful in the design of new projects and in the application of existing ones. More specifically, effectiveness research can help to advance remembrance education in at least three ways. First, it should stimulate debate about the long-term goals of remembrance education. Second, it tests the empirical validity of existing theories regarding the mechanisms behind remembrance education, and thus bears the potential to integrate these theories back into practice. Effectiveness research can improve remembrance education, yet this requires time, resources and expertise. Precisely for this reason, the existing research must be carried out to the highest standards possible. Third, this type of effectiveness research may eliminate various misunderstandings and persistent myths.
Considered in this way, and when embedded in a process of continuous reflection, dialogue and optimization process, effectiveness research concerning remembrance education is primarily aimed towards the future.
This manual is the first of its kind to offer a comprehensive foundation for minimum structural standards in Connecting Law and Memory, and constitutes a starting point for further developments and expert exchange.
During the Connecting Law and Memory conferences a number of key insights were distinguished. These include:
The manual also offers the possibility to assess projects as regards their structural quality, to set priorities on a number of levels and to make more conscious, strategic selection decisions. Particularly as concerns the running of such programs and the administrative-bureaucratic realities associated with them, it is essential to be able to draw upon a structural guidebook when developing or assessing processes.
Although spreading areas of work amongst several implementers, teams or projects can safeguard specializations in the respective areas, it can also lead to additional burdens, such as loss of information and time depletion stemming from coordination and co-operation. Structural standards are therefore a highly effective means for avoiding such drawbacks, provided such standards align with the needs of the practical work and reflect the respective substantive and factual realities.
As a final note, it should be emphasized that this manual marks only the beginning of a debate about the standards and evaluation of training courses in human rights for law enforcement and civil servants. Effective process and impact evaluation can only be carried out on the basis of coherent structures and defined processes.
– Auschwitz Jewish Center, Oswiecim, Poland
– Auschwitz Institute for Peace and Reconciliation, New York, United States
– Bart Brandsma, Schoonrewoerd, The Netherlands
– De Nieuwe Onrust, Nijmegen, The Netherlands
– Flossenbürg Memorial, Flossenbürg, Germany
– Kazerne Dossin, Mechelen, Belgium
– Mémorial de la Shoah, Paris, France
– POLIN Museum of the History of Polish Jews, Warsaw, Poland
– United States Holocaust Memorial Museum, Washington, D.C., United States
[1] This chapter is written against the background of an optimization study (including effectiveness research) the two authors conducted in 2017 – 2018 in close collaboration with the organizers of the Holocaust, Police and Human Rights program offered to police officers and trainees at Kazerne Dossin. Part of the text is based on Spruyt et al. (2014). Can peace be thought? Researching the effectiveness of peace education. Brussel: Flemish Peace Institute.