Go to content

3. Data collected by the research team on the evaluation and reporting practices of youth organisations in the Nordic countries

Writers: Anu Gretschel, Viola Särkiluoto & Kai Tarvainen

3.1 Literature review and interviews

Our research team investigated the means and tools that Nordic youth organisations have in place to evaluate their activities. We familiarised ourselves with the subject by reading publications and conducting interviews. For the purposes of this report, we have compiled the most interesting examples in cooperation with the steering group. In order to avoid repetition in the text, each example highlights issues that have not been raised much elsewhere in this publication. In particular, we aim to answer the following three  questions:
  1. What kind of routines related to evaluation and reporting have been developed in youth organisations in the Nordic countries?
  2. What themes in the activities are highlighted through evaluation?
  3. What kind of challenges are seen to be involved in evaluation?
With regard to publications, we focused our search on reports with an investigative approach and guidelines that are given to organisations in connection with evaluation and reporting. The publications were commissioned by the organisation itself, or they were conducted as a thesis or purchased service, for example. By evaluation guidelines, we refer to the guidelines issued by funding bodies, such as the public administration, and the evaluation guides produced by the umbrella organisations of youth organisations, for example. We surveyed publications from the last ten years. We went through over 70 titles from different Nordic countries. As a result of this work, the research team and the steering group can be relatively confident that a significant part of the important evaluation literature from the youth organisation sector has been taken into account in this publication, albeit limited to the last few years. The purpose of this limitation was to ensure that we would provide the public with the most up-to-date information on the activities of each organisation.
The research team was already somewhat familiar with the activities of youth organisations in Finland. For the other Nordic countries, the team first utilised the Youth Wiki database, which provides country-specific information on the activities of youth organisations. This way, we gained an understanding of the role of youth organisations in relation to municipal youth work and youth leisure activities, as well as information on the funding of youth organisation activities.
We looked for surveys, reports and studies on the evaluation and impact of the activities of youth organisations in a number of databases. However, most of the studies were found through Google and Google Scholar searches. We also searched studies in academic databases (EBSCOhost, SocINDEX and ProQuest), ResearchGate and the catalogue search of the University of Helsinki Library. We started the literature search with the keywords youth organisation (nuorisojärjestö, ungdomsorganisation) combined with various related keywords, such as effects (vaikutukset, effekter), quality (laatu, kvalitet), evaluation (arviointi, evaluering), method (menetelmä, metod), social impact (vaikuttavuus, effektivitet) as well as different combinations (AND/OR) and compounds of the above (e.g., effektvaluering). Since the search results were very limited, we included Swedish search terms of utvärderingsrapport and effektrapport in combination with search terms such as ungdom and unga. The aforementioned and corresponding keywords were used in Norwegian, Swedish, Finnish and Danish, as well as in English in a wide variety of combinations. As the process progressed, we extended the list of keywords, for example, by adding the names of different youth organisations. As a result, we found a significant number of bachelor’s theses particularly from Finland on Google Scholar. However, several of them dealt with, for example, the impact of youth social welfare, mental health, substance abuse support services and projects rather than the activities of youth organisations. The master’s theses found on Google Scholar that included concrete themes related to evaluation or social impact in youth organisations were examined more closely.
Only a few scientific surveys and reports specifically on the evaluation of the activities of youth organisations were found. Also, Hanna Laitinen (2018, 28) notes in her doctoral thesis that only a few academic studies on youth organisations have been carried out in Finland, for example. It was also apparent that the keywords in publications rarely included evaluation, effects and impact, but various other words related to youth organisation activities. Thus, the keywords used in this study did not necessarily lead to matches. We complemented the literature found in the databases with the help of representatives of the umbrella network of youth organisations established for the project, as well as tips provided by both the interviewees and Nordic Youth Wiki correspondents. We also browsed the web pages of organisations and authorities and examined the works of organisations that implement research and development processes.
We conducted a total of 13 interviews in four Nordic countries. The organisations in question are presented with their respective interview. The interviewees had the opportunity to revise the accuracy of their interview citations. A total of 15 people participated, representing 16 different organisations. These organisations were youth organisations or other organisations working with young people, their umbrella organisations, organisations carrying out research and development processes in youth organisations, ministries and their committees, and agencies set up by ministries. The interviews lasted one hour and were conducted online through Teams.

3.2 Glasses for the reader: how evaluation relates to a better understanding of one’s own activities and operating environment 

3.2.1 Evaluation as reflection

In this report, we consider evaluation as an activity that systematically examines whether an activity met the expectations that were set or could be set during or after the activity. Next, we will describe the timeless nature of reflection by quoting Donald A. Schön’s (1987, 1983/2007) classic texts. To be specific, Schön does not use the word ‘evaluation’ in his texts – he associates reflection with the design of activities and the structuring of design needs. However, in our opinion, Schön’s way of thinking also serves as a good depiction of the reflection involved in evaluation.
Schön uses the term “knowing-in-action” when a person knows how to act highly professionally in a practical situation, but does not consciously make choices between different actions. With the term “reflection-in-action”, Schön refers to a more advanced reflection in the sense that the agent is able to and truly does reflect on the choices related to how to act. Reflection on “reflection-in-action”, on the other hand, relates to examining the depth of the reflection. According to Schön, the action, situations and the possible problems associated with them are not, as such, well-established to be solved. According to him, they are rather uncertain, unique or conflicted. . In this case, it is not enough to consider the type of road that should be built if the question of whether the road is going to be built have been ignored in the first place.   For example, economic, environmental, political and more extensive value-related issues may influence the decision. Schön regards reflection not as a single event of optimisation, but as a process that takes into account uncertainty, the unique nature of situations and the possibility of conflict. (See Schön 1987, 1, 4, 6, 39, 41.)
According to Schön (1987, 5), there is often competition over which perspectives should be taken into account in the development of activities. When it comes to the evaluation examples of Nordic youth organisations, this is exactly the case: they have to consider whether they should focus on the number of activities or, for example, on accessibility and equality. According to Schön, in a technical examination that is too narrow, for example, the unintended side effects, risks or even the environment are ignored (Schön 1987, 6). When working with young people, it is possible to ignore such an important issue as looking at things with young people and from their perspective.
Paraphrasing Schön’s views, it could be said that a person reflecting on their actions in a somewhat uncertain practical situation acts in the same manner as a researcher. Reflecting is not limited to considering the means of action in relation to predefined objectives of the action, but responds to the challenges encountered by considering both the objectives and the means flexibly. The thought also entails that if a certain person could be considered to have knowledge on a certain matter, for example based on their position, it is through wisdom that this person acknowledges that they enable learning for themselves and others, as they acknowledge the knowledge of others and include them in the conversation. (See Schön 1983/2007, 68, 300.)
According to Schön, the struggle over how to define the activity, situation and problem and the political measures needed to find solutions is always a mixture of investigation and politics. Views on the nature of the challenges are cognitive constructions built from a certain perspective, the birth of which creates space for the use of political force. In the wider societal debate, the positioning of the activity, situation and problem, the definition of the policy, and the interpretation of the backgrounds that led to the situation include both investigation and political perspectives. (Schön 1983/2007, 348.) In terms of the autonomy of youth organisations, it is important that the youth organisations themselves are also pro-actively involved in identifying the needs for their activities and determining their role from the local to the global level. A good example of this type of activity was found in Denmark, where the umbrella organisation for youth organisations, DUF, organised a survey on the political activity of young people and DUF’s own efforts to make Danish society more democratic. In Norway, the umbrella organisation LNU considered the publications of the research institute Senter for forskning på sivilsamfunn og frivillig sektor (Centre for Research on Civil Society and Voluntary Sector) an important literary example with regard to evaluation. The publications discuss the accessibility of leisure activities organised by the organisations, the correlation to the needs of young people, and the need to develop attractiveness, taking into account the different backgrounds of young people.

3.2.2 Theory/Theories of Change

This study on the evaluation of Nordic youth organisations found a few examples that use the theories of change in youth organisations in Sweden and Finland. These examples are described respectively for both countries in this report. However, the main features of the theory of change will next be presented briefly.
According to Sue Cooper (2018, 18–19), who studies the impact of youth work carried out in organisations and by the public sector in England, the purpose of using the theory of change is to be able to construct an evaluation system suitable for evaluating social activity, which can take into account the characteristics of the operating environment and the complexity of the activity.
According to Cooper, the use of theory of change has been popular both among the evaluators and among the funding bodies in various countries for more than half a century. Despite the word theory in the title, the theory of change is not in fact a single theory, but rather a set of ideas, with which each organisation can highlight how exactly the implemented activity is expected to bring about change and what this change can be. Cooper has aptly stated that these assumptions are in fact the actual “theories” of the theory of change. At this point, Cooper uses the word theories in the plural, because every organisation, or even every action of the organisation, has its own theory. (See Cooper 2018, 18–22; Fox & Grimm & Caldeira 2017 also write about the theories of change in the plural.) For example, the section of this report focusing on Sweden refers to the prevailing practice in the country where organisations define “Our Theory of Change” (“Vår förändringsteori”) as part of their efficiency reports. Fox et al. (2017, 49) state that writing an organisation’s own theory of change is an issue in which the organisation’s employees and various stakeholders must be widely involved. They also stress that not only must the intended effects be achievable, but they must also be comprehensible to all those involved in the process.
Serrat (2017, 237–241) begins by defining social change as any change in the social order of society. According to him, applying the theory of change to the evaluation of social change is a purposeful activity, in which, for example, changes triggered by an activity or project are often depicted as a branching chain. The branching chain demonstrates how intermediate outcomes lead to a more extensive intended result through development. According to Serrat, some of the changes required along the way to the goal can also occur as a domino effect, without new measures.
Cooper (2018) uses the term “precondition” to describe the so-called intermediate outcomes, thus emphasising their absolute necessity for reaching the final results. In her study, Cooper examined the impact of youth work using the theory of change. According to Cooper, young people experienced a sense of belonging and increased trust in youth work. According to her, meeting these two conditions made it possible for young people to take responsibility for their activities in the medium term and predicted the realisation of the pro-social behaviour of young people, which was the main outcome in the longer term.
As shown in the next subchapter, focusing on Finland, constructing indicators is essential in the theory of change so that the change can be monitored – on the other hand, the theory also includes learning from the change. According to Serrat (2017, 241), applying the theory of change is a more conscious, reflective and creative visualisation exercise, rather than something that is completed at once. He notes that an evaluation based on the theory of change follows the Plan–Do–Check–Act cycle.

3.3 Snapshot of the evaluation debate in youth organisations in various Nordic countries

3.3.1 Finland

3.3.1.1 The organisation sector is waking up to the challenge to make impact visible

In recent years, the debate on promoting effects and impact evaluation of Finnish youth organisations has intensified as funding bodies wish to make efficient use of limited resources. Laitinen (2018, 170) also notes that reporting requirements have increased, and the availability of funding has tightened in general. The underlying causes include the interests of funding bodies, such as ministries, but making activities more efficient is also considered important in the organisation sector. A good example of this is the Hyvän mitta (Good Measure) project coordinated by the Finnish Association of Social Enterprises (2016–2019), where the impact chain application is still widely used as a basis for evaluation development work. The project’s main partners and funding bodies were Kela, We Foundation, the Finnish National Agency for Education, the Ministry of Education and Culture, Sitra, the Funding Centre for Social Welfare and Health Organisations (STEA) and the Prime Minister’s Office. The publication Vaikuttavuuden jäljillä – opas yhteiskunnallisen vaikuttavuuden kehittämiseen (Seeking social impact – a guide to developing societal impact), produced in the project, states that peer learning between organisations has played an important role in creating the impact chain mental model. For example, the theory of change discussed in the previous subchapter has also been used in impact chain development. (See Anoschkin 2019.)
According to the aforementioned publication Vaikuttavuuden jäljillä, the impact chain is a tool and a mental model that can be used to define the social impact of activities both with regard to individual actions and at different levels of the entire organisation. The chain consists of seven links: need, vision, goal, resources, measures, results and social impact. The parts are closely linked to each other during the working stage. Impact, need and vision describe the social level, while the other links describe the concrete action level. Since the links in the model have a cause-and-effect relationship with each other, every aspect of the chain is important. However, the importance of the beginning of the chain is particularly emphasised, i.e. the need and vision. The first step is to identify the social need and to come up with a vision for the activities derived from this need. Only after these steps the goals for the activities can be considered. At this stage, the aim should be to describe and evaluate the kind of concrete changes needed in the target group in order to achieve the vision. Then, the chain can be proceeded along to the stages where solutions and the required resources can be innovated. The impact chain thus creates a depiction of the desired change in a social phenomenon that the activity seeks to bring about. When data collection and the right indicators are combined with the above-mentioned concept, the social impact of the activity by utilising the model can be verified. (Ibid., 7.)
Two larger evaluation development trends can be identified from the results of the Hyvän mitta project. The first is the need for a more precise and broader definition of what society needs and by what types of actors, because the social vision that guides the activities of organisations would then also become clearer. The second trend that arises is the need for a common set of indicators for the organisation sector. (See Anoschkin 2019.) The latter work, in particular, is now being carried out in Kentauri, which is a youth work centre of expertise appointed by the Ministry of Education and Culture (see Fields 2021). The tasks of the Youth Work Centre of Expertise Kentauri are diverse, but one of the priorities is to develop common indicators to be used by youth organisations. The Ministry aimed to link the indicators to the themes of sustainable development. The primary focus is on the social sustainability perspective, for which age-specific inclusion indicators have been developed. (Group interview 15 February 2022, Kentauri.)
Kentauri also provides evaluation training in the organisation sector. In 2021, there were six separate social-impact-related training sessions that covered the impact chain and concepts, goals and indicators, the collection and analysis of evaluation data, the management and coordination of social impact and evaluation, social impact communication and utilisation of evaluation data. A representative of Kentauri states that the challenge identified in these training sessions was that the representatives of the organisations did not have enough time to carry out the development tasks that were part of the training, even though the webinars had enough participants and feedback was good. In the future, the way in which the educational process is offered should therefore be reorganised. (Group interview 15 February 2022.)
According to the representatives of the Finnish National Youth Council Allianssi and Kentauri, youth organisations have become increasingly interested in using the impact chain in recent years (Group interview 15 February 2022). One of its users is The Finnish Youth Association (Suomen Nuorisoseurat), which offers cultural hobbies, such as dance, theatre, circus, music and sports, for all ages. In the interview, a representative of the organisation says that the impact chain and its structure of activities are utilised in the organisation, firstly as a standard to ensure that the activities are implemented as planned in their respective stages. Secondly, the chain is considered a framework to ensure that all parts of the activities are in place and realised in relation to the whole. Overall, the organisation feels that utilising the impact chain has increased reflection on the goal orientation of the activities, which in turn is considered to be good for the development of the organisation. (The Finnish Youth Association, interview 8 April 2022.)
Similarly to many other organisations, the Finnish Youth Association also monitors the numbers of members, participants, participations, events and gatherings, among other things. In addition, they have started developing their own method of impact evaluation with determination, using both quantitative and qualitative indicators. In their own development work, the Finnish Youth Association combines elements from two main trends: the association cooperates with organisations such as Kentauri in developing inclusion indicators for the youth organisation activities mentioned above, and with the adult education institution Citizens’ Forum with regard to raising awareness of the impact of young people’s cultural activities. In connection with the evaluation of the impact of cultural activities, a questionnaire was created for the organisation to identify the impact of the activities on the participants’ physical, mental and social well-being, as well as on learning and cooperation.
The background material used in the development work of the Citizen's Forum is the WHO report on the role of the arts and culture in improving health and well-being (Fancourt & Finn 2019), in Finnish see also Taikusydän 2021; e-mail information provided by H. Pulkkinen Citizen’s Forum 15 August 2022.
The organisation has aimed to include both development work projects in the impact surveys that the organisation provides to its stakeholders on the Pokka platform
The Pokka assessment tool allows you to compare your own results with those of other operators in the same field, or of those using the same form of activity. It provides ready-made, customisable questionnaires that enable an organisation to collect information on the performance, impact and development needs of its activities. Kukunori is responsible for the Pokka assessment tool and Vertics for the technical implementation. Source: Pokka (kukunori.fi)
. (The Finnish Youth Association, interview 8 April 2022.)
According to the assessment of an Allianssi representative, evaluation support and assistance are available to the organisations, but it is debatable whether the organisations have the resources to receive these services. As an umbrella organisation, Allianssi aims to encourage organisations to acquire comprehensive evaluation training. Meanwhile, an organisation may only engage in brief sparring sessions that address specific issues, but do not form a picture of the activities of the organisation as a whole. For example, an organisation’s values may be well thought out and reflected in its strategy, but at the same time, they may not yet be reflected in its evaluation. (Group interview 15 February 2022, Allianssi.)
According to an Allianssi representative, particularly the open activities of organisations (such as clubs, camps and excursions) are still mostly reported as the number of activities, even though it is a known fact that these activities can, for example, reduce loneliness. This aspect is not mentioned in reporting. However, this type of visibility is already expected from the organisations. When it comes to developing the indicators, Allianssi considers it important that different youth organisations along with diverse young people of different ages are able to use them, and this way the resulting information would be comparable. (Group interview 15 February 2022, Allianssi.)
In an interview, the representatives of Kentauri and Allianssi emphasise the importance of evaluation in the organisation sector. However, the development of evaluation, indicators and tools takes time. There is an increased need for open discussion on how much can be expected of the civic society actors in terms of evaluating quality and social impact. The funding-related evaluation requirements are also connected to the issues of the autonomy and intrinsic value of civil society activities. On the other hand, it is understood that it is important to ensure that the funds allocated for civil society activities are also used for the right activities. A fear of a decrease in funding is often associated with the development of evaluation. The fact that this could be used as an opportunity to update operating models, which can be decades old, is not given the same attention in the debate. According to the representatives of Allianssi and Kentauri, common indicators could provide information on the joint effects and impact of the work carried out in the youth organisation sector. (Group interview 15 February 2022.)
In addition to the impact chain, there are many different evaluation methods in the Finnish organisation sector. Next, we will highlight the development work of the association Non Fighting Generation (henceforth NFG) concerning evaluation, and Sovari, which collects national evaluation data on outreach youth work and workshops. Both examples highlight the role of young people as the evaluators of activities.

3.3.1.2 Already more than ten years of systematic evaluation

Timo Purjo and Maria Tervahauta have written a study on the activities and results of the NFG over several years entitled Nuorten väkivallattomuuskasvatus - Tuloksia, vaikutuksia ja vaikuttavuutta (Education in Non-Violence for Young People - Results, Effects and Impact) (Purjo & Tervahauta 2014). Purjo is the founder of the organisation, and Tervahauta is the developer and director of the organisation. However, the study was more laborious and unpredictable than anticipated, and the first page of the study states that without grants allocated for evaluation, organisations can find it challenging to carry out a large-scale evaluation. This is due to the fact that, for example, a general grant is often insufficient to cover the necessary time and financial expenditure. The organisation in question operated for almost 20 years before writing a publication on the results, effects and impact of its activities. The thought of writing a publication was there, but the actual writing was long postponed not only due to the lack of allocated funding but also because the evaluation was not seen to be highly expected. The results of the activities of the organisation and its partners were known, and it was considered important to spend the time and use the received resources on the activities for young people (see T. Purjo 2014).
Non Fighting Generation is a national youth organisation that has been active since 1996, and at the core of its activities is the non-violence education of young people. The organisation cooperates with schools in particular, but also with other bodies, such as prisons and youth and social work institutions. One of the organisation’s key operating models is organising discussion groups for violent young people and young people whose symptoms emerge as bullying. (T. Purjo 2014). Today, the organisation also carries out extensive individual counselling for almost a hundred young people each year. Approximately half of the young people who receive individual counselling come through the child welfare outpatient services, while the rest are found through contacts in schools, the police and guardians.
The NFG systematically monitors and evaluates the success and impact of its activities (Tervahauta 2014, 38; T-L. Purjo, e-mail communication, 28 February 2022). The results and impact of the activities have been monitored for a long time through feedback collected from partners such as schools and child welfare services. (T. Purjo 2014, 27–30.) The participating young people, the instructors of the discussion groups and those who direct the young people to the discussion groups (especially schools and the social services) are currently asked to fill in evaluation forms electronically. Evaluations are often requested at the beginning and at the end of the activity, but in the case of long-term activities, interim feedback is also requested. (T-L. Purjo, e-mail communication, 28 February 2022.) The most important issue to be established is the change in the young person during the group activities and under their influence. This is evaluated, for example, through interviews with young people after group activities have ended, as well as through observations and reports by the instructors. (Tervahauta 2014, 38.) According to e-mail communication from NFG’s Executive Director Tuija-Leena Purjo, the response rate of the electronic evaluation forms is high, approximately half of the partners and up to 70% of the young people respond to the evaluations.
The organisation has also carried out long-term monitoring. In her article, Tervahauta (2014) examines the impact of activities based on interview material collected both soon after and no more than two years after the group activity. (Ibid. 46–50.) In addition to the evaluation studies and reports produced by the organisation, the organisation cooperates with students who are writing their theses and publishes studies on the organisation’s theoretical framework (Tervahauta 2014, 38; e-mail communication T-L. Purjo, 2022). The volume of demand has also been considered an indicator of the quality of the organisation’s activities. The level of demand is currently approximately 30% higher than what the organisation can meet with the resources provided by the funding bodies (T. Purjo 2014, 7; T-L. Purjo, e-mail communication, 28 February 2022).

3.3.1.3 Key figures and experimental knowledge of targeted youth work by organisation, region, and nationwide 

Unlike the organisations engaging in open youth work, the institutions organising outreach youth work or workshop activities, such as youth organisations, have been using a common evaluation system since 2016. The results of national Sovari surveys on workshop activities and outreach youth work published in 2021, namely ”Pystyn asioihin, joihin en uskonut pystyväni” Työpajatoiminnan valtakunnalliset Sovari 2020 -tulokset (“I can do things that I didn’t think I could” National Sovari Survey Results 2020 for Workshop Activities) and ”Tuntuu, että on pidetty huolta” Etsivän nuorisotyön valtakunnalliset Sovari 2020 - tulokset (“I feel looked after” National Sovari Survey Results 2020 for Outreach Youth Work), assess the impact of workshop activities and outreach youth work (Kinnunen 2021a; 2021b). Sovari is an indicator of social empowerment developed and maintained by Into – Association for Outreach Youth Work and Workshop Activities, which measures the quality of services and their socially empowering impact. The indicator has been developed in cooperation with the actors of workshop activities and outreach youth work, as well as with the customers. (Kinnunen 2021a, 3.) Its user base has since grown annually, and it has become an established practice in workshop activities (Kinnunen 2021a, 4; Kinnunen 2021b, 5). Sovari measures the success of the key goal of workshop activities and outreach youth work: the social empowerment of the customer.
In addition to Sovari, statistical data on social impact (PAR statistical system for tracking how customers are placed in education, employment or other services, for example) is used in workshop activities and in outreach youth work. In her report, Kinnunen highlights the benefits of using the indicator on a large scale: uniform and nationally comparable key figures and data on the social impact. The Sovari indicator presents the data that can be used to demonstrate the impact of services on customers’ life situations and, thus, highlight the importance of and the need for services in the service system. The information obtained from the indicator can also be used to develop services. (Kinnunen 2021a, 3.)
In Sovari, the quality and impact of outreach youth work and workshop activities are measured in two different anonymous online surveys. In the first evaluation section of workshop activities, the coached assess the implementation and quality of the workshop activities in three key areas: 1) work tasks and participation, 2) communality, and 3) guidance and support provided in work and individual coaching. In the evaluation section of outreach youth work, the following areas are included: 1) the accessibility of the service, 2) the confidentiality of the service and the fluidity of the dialogue between the young person and the actors, and 3) the help and support received from the service in promoting the life situation and finding a suitable path for the young person. In the second section, both workshop activities and outreach youth work surveys assess the perceived changes in social empowerment in five areas: 1) self-knowledge, 2) social skills, 3) life management, 4) everyday life management, and 5) study and working life skills. According to Kinnunen, the surveys also provide a channel that promotes the participation of young people and the coached. In the surveys, young people and coached individuals can express their own views on the implementation of outreach youth work and workshop activities, and thus influence the development of the activities. (Kinnunen 2021a, 3–4; Kinnunen 2021b, 3.)
According to Kinnunen (2021a), Sovari produces key figures and compiles experience data very extensively, both on an organisational, regional and national level. The outreach youth work or workshop activity organisers are not responsible for producing information alone, since Into, as the centre of expertise in the youth sector appointed by the Ministry of Education and Culture, draws up annual summaries by region, coordination area or organisation, in addition to national reports. (Aforementioned study 3–4.)

3.3.1.4 Not everything can be measured through causality, but this does not mean that the importance of developing evaluation skills should be compromised

In 2022, the Ministry of Education and Culture will allocate approximately EUR 18.5 million in state aid to national youth organisations (Ministry of Education and Culture, national youth sector organisations n.d.). To qualify for state aid, an organisation must first be approved as eligible for such aid. When applying for eligibility for state aid, the nationwide coverage, quality, scope and social impact of the activities are assessed, as well as how the organisation promotes non-discrimination, equality and social inclusion of young people. The conditions for eligibility for state aid, as well as the criteria for the distribution of the aid, are laid down in the Youth Act and the Government Decree on Youth Work and Policy. The criteria for the distribution of grants are the quality, scope, social impact, financial management and the promotion of the inclusion of young people (Ministry of Education and Culture, acts and decrees: Youth n.d.).
In their interview, the representatives of the Ministry of Education and Culture and the Assessment and State Aid Commission operating under the Ministry note that the Ministry evaluates the social impact of organisations based on the organisations’ self-evaluation of the relevance of their activities both in youth work and policy, as well as in the sector of the organisation. The impact evaluation is also based on the Ministry’s evaluation on the significance of the organisation’s activities in relation to those of other youth sector organisations, as well as on what the impact of the organisation’s activities are on young people and the sector itself. With regard to promoting inclusion, organisations are invited to specify the opportunities they offer to young people to promote their participation in decision-making and how the organisations contribute to young people’s ability and capacity to function in society in other ways. Organisations may set both annual and long-term objectives for their activities. They can target a limited number of people or be more relevant to society as a whole. On the basis of the value system of its own activities, an organisation can largely determine which matters it monitors when evaluating its own activities, what results are expected and what data is used to report the outcomes. However, the same quantitative activity data is collected from all organisations. (Group interview 24 March 2022.)
Based on the same interview, the lack of systematic evaluation in organisations in relation to their own activities may be reflected, for example, in the fact that organisations ask their members for feedback, but ultimately do not channel it into the development of their activities. However, it is evident that the level of planning and evaluation of the organisations’ activities is constantly improving. This development may be the result of rating the organisations’ actual and planned activities in accordance with previously determined criteria and presenting the rating results publicly on the Ministry’s website, and the fact that the Youth Work Centre of Expertise Kentauri has been given the task of aiding organisations in developing their evaluation systems. The Ministry also provides seminars for organisations on topical evaluation themes, such as sustainable development. The Ministry also commissions a certain number of theses or internships in relation to funding organisational activities. Currently, a thesis on the subject of the quality of non-discrimination and equality planning in organisations is being carried out. (Group interview 24 March 2022.)
Based on the same interview, the Ministry does not oblige organisations to use any particular method of evaluation as the basis for the documentation of their activities. In general, the quantitative documentation of activities is carried out more systematically in organisations than the evaluation of achieving the set objectives. According to the representatives of the Ministry and its Assessment and State Aid Commission, the impact evaluation in organisational work is still in its early stages of development. On the other hand, the Ministry is also aware that the connection between successful organisational activities and, for example, the development of the growth and living conditions of young people in the wider society can never be fully “causally reported”. (Group interview 24 March 2022.)

3.3.2 Sweden

3.3.2.1 Challenging the evaluation criteria established by the funding bodies

In Sweden, Fryshuset, which operates in more than 60 localities in the country
Fryshuset has projects and activity houses of different sizes in Gothenburg, Hammarby Sjöstad, Helsingborg, Husby, Malmö, Nacka, Nybro, Torsås and Östra Göinge.
, was the main partner in this project. Fryshuset operates in four areas: Work and entrepreneurship, Role models & belief in the future, Schools and Youth culture. Fryshuset is also active in other countries, for instance in Norway, Cuba, the Netherlands, Armenia and Denmark. The activities are funded, for example, by donations from foundations and companies, as well as by public funding from cities and the state. Fryshuset’s fundamental principle is to base the activities on the needs of young people. The strategy is to first find out how the young people are doing and what they need, and then to apply for funding for this activity. Fryshuset publishes plenty of literature about needs, requirements, and project results. (Fryshuset, interview, 25 March 2022.)
In Fryshuset’s view, evaluation requirements have tightened in the last five years as a condition for obtaining funding, for example, from companies. The evaluation must be planned, and the intended impact and the type of evaluation data used to verify this impact must be identified in advance. In the past, funding bodies were more interested in the numbers of young people, for example. Since 2019, the organisations’ personnel have been instructed to use an evaluation method based on the Theory/Theories of Change. The theory of change is commonly used in Sweden as the basis for organisations’ evaluation, and its use is part of the objective of Giva Sverige (the Swedish Fundraising Association), which guarantees the ethical nature of fundraising and requires its organisations to follow the evaluation principles (Giva Sverige 2020). In practice, this means, for example, that each organisation describes its own theory of change as part of its impact report, which is linked to Giva’s website every year. Giva’s member organisations include organisations for fighting cancer, animal welfare, the youth sector and elderly services. The impact reports of all member organisations can be found on the Giva membership page (Giva Sverige, members n.d.; see e.g., Fryshuset 2020; the same pages also contain the impact reports of almost all the organisations presented next in this report: Svenska Röda Korset 2021; Raoul Wallenberg Academy 2021; Mind 2021; Tjejzonen 2021).
Fryshuset is part of a committee in which organisations, municipalities, various research and development organisations and companies work together to develop evaluation systems, with the aim of standardising the indicators for measuring impact (see Svenska institutet för standarder n.d.). According to Fryshuset, the committee aims to provide actors and funding bodies with information on the types measurements and terminology that can be used. This work is still in progress and will take at least one more year to complete. (Fryshuset, interview, 25 March 2022.)
Fryshuset aims to carry out evaluations at an appropriate level. If a project aims to bring about major societal changes, this must also be made visible. Fryshuset’s publication Civilsamhallet i Orten (Nilsson Lundmark & Nilsson 2020) describes how projects seeking change aim not only to solve a specific problem quickly at an operational level, but also to develop an experimental and learning approach with confidence-building measures in cooperation with the participants as well as any other organisations involved. For example, when looking for a solution to avoid young people’s exclusion from education and employment, the sense of pride and hope among young people living in a particular neighbourhood could be increased by paying attention to the structural aspects that affect the atmosphere and culture of the region. One of the expected outcomes will measure the future growth of young people’s faith and highlight the steps that helped achieve this goal. In terms of changes on a structural level, the planning and evaluation of activities will above all monitor the need for Fryshuset’s activities in a field that is not covered by other actors in the area. Secondly, the publication highlights the steps taken to ensure that the impact also reaches the strategic level that will bring about the necessary structural changes. (Nilsson Lundmark & Nilsson 2020, 9, 21, 33–51.)
From Fryshuset’s point of view, the challenges of evaluation are related to the fact that it is sometimes impossible or at least difficult to ask the participants and organisers how they feel, because responding to surveys is time-consuming, and not everyone is able or willing to respond. On the other hand, the fact that the same people tend to participate in Fryshuset’s activities again and again facilitates evaluation. This shows that the young people usually feel safe with the instructors and in the situations, which again facilitates discussion. It is also worth noting that a lengthy questionnaire in itself may reduce the motivation to respond. In order to develop and evaluate the activities, a wide range of information is needed from the participants. However, there is no need to ask all the questions at once. Fryshuset has asked about different issues at different times, for example, information on the satisfaction or self-esteem of young people is collected for evaluation in certain months. The respondents are not always the same, and sometimes the questions are directed at everyone present or only certain groups. Fryshuset also considers the flexibility of evaluation to be an important factor. According to Fryshuset, funding bodies should accept that evaluation is adapted to the nature of the activity. Primarily, evaluation should help the organisation itself to function better and not serve the informed interests of the funding bodies. After all, young people are the most important resource in the activities, and Fryshuset is grateful for the fact that they still want to join in on the activities in such large numbers and state what activities are needed. (Fryshuset, interview, 25 March 2022.)

3.3.2.2 Evaluation cannot be developed only from the outside

The Swedish Red Cross Youth (Svenska Röda Korsets Ungdomsförbund) organises activities for and with young people on a broad scale, such as groups for children’s schoolwork, discussions on children’s rights, supervision of interests and reaching out to marginalised young people. As a representative of the organisation says in an interview, the impact also varies from an individual to a national level.  In 2021, a new, strategically planned addition was introduced: the organisation will increase its physical presence and organise activities in community neighbourhoods in different parts of the country. The organisation finds that funding bodies have higher expectations when it comes to evaluation. The funding bodies can define what should be measured, but not how the information is collected. They are interested in the impact, but this is more reflected in the application phase as a prediction requirement, rather than in the detailed reporting on the use of funding. In an organisation that has been operating for a long time, predicting the impact is considered easy in the sense that the operating methods used have been proven effective over the years. (Swedish Red Cross Youth, interview 27 April 2022b.)
Every six months, the Swedish Red Cross Youth evaluates what needs to be developed, and this can be considered the largest evaluation process. An external evaluation is also carried out every few years, asking both staff and volunteers how well the objectives have been achieved. Participants in voluntary activities also play an important role in the evaluation, as they are also regularly asked for their views on what things a given activity may have introduced to the participants’ lives. The most important future challenges in evaluation are linked to how the organisation can better establish the participants’ own perspective. If many participants take part in workshop activities at the same time, they are asked questions via a questionnaire. However, if the number of participants in some activities is small, qualitative data collection methods are used. On the other hand, the organisation does not have the capacity to evaluate all its activities. The organisation feels that, in the future, more funding will become available as more organisations participate in evaluation. (Swedish Red Cross, interview 27 April 2022b.)
The view of the Swedish Red Cross Youth is that different types of organisations cannot apply the same evaluation solutions, because the target groups, operating methods and sectors of the organisations are different. Furthermore, evaluation cannot be developed entirely from outside, as precise knowledge of each organisation and its operating environment is required. The Swedish Red Cross Youth feels that evaluation is so important and useful that the organisation would benefit from more extensive evaluation. However, this is likely to take up too many resources. (Swedish Red Cross Youth, interview 27 April 2022b.)

3.3.2.3 You cannot put a price on changes in the value base and in thoughts

Since 2001, the Raoul Wallenberg Academy has been working with young people and teachers to combat injustice and inequality, following the example of Raoul Wallenberg. The methods include a leadership programme for young people, various school programmes, teacher training and material production. The activities of the Academy relate to the internalisation of values related to human rights and democracy and the formation of knowledge with young people on how they can obtain empowered positions and influence the world at any given time. As in the case of youth organisations in general, the challenge for evaluation is to verify the wider and longer-term impact of the activities.  The activities of the organisation are not aimed, for example, at reducing the costs to society, like the fight against drug use in many organisations, for example, can be seen as such an activity to some extent. Instead, they seek  to highlight a change in values and views, despite the fact that some of the funding bodies still seem to be more interested in, for example, the number of organised activities and encounters. The interviewee points out that this type of attitude towards evaluation can mean that the evaluation does not take into account how young people feel about the activities and encounters, and whether they are good or bad. Organisations do not only compete for funding from foundations, for example, but also individuals who donate will compare the organisations’ impact reports, as they hope that the donated funds are used as effectively as possible
An example of the Swedish debate on so-called effective altruism, i.e. effective selflessness (see Anderljung 2018). (Raoul Wallenberg Academy, interview 13 May 2022.) 

3.3.2.4 Adding value in relation to the Sustainable Development Goals of the United Nations

The Mind organisation promotes reading evaluation literature and learning about evaluation from other organisations. At the same time, they want to maintain a strong ownership to the manner in which evaluation is assessed and how evaluation is guided. The measured subject provides evidence of social impacts that can then be reported further. The practical evaluation work also provides a learning experience in interpreting the indicators and their functions. This provides information on how the matter can be managed and handled in the future. According to the organisation, non-profit activities are becoming more professional as evaluation develops, meaning there is more paid workforce, and the realisation of values can be verified more efficiently. Of course, there is also a fear of losing the organisation’s identity or that evaluation is only done for the purpose of obtaining funding and controlling its use, not so that it would benefit the target group of the activity or become a learning experience. In Mind, the structure of evaluation is based on the theory of change, which has been polished with the help of the value chain thinking (in Swedish, värdeskapandekedjan) (see Hanh & Hök & Jannesson 2016). According to value chain thinking, it is important to keep the objectives of the activity in perspective and make them visible, for example, in the context of the United Nations Sustainable Development Goals (see United Nations n.d.). (Mind, interview 27 April 2022a.)
One of the most important services offered by Mind is the Suicide Help Line, which anyone who needs help can contact by phone or chat. In 2019, an evaluation process related to the chat contacts was conducted. There were 27 000 chat contacts, of which 4 500 were made by young people under the age of 24. It should also be noted that 135 of these were made by children aged between 7 and 12. The person who contacted the service is asked to participate in the evaluation of the service at the end of the discussion. With a ready-made list of questions, the person is asked what they gained from the discussion and what they would have done if the discussion had not taken place. The challenge with evaluation is that it is often difficult to motivate both the people who feel the weakest and those who feel strongest to participate in the evaluation. Evaluation participants will always have freedom of speech, and they can exercise it as they wish. The organisation points out that evaluations should always be planned to suit the target group. It is impossible to ask about everything, especially from those in vulnerable situations. The evaluations should be limited so that also this part of the discussion benefits the users of the service. (Mind, interview 27 April 2022a.)
The volunteers who participate in the discussion also take part in the evaluation, assessing, among other things, whether they feel they were able to help those in need. Discussing the extent to which it is possible to help helps both the organisation to develop their service and the volunteers themselves to cope at work. The activities may have been correct and the call successful, even if a change was not achieved or visible. And even if an impact was made, it is important to keep in mind that people who contact the services often have other contacts as well. This means that any change that occurs may not be attributed to a specific organisation. Mind’s activities have significant value at least in forming an additional channel for discussion, and Mind works in cooperation with other actors in building a safety net. (Mind, interview 27 April 2022a.)

3.3.2.5 There are several evaluation methods and finding the best one for one’s activities may require a lot of effort

In 2018, Mötesplats Social Innovation, or the MSI, published a study entitled Alla pratar om det men få gör det – en handbok i effektmätning (“Everyone talks about it, but few people do it – impact measuring manual”), which discusses social impact evaluation models and the actors who have helped to develop them and keep developing them for the needs of companies and organisations. The study is a guidebook that presents perspectives on measuring social impact and the meaning, backgrounds and forms of measurement. The aim of the guide is to decode the impact measurement process to the reader step by step. Editor of the guide, Erika Augustinsson, mentions the same phenomenon in the preface to the guide that we have encountered in our discussions with the interviewees: impact measurements are a good way to clarify and strengthen the organisation’s own goals and to communicate the impact and development of the work to the funding bodies. At the same time, impact measurement requires resources and expertise, which is why this may be difficult to implement alongside the main activity. (Augustinsson 2018, 5.)
The MSI is an expertise and cooperation platform established by the University of Malmö that combines social innovation and social entrepreneurship. Its partners include several universities and the City of Malmö. The objective of the MSI is to build innovation capacity with higher education institutions, companies as well as public and third sector actors, and thus improve opportunities to face today’s societal challenges. (Augustinsson 2018.)
The guide reviews various methods of impact evaluation, such as the Pestel analysis (analysis of political, economic, social, technological, environmental and legal factors) for evaluating business environments, and SWOT (analysis of strengths, weaknesses, opportunities and threats). In evaluation that is based on the theory of change, it is recommended to measure the initial situation of the activity, the number of activities, the results and social impact, and use these measurements in the verification. The measuring methods can be surveys or interviews, depending on the nature of the activity and the target group. The guide emphasises the preparedness to modify the questions used based on previous measurements and experience gained from the activities. In addition, the guide emphasises the important role of reflection in determining the impact in relation to the baseline and the data collected during the activity. (Augustinsson 2018, 13–29.)
The MSI guide presents practical examples of impact measuring and actors that provide services to help companies and organisations with issues related to quality development and impact evaluation. Case examples include Tjejzonen, whose current evaluation model was created in cooperation with the Social Initiative organisation (see also Prine 2018).
Tjejzonen is a non-profit organisation based in Stockholm, whose mission is to provide safe spaces for girls between the ages of 15 and 25 to meet up and interact, thus supporting the psychological well-being of girls and young women. The most important services of the Tjejzonen organisation are an online 24/7 chat, and big sister and little sister activities. (Larsson et al., 2016.) Efforts are constantly being made to develop activities in interaction with the reference group and based on their needs (Tjejzonen 2016). In 2011–2014, the University of Örebro conducted a research project on Tjejzonen’s sex-oriented preventive substance abuse work, which was carried out in the form of sister activities for girls.  Among other things, the study looked at the importance of the sister activities for the girls and volunteers and the impact of the activities on them. Data was collected in the form of surveys and interviews from the young people and volunteers who participated in the activities over several years. The challenge in the longitudinal study was reaching the participants. The initial survey was completed by 109 participants in the role of little sisters, 17 of whom chose to respond anonymously, meaning that further questionnaires could not be sent to them. The response rate was low in the second and third survey, and therefore, no general conclusions could be drawn. (Larsson et al., 2016.) After the research project, Tjejzonen has continued to develop its own impact evaluation tools based on the theory of change since 2015 (Tjejzonen 2018).
An example of the evaluation challenges mentioned in the MSI guide are the special features of the sister activities organised by Tjejzonen, which challenge the organisation in terms of monitoring activities and evaluating impact. For many young people, anonymous participation and the strict protection of privacy are prerequisites for them to seek out activities, which is why it can be challenging to carry out comprehensive monitoring and impact studies (Tjejzonen 2016). Currently, the impact of the organisation is measured by the scope of its activities, which is based on the number of interactions and chat conversations that take place online and in close contact, as well as voluntary satisfaction surveys that participants can respond to if they wish (Tjejzonen 2020). Tjejzonen reported that evaluation and the ways to actively study the impact of activities are still being developed (e-mail communication, 17 January 2022.)

3.3.2.6 In addition to developing evaluations, other development actions may also be necessary

An interviewee from the Green Students (Gröna Studenter) describes a situation where the main development needs are related to increasing meagre activities, rather than considering evaluation. The organisation receives funding from the party in accordance with a certain process proportion, although there has been pressure to reduce this share. In election years, the activities are largely related to the elections. At other times, the activities are more extensive, including familiarisation with decision-making, taking initiatives in connection with the party policy, and enabling members’ empowerment opportunities in different ways. The organisation points out that the results of the activities are highly contextualised, and that when operating with limited resources, even the smallest result may be significant. When assessing the volume of activities, it should be taken into account that the attractiveness of parties and charities as a source of funding can vary considerably over time, for example due to political cycles or public debate, and the amount of funding affects the number of activities. (Green Students, interview 13 May 2022b.)

3.3.2.7 Youth organisations’ experimental knowledge is taken seriously in authority-driven development of impact evaluation instruments 

The Myndigheten för ungdoms- och civilsamhällesfrågor (MUCF) (Swedish Agency for Youth and Civil Society) is an authority that aims to ensure that the government’s youth and civil society policy objectives are met. These statutory objectives may include making a positive contribution to the agency of young people or improving the quality of services. Due to the mandate given by the government, the impact of MUCF funding is assessed in more detail. Impact evaluation is developed by conducting interviews in organisations that have received funding from the agency with the aim of gathering views on how the impacts of certain activities can contribute to achieving broader objectives
These interviews have been reported, for example, in a publication entitled Organisationsbidragens användning och effekter 2021. Återrapportering av tre statsbidrag till ideella organisationer (Löving 2021).
. (MUCF, interview 28 April 2022.)
The aim of the agency's evaluation development work is to find an evaluation method that would allow the comparison of different activities by going beyond, combining and challenging existing methods of evaluation. The task is described as challenging, in the sense that the agency allocates several types of funding. They offer aid for projects working to combat racism, and organisations working with different age, language and cultural groups, such as youth organisations. The financed activities are based on a wide range of laws that describe the objectives to be pursued. It is also challenging to show how short funded periods can result in long-term effects, especially when organisations, activities and society are constantly changing. (MUCF, interview 28 April 2022.)

3.3.3 Norway

3.3.3.1 Keeping civil society vital

The Norwegian Children and Youth Council (Landsrådet for Norges barne- og ungdomsorganisasjoner, LNU) is the umbrella organisation of 102 organisations working with children and young people. As in Denmark, the umbrella organisation in Norway is also involved in distributing state funding to its member organisations. In practice, funding for youth organisations can mainly be obtained from two sources. The national operating grant consists of an annual basic grant determined by the Ministry and a changing rating-based grant. The financial proposals are submitted by the Fordelingsutvalget (Allocation Committee), an administrative body under the Norwegian Ministry of Children and Families. It is composed of seven members appointed by the Ministry, three of whom are elected by the organisations and at least two of whom must be appointed by the LNU, the umbrella organisation for children and youth organisations. Of course, some of the organisations also apply for funding directly from ministries. (See Fordelingsutvalget for tilskudd til frivillige barne- og ungdomsorganisasjoner. n.d.; LNU, interview 21 March 2022.)
Youth organisations (excluding, for example, sports organisations with their own committees), which work nationwide with children and young people, with issues affecting them, and which involve children and young people, can apply for national basic grants. The organisations must also meet other funding criteria, including accessibility and non-discrimination, clearly recorded rules, budgets, financial statements and annual reports for both central and local organisations, a democratic structure allowing participation of all persons aged 15 and over, national elections held at least every four years, a central membership register and a minimum of 700 members. There must be at least five local organisations. In the case of smaller organisations, the membership conditions are more flexible, and organisations may also apply for a separate grant to cover the start-up costs for the organisation. (Forskrift. n.d.; see also report Fordelingsutvalget 2019.)
Significant funding that the LNU provides in the field concerns various free activity groups, which have the possibility to apply for a grant from Frifond, a fund established by the Norwegian Parliament in 2000, which channels the surplus generated by gambling. Frifond funding is intended to enable children and young people to carry out independent projects and activities, both in the form of operating grants and project grants. Frifond project grants are available for theatre and music projects, as well as projects organised by children and young people. Music grants are managed by Norsk musikkråd and include a restart grant to revive children and young people’s music groups and activities after the pandemic. The theatre grant, as well as the grants for children and young people, are coordinated by the umbrella organisation for youth organisations LNU. Frifond funding is advertised as the easiest and least bureaucratic application process in the country. (LNU, støtteordninger. n.d.; Frifond, om Frifond. n.d.; Frifond, om Frifond-ordningene. n.d.; Frifond, retningslinjer for støtte. n.d.; Norsk musikkråd, frifond. n.d.; LNU, interview 21 March 2022.)
In Norway, the central organisation plays an active role in supervising the interests of the youth organisation sector when it comes to the public administration deciding on the funding criteria. According to an LNU representative, there is no requirement to actively itemise the impact of the activities in regard to the allocation of funding and reporting, because the activities of many organisations, such as student and political organisations, include activities that require years, even decades, of work. Implementation is affected by the interests of the decision-makers, the government’s relative strength and the schedules. In addition, there can always be aspects of learning and success through error. The LNU believes that evaluating which organisation makes the biggest impact is a slippery slope. It is difficult to predict and report an achievement. It is also difficult to interpret whether absolute causality can be established between what has been done and what can be measured. If impact measurement is emphasised, institutionalised and large organisations that have the resources to develop measuring activities may gain the upper hand. (LNU, interview 21 March 2022.)
In the LNU, the main focus of evaluation is on the democratic management of activities that are organised through funding. With this in mind, the main questions of evaluation are as follows: How are choices made between different activity options, are young people involved in the decision making, how satisfied are the members, and are the organisational culture and practices regularly evaluated? The LNU also helps ensure that the criteria for the funding and the application and reporting forms are kept simple, so that organisations of all levels can apply for funding, and that the applicants do not have to spend too much of their resources on the application. According to the LNU, preserving the diversity of civil society has an intrinsic value, as does working together in civil society, as well as influencing fellow human beings and creating a democratic society. (LNU, interview 21 March 2022.)
The following examples from Norway discuss the challenges of evaluation in terms of improving the world, the possibilities to justify the need for activities through scientific data, and developing accessibility and attractiveness and how well the activities match the needs of young people.

3.3.3.2 How to evaluate whether our actions have resulted in a decrease in inequality

Changemaker Norway works to remedy the injustice in the world through constant meetings with politicians, lobbying, maintaining press relations and organising demonstrations, school visits, environmental projects, weekend meetings and leadership studies. The evaluation is kept as diverse as the activities.  There is a social media campaign analysis about how the activation of young people went.  Or they focus on how the knowledge of young people was improved through education and training activities. Or they can verify how they manage to deal with local events.
According to the organisation’s representative, the debate on how to highlight the impact of the activities is not yet very advanced. However, there is one concrete example. When international taxation was discussed in the government, the organisation received feedback stating that without their activities, some of the changes would not have been achieved. The organisation has been in operation for 30 years, and it is possible that some of the things that were done 20 years ago are only now starting to have an effect. (Changemaker Norway, interview 6 May 2022.)
The interviewed representative of the organisation notes that it is difficult to see the change in the policy or in children and young people. According to the representative, it must be kept in mind that even the smallest contact with a 14 or 15-year-old can be the start of a significant interaction. Young people do not automatically accept everyone as an interlocutor. This position must be earned by one’s own actions from the first encounter. (Changemaker Norway, interview 6 May 2022.)
Changemaker Norway expects evaluation to develop. One of the challenges is that not many of the young people involved are members of the organisation and are only involved for a short period of time. Young people are interested about the activities, not their evaluation. There would be no evaluation culture without the organisation’s employees. The quarterly discussion with the board of the organisation, which evaluates how things are going and what more is needed, plays an important role. Funding comes from a number of different sources, and different parties request reporting in different ways. The organisation is fortunate in that the main funding bodies have a clear idea of what can be expected from the activities of this type of organisation, and how they can be evaluated and reported. (Changemaker Norway, interview 6 May 2022.)

3.3.3.3 No need to measure everything yourself if there are other trustworthy bodies offering a basis on which to build

The organisation The Psoriasis and Eczema Youth Association of Norway (Psoriasis- og eksemforbundet Ung i Norge, hereinafter PEF-youth) recognises the importance of meeting other people of the same age who are in the same situation, for example, with a specific disease-related condition. It is important to participate in activities that allow the feeling of shame and stigma to subside. However, it is almost impossible to measure the improvement in self-esteem as a result of a particular activity. For this reason, the organisation in question has found the section “Background information” particularly important when making funding applications. In this section, an organisation can refer to the results of various research groups, according to which people with a specific condition can easily suffer from issues, such as mental health problems and fears, which can be prevented, for example, by having friends and an atmosphere of trust in which one can speak and peers can meet up. It may not be as easy to talk to other peers who may not truly understand the issue. The organisation builds up its expertise on exactly what its members need and its ability to organise it for them. Funding applications must highlight the needs, translate them into objectives and determine how they will be achieved. The organisation itself does not have the possibility to carry out in-depth scientific research, and it is therefore good that there are reliable bodies whose research can be referred to. (PEF-youth, interview 2 June 2022.)
Although the PEF-youth has not yet attempted to make a comprehensive impact evaluation, they are still carrying out evaluations. After the meetings, the participants are asked whether they are satisfied with the organisation of the meetings and, for example, whether the visitors (such as doctors specialising in dermatology or rheumatology etc. or physiotherapists) highlighted the chosen topics clearly enough and if the participants found the topics relevant. When implementing projects, the organisation evaluated the outcome of the activity together with the participants, such as comics about specific conditions and whether they are satisfied. When the completed works, such as comics, are published on social media, visibility is created, which can contribute to raising awareness. In this case, for example, the number of clicks the comics have received can be documented. The organisation also received qualitative feedback on the cartoons from professionals, for example, who praised the cartoon as appropriately informative, taking into account the target groups. The professionals also stated that they would be happy to share it with children and young people with the specific condition. (PEF-youth, interview 2 June 2022.)
According to a PEF-youth representative, it is possible that the increased requirements for evaluation will increase the professionalisation of activities in a way that will distance them further from grassroots activities. The PEF-youth considers that the funding applications offered by different entities have very different standards, some of which are very difficult to meet. According to the interviewed PEF representative, you can get the wrong impression of the activities in the sector by reading funding applications and lists of the applicants, because some organisations cannot apply for certain funding due to a lack of resources. The PEF-youth has managed to find the funding channels that are accessible to them. These funding bodies have the previously mentioned “Background information” section in their application forms, where research data produced by other bodies on the needs of the organisation’s members can be added. The interviewee states that without this section, it would be quite impossible to submit an application. (PEF-youth, interview 2 June 2022.)

3.3.3.4 Cumulating knowledge related to those who participate and how to help engage those who do not 

In their research report Samfunnsengasjert ungdom. Deltakelse i politikk og organisasjonsliv blant unge i Oslo, published in 2017, Guro Ødegård and Adun Fladmoe examine the participation of young people in voluntary work and political activities in Oslo. The purpose of the study was to find out how many young people participate in the activities of organisations or political youth organisations, and whether there have been any changes over the past twenty years. In addition, the study aimed to identify the differences in participation between different genders and between young people from different ethnic and socio-economic backgrounds. The variables taken into account in the comparison were, for example, ethnic background, residential area, socio-economic status and the organisational activity of the parents. In addition, the study examined whether the same young people are always involved in the activities of organisations and political activities, what kind of activities are perceived as attractive and whether some groups are excluded. (Ødegård & Fladmoe 2017.)
This study uses the results of the Ung i Oslo 2015 survey, to which more than 24 000 lower and upper secondary school students (aged 13–19) from the Oslo region responded. The data from 2015 was compared with the data from 2006 and 2012 to see the development in different decades. Around two-thirds (65%) of all respondents were involved in the activities of at least one organisation. Among the young people, sports organisations were the most popular. This sector also displayed the greatest differences in the minority and socio-economic backgrounds of young people. Girls from minorities were under-represented in the organisations. The lowest participation rates for these girls were in the activities of sports organisations. The children of parents who were active in organisations were most actively involved, while the differences in political participation between minority and majority youth were not significant. On the basis of the data, young people appear to be at least as active, if not slightly more active, in the organisation sector than they were 10–20 years ago. The study was published by the Centre for Research on Civil Society and Voluntary Sector (Senter for forskning på sivilsamfunn og frivillig sektor). (Ødegård & Fladmoe 2017.)
In the same year, the same research centre also published another interesting publication related to the theme: Norsk frivillighet: Utviklingstrender og samfunnseffekter (Kraglund & Enjolras 2017). It examines the participation of minorities in civil society, how organisations work, what challenges they face in their work, and how cooperation with municipal authorities is perceived. The publication states that the social impact of organisation and voluntary activities can be diverse: direct or indirect, short-term or long-term, affecting individuals, local communities and society at large. Impact on public health, integration work, emergency operations and preventive work against radicalisation can be used as examples. According to the publication, people with a migrant background in Norway are relatively well integrated into the labour market and education compared to other OECD countries. However, when compared to the majority of the population, the involvement of minority representatives in civil society is more limited. The publication notes how organisation and volunteer work can create and strengthen networks, create a culture of reciprocity and increase social inclusion, which would be of benefit to immigrants in particular. (Kraglund & Enjolras 2017, 63–65.)
The same publication Norsk frivillighet: Utviklingstrender og samfunnseffekter also looks at the longitudinal survey from 2012 and 2015. An article originally published in 2012, Rekruttering av barn og unge till frivillige organisasjoner. Barrierer, tilltak og institusjonelt samarbeid, the starting point of the study, examined youth participation in organisation activities in four Norwegian cities (Ødegård, Steen-Johnsen and Ravneberg 2012). The article highlights the obstacles young people have in participating in organisation activities on an individual, organisational and social level, and how the organisations have removed the obstacles. Particular attention was drawn to the participation of young people from ethnic minorities, who are less likely to participate in youth organisation activities, such as sports and leisure organisations, than the rest of the population. In addition to documentary analyses, material was collected by interviewing adult representatives of schools and sports clubs. In the study, sport organisations were considered to have the lowest threshold for participation as they do not promote religious or political agendas in their activities that could prevent young people from immigrant backgrounds from participating. The study found that, for example, girls belonging to ethnic minorities had a particularly low level of participation and often quit their hobbies at the beginning of puberty. At the individual level, obstacles to participation included socio-economic resources and challenging family situations, poor Norwegian language skills, and inexperience of local organisational culture, and religious and/or cultural practices. At the organisational level, for example, the lack of volunteers and the continuity of activities were challenging. (Ødegård et al. 2012, 88.)
The article also reviews the strategies and measures taken to involve young people in these organisations. Measures in different areas included communicating with young people’s parents orally and face-to-face instead in writing, involving young people in the planning of activities, providing role models, diversifying activities, and cooperating with municipal actors and organisations in relation to communication and reaching out to young people. The article states that clearer coordination of the recruitment and involvement of young people and setting long-term goals from these perspectives are required to involve young people in the activities. (Ødegård et al. 2012, 90–93.)
In a further study published in 2017, Levende drabantbyer: Ungdoms deltakelse i organiserte fritidsaktiviteter i flerkulturelle lokalsamfunn, Ingunn Eriksen and Lars Frøyland examined the participation of young people in leisure activities. One area of interest in the study was the extent to which young people participate in leisure activities. The study also examined the means and strategies used by organisations to recruit young people into their leisure activities. The same municipalities were examined as in the previous study. They all share a relatively large proportion of residents with an immigrant background, up to 50% of the total population of the city. As time had elapsed since the previous round of the study in 2012, it became possible to identify the actual impact of the previous recruitment measures in the monitoring. Each city had initiated or taken physical and social measures in the past five years to develop the area and improve services. With regard to the development of organisational activities, the measures strived to develop a system that allows young people’s perspectives to be heard more systematically and to start new leisure groups based on young people’s wishes. (Eriksen & Frøyland 2017.)

3.3.4 Denmark

3.3.4.1 Democracy as an important evaluation criterion 

In Denmark, the partner of the project is the umbrella organisation of child and youth organisations the Danish Youth Council (Dansk Ungdoms Fællesråd DUF), which has 80 member organisations. The objective of the DUF is to promote opportunities for young people to influence matters, strengthen their inclusion and opportunities for dialogue, and to involve young people in democracy, society and organisations at a national and international level. (DUF, information about the organisation n.d.)
Like the central organisation in Norway, the DUF is in an important position in Denmark between the central government and the organisations. Its role can be described as a financial distributor, funding body, lobbyist and gatekeeper (DUF, interview 15 March 2022). We will first outline these roles from the point of view of funding and funding-related criteria, such as democracy. This will be followed by a brief review of the role of the DUF as a lobbyist and gatekeeper, and we will provide a few evaluation examples from youth organisations.
The DUF not only finances the activities of the organisations, but also contributes to the distribution of state grants and the gambling surplus to its member organisations. The criteria for funding for youth organisations include the organisation’s active activities in different parts of the country, democratic governance and working methods, community spirit, active youth participation and agency, access for all, development of activities, feedback from young people, and the systematic nature of reporting. In addition, organisations may choose to identify other objectives that are intrinsically linked to their activities and report on whether these objectives have been achieved. (DUF, interview 15 March 2022.)
Organisations can apply for funding for both basic and specific activities. Many organisations have a number of sources of funding, both public and private. Applications for funding from the DUF are assessed in cooperation between the DUF and the Ministry of Culture. Youth organisations, as well as sports organisations and civil society actors in general, are often funded by the surplus generated by the gambling industry. Then again, many other ministries also fund the activities of organisations. Organisations that are not members of the DUF apply for funding directly from the Ministry of Finance, for example. (DUF, interview 15 March 2022.) 
The interviewee states that the DUF nurtures democratic ideals in its activities and considers there to be a need to “train democratic muscles in organisations”. According to the interviewee, it is easy to think that the increase in democratic activity only affects, for example, political youth organisations, when it actually affects everyone, including student organisations and peace movements. The DUF has noticed that when actors are brought together to reflect on democratic practices, actors from very different backgrounds tend to find common ground. (DUF, interview 15 March 2022.) Among other things, the DUF has influenced the criteria for distributing the proceeds from gambling, which require the supported organisations to have a democratic structure, which again is considered to prove that the organisation is promoting the democratic commitment of its members and the understanding of democracy. There must also be elements in the activities of organisations that support social inclusion. In addition, young people must be given opportunities to develop competencies, such as responsibility, cooperation skills and leadership skills, through the activities of organisations. Aid from the same funding body for the initiative of organising independent activities for children and young people can also be considered to emphasise democracy (see Tipsungdomsnævnet 2016 Tilskudsbekendtgørelsen §1, §2, §27.)
The DUF notes that the indicators it provides are not typically strict, but rather indicative, since the most important objective in the “civil society DNA” is to create democratically functioning communities. Large projects do not seek to report, for example, the growth of individuals, but rather the volume of activities and other general indicators. In its own activities, the DUF, for example, operates in cooperation with schools, where young people are encouraged to make an impact. For example, the DUF surveys voter enthusiasm both before and after the activities. (DUF, interview 15 March 2022.)  
For its own knowledge base, that of its member organisations and society at large, the DUF has since 2011 carried out the Demokratianalysen survey to identify young people’s attitudes, social inclusion and participation in organisations, as well as digital skills. The 2021 results report presents, among other things, the membership of young people in various organisations and the forms of their participation in political life. The report also examines the opinions of young people regarding developments in Denmark and the world, as well as their interest in politics and social issues. The evaluation is conducted annually, and the comparison gives an idea of the changes taking place in young people’s attitudes and participation. The data used in the Demokratianalysen survey of 2021 was collected through online panels over a three-week period. The data consisted of a total of approximately 1 700 online interviews, of which nearly 600 were conducted with persons aged 16 to 25, while the remaining 1 100 with persons aged 25 or over. (Dansk Ungdoms Fællesråd 2021.)
In the interview, a DUF representative says that organisations use an infinite number of different evaluation methods, such as story-formatted reporting or quantitative figures. According to the representative, there are also organisations whose evaluation is more reminiscent of reporting rather than evaluating. However, many of these organisations strive to set themselves measurable objectives. For party-political youth organisations, this type of objective may be to increase the voter participation of young people in a given area, for example. On the other hand, the interviewee notes that not everything can be measured. It is particularly difficult when an organisation is trying to implement something new, and things are not going according to plan. With this in mind, the representative suggests that the core of youth organisational activity eventually comes down to learning by doing. (DUF, interview 15 March 2022.)
Together with its member organisations, the DUF has compiled a toolkit from various sources for evaluating international projects. The package includes a section supporting the self-evaluations of the team that carried out the project, where the team can think about whether the planned measures were implemented, if the work had an impact and if the right things were done. Several tools are available for embellishing the project, in terms of both highlighting the positive and negative effects that have occurred during the project, specifying whether they are thought to have been caused by the project or by some other reason, recording the biggest changes made and placing the project in a timeline. They also provide guidance for building a logical chain from the project’s main objectives, sub-objectives and functions, in such a way that, for example, unexpected effects are also taken into account (DUF, Toolbox til internationale projekter. n.d.)
The role of the DUF as a lobbyist and gatekeeper for youth organisations is to act in close proximity to them. According to the representative of the organisation, it is, in many ways, able to act closer to the organisations than officials in ministries. In Denmark, the development of indicators for evaluating youth organisation activities has been delegated to the DUF in the same way as sports organisations under the same ministry have been entrusted to their umbrella organisation. The DUF sees a wide range of challenges in the development of evaluation. Applying for funding and reporting on its use must not become too complicated and difficult for organisations. DUF’s helpline advises organisations on the interpretation of, for example, the terms “activity” or “active member”, or what is the minimum requirement for each criterion. The DUF can easily answer these questions on a practical level, as it is familiar with the realities of field work. On the other hand, the role of the DUF is to keep the ministry aware of the challenges of practical work and to steer the evaluation obligations in the right direction. (DUF, interview 15 March 2022.) 
Next, we will examine other highlighted evaluation examples from Denmark. The first concerns all the Nordic organisations that utilise Erasmus+ funding. An example of sporting activities is the need to tailor activities and evaluation to the needs of the target group, in this case girls. The Scout Movement example, in turn, explains how evaluation is linked to everyday activities and how it is important to highlight different aspects in the evaluation without forgetting the core of youth work: having fun and being together. The Denmark section ends with an emphasis on democracy, where it all began. The Youth Island provides inspiration due to its operating model and also because of the kinds of issues they wanted to address in evaluation.

3.3.4.2 Erasmus +: Danish youth associations regularly taking part in a Europe-wide survey

In the report Monitorering af ungdomsusdvekslinger under Erasmus+: Ungdom, published in 2020, Carsten Yndigegn examines the results of the Erasmus+ exchange programme for Denmark. The study was conducted by RAY, or Research-based Analysis and Monitoring of Erasmus+ Youth in Action Programme, an autonomous European research network. It contributes to the development and quality assurance of the Erasmus+ Youth programme and to the development of youth policy based on evidence and research. It also contributes to identifying learning in the context of transnational mobility in the youth sector and to the dialogue between research, policy and practice in the youth sector. (Yndigegn 2020, 18.)
The RAY network carries out the RAY Monitoring survey to monitor the quality and impact of Erasmus+ Youth activities. The survey is targeted at young people participating in Erasmus+ Youth projects as well as project organisers and people in charge. Its purpose is to survey the impact of participation on participants and organisations in a variety of ways. It also examines the practices, administrative framework and funding of projects, as well as the opportunities for their development. In the past, the priorities of the monitoring survey included changes in the values, attitudes and behaviours of the participants, experience of inclusion, and how international projects promote the participants’ civic and social skills, language skills and intercultural competences. (See Yndigegn 2020, 19.)
The survey has been developed over the years, and it currently consists of 30–40 multiple-choice questions, 5–10 open questions and individual questions in relation to more specific background information. Overall, the question template contains approximately 200 sub-questions. The Youthpass tool is utilised in the participants’ self-evaluations. As the reference group of the survey is located in several countries, for reasons of practicality and accessibility, the Monitoring survey takes the form of a post-project online survey and is sent simultaneously to all participating countries. As a result of the evaluation, Yndigegn states in his report (2020) that the impact of Erasmus+ Youth projects and the project activities carried out in the framework of the programme have so far contributed to the objectives of the Erasmus+ Youth Exchange Programme, such as strengthening cultural and European understanding and recognising the development of informal learning, such as values and attitudes. 

3.3.4.3 Modifying sports offering and evaluation methods to meet the needs of girls 

Since 2016, the National Olympic Committee and Sports Confederation of Denmark (Danmarks Idrætsforbund DIF) has been working with the Center for Ungdomsstudier and ten sports associations on the project Gode idrætsmiljøer for piger (“Good sports environments for girls”) (Center for Ungdomsstudier, n.d.). This research and development project was based on the fact that, among young people, girls are under-represented in sports clubs. School and sport are seen as a challenging combination, and there is a significant drop in the number of sports club members as young people move from lower to upper secondary school, with girls being the most likely to drop out (see the guide created during the project Jensen & Østergaard 2018.) The purpose of the project was to study, identify and increase understanding of the specific characteristics of girls in relation to sports activities, such as their tendency to perfectionism, and to find ways to strengthen girls’ participation in sports clubs. The project looked for tools, for example, to communicate the performance and expectations required in sport to girls, to create safe spaces and operating cultures, and thus to better engage young girls in sports club activities. (Center for Ungdomsstudier, n.d.)
In addition to the guide, the project produced a collection of material, which is published on the Center for Ungdomsstudier website. Both the guide and the material collection are intended to be used in girls’ sports activities, and the collection also includes evaluation models created in the framework of the project, the shorter of which is presented in more detail in the Gode idrætsmiljøer for piger guide (Jensen & Østergaard 2018).
This self-evaluation model can be used either during training or after sports competitions, with the team or individually. Young people can first complete the evaluation form independently or in small groups, after which the answers will be reviewed together with the coach. Self-evaluation consists of five open questions about successes and the development of the young athlete. The evaluation focuses on sport-related factors that the young people themselves can influence. A particular feature of the self-evaluation model is that the focus is on the athlete’s successes and development process, rather than on the sporting achievements or the results of a competition. The evaluation model serves as a tool that supports girls to identify and reflect on their own development. The template mentions that evaluation can be used to support the athlete or team in formulating future goals and in monitoring the goals. It is recommended that the evaluation is carried out in varying ways independently, in small groups or with a team, at home, during training or competitions, so that the topic is examined in a variety of ways. (Jensen & Østergaard 2018.)

3.3.4.4 It is important that having fun and doing things together receive respect as evaluative aims

The interviewee from The Danish Guide and Scout Association (Det Danske Spejderkorps) states that the measured objectives, indicators and the method of measuring depends on the funding body. Sometimes the focus is on quality, sometimes on quantity. The number of new members and volunteers is key, especially in organisations where the young people have to be waitlisted in large cities due to high popularity of the activities. This has been taken into account in the organisation’s long-term plan by launching new subunits and thereby shortening the waiting list as much as possible. (The Danish Guide and Scout Association, interview 26 April 2022.)
The interviewee states that they are not concerned of losing the organisation’s own will amidst the evaluation objectives of funding bodies. The problem is to have enough employee resources to develop evaluation, despite the large organisation having 25 000 members. On the other hand, a particular type of evaluation can be carried out, if necessary, as long as there are solid reasons for it. (The Danish Guide and Scout Association, interview 26 April 2022.)
The Danish Guide and Scout Association recognises the importance of increasingly seeking the perspectives of young participants in the evaluation of quality. For example, opinion polls with different coloured balls are used for this purpose. It is easy to take a picture of the distribution of the balls and attach it to the reporting. The interviewee also highlights the research consent issues that need to be considered when working with young people. When using balls, all that is needed is the young people’s willingness to take action. However, the same does not apply to interviews. Surveys are also often used, but the survey process is more demanding for young people as they need to prepare to read and write. Scouts often stay scouts throughout their lives. The organisation’s magazine features biographies to bring perspective to understanding the meaningfulness of the activity. On the other hand, these stories have not yet been used as material to evaluate impact.  (The Danish Guide and Scout Association, interview 26 April 2022.)
As the interviewee noted, evaluations of the nature of the activities can highlight many things: having fun, doing things together, spending time in nature, utilising natural resources sustainably, cooking, being away from devices, being happy, which is especially important during the coronavirus pandemic. The organisation also has a large project involving 12 000 children and young people doing activities in the natural sciences. The aim is to arouse interest in studying natural sciences, for example. (The Danish Guide and Scout Association, interview 26 April 2022.)

3.3.4.5 Youth Island and space for communality

In a report published in 2021, Et fællesskab i bevægelse. Unges perspektiver på Ungdomsøens fællesskab, Benjamin Henriksen examines young people’s experiences of community and communality in the Ungdomsøen youth island, as well as young people’s ideas on what the island’s activities should focus on. The report also presents young people who are actively involved and their experiences of the island’s community. The third focal point of examination is how young people find their way to the community and the island’s activities. (Henriksen 2021.)
Ungdomsøen (Youth Island) is an island off Copenhagen, serving as an experience and development centre for young people. On the island, young people can create and carry out activities and events, participate in courses, meet other young people and create new communities. The aim is to support young people’s own initiative and social responsibility, and to strengthen their belief in the fact that they can create positive changes in their own and other young people’s lives. (Ungdomsøen n.d.) The island is an old fortress, which was acquired and donated to the Danish Scout Movement in 2015 by the A. P. Møller Foundation and the Nordea Foundation. The foundations financed the renovation of the island’s premises and outdoor areas, and Ungdomsøen was opened to young people in its current form in August 2019. (Nordeafonden n.d.)
The data in the evaluation report was collected with a questionnaire (n=50) on how young people experienced the island’s communality. Individual interviews with active young people (n=8), and a focus group interview with young people from GEÅ were also conducted. Young people from GEÅ, or Giv-et-år volunteers, live on the island for a year and are involved in, among other things, the island’s repairs and maintenance, activities and organisation. In addition, the informal discussions that took place during field work were used in the evaluation, and being present in the field provided a practical insight into the dynamics of the community. (Henriksen 2021.)
90% of the survey respondents felt that there was room for everyone in the community, while 50% felt that there was still room for improvement in the diversity of activities and engaging different groups. Some teams and small communities on the island are more close-knit than others. The island and its community are also constantly changing because the island has been created with and from young people to young people. The report therefore stresses the particular importance of creating a safe community by developing the opportunities for all young people to experience inclusion. There are many ways to participate in the activities of the island. The first step can be through social media, a school visit, meeting an island ambassador or attending a course. Joining the community requires initiative from the young people, but there are many routes, and the island and its community are accessible both online and physically. (Henriksen 2021.)

3.4 Summary of the research results

In the above section written by the research team at Finnish Youth Research Network, we looked at several examples of evaluation found in Nordic youth organisations from three perspectives. Firstly, we examined the evaluation and reporting routines that the organisations have established. Secondly, we were interested in the themes that they want to highlight through evaluation. Thirdly, we explored the challenges experienced by youth organisations with regards to evaluation. Examples from literature and interviews were used as research data.
For the purposes of this study, evaluation is understood as the examination – reflection – of one’s own behaviour, which should be present to some extent in everything which is done. Ideally, evaluation is the act of critically examining not only the outcomes and the validity of the goals set for the evaluated activities, but also the rationality of the evaluation methods and the set criteria in relation to the activities (see Schön 1987). Although the period of getting to know Nordic youth organisations was fairly brief, the examples from the organisations showed a good level of critical reflection. The evaluations conducted in the organisations examine, for example, whether the quality of the activities is high enough and if the activities meet the needs of young people. Some of the organisations monitor their own activities and the activities of society specifically from the perspective of promoting democracy, whether the activities allow young people to participate in the debate and whether young peoples’ opinions have an impact. Other evaluation criteria that are used include aspects of community, inclusivity, equality
The equality aspect can be considered to include, for example, evaluating and improving the accessibility of activities from the perspective of different minority groups. and sustainable development.
and sustainable development.
Evaluation criteria were adjusted based on, for example, ideas on Social Return on Investment.
Based on the examples, the relationship of the organisations with evaluation can be described as realistic. By this, we mean that the youth organisations do not seem to believe that evaluation will ever portray the whole truth about the success of their activities. Consequently, the aim seems to be rather to conduct evaluations in the best possible way in each situation, given the resources allocated to evaluation at the time. Despite this realistic approach, many organisations want to learn more about evaluation. Some of the representatives of organisations said they obtain more information by reading and participating in trainings and various working groups that develop evaluation. Learning about the evaluation processes of their organisation and those of others is considered important.
Many organisations learn about evaluation by writing their own theory of change. This theory of change is composed to show which goals should be achieved first in order to achieve the next ones and what are the values underlying the activities. When defining their goals and how to achieve them, the organisations make use of a wide variety of knowledge about their operating environment: research data is obtained, for example, from literature, but also by conducting their own studies, whereas experiential knowledge is collected, for example, by asking questions from young people or volunteers working with young people. At the same time, the organisations seem to understand the need of funding bodies to collect key figures that are comparable with other organisations to demonstrate, for example, the volume of activities, such as the number of participants, interactions, conversations with politicians, projects, programmes or clicks on social media, and the satisfaction of participants. When speaking about the important goals of evaluation and reporting, the organisations mention the importance of promoting transparency in financial management, also in the civil society.
In some Nordic countries, organisations have clearly set out to improve the measurement of the impact of their activities. Some organisations are trying to communicate their impact on young people’s thoughts, emotions and values, as difficult as this may be. For some organisations, impact means the changes they have achieved together with the young people in the young people's lives. Another goal may be to change the roles adopted by young people in society. Or change the circumstances in which young people live, such as the atmosphere of their neighbourhood. Other measured impacts include changes in physical, mental and social well-being and learning various skills. It was also pointed out that, when working with young people, an important impact is enabling peer activities and producing joy. The need of the organisations to consider elements that are required for systematic social change was also brought up. This may trigger the need to develop cooperation between different parties, for example.
In some Nordic countries, the requirement to measure impact was to some extent considered a risk to maintaining a diverse civil society. This is because, for example, achieving a specific goal can often depend crucially on external factors: the will of politicians, the division of power at the time, chance, the timespan of decision-making, the complexity of the issues related to young people’s lives and other issues that happen to be on the agenda at the same time.
According to many of the interviewed representatives of organisations, data collected from the young people will be even more important in evaluation in order to receive funding. Young people are asked to assess the quality and impact of the activities using interviews and questionnaires, for example. In addition, some organisations have developed simpler ways to measure the satisfaction of young participants, such as voting with different coloured balls during the activities. The collection of feedback from young people has also been distributed across several months so that the evaluation does not overburden the core activities or the young people themselves.
When summarising, there are eight steps highlighting the messages heard about where to put more emphasis in evaluation in youth organisations, see Table 1.
Table 1. 
When evaluating, there is a need to …
1. Connect the values, aims and evaluation
2. Understand that developing evaluation never ends
3. Search for balance between “your theory” and (comparable) quantitative facts
4. Hear the opinion of young people and those working with them
5. Understand the role of research data
6. Build up group-force of NGOs in challenging the funders about what kind of results can be expected
7. Know that evaluation always has its limits
8. Ensure civil society being civil society
And of those eight steps, it is especially important to keep in mind the first one, which is about the  value world the youth organisations are working with, see also Table 2. 
Table 2. There are lot of aims and values which need to be taking account in evaluation when talking about youth organisations:
There are lot of aims and values which need to be taking account in evaluation when talking about youth organisations:
  • Joy
  • Connection with peers
  • Being there; answering the needs of young people (with young people)
  • Democracy
  • Communality (engagement)
  • Sustainability
  • Equality
  • Accessibility
  • Outreachness
  • Transparency
  • Use of money
  • Results of evaluation processes
The study also revealed other perceived challenges to evaluation which were more related to how evaluations are organised. How can each organisation ensure the required expertise and temporal resources for evaluation? How can young people be motivated to take part in the evaluation? How can the organisations’ expertise be taken into account when coming up with evaluation criteria? How can evaluation be organised so that it does not disturb the activities, for example in sensitive situations where help is given to a young person? How funding applications and the related evaluation and reporting can be made so simple that even small organisations have the chance to apply for funding from a variety of sources? How can the standard of evaluation be guaranteed so that it helps organisations achieve their goals as fully as possible?
Quotes
The language of the statements has been edited by the research team; the meaning is the same as in the interviews.
from the interviews concerning the opportunities and threats involved in evaluation in youth organisations:
  • “Building strength by including the evaluative voice of young people more often in the reports.”
  • “There is not enough workforce for evaluation.”
  • “There is help available for NGOs wanting to develop evaluation, but NGOs don’t have the resources to accept that help.”
  • “Due to the anonymity of the conversations, we find it difficult to examine the long-term effects of a conversation or to link the conversation to effects at the societal level.”
  • “The NGO knows its values, but they are not yet related to evaluation.”
  • “Evaluation means a possibility for the NGO to update its practicalities.”
  • “Time for evaluation may mean less time for young people.”
  • “How much does civil society need evaluation? Let’s also ensure that civil society can be civil society.”