Egyesek - ProfessionalED

Research Report

Project result 3

Summary

Our research aims to provide insights into whether our training methodologies used in youth projects are effective in developing core skills. Through aggregated data and comprehensive assessments, the research seeks to establish the validity and impact of the proposed training methodology for future youth projects.

A total of 55 participants from the three Erasmus+ projects (ProfessionalED, Facilitate the Change, and KFC Reloaded) were involved in the research. All participants were active youth workers and/or trainers. The analysis showed a consistent improvement in participants’ skills across the different training courses. The aggregate score, which averaged 10.4 before the first training, increased to 12.6 after the final training session, reflecting an overall improvement of 21%. This suggests significant development in participants' essential skills through the course of the training program. Most participants demonstrated statistically significant improvements in essential skills after each training courses.

The data strongly supports Hypothesis 1, that participants developed essential skills throughout the training cycle. The 21% improvement in the aggregate score underscores the efficacy of the training and assessment methodology in fostering skill development in youth workers and trainers. The findings suggest that the training methodology, particularly when coupled with both self-assessments and external assessment (as seen in ProfessionalED), is effective in competence development.

While Hypothesis 1 was strongly supported by the quantitative data (showing significant skill development over the training cycle), Hypothesis 2—which proposed a correlation between training outcomes and professional performance—was not supported by the data. The lack of significant correlations could be attributed to a combination of small sample size, methodological differences. While Hypothesis 2 was not supported by the quantitative data, interviews revealed that participants were actively applying the skills they learned in both their professional and personal lives.

The qualitative research in this research report reinforced the idea that participants in ProfessionalED project experienced significant personal growth, which also had a positive impact on their professional lives. Furthermore, holistic approach of the methodology, which focused on both personal and professional development, proved to be highly effective in creating lasting change in participants. The strong emphasis on self-reflection, emotional intelligence, and community building were key factors in making the experience transformative and impactful for the participants.

In future studies, it would be valuable to expand the sample size and refine assessment methods to better capture the connection between personal growth and professional performance.

Quantitative Research and its Findings

This project aims to validate a novel training and assessment methodology as an alternative to traditional approaches for evaluating training success in Erasmus+ and - in a wider perspective - youth projects. This report is focused on drawing conclusions from ProfessionalEDs’ training and assessment methodology, however, the decision was made to aggregate ProfessionalEducations’ data with two other similar projects: Keys for Change (KFC) Reloaded, and Facilitate the Change. The decision to aggregate the data from these three projects was based on two main considerations.

First, to draw meaningful conclusions from the data, it is essential to meet certain preconditions, the most important of which is having a sufficiently large sample size to detect the significant impacts of the training programs. Since each Erasmus+ project has a limited number of available spots, and considering the long-term nature of the ProfessionalED program (3 trainings spread out over one year), it was anticipated that the number of participants completing the full training would be too small to effectively capture its impact.

Second, the training methodologies used in Keys for Change (KFC) Reloaded and Facilitate the Change closely align with the methodology used in ProfessionalED. For instance, participating in the first training session of KFC is expected to yield similar outcomes to participating in the first training session of either Facilitate the Change or ProfessionalED.

These two factors - the need to ensure the scientific rigor and generalizability of the statistical analysis, and the methodological similarity between the projects - led us to aggregate and analyze the data from all three projects as a whole. However, there were some differences between the projects, which are outlined below.

ProfessionalED and KFC both consisted of three training moments, interspersed with fieldwork sessions. Facilitate the Change, by contrast, included only one training session and did not involve fieldwork. Among these, ProfessionalED featured the most robust assessment methodology, incorporating both self-assessments and external evaluations (conducted after each fieldwork activities), while for the others, there were no external evaluations, only self-assessments. This dual approach provided an additional layer of evaluation to compare self-perceived and externally observed skill acquisition, making it the most comprehensive assessment model of the three projects.

Before starting the program, participants in all projects completed a self-administered diagnostic questionnaire based on the SkillsBuilder Universal Framework, assessing eight core skills critical to youth work: listening, speaking, problem-solving, creativity, staying positive, aiming high, leadership, and teamwork. After each training moment, participants completed a self-assessment questionnaire to evaluate their progress in these skill areas.

The goal of this report is to evaluate the development of essential skills for youth workers and trainers, focusing on whether the training methodology contributed to measurable improvements in skills, as assessed through both self-assessment and external evaluation. Therefore, we tested the following hypotheses:

Hypothesis 1

Participants developed essential skills throughout the training cycle.

Hypothesis 2

There is a correlation between an individual's performance in their professional life and the training cycle, thereby providing insights into the reliability and validity of the proposed methodology.

Participants or target group

A total of 55 participants took part in the three Erasmus+ projects: ProfessionalED, Facilitate the Change, and KFC Reloaded. The selection criteria required participants to be active youth workers or trainers. The breakdown of participant pool is presented as follows:

⁠

Map 1: Distribution of participants by nationality.

⁠

ProfessionalED

⁠

26 participants, 15 female and 11 male;

Participants came from Netherlands (4), Hungary (7), Italy (7), Portugal (5), Czech Republic (3).

Facilitate the Change

⁠

17 participants, 11 female and 6 male;

Participants came from Romania (2), Hungary (5), France (3), Spain (2), Czech Republic (2), Greece (2), Belgium (1).

Keys For Change Reloaded

⁠

12 participants, 11 female and 1 male;

Participants came from Turkey (1), Austria (1), Belgium (1), Hungary (1), France (2), Portugal (1), Serbia (1), Slovakia (1), Latvia (1), Netherlands (1) and Italy (1).

The participants' diverse backgrounds provided a rich cross-cultural perspective on the training methodology. The gender distribution within the projects ensures a balanced representation, with variations observed across projects. Notably, ProfessionalED maintained a more balanced gender distribution, while Facilitate the Change and KFC Reloaded had a higher proportion of female participants.

Research design and methodology

The research design was structured to assess the effectiveness of a training and assessment methodology implemented across three Erasmus+ projects: ProfessionalED, Facilitate the Change, and KFC Reloaded. The data available allows for a longitudinal design by comparing the different moments of both self-assessment and external evaluation (in the case of ProfessionalED). The different moments are described as follows:

Training moments and self-assessments: Before the first training moment, participants completed the diagnostic questionnaire using the SkillsBuilder framework. This happened again three more times after the completion of the training moments. Therefore, there are four different data collections regarding the self-assessments. Exceptionally, the program Facilitate the Change only had one moment of training, which then corresponded to the first moment of training of all the programs.

Fieldwork: Participants of ProfessionalED and KFC reloaded completed self-assessments after their fieldwork. This moment happened after the training moments, as it was an opportunity for participants to apply what was learned previously in the training sessions.

External assessment: Only ProfessionalED involved an external assessment, which took place after the fieldwork moments. Experienced assessors observed the participants' fieldwork performances and evaluated their skills in real-life scenarios. These assessments were made by using the same SkillsBuilder framework.

Scale used for assessments

As mentioned before, the quantitative scale used to measure the target skills was the SkillsBuilder universal framework. This framework assesses the abilities in eight key skills: listening, speaking, problem solving, creativity, staying positive, aiming high, leadership, and teamwork. For each skill, participants evaluated themselves using a scale of 0 to 15. The scale for each skill was composed of 15 questions, each addressing a component of the skill. For example, one question for the “Speaking” skill was: “I speak engagingly by using tone, expression and gesture to engage listeners”. The possible answers for these questions, range from "Never" to "Almost Always." The score for each skill is computed depending on the answers to each of the questions. Additionally, and as mentioned before, external evaluators used the same scale during fieldwork.

Hypothesis 1

Participants developed essential skills throughout the training cycle.

Summary

To evaluate the effectiveness of the training methodology and assess whether participants developed the targeted skills over time, a series of paired t-tests were conducted. These tests were used to compare participants' self-assessed skill levels at different stages of the training program.

The following comparisons were made:

Before Training 1 to After Training 1 (N = 53):

This comparison aimed to assess whether participants’ self-assessed skills improved immediately following the first training moment, when compared to before the training started.

The number of participants in this analysis slightly differs from the full sample, as there were two participants who completed the first training but did not complete any self-assessment moments.

After Training 1 to After Training 2 (N = 26):

This comparison involved participants who had completed both Training 1 and Training 2, after which their self-assessed skills were compared. Note that Facilitate the Change (FTC) participants were excluded from this comparison, as they only participated in Training 1.

After Training 2 to After Training 3 (N = 17):

This comparison examined participants who completed the self-assessment after training 2 and after training 3, the final training moment.

Before Training 1 to After Training 3 (N = 20):

This comparison assessed whether participants showed overall skill development from the beginning of the program (before Training 1) to the end (after Training 3). This analysis included only participants who completed the first self-assessment, before the first training and the final self-assessment, after the final training of the program. Counterintuitively, there are more participants in this comparison than in the previous one, as there were some (3 participants) that participated in the first and last moments, but not in the self-assessment post training 2.

This rationale for comparisons, consisting in comparing only those participants that were present at each moment, allows us to control for survivorship bias, due to dropout during the program and analyze only the differences related to the participants that completed each of the training moments.

Method of analysis

Each of these comparisons was made for each of the eight key skills, as well as the aggregate score across all skills. The goal was to evaluate whether participants’ self-perceptions of their skill levels changed significantly across the different stages of the training program.

Paired t-tests were run to compare the mean self-assessment scores at each time point. For each test, the null hypothesis (that there is no significant difference between the two assessments) was tested against the alternative hypothesis (that there is a significant difference). If statistically significant differences are found for each comparison, lends support to hypothesis 1: “Participants developed essential skills throughout the training cycle.”

The analysis considered both individual skills and the aggregate score, allowing for a comprehensive understanding of overall skill development as well as progress in specific areas.

Rationale for grouping

It is important to note that the training methodology was consistent across the projects, so the comparisons were made regardless of project membership. However, the number of participants in each comparison varied, as only those who participated in both assessment moments of each comparison were included in the paired t-tests. This explains why the sample size (N) differs between the comparisons (e.g., N = 53, N = 26, N = 17, and N = 20).

By conducting these paired comparisons, we aimed to determine whether there were significant improvements in participants' self-reported skills throughout the course of the training program. If significant changes were observed, this would suggest that the training methodology effectively fostered skill development.

Results Hypothesis 1

Before Training 1 vs. Post Training 1

⁠

Graph A: This graph illustrates the changes between the assessments done before training 1 and after training 1 (N = 53) Every change between assessment moments was statistically significant, having the skill “Listening” with the greatest improvement (1.3), and the skill “Problem Solving” with the least improvement (0.6). Overall, the trend of results indicate an increase in the self-assessment scores, with an average increase of 0.9 points.

Post Training 1 vs. Post Training 2

⁠

Graph B: This graph illustrates the changes between the assessments done after training 1 and after training 2( N = 26). “Problem Solving” and “Creativity” were the skills which improved the most (0.8) and “Listening” was the skill which improved the least (0.6). As for “Leadership” and “Teamwork”, the increases were not statistically significant, therefore we cannot draw conclusions regarding their improvement. Overall, the mean change for all skills (0.7) suggests that, even if more modest than the previous moment, there was an improvement.

Post Training 2 vs. Post Training 3

⁠

Graph C: This graph illustrates the changes between the assessments done after training 2 and after training 3 (N = 17). “Problem Solving” was the skill which improved the most (1.5) and “Creativity” and “Leadership” were the skills which improved the least (1.1). Even though the differences range from 0.6 to 1.0, the changes in self-assessments for “Listening”, “Speaking” and “Teamwork” were not statistically significant. It is important to note that, since the sample size varies for each moment of analysis, this means that the same degree of change in different moments may in one instance be statistically significant, while not in other cases. For example, in Graph B, all statistically significant changes were smaller than 1.0 points. However, in this case, “Speaking” increased by 1.0 points and there is not enough sample size to allow us to conclude that this increase is significant.

Before Training 1 vs. Post Training 3

⁠

Graph D: This graph illustrates the changes between the assessments done before training 1 and after training 3 (N = 20). In essence, it shows the magnitude of change throughout the whole project. Every change between assessment moments was statistically significant, having the skills “Listening” and “Staying Positive” with the largest improvements (2.5 and 2.4). “Problem Solving” and “Leadership” were the skills which improved the least (2.1). Overall, the trend of results indicate an increase in the self-assessment scores, with a mean of 2.3.

Trendline

⁠

Graph E: This graph illustrates the aggregate score for every self-assessment moment during the project, considering only the participants who took part in every single self-assessment moment (N = 17). Overall, the trend of results indicates a constant increase in the self-assessment scores throughout the whole training.

⁠

Discussion Hypothesis 1:

This pattern of results clearly shows a positive development of the self-assessed skills in the participants of this program. More specifically, for each training moment there was an increase in the self-assessed skills when compared to the previous one. Except for very specific cases, which did not show statistically significant improvement from one moment to the next, there was a consistent improvement over training moments.

It is important to notice the overall improvement: the aggregate score of the first moment (before training) was 10.4 and the final aggregate score (after the last training moment) was 12.6. . This shows an improvement of approximately 21% during the course of the program. A 21% improvement in the aggregate of such fundamental skills, through the participation in just 3 training moments is a relevant increase that needs to be taken seriously. It is possible to affirm that during the course of this project, participants experienced a significant improvement in their self-assessed skills, supporting the hypothesis that participants developed essential skills throughout the training cycle.

Hypothesis 2

There is a correlation between an individual's performance in their professional life and the training cycle, thereby providing insights into the reliability and validity of the proposed methodology.

In order to test hypothesis 2, participants were subject to external assessments in the implementation of field works in their professional contexts. Pearson Correlations were used, as this method is used to quantify the strength and direction of a linear relationship between two continuous variables.

In order to measure possible correlations, the self-assessment changes in the previous section were compared towards changes in field work external assessments scores. In essence, there was a correlation of changes for each moment available. Professional Education participants were the only ones considered for these analysis, as they were the only ones subjected to external assessment.

The following correlations were assessed:

Correlation between the change post training 1 to post training 2 with the change from fieldwork 1 to fieldwork 2

This correlation aimed to test whether there is a correlation between the skill improvements observed between the first and second trainings and the external assessment changes between the first and second fieldworks.

Correlation between the change post training 2 to post training 3 with the change from fieldwork 2 to fieldwork 3

Method of analysis

Each of these comparisons was made for the change of each of the eight key skills, as well as for the aggregate score across all skills. The goal was to evaluate whether the changes in participants’ self-perceptions of their skill levels, after each training, were correlated to the changes in the external assessment of their skills, observed in the implementation of field works in their professional contexts, for the different stages of the training program. changed significantly across the different stages of the training program.

Pearson Correlations were run to assess the presence of significant correlations between the changes for each skill and aggregate score between the training self-assessments and the field work external assessments. For each test, the null hypothesis (that there is no correlation between these assessments) was tested against the alternative hypothesis (that there is a significant correlation). If statistically significant correlations are found for each comparison, it points to the fact that improvements in the self-assessed skills developed during the trainings, translate to the implementation of field works in the participants’ professional contexts, supporting hypothesis 2.

Results for Hypothesis 2

For all Pearson correlations assessed, not a single significant relationship was found between the changes in self-assessed skills and the changes in external assessed skills observed in the field works (Consult Annex with Pearson Correlation results).

Discussion of Results Hypothesis 2

There may be several different explanations to account for the lack of significant correlations being found in this study.

Competing explanations:

No effect: Firstly, and the most straightforward one, is that the hypothesis simply is not supported by data. That, in fact, the self-assessed skills developed during the trainings, do not translate to the participants’ professional contexts. This is a possibility that needs to be considered. However, considering the participant testimonials presented later in the report, it is important to consider other possibilities that could explain this pattern of data.

Small sample size: The sample size of the Pearson Correlations was extremely small, considering that only Professional Education participants were subjected to external assessments. According to statistical power/sample size calculators, in order to find a moderate correlation (r > 0.4), considering standard assumptions, a sample size of 62 participants is required, which wasn’t the case. This indicates that, unless there was an extremely strong correlation that with the current sample size, it would not be possible to assess whether there is a significant correlation between participants’ performance in their professional life and the training cycle.

Different assessment methodologies: It is also possible that the utilization of different assessment methodologies - self and external assessment - leads to different components of the skill being assessed, which could lead to the appearance that the changes were not correlated. In fact, despite the underlying utilization of a common framework - Skillsbuilder - with a clear operationalization of the building blocks of each skill it is very likely that the way that a participant self-assesses doesn’t exactly match the way that external assessors evaluate the manifestation of these skills. This possibility brings the question? Is there one of these methodologies (self vs. external) that is better than the other? Or are they complementary in the sense that they observe different components of each skill?

This study protocol had to conform to very specific implementation challenges that limited our ability to develop the most rigorous assessment methodologies possible. However, considering the promising results regarding hypothesis 1, and the fact that a robust longitudinal methodology was used, it is important to consider how to proceed further and how future protocols could mitigate specific limitations of this study.

Sample Size (Limitation): While the sample size concerning hypothesis 1 was sufficient to identify statistically significant improvements in self-assessed skill development, this was not the case for the comparison to external assessments. A very clear way to address this limitation is to extend the protocol of external assessment to different programs and, consequently, to more participants. Additionally, larger sample sizes will allow us to draw conclusions from different demographic variables. Are the effects felt the same way for different sexes, age groups or nationalities?

Lack of Control Group (Limitation): In addition to the sample size, this program did not include a control group. The comparison of the effects of an intervention to control groups are the gold standards to measure the effectiveness of any intervention, from medicine to psychology. In this specific case, it is not unreasonable to consider that any individual, even if they do not participate in trainings will have a baseline improvement in their skill development. This means that the improvement observed in the participants of this program should be compared to this baseline improvement, at the risk of overestimating the impact of this particular training program. It is also possible that there is not a baseline improvement in skills and that the impact of this training is being accurately estimated. However, for now, in the absence of a control group, that is still to be determined.

Assessment Methodologies - External vs. Self (Limitation): It is possible that self and external assessments are evaluating different components of the skills being measured. While this aspect points to the importance of considering both these types of assessments in the youth work environment (which usually only relies on self-assessment), it begets a clearer definition of what each type of assessment actually measures. The present program attempted to do this by utilizing a universal skills framework, creating a common ground by which different types of assessments would be based on. However, some questions arise. Considering that a self-assessor can base their evaluation on a very broad experience of these skills on their professional lives, external assessors are only basing their evaluation on a narrow assessment window. Furthermore, some skills, by definition are more observable than others. For example, it seems to be much easier to observe whether someone has good “Speaking” skills, than whether someone is capable of “Staying Positive”. From this, next projects should aim to have a clear protocol when it comes to defining how the external assessment should be conducted. This will help harmonize the results and creating a clearer picture of these kinds of assessments.

Longitudinal Aspect (Strength): It is often the case that training programs do not have the ability to measure their impact in the medium/long term. Most trainings only evaluate their immediate impact on participants. However, the importance of assessing whether the impacts of a training are maintained and evolve over time cannot be understated. One of the main ways to know this, is by conducting longitudinal studies, which consist in examining changes in the participants over a period of time. This was a longitudinal study, where participants were assessed in a period of time over one year. This is what allowed us to identify trendlines of skill development in a clear and robust way;

Unified Framework (Strength): Despite the assessment aspects mentioned before, a strength of this project lies in the utilization of a unified skills framework. This framework was utilized by every single participant to conduct their self-assessment and by each external assessor to conduct the external assessments. This allowed to create a basis for a common understanding of the skills being developed and assessed. A particular aspect that is worthy of note is the operationalization of the concepts. For each skill, there were 15 clearly determined components that were agreed upon, based on the Skillsbuilder Framework. Additionally, this specific Framework, due to its Universal approach, is able to capture the broad range of skills that are needed and developed in the field of youth work. In our view, the utilization of this framework was a clear strength of this program and more attention should be given to it, as a means to measure, develop and report skills in this field, as an alternative to currently available options.

Qualitative Research and its Findings

Impact research on the field of youth work and trainings include particular challenges.

In ProfessionalED we also had several, out of two were major and led us to conduct the qualitative part as well:

The small size of the sample;

The long term impact (5-10 years), that might be there, does not fit into the time frame of Erasmus+ KA2 project.

We aimed to support the quantitative results with interviews. We randomly chose one representative from each country from where we had participants go through the whole cyle. We had 1 female and 3 male to have an interview with.

We chose the semi-structured interview technique: 5 questions were set that were covering the hypothesis. It is also important to claim that interviews have their own specific distorting feature based on the impressions of the people conducting them.

Hypothesis 1

Participants’ of ProfessionalED project developed essential skills throughout the training cycle.

Hypothesis 2

There is a correlation between an individual's performance in their professional life and the training cycle. (It is a known fact that they work in the field of youth.)

The subjects of the interview are chosen randomly. They participated in all 3 training courses, did the assessment cycle (self- and assisted) throughout the field work as well and fulfilled the criteria to join the project (working on the field of youth).

Questions:

What do you think about the training courses?

What do you think you could have learnt/developed by participating in the ProfessionalED training cycle?

How do you think the self-assessment influenced your learning process?

How do you think the assisted(external) assessment influenced your learning process?

Do you use anything learnt throughout the learning cycle in your professional life? If yes, what and how?

Do you use anything learnt throughout the learning cycle in your personal life? If yes, what and how?

Are you going to further use Skills Builder in your personal or professional life?

Anything else you would like to say?

The interviews were conducted online, with durations ranging from 45 to 80 minutes. Participants in the ProfessionalED project were generally enthusiastic about participating, and three out of four explicitly expressed gratitude for the opportunity to be involved. This positive attitude likely influenced the tone of the interviews. Only one participant maintained a more neutral stance.

It's important to note that the interviewer and participants were already professionally acquainted and shared a common understanding of the ProfessionalED project. This prior relationship facilitated communication during the interviews.

After the interviews were recorded, they were transcribed and coded. Since both the interviewer and participants were familiar with the ProfessionalED project, the coding process was informed by their shared knowledge and professional experience. The coding was performed using Dedoose, and the analysis was carried out using both Dedoose and Excel.

The codes and their applications in all the interviews:

communication: 4

community: 4

complexity: 7

consistency, repetition

cultural differences: 1

enjoyment: 1

facilitation: 7

program design

long term impact

moderately useful

not useful