Deliverable 2 (Methods Training): Organisation of retreats for survey methodologists in the HBSC network and a mentoring scheme EC Operating Grant Deliverable

Aim

“Improve methodological skills within the HBSC network”

“Increase the connectedness between methodologists within the HBSC network”

“Facilitate scientific cooperations between several entities within the HBSC network which are working on methodological development of the study, e.g. Data Management Centre (University of Bergen), Methodology Development Group (network-wide group of people working on the improvement of data quality and output).”

“Provide structures that allow a better visualisation of methodological resources within the HBSC network.”

Activities

A) Methods Training

The methods training sessions were based on a skills and training needs questionnaire circulated within the network. Different topics were ranked with respect to the difference between perceived average skill level and perceived relevance of the topic. The emerging topics for the workshops were

● Item Response Theory (IRT) ● Trend Analysis for Cross-Sectional Survey Data ● Structural Equation Modelling (SEM) ● Sampling and Weighting Techniques ● Introduction to Curve Fitting ● Differential Item Functioning (DIF) ● Multilevel Models

Softwares covered within the workshops were

● Mplus ● R ● SPSS ● Stata ● Excel

The activities included a capacity building plan, 3 training sessions (2 trainers per session) and were based on a methods action plan.

B) Mentoring Programme

A second aim of this deliverable strand was a “Mentoring Programme for Survey Methodologists”. The aim of such a scheme was to strengthen the capacity in survey methodology in the network, and to provide means to facilitate exchange and knowledge transfer between network members. Two main activities took place: (1) With respect to skills similarity, different patterns of collaboration between people were tested during the methods workshop in Sofia and Tallinn. (2) An “online Mentoring Scheme” was installed that allows people to search for methodological mentors within the HBSC network. The website is based on a self-organizing questionnaire.

Deliverables

A) Methods Training

● 3 training sessions ○ including a 2-day methods retreat for 43 HBSC methodologists

■ two 3-day methods retreats were organized. The total number of

participants was 65 (including 6 trainers)

■ The number of people attending at both retreats was 16 ○ 2 trainers per session

■ overall 6 trainers were involved (at least two per session) ● number of participants in training (20% of target audience)

○ According to the HBSC members directory 68 people are in the methodology development group. With a total count of 47 HBSC members the coverage of this audience was 69%. Considering that also people from other groups may be counted as target audience and the members directory might be partly incomplete the coverage may be reduced. However, the participation can be

considered highly satisfactory ● qualitative assessment of training opportunities by participants

○ Both methods retreats were evaluated with qualitative and quantitative

methods

○ The evaluation can be found below

○ The dataset of the evaluation is available to the network

B) Mentoring Programme

● uptake/satisfaction of methods mentoring system for teams (10%/80% of target audience)

● more efficient use of network resources and knowledge transfer within the network ● mentoring of future strategic leadership positions

Outputs

A) Methods Training

○ “Methodology workshop”; alongside “HBSC Autumn Meeting 2014” (Tirana, Albania, 13th of November, 2014)

■ Trainers: ● Dorothy Currie (University of St Andrews) ● Dr Martin Steppan (University of St Andrews, IFT Institut fuer

Therapieforschung, Munich) ■ Evidence

● See conference programme here ● Participation: about 40 (no evidence of attendance available)

○ “Methods Training Workshop Conference I” (Tallinn, Estonia, 20th-22nd of May 2015)

■ Trainers: ● Prof. Emmanuel Kuntsche (Addiction Swiss Lausanne, Eötvös

Loránd University, Budapest) ● Prof. Torbjørn Torsheim (University of Bergen, Norway) ● Dr Franziska Schwabe (University of Dortmund, Germany)

■ Evidence: ● See conference invitation here ● See workshop programme here ● See presentation on workshop and pictures of conference here

(HBSC Spring Meeting, June 2015, Vienna - Austria) ● See registration form here ● Participation: 36 attendees and 3 trainers (see conference

registration form)

○ “Methods Training Workshop Conference II” (Sofia, Bulgaria, 24th-26th of June)

■ Trainers: ● Prof. Alberto Maydeu-Olivares (University of Barcelona) ● Prof. Ali Uenlue (Technical University of Munich, Germany) ● Dorothy Currie (University of St Andrews, United Kingdom)

■ Evidence: ● See conference invitation here ● See workshop programm here ● See pictures of the conference here ● See registration form here ● Participation: 23 attendees and 3 trainers (see conference

registration form)

B) Mentoring Programme

● Collaboration “Experiment”

A crucial question when it comes to matching people for knowledge exchange is if rather similar or rather dissimilar people shall be matched and what the consequences on the quality of collaboration are. Hence, a little experiment was conducted during one of the methods workshops (Tallinn, May 20-22). People were assigned to different sitting orders each day. The sitting order was related to people’s answers to a “Skills and Training Needs Questionnaire” they had submitted earlier. On the first day the sitting order was arranged with respect to “maximum skills heterogeneity” of people sitting next to each other, i.e. neighbours were as maximally different regarding their methodological skill set as possible. On the second day, the sitting order was “random”, i.e. only determined by a random generator (Excel). On the third day people sat to people who had maximally similar methodological skills. The purpose of this experiment was to find out what kind of matching of individuals might lead to most beneficial outcomes for future mentoring programmes .

● Online Mentoring Scheme

An online mentoring scheme was set up on the HBSC members website. The website operates as a browsable directory of people within the HBSC network. People can edit their profiles with respect to their own skills. People within the HBSC network can search other HBSC experts with a certain skillset they require. The questionnaire of this website is self-organizing, i.e. the fields that show up in the skills questionnaire are not pre-determined, but can be added by all users. This leads to a self-organizing platform that allows highly flexible adaptation of the directory to the needs of the people who use it. The online mentoring system can be found here.

Dissemination

● Twitter ● Facebook ● HBSC website (Training Sessions) ● Online Mentoring Scheme ● HBSC newsletters (internal and external stakeholder) ● Conference Discussion (HBSC Autumn Meeting, Tirana, Albania, November, 2014) ● Conference presentation (HBSC Spring Meeting, Vienna, June 2015)

Evaluation

Methods Training

Evaluation was an integral part of the organized methods workshops. According to the meeting agendas (see example here) at the end of every workshop day time was reserved for evaluation only. The anonymized responses to the evaluation forms to each workshop can be downloaded here. In this report only examples of responses to the evaluation are shown. People were allowed to use pseudonyms in their evaluation and were explicitly instructed to give “honest” and not “polite” answers. The aim of this instruction was to get most useful feedback.

Figure 1 depicts the impact of different workshops on future work of the attendees. “Weighting and Sampling Techniques” (Dorothy Currie) had the highest impact. “Structural Equation Modelling” (SEM, Alberto Maydeu-Olivares) ranked second, and “Curve Fitting” (Martin Steppan) ranked third on this question. Compared to these higher ranking topics, Item Response Theory (IRT) and Differential Item Functioning (DIF) had lower perceived impact on the future endeavours of workshop attendees. The analysis shows that different topics can have very different practical relevance for attendees although all of the topics had high relevance for the network according to a skills and training needs questionnaire.

Figure 1. The impact of workshops on future work of attendees.

Figure 2. Different evaluation questions (all workshops) with an agreement higher than 50%

Figure 2 illustrates the average response (and standard deviations using whiskers) for questions with high agreement rates (>50% on a scale between 1 and 5) by decreasing endorsement (for all workshops). With an average score of 4.9 on a scale of 1 to 5 there was high agreement regarding the commendability of the workshop to other people. Also the relevance of the topics (4.3), perceived learning achievement (4.0), and the structure of the workshops (4.0) had very high agreement rates (>80%). The correspondence with attendees interests (3.8), the effect on the perception of other people’s work (3.7) and the perceived gain of new practical skills (3.3) were evaluated positively.

Figure 3. Evaluation results for two methods workshops (school mark scale)

Figure 3 illustrates the overall evaluation results for the methods workshops in Tallinn and Sofia on a scale between 0 (negative evaluation) and 10 (positive evaluation). The average score for Tallinn (8.0) was lower than the evaluation for Sofia (8.6). However, the difference was not significant from a statistical point of view (t-test; p<.224). Hence, the overall evaluation for both events was positive. 51.6% of the participants stated that they made “very interesting professional contacts” at the workshop, 41.9% said that the professional contacts they made were “positive”. The question with the highest agreement was about “social atmosphere” which was highly appreciated by the participants (m=4.8, scale from 1 to 5). The question with the lowest agreement was about “whether the different topics of the workshop fitted to each other”. However, also this lowest ranking question was still evaluated positively (m=3.6; scale from 1 to 5).

Mentoring Programme

Figure 4. Social atmosphere with respect to sitting order (“collaboration experiment”)

Figure 4 illustrates the results of a collaboration “experiment” during the workshop in Tallinn. The aim of this “experiment” was to find out which ways of matching individuals in a mentoring scheme is more beneficial: people with very similar or very different skill set? The results show that the “chemistry” with sitting neighbours in the workshop was evaluated more positively when people had high skills diversity (heterogeneity). On the other hand, high similarity of skills led to a lower evaluation of the atmosphere (“chemistry”) with collaborators. No statistical tests were performed for this little experiment, due to a very high potential influence of confounding variables. For example, the three different “treatment” conditions took place on consecutive days (Wednesday, Thursday, Friday). The found decrease of a positive atmosphere could simply be due to “getting tired” during a workshop conference. However, there were several comments of attendees that there was indeed a remarkable difference regarding the atmosphere between different days, and that the last day (“minimum skills heterogeneity”) felt less appealing than the other days. Future work on a mentoring scheme might use the outcomes of this little experiment again in a more systematic way and corroborate the results. So far, the results can be taken as a slight indication that a matching of people with different skill sets is a more promising endeavour.

Figure 5. Screenshot of the browsable Mentor’s Directory on the HBSC website. Experts can be searched based on skills required.

Figure 5 show the interface of an “online mentoring scheme” which can be found on the HBSC website. People can edit their own methodological skills, and other people in the network can search for “mentors” within the network based on skills they are looking for. The scheme is aimed at an increase in knowledge transfer and exchange between different people within the HBSC network.

Figure 6. Screenshot of the browsable Mentor’s Directory search functionality on the HBSC website. Experts can be searched based on specific methodological & software skills, language skills etc.

Figure 6 shows the search functionality within the “online mentoring scheme”. The website was programmed in Visual Basic.NET works “self-organizing”, i.e. fields and skills people can tick off, are not centrally pre-determined, but every individual can submit his / her own skills (or what they think a skill is). People who submit their skill sets later find the categories

other people have submitted earlier which facilitates a more dynamic growth of what people consider as necessary skills for their work.

Reflections

A) Methods Workshops

The organized methods workshops were well-received and evaluated positively. There was high overall content with the format, content and procedure of the organized events. However, there was also positive critique and suggestions to make similar events better and beneficial to all participants and the whole network. These evaluations are important data and can be used for the future (see Annex for links to datasets). Workshops can be used for testing new formats of collaboration, to introduce new social media (e.g. the online mentoring scheme) and to engage people in getting involved in collaboration, co-authorship and other ways of professional interaction.

B) Mentoring Scheme

Facilitating exchange between individuals who live hundreds and thousands of miles apart from each other is a challenging endeavour. Traditional concepts of linking a mentee with a mentor might not be suitable for an international network of people with already high scientific profiles and expertise in their own fields. Hence, more horizontal relationships between researchers with different research profiles might be useful for an international network like the HBSC study. When using the word “researcher” here, the focus of such a mentoring scheme should not only be set to “scientists” in the traditional sense. Very often, required skills lie without the usual academic world (e.g. database or programming skills). It would therefore make sense, to include all members of the HBSC network within an online mentoring scheme, i.e. also “administrative”, “technical” staff and apply the term “horizontal” also to the variety of different job families. Using methods workshop to introduce mentoring activities has proved to work very well. Future endeavours may focus on how to make the now established technical infrastructure more popular among network members. It is worthy to think about ways how to increase seeking external assistance within the network (through the mentoring scheme) is beneficial for both sides: the information seeker and the information provider. Incentives like co-authorship and reciprocal help might support this process.

Future plans

● Edit methods workshop video material and make it available to the HBSC network ● Include methods workshop material within the HBSC e-learning platform ● Link the online mentoring scheme with the browsable members directory ● Increase the popularity of the “online mentoring scheme” ● Introduce a “currency” within the mentoring scheme in order to increase the

likelihood of co-authorship, knowledge transfer, intellectual exchange, likelihood of seeking help

● Attach a forum to the mentoring scheme in order to discuss topics of collaboration, mentoring, knowledge transfer etc.

● Organize future methods workshops or “summer schools” using the experience from these workshops and evaluation (e.g. see figure below)

Figure 7. Evaluation of workshops can be used to organize similar events in the future

Appendix

Further Evidence:

Example of an evaluation form for different topics

Example of an evaluation form for whole workshop meeting

Evaluation data for single topics

Overall evaluation data for whole meetings

Free-format feedback in workshop evaluation forms

The following statements represent anonymized free-text feedback of people attending workshop meetings in Sofia and Tallinn. The list is comprehensive and no comments were deleted. People were allowed to use pseudonyms in their evaluation and were explicitly instructed to give “honest” and not “polite” answers. The aim of this instruction was to get most useful feedback.

OVERALL WORKSHOP EVALUATION

“When trying to answer the question about the number of topics and number of speakers, I will note that it all depends on the topic. There might be topics that whole 2,5 days should be devoted to and there could be others - less complicated ones.”

“I liked the way Torbjorn and Franziska worked, with the example database and syntaxes. You could really start working with the new program (Mplus and R) and learn by adaptation of the syntax.”

“I think it is necessary to have more lab sessions. It could be nice also to have more reading materials distributed beforehand. It could be very useful to have a list with the participants' names, institutions and e-mail adresses for networking later on. And some technicality (which sometimes becomes very important for evaluation) - it is good to have certificates for attendance. Otherwise, I enjoyed very much the workshop and learned interesting new things about the methods taught.”

“Keep doing like this!! It would help to increase research capacity and enhance personal relationship within the HBSC network.”

“Simply more time for practical exercises. That could be reached by increasing the duration of the work shop or by decreasing the number of topics covered.”

“I think that the workshop was very good: well-organized, interesting topics. In my opinion, it should have necessary a larger screen. Some presentations were poorly visible from the last line.”

“Overall, it was a fantastic and very productive experience. It would be probably better to have one topic less and more in-depth, i.e. to have more time for practical examples.”

“Thank you for organizing useful and pleasant event! I am looking forwards to future workshops with more topics !”

“Just a social thing - make everyone present themselves at the very start of the workshop. I fully understand, seeing that everyone is sitting on rows (not facing eachother), how this may be difficult. This issue was discussed among many of us throughout the workshop - so I would hope we could do such a presentation round in Sofia.”

“Thanks to Martin for the perfect planning!”

“No, I think it was a perfect organized workshop, I'm under impression. Just one remark: we all learn in action, it would be great to start with action, not with a lot of words, especially if we talk about such a subject like statistics”

“More practical work with the software. More time to read the preparatory material. I have never used the different software packages before so it would have been useful to see how the data is set up for analysis - we could have even done that ourselves if it was sent out with the preparatory material. More of a workshop style and less theory/lecture. Less participants in order to be able to run a true workshop style. Food was excellent, venue was excellent and social aspect was excellent. Topics were very relevant too.”

“more focus on hands-on session where we could apply the new methods/approaches on HBSC data. - more team-work activities. - organize a follow-up, small quiz or task just to see if people still remember/use the knowledge a few months after the event.”

“to create a small (3-4 person) working teams and change them every day and every morning/afternoon to let everyone to work with everyone.”

“I would suggest to have reading materials beforehand: papers introducing the method, papers with examples of studies in which the methods was applied and manuals about the software. I would suggest also longer lab sessions with individual and group work. And of course - more workshops in the future! :)”

“A workshop like this is one of the best possible introductions to HBSC I could have asked for.”

“The workshop was adapted to "experienced" users of R or M-plus. Difficult to follow one topic from the very basic concepts to the most elaborated ones in only one day training.”

“everything was perfect. excellent organisation, presentation, social events, etc. Bravo! :-)”

“Maybe a more "active" workshop would make the practical benefits of learning about new methods more usable. I really like the idea of including a quiz during the workshop to stimulate discussions and interaction.”

“To have longer seminar, for 5 days To have more exercises using HBSC data”

“Try to find a country which is more accessible and central in terms of transportation, and therefore less travel expensive. No need for 5 star hotels. The workshop room needs better wireless internet and I would suggest different siting organization.”

“Match up level of attendees carefully with level of course - clear guidelines for previous experience/skills necessary . Too much to learn new stats package and new analyses in same session -both really valuable, but one held up the other. Separate these another time perhaps or run a pre-course R familiarisation.”

“Thank you for all the work!”

“Center processes on countries need (based not on frequency of response but importance), Not only PIs opinion but overall level of productivity”

“1) Smaller groups in order to make more close contact between the trainer and audience. 2) More emphasis on software management, including software alternatives. 3) Majority of examples on HBSC as a

common denominator for learning audience.”

“I think you did a good job. It was very interesting, sometimes difficult but this can also be linked to my own limited skills of the used programs. Maybe in the future it could be a good idea to ask the participants to already try to learn the basics for the programs used. This will save some time and can make room for other discussions/focus on the topic.”

REGISTRATION PROCESS

“the registration process was very professionally organized. Thank you for that!”

“Registration process was clear.”

“There was no problem.”

“I even don't know where the problem was - I got to know about the workshop when the last call from Dorothy was made....I had to decide for my participation in very short time but I was aware I wanted to go…”

“For this workshop everything was ok regarding the registration process.”

“everything was fine”

“It was ok”

“Important information received a little bit in the last minute”

“perfect”

“Nothing”

“The registration process was very simple, intuitive, and user friendly. Thank you, Martin!”

“I missed the possibility of screening who is already registered”

“It was ok.”

“Everything is perfect”

“delayed confirmation of registration”

“everything was perfect”

“I answered 4 on the previous question, I have nothing else to add. :)”

“Flexible, excellent”

“I liked everything.”

“I was ok”

“Would have liked more details of course content/level before registration”

“The registration was very intuitive and straight forward. Nothing to complain about.”

“Anything, I do liked”

“Everything was OK.”

TIME STRUCTURE FEEDBACK

“Shorter time for effective learning and more breaks…”

“Very intense but also very productive. Really well done!”

“To devote more time for each topic”

“Timetable was good - sufficient breaks and good start and end time.”

“The time structure was well designed.”

“I think it was ok.”

“Timetable was good the way it was.”

“Maybe a longer break mid-day and go out for lunch? For some PA and fresh air”

“The time was very short for the nature of the workshop”

“More time to cover the topics - more time to do practical examples”

“giving attendance mooooore time”

“Nothing”

WORKSHOP PROGRAM

“I would decrease the number of topic per seminar and have more time devoted to the those left. It would be better to have more practical implementation during the workshop - it would be great!”

“It would have been better to have the trends first day and second day the multilevel. In that way, we already would have a notion of Mplus, to work further on the next day.”

“It might be better if the workshop covers one or two methods with more practical excersises and lab sessions.”

“Too much information for short time course. I would like to understand theorethical issues more before trying to do this practically.”

“More time for practical exercises with eventually fewer topics .”

“I am not sure that the differential item functionning was very useful for everyone. However, it was still interesting, particularly the introduction into the R-software.”

“To arrange more practical exercises”

“more practice”

“The structure - This was not a true workshop style. The only part of the session that I would have classed as a workshop was Franziskas session.”

“I would offer more time to team work and practical exercises mostly with HBSC data.”

“It was well structured and the topics were closely related.”

“Quality of the program was good. My only remark is that during the IRT workshop, the level of R knowledge required of people did not match with our expertise.”

“More examples on HBSC data for understanding how the new knowledge can be used in the survey”

“I think the first program that included systematic reviews and meta-analyses was perfect. And I was a little bit disappointed that those areas didn't happen in the final program. It would be very interesting to have it in a future workshop.”

“As Martin mentioned, a different type of setting might make it possible to bennefit even more from the workshops”

“giving attendance mooooore time”

“I think the quality of the program is good and the purpose as well. However, I think that by providing one-day workshops where most of the experts want to provide as much information there is about the topic, was sometimes overwhelming. Maybe it would be a better idea to dedicate one workshop on one topic and to organize several ones (if possible of course). It would also be interesting to work on the data after learning the methods. If the same structure will be organized again in the future, I think that teachers should try to limit the given information as it is better to spend more time on a limited amount of things then to go too fast and have an audience that cannot follow it anymore.”

BOOKING PROCEDURE

“everything was fine”

“this question is not relevant for me as I arranged the travel myself”

“Nothing - it was very easy and well looked after.”

“I think Linda Banks is doing a wonderful job in managing our(mine) requests.”

“Some difficulties for booking the flights we asked for. It took longer and therefore the rates increased. Also the process of going through a travel agency made the flights more expensive than if we have bought them ourselves, but I understand that is the regular procedure.”

“Bulgaria is too far from Portugal and the travel expenses should take into account from where people are travelling and also if they have financial assistance or not. In my own case, I had to pay for most of the travelling expenses by my own means because I am only a PhD student and the university does not provide financial assistance.”

“I think Linda Banks did an excellent job.”