Examination
As the Forum aims to keep the certification free for practitioners, an online examination process has been chosen. The summative assessment will be conducted primarily using multiple-choice questions. For each skill a pool of questions and answers will be created, drawing from both internal and external contributions, and an examination will consist of randomly selected questions. Future developments will include questions beyond the multiple-choice type.
We believe the incentive to deliberately cheat during the assessment, e.g. by having the exam filled by someone else, is low. Therefore, we address this issue in a lightweight and cost-effective fashion. Our process deploys several strategies to minimize the risk of cheating, such as raising the examinees’ awareness, using a large pool of questions, setting time limits for each question, and a delay between registering for and taking the actual examination. Since knowledge can quickly become obsolete, each certificate needs to indicate when (month and year) the qualifying examination took place. Also, because the examination of a single fine-grained leaf-level skill would be too easy to pass with short-term memorization and more prone to cheating, the certificates bundle multiple skills together. To ultimately provide trust, the Forum hopes to provide the automatic generation of short tests for prospective employers that would allow validating the knowledge of applicants under their own supervision or in assessment centres.
We manage exam questions internally within the HPCCF consortium. To facilitate external contributions, we provide on our wiki-webpage a light-weight interface for suggesting a question for each skill-set which can be used by anyone – those skill-related questions are reviewed and ultimately managed by the exam curator who is part of the steering board. A well-defined process is being created for prospective contributors to verify the suitability of the contributed questions. For a question to be approved, it will need to undergo a review process where comprehensibility, logic (does it have at least one clear answer?) and rigor (does the question lead to the expected answer?) have to be met. Particularly, each exam will be designed considering a variety of aspects such as competency assessment, learning goals, exam objectivity, and taxonomy of educational objectives \cite{krathwohl2000}.
As HPCCF exams are potentially interspersed between courses for a particular examinee, examinees may profit from the forward learning effect, i.e., the outcome of such a lightweight test may influence how a practitioner will study the material further. Lecturers, who will be provided with “their” examinees summarized outcomes, may profit from insights to conceptual issues course participants may exhibit.
The questions are the only proprietary component of the HPCCF – using restrictive license terms for authors while giving them credit. By providing potential employers with our tests, too, will have the means to test how an exam is seen and complete it from a student’s point of view; this is a measure with established relevance \cite{Brink_2011}.
Related Efforts
The Certification Program is a new community-wide effort and to the best of our knowledge nothing similar was attempted at such a scale, both with respect to the comprehensibility of the covered content and the international reach of the effort. The other related efforts mainly focus on either providing a comprehensive catalogue of existing training materials and opportunities, providing simple badges confirming participation in a specific training event, or establishing a branded well-defined content and teaching practices recognised by the community. A number of institutions and organisations attempted to catalogue the existing training materials, keep a list of training events and bring the HPC training community together, the ongoing efforts include:
PRACE Training Portal, the
HPC University and the
ACM SIGHPC Education Chapter. An example of a badging effort is
XSEDE Training Badges Program. Finally, the last example refers both to the
Carpentries initiative, and
HPC Carpentry which was developed in recent years. The HPC Certification Forum recognises the importance of these efforts and is actively engaging with their contributors.
Conclusions
The program of the HPC Certification Forum allows the existing content to be re-used but also makes it possible to create a new ecosystem in which HPC centres, research labs, academic institutions, and commercial companies could offer the best of their teaching material. The HPCCF aims to support existing activities and complements them by providing a unified and clear way of mapping out the relevant HPC competencies. It should be emphasised that the HPCCF does not regulate the content of training material; we purposely separate the definition of skills, the examination, and the certification from the content delivery. The program does not prescribe a curriculum or any fixed order in which skills should be obtained, thus providing flexibility. It eases the navigation between different competencies without being overly restrictive. We are hoping that a majority of the existing and newly created teaching resources can be branded indicating the skills they cover.
We believe the program will bring multiple benefits to everyone involved in HPC teaching and training. Making clear what skills are required or recommended for a competent HPC user would be helpful to both the HPC service providers and practitioners. Training providers could bundle together skills that are most beneficial for specific user roles and scientific domains, which would allow practitioners to browse through skills to quickly identify and learn the skills required to perform their tasks. The variety of training offered within the HPC community makes finding the right resources more complicated than it should be. We hope that the certification program will eventually provide useful information on where the desired skills are taught. The examination confirming that a certain set of competencies has been acquired makes the learning process more complete and meaningful.
By participating in the program, HPC training providers can increase the visibility of their teaching opportunities and share their resources more effectively. The mapping of the skills defined by the program onto the existing training materials should also help to identify any potential gaps and improve the integrity of the offered training. Finally, the certificates recognized by the whole HPC community will simplify the inter-comparison of independently offered courses and provide additional incentives for participation. Overall, the flexibility of the program allows for the construction of more personalized and just-in-time pathways to learning about HPC.
To achieve these goals, the forum welcomes contributions from volunteers. For this initiative to truly fulfill its role the involvement of the members of the HPC training community with diverse backgrounds and experiences is required. As the HPCCF community is managed collaboratively according to self-managed evolving rules \cite{Ostrom_2015}, it is welcoming and the expected contribution is not demanding, due to the informal nature of the forum.