Skills for the Digital Economy - Final Programme Evaluation

Descriptions

The Skills for the Digital Economy programme develops and delivers flexible, industry-led training to meet the needs of creative media employers and freelancers working or living in west Wales, north west Wales the Valleys.The programme is backed with £2.6m from the European Social Fund (ESF) through the Welsh Government, with the rest of the funds provided by Welsh broadcaster S4C, the trade association for independent producers in film and television in Wales, Teledwyr Annibynnol Cymru (TAC) and Creative Skillset Cymru.The Skills for the Digital Economy is a four and a half year programme which started in 2011 and will run through to June 2015 with the prime objective of supporting the creative economy in Wales which was identified by the Welsh Government as a priority sector for business support.The Skills for the Digital Economy programme is made up of two main stages:1. ResearchCreative Skillset Cymru conducts research on the skills needs for the creative media workforce in the convergence region, which has informed the development and delivery of training provision.2. TrainingBased on the skills priorities identified in the research, training is tendered out for training providers to deliver. The Skills for the Digital Economy team work with training providers to run full and part time courses as well as short courses. These courses are mainly aimed at industry professionals looking to upskill or make a career change, as well as new entrants to the industry. The training schemes continuously adapt and develop according to industry needs.All our training courses are guided by the Skills for the Digital Economy Industry Steering Group. A further key aim of the programme is to address the diversity and environmental sustainability issues the sector may currently face in the convergence region. This can include increasing gender equality, helping companies ‘green' their productions, or adopting green ICT.We now wish to commission a final report to provide a robust quantitative economic impact evaluation of the Skills for the Digital Economy programme to assess its effectiveness and performance including: achievement of gross outputs and expenditure against its approved targets; adjustment from gross to net outputs after accounting for deadweight effects, as appropriate; assessment of the contribution of outputs to attributable outcomes including any unintended outcomes/effects; assessment of additionality and an assessment of the wider impacts on west Wales, north west Wales and the Valley's economy or more broadly if significant.More specifically to assess whether or not the programme has achieved its objectives and what has and has not worked well, for whom and under what circumstances. To assess the original rationale for the project and whether this rationale is/is not still valid (including how it fits in with ESF priorities and those of other local, regional and national strategies).Assessment of Strategic Added Value (SAV) - broadly, the added value realised through coordinating and influencing activities that result in project outcomes being delivered by other, in addition to outcomes directly delivered through funded activities.A value for money assessment will also be required to demonstrate the cost effectiveness in terms of input/output cost ratios and a cost-benefit analysis to monetise the social and economic return from investment.Confirmation that the programme is on track to deliver against the Creative Skillset Cymru 8 'tests' which can help us recognise where SAV is being generated namely:1. Creating confidence in the prospects for regional growth2. Providing strategic leadership (of regional partners and stakeholders)3. Exerting a strategic influence (over key partners and stakeholders)4. Leveraging investment from other sources5. Developing synergy (in activity of sector partners, stakeholders and target audiences)6. Stimulating a 'scaling up' of beneficial activity7. Enhancing the quality of a desirable regional economic activity8. Encouraging engagement in the regional strategyFinally, to assess the effectiveness of the process of delivery including management, administrative and delivery mechanisms and operational characteristics that have had a significant effect on the scale and nature of the outcomes and impacts realised through the project.1. Industry Led Researcho Has the research led to a greater understanding and detailed knowledge of the regional skills priorities of the Creative Industries workforce in west Wales, north west Wales and the Valleys?o Have industry needs been articulated clearly in the research to influence policy change for the ESF Defined Convergence areas of Wales?o As a result of the research, has training provision been developed for this project that meets industry needs? How well has Creative Skillset Cymru and its ESF Industry Steering Group utilised the findings of the research?o Have employers and stakeholders reported satisfaction with the research?o How well has Creative Skillset Cymru disseminated the findings of this research to a wider audience?2. Training DeliveryThis section will be the main focus of the Evaluation. An overall evaluation of each scheme should then be presented. Below some guidelines have been provided but these are not exhaustive:Evaluate the extent to which this programme is delivering against its targets/indicatorso Is this programme on track to deliver training for 975 participants?Evaluate the quality of the training course /providero % of participants reporting that they would recommend the provider/the course to others;o % of participants reporting that the course met their expectations;o % of participants reporting that the facilitator’s knowledge and expertise was of a high standard;o % of participants reporting the course facilitator’s responsiveness and question handling was of a high standard;o % of participants reporting that course handouts and materials were pertinent and useful;o % of participants reporting that the course represented value for money (participants will be expected to contribute to the costs of the course);o % of participants reporting satisfaction with the application process and pre-application support given by the provider and investigate reasons if there was any dissatisfaction?Evaluate the impact on the participants’ themselves and the extent to which their employability/career prospects have been improved as a result of the training attended:o % of participants reporting the extent to which they have so far been able to apply the skills and knowledge gained from the course in their work;o % of participants reporting the extent to which the course has helped to increase their confidence in a professional context;o % of participants reporting the extent to which the course has helped them to secure new employment or contracts;o % of participants reporting the extent to which the course has enabled them through whatever work they have done, to operate at a higher level than they were previously;o Participants should also be asked to give details of work they have completed since undertaking the course, providing details of the most recent production(s) on which they have been credited.Evaluate the experience of the employers whose staff have attended the training and in particular, the extent to which the training will have impact on the business:o % of employers reporting new contract gained whether at Wales, Network or International level;o % of employers reporting jobs created;o % of employers reporting increased sales;o % of employers reporting increased turnover and profitability;o % of employers reporting their performance has improved;o % of employers reporting increased workforce productivity;o % of employers reporting closing of the skills gap;3.Equal Opportunitieso How successful was the recruitment and marketing campaign to promote the various training schemes to diverse audiences?o How did the training provider publicise the training scheme to ensure access to and participation from the widest possible range of applicants across west Wales, north west Wales and the Valleys?o How many applications did training providers receive and how did they ensure they shortlisted and interviewed the most appropriate cohort?o In what other ways (other than marketing activity) did the training providers ensure the participation of representatives from a wide range of backgrounds?o Is the programme on track to deliver against its diversity targets:o 429 female participants?o 28 BAME participants?o 22 disabled participants?o 400 Welsh speakers?o 68 employers adopting and improving equality and diversity strategies and monitoring systems?4. Environmental Sustainabilityo Did the training provider ensure that there was an element within their actual training programme specifically addressing environmental sustainability?o Did the training provider recruit and market their training in a way that was compliant with our Environmental Sustainability specifications?Whilst we have listed above several examples and questions to consider. However, we cannot stress the importance for those tendering for this work to come up with some innovative evaluation models of their own.THE INTENDED AUDIENCE FOR THE INTERIM EVALUATIONo Creative Skillset Cymru as deliverer of the Programme;o The Principal Funders WEFO;o Match Funders S4C and TAC;o Welsh Government Creative Industries Sector Team and Panel;o Other Welsh Government departments including DFES and ESTo Employers within the sector including the main broadcasters (S4C, BBC Wales and ITV) as well as independent production companies from across our sub-sectors;o Unions in Wales – Equity, BECTU, National Union of Journalists;o Creative Media support agencies and trade associations including Film Agency for Wales (FAW), Welsh Screen Commission (WSC), Teledwyr Annibynnol Cymru (TAC); Producers Alliance of Cinema and Television (PACT); National Union of Journalists; Welsh Books Council; Ffotogallery;o Higher Education Funding Council for Wales (HEFCW);INTENDED USES OF THE EVALUATIONStrategic Intelligenceo Inform internal strategic and business planning e.g. identifies areas for further and new development, share lessons of good practice, improve processes, isolate weakness;o Inform resource allocation e.g. invest in what works and allows disinvestment in unproductive areas;o Maintain and continually improve performance;o Remain responsive.Assurance on performance and impacto Accountability e.g. responding to the funder’s wish to fund impact not activity;o Transparency and trust;o Inform future funding bids;o Inform funders.Evidence of achievementso Accountability to WEFO, match funders and stakeholders;o Demonstrate value added to WEFO, match funders and stakeholders;o Demonstrate, promote and reward success.THE AVAILABLE INFORMATION/EXISTING EVIDENCE BASEData to be collatedWhat type of data will be collated;Data relating to the participants; Diversity Monitoring Forms, Participant Enrolment Forms, Participant records, attendance records, evaluation forms on completers, questionnaires/follow-up information;Data relating to de-minimis;Copies of signed de-minimis forms;Data relating to participant fees; Copies of invoice and bank statements;Data relating to employers assisted or financially supported; Timesheets, beneficiary records which include information on the levelIf financial assistance is given, then adequate supporting documents from the provider to the employer recording grant assistance which includes grant application form, criteria, signed agreement of loan/grant etc.Data relating to projects delivering specialist training in sustainable development; Course materials e.g. handouts and presentation slides,Participant records, attendance records, evaluation forms on completers, questionnaires/follow-up informationData relating to research studies; Research reports, notes or records from dissemination methods (e.g. bulletins, published papers, meetings, seminars, conferences, workshops, online Data relating to employers adopting or improving equality and diversity strategies and monitoring systems; Copies of Annual Report on progress of the integration of equal opportunities within the project. Report would include monitoring data and any actions to be taken forward;Data relating to the procurement of consultants and training providers;We will ensure we keep records of which suppliers have been invited to tender, when the invitations were sent and the replies received.• Copies of all tenders received, together with all correspondence with those invited to tender, will be retained on file, and submitted to WEFO.Data relating to the course impact;Creative Skillset Cymru Participant Evaluation Forms• Evaluation Reports from Training ProvidersNOTE: To register your interest in this notice and obtain any additional information please visit the Sell2Wales Web Site at http://www.sell2wales.gov.uk/Search/Search_Switch.aspx?ID=15610.