SVA Qualification: Approach to Witnessing |
Guidance Note |
This guidance note provides an overview of the approach to witness testing that will be taken by Elexon when reviewing Qualification applications. It should be noted that the information contained within this document is for guidance only. The Qualification Process is designed such that the responsibility for meeting the Qualification Requirements and completion of the Qualification Process (including the planning, design and execution of relevant testing) is that of the Applicant.
Elexon will provide support and guidance to Applicants throughout the Qualification Process and will review relevant documentation and evidence (which may include witnessing) in order to assess whether the required standards have been met, and in order to make a recommendation to the Performance Assurance Board (PAB) regarding an Applicant’s Qualification.
Key Features of this Approach
Key features of the approach to witness testing are:
Flexibility – the Qualification Process has been designed to be a flexible process that can adapt to meet the needs of a particular Qualification Applicant. In order to retain this flexibility, the approach to witness testing is also flexible.
Scalability – the Qualification Process has been designed to be scalable depending on the risk presented by a particular Qualification Applicant. The approach to witness testing establishes a common framework whereby the use of witness testing is scalable depending on an assessment of an Applicant’s risk.
Transparency – by agreeing and publishing its approach to witness testing the PAB is ensuring that the Qualification Process remains transparent. Applicants and prospective applicants will be able to form a reasonable expectation of the amount of testing that will be witnessed during their application and also how this might vary if the risks that their application presents are not appropriately mitigated.
Consistency – any witnessing approach employed needs to be consistently applied across applications and any variations in approach (due to insufficient risk mitigation) need to be consistently applied to applications in similar circumstances where applicable.
The diagram below outlines the approach that will be taken to witnessing as part of the Qualification Process.
The
Qualification Process has been designed to enable applicants to complete and submit the
Self Assessment Document (SAD) either as a whole or section by section. In either case Elexon will review initial drafts and accompanying evidence. This initial review will assist in informing the
Applicant on the amount of testing that may be witnessed. Section 1 of the SAD provides an overview of the
Applicant and section 3 provides information on the testing that the
Applicant intends to conduct. The initial drafts of these sections will provide much of the key information that will enable Elexon to assess the risk of the
Applicant and provide initial guidance on the extent and coverage of tests that may be witnessed during the application. Information provided in response to the other sections of the SAD may also inform the programme of witnessing.
Elexon will co-ordinate its review of SAD sections with the Applicant in order to determine the most practical method of witnessing for both Elexon and the Applicant’s organisation and the most efficient witnessing schedule should it be determined that witnessing is required for the Qualification application.
For those Applicants who are also required to complete market entry processes under the Retail Energy Code (REC) Elexon will endeavour to liaise, co-ordinate and align any requirement for witnessing of testing with REC such that the schedule can be optimised for all parties and duplication avoided. It should be noted that, where applicable, Elexon will take into account information and/or evidence provided for the Applicant’s REC entry process application.
The phases involved in witness testing are described below.
Objective of Phase: To review and assess the risk of the Qualification Applicant based on an assessment of the information provided in the Applicant’s responses to the SAD and supporting evidence.
Description of Phase: Following submission of a draft response to a section of the SAD (including evidence where provided) Elexon will make a determination about the risk presented by that particular Applicant. This assessment of risk will be based on tangible and quantifiable factors. Particular attention will be paid to the Applicant’s project management approach, process, systems & technical environment design approach, testing documentation and procedures, specifically:
Approach to testing;
Testing specification;
Test plan;
Expected results of tests;
Defect/issue management process;
Test evidence, sign off or authorisation process for test phase exit reports; and
Details of any relevant testing previously performed by the Applicant.
Other factors that may impact the assessment include, but are not limited to:
The role that the Applicant is seeking Qualification for;
Whether the Applicant is currently operating in that role or a similar role;
The intended dimensions of operation;
The scope and nature of the planned testing;
The control environment in which testing, transition and implementation of the live service will operate; and
Whether the Applicant is using systems and/or processes that either it already uses when providing other agency services or are used by other Qualified Persons operating in the market.
Elexon will review this information during the initial stages of the application in order to provide assurance that the Applicant has controls in place to ensure that the testing process will be well managed and robust. Elexon will assess whether the documentation provided meets good industry practice (e.g. that the project is being managed in accordance with standard project methodology, for example PRINCE2 methodology). Elexon will discuss with the Applicant any issues that it has identified at this stage so that any necessary amendments can be made.
Elexon will discuss the results of any risk assessment with the Applicant. It may be that the Applicant wishes to re-evaluate its approach or to clarify the information provided. Alternatively, the Applicant may wish to provide additional information in support of its application. The submission and review of the SAD and supporting evidence may be an iterative process, allowing for the results of ongoing risk assessment to vary throughout the lifespan of the application.
Objective of Phase: To determine, in partnership with the Applicant, the most practical and efficient witnessing schedule based on the assessment of the responses provided in the draft SAD.
Description of Phase: The findings of phase 1 will drive the initial level and type of witnessing activity that may be performed by Elexon during the application, in order to assess the information provided in the SAD, and will inform the balance that is struck between witnessed testing and evidence review. For example, where the overall approach to testing, including the tests, expected results and defect management is well documented, more emphasis may be placed on evidence of this rather than witnessing of the actual tests themselves.
Elexon will liaise with the Applicant in order to establish the optimum witnessing schedule. Timely submission of section 3 of the SAD and open communication between Elexon and the Applicant will help to ensure that witnessing can be scheduled to meet the requirements of the planned testing.
Objective of Phase: Through the witnessing of a sample of tests and further tests if deemed necessary to provide assurance that the Applicant is following its processes and procedures when performing tests.
Description of Phase: During this phase of the application the Applicant will perform their testing in line with their test plan. Elexon will review the execution of a test at the beginning of this phase to provide assurance that testing is being performed in line with the approach provided and reviewed during Phase 1. Any issues identified during this witnessing will be provided to the Applicant on the day. If issues are identified during witnessing, Elexon may choose to re-evaluate the risk assigned to the application. Where issues have been identified the risk of the Applicant will be re-assessed and the performance of additional tests may be witnessed by Elexon. Similarly, if the Applicant demonstrates a good approach to testing during the planned witnessing activity, the risk associated with the Application may be assessed as lower, and the extent and coverage of witnessed testing may be reduced.
Phase 4: Collating Evidence
Objective of Phase: To confirm, through provision of evidence collated by Elexon (as witness to testing) that testing has been completed in line with plans/expected results and that any outstanding issues have been documented and submitted into a stated defect management process (and, where appropriate, resolved).
Description of Phase: Following completion of Phase 3 Elexon will ensure that any witnessing that has taken place has been documented and will capture the findings of the witnessing that has been undertaken. This information will form part of the evidence to support the review of the SAD and the final report to the PAB.
The Applicant will be able to view and comment upon the findings of the witness and will be aware of the information that form part of the application supporting evidence as a result of the witnessing that has taken place.
What if an Applicant has already performed some testing prior to commencing the Qualification application or prior to submission of the SAD (particularly section 3)?
Elexon will always discuss testing with the Applicant as part of the guidance provided in the early stages of an application and at the initial planning meeting. This may identify testing which is planned to take place prior to the Applicant submitting any SAD responses or evidence. In this case Elexon may determine that some or all of these tests should be witnessed in order to avoid any potential need to repeat the tests at a later stage in the process. Elexon will also discuss with REC the timing of tests that are required under the REC entry processes for the purposes of Qualification applications from Suppliers and SMRAs.
How will Elexon ensure that risk is assessed fairly and consistently?
Elexon will follow processes and procedures that detail how risk should be assessed. Elexon may vary the risk associated with an application (either up or down) depending on other information that is available at the time of the assessment. A precedents register will be maintained internally by Elexon to record any decisions, including the rationale, when Elexon varies the extent/coverage of Applicant testing that is witnessed testing based on risk assessment.
What if an Applicant disagrees with the assessment of risk and the level/extent of witnessing that has been assigned to its application?
In the first instance, if the
Applicant disagrees with the risk or witnessing schedule that has been assigned it should discuss this with Elexon. Any information that can be provided to support the
Applicant’s view should be provided. Where the
Applicant and Elexon cannot reach agreement, the PAB will be asked either by Elexon or the
Applicant (via Elexon) to make a decision. If required, thereafter the
Applicant can further appeal in accordance with
BSC Section J3.7.
What testing can be witnessed?
Any testing performed by an Applicant may be witnessed by Elexon, including, but not limited to:
Arithmetic accuracy testing;
Unit testing;
System testing;
Integration testing;
Regression testing;
Volume and performance testing;
Business process testing; and
Resilience testing.
What happens if the Applicant has identified faults during testing?
Any faults identified during testing that will impact the Applicant’s ability to meet the Qualification Requirements should be resolved prior to go-live of the Applicant’s service.
What if the Applicant has not completed all aspects of testing before the final review of the completed SAD (including evidence) is scheduled to take place?
Normally the Applicant will have completed all aspects of testing before the final SAD and evidence review takes place. However, it is recognised that this is not always possible. When completing its SAD response the Applicant will complete this as if all testing had been completed. That is, the Applicant will detail all testing that it intends to perform, the expected results, and the acceptance criteria and approval process that will be followed when determining that all necessary testing has been completed successfully. Qualification or re-Qualification by the PAB will be based on the Applicant completing all of the actions detailed in and in line with processes and procedures referenced in its SAD response. The PAB will be informed of any testing that is outstanding when asked to make its decision. The PAB may decide to defer its decision until such time as the Applicant has completed any or all further testing.
What criteria will Elexon use for assessing whether all required functional testing has been completed?
An Applicant is expected to perform functional testing to demonstrate that its systems and processes will enable it to deliver all of its obligations in a compliant manner without adversely impacting other participants. Testing requirements are detailed in Appendix 1 of the SAD. Elexon will review an Applicant’s test plan and test schedule to ensure that there is full coverage of the testing requirements. Applicants are expected to test business processes in isolation and concurrently, e.g. change of agent concurrent with change of Measurement Class. Test plans that are based on the Storyboards will be anticipated to cover at least the minimum level of functional testing required. When reviewing test plans Elexon will also be seeking assurance that testing will confirm that data flows will be sent within appropriate timescales and that any flows produced are of the necessary quality and will not fail either data flow or data file validation.
Intellectual Property Rights, Copyright and Disclaimer The copyright and other intellectual property rights in this document are vested in Elexon or appear with the consent of the copyright owner. These materials are made available for you for the purposes of your participation in the electricity industry. If you have an interest in the electricity industry, you may view, download, copy, distribute, modify, transmit, publish, sell or create derivative works (in whatever format) from this document or in other cases use for personal academic or other non-commercial purposes. All copyright and other proprietary notices contained in the document must be retained on any copy you make. All other rights of the copyright owner not expressly dealt with above are reserved. No representation, warranty or guarantee is made that the information in this document is accurate or complete. While care is taken in the collection and provision of this information, Elexon Limited shall not be liable for any errors, omissions, misstatements or mistakes in any information or damages resulting from the use of this information or action taken in reliance on it. |
