2 DIP Interfaces

2.1 DIP landscape

1. validate message/event headers;

2. validate the message/event structure; and

3. address and route messages based on content.

4. to secure the mTLS channel between their PEP and the DIP; and

5. to digitally sign Messages that are sent to the DIP.

2.2 DIP Core Services

1. Message Brokering – the core component of the DIP responsible for relaying messages between Market Participants includes the following functions:

1. Message Validation, Security and Repudiation;

2. Message Archiving;

3. Message Replay;

2. Message Brokerage Management – this includes the following functions:

1. Manage Message Channels;

2. Add/Remove Message Channels;

3. MPAN based addressing;

3. Participant Management:

1. DIP On-boarding and DIP Off-boarding;

2. Participant administration including DIP Role maintenance;

4. Service Management:

1. Application management;

2. Incident Management;

3. Change Management;

4. Availability Management;

5. Problem Management;

6. Data Management;

7. Capacity Management;

8. Certificate Management – the possibility exists that this function could be carried by the DCC SWIKI services – yet to be agreed;

5. Reporting:

1. Tracking workflows across MHHS business process.

The Admin Portal is the DIP Manager’s means of administering the DIP, the user portal is for DIP Users

2.3 DIP Connections

1. Message integrity – to ensure that the Message received is the same as it was when sent and has not been accidentally or maliciously changed in transit;

2. data origin (Message) authentication – assurance that a given entity is the original source of the received data – the Message is authenticated using a cryptographic key. Where required, each Message is individually authenticated to allow the recipient to verify the claimed identifier of the Message;

3. Mutual Authentication – both parties authenticate each other such that both are assured of the identity and authenticity of the other; and

4. Confidentiality – ensures compliance with Data Protection Legislation.

2.4 DIP Users

2.4.1 There are primarily three different types of DIP Users:

1. Active Market Participant;

2. Non-Active Market Participant; and

3. DIP Connection Provider (DCP).

2.5 Active Market Participant

2.6 Non-active Market Participant

1. the DCP will connect to the DIP using their own Digital Certificate to secure the mTLS connection;

2. the DCP will sign the messages sent on behalf of the Non-Active MP using the DIP User’s Digital Certificate (see paragraph 6.3.2).

3 DIP Access and participant engagement

3.1 The DIP Portal

1. reporting on audit and monitoring data;

2. manage users and Digital Certificates; and

3. perform DIP functions such as replaying Messages.

1. DIP User representatives are invited by the DIP Manager or their DIP User Administrator;

2. the user receives an email welcoming them to the DIP. Within this email is a redemption link;

3. the user clicks upon the link, and they are taken to the DIP Portal where they are asked to sign-in with their company credentials;

4. upon successful sign-in the user is asked to configure Multi-Factor Authentication (MFA). This involves configuring one or more second factor authentication mechanisms (the first factor being the password) such as:

1. Microsoft authenticator app;

2. mobile phone (text or phone call);

3. Google Authenticator (Chrome extension);

5. the user will then review and accept the terms of use for the DIP Portal; and

6. once accepted the user will be able to access the DIP Portal.

3.2 Registration

3.3 PKI Role Governance

3.4 Interfacing with the DIP

4 Public Key Infrastructure

4.1 Overview

1. a Public Key that is made widely available and acts as authentication anchor;

2. a Private Key that is used to produce digital signatures on Messages.

4.2 DIP Certificate Authority

4.3 Chain of trust

4.4 Digital Certificate usage

1. the creation, sending, receiving, and processing of communication within the DIP environments;

2. Symmetric key generation (Digital Signature, Key Agreement);

3. TLS communication (Digital Signature, Key Agreement, TLS web client authentication, TLS web server authentication);

4. Authentication and Non-Repudiation (Digital Signature, Non-Repudiation, key encipherment, data encipherment, Key Agreement, TLS web client authentication. TLS web server authentication).

Digital Certificate Path

4.5 Key management

4.6 Digital Certificate Revocation

1. when any of the information in the Digital Certificate is known or suspected to be inaccurate;

2. upon suspected or known compromise of the Private Key associated with the Digital Certificate;

3. upon suspected or known compromise of the media holding the Private Key associated with the Digital Certificate.

1. immediately upon becoming aware that the Digital Certificate has been compromised, or is suspected of having been compromised; or

2. immediately upon ceasing to be the DIP User in respect of that Digital Certificate.

5 DIP Security Requirements

5.1 Security requirements

The security requirements in the DIP Rules are based on National Cyber Security Centre Cyber Assessment Framework (NCSC CAF) recommendations that organisations adopt a recognised baseline of security controls such as those prescribed in ISO 27001 (or equivalent, e.g., NIST 800- 53), supplemented with a set of enhanced level controls on which DIP Users are reliant to protect the DIP Ecosystem.

1. Mutual TLS (mTLS) for transport layer security and authentication; this should be TLS 1.3 but shall be TLS 1.2 as a minimum;

2. Message signing for non-repudiation;

3. mTLS for strong security authentication/authorisation at the API; and

4. API key (ingest only) for rate limiting and usage tracking.

5.2 Information Security Management System certification

The DIP Manager is responsible for ensuring the DIP is certified against ISO 27001. Certification shall be provided by a UKAS-certified auditing body.

DIP Users shall carry out regular risk assessments of their systems and interfaces. Details of how to conduct risk assessments can be found in well-established assessment models, such as ISO 27005 or HMG’s Information Assurance Standards 1 & 2.

5.3 TLS requirements and configuration

5.4 TLS key generation and Certificate Signature Requests

1. any information that constitutes a trademark unless it is the holder of the Intellectual Property Rights in relation to that trademark; or

2. any confidential information which would be contained in a Digital Certificate issued in response to that CSR.

1. shall reject it and refuse to issue the Digital Certificate which was the subject of the CSR;

2. shall inform the DIP User which made the CSR why it was rejected.

6 Managing Digital Certificates

6.1 DIP Users responsibilities

1. Issuing Digital Certificates;

2. Renewal of a Digital Certificate – prior to expiry;

3. Reissue of a Digital Certificate; and

4. Revocation of Digital Certificates.

6.2 DNS Domain creation/verification

6.3 DIP Digital Certificate requirements

1. One for non-production environments; and

2. One for production environments.

1. The DIP User owns the Digital Certificate required for Digital Signing and will need to authorise the DCP to use this Digital Certificate on their behalf; and

2. The DCP will need their own Digital Certificate for securing the mTLS connection.

1. Scenario One – DIP User A direct to DIP:

1. DIP User A manages their own connection to the DIP using the same multi-use Digital Certificate for TLS and digital signing;

2. They have one Digital Certificate for non-production environments, and one Digital Certificate for production environment;

2. Scenario 2 – DIP User B uses DCP X for connection to DIP;

1. DIP User B authorises DCP X to sign Messages on their behalf using DIP User B’s Digital Certificate (this certificate does not allow mTLS);

2. DCP X uses their own Digital Certificate for mTLS;

3. Scenario 3 – DIP User C uses multiple DCPs;

1. DIP User C requests multiple Digital Certificates for DCP X, DCP Y, etc.;

2. DIP User C authorises both DCP X and DCP Y to sign Messages on their behalf;

3. DCP X uses their own Digital Certificate for mTLS;

4. DCP Y uses their own Digital Certificate for mTLS.

6.4 Certificate Subscriber obligations

6.5 Relying Party Obligations

1. Submit a query in search of a Digital Certificate;

2. Verify a Digital Signature created with a Private Key corresponding to a Public Key contained in a Digital Certificate;

3. Rely on a Digital Certificate, or on the information contained within a Digital Certificate in support of a transaction;

4. Otherwise rely on, or use any information or services provided by the DCA.

6.6 Certificate Revocation Request

6.7 Digital Certificate renewal

1. 90 days prior to the Digital Certificate expiring;

2. 60 days prior to the Digital Certificate expiring;

3. 30 days prior to the Digital Certificate expiring;

4. 1 day prior to the Digital Certificate expiring.

6.8 Private Keys

6.9 Encryption Secrets/Password Guidance

7 DIP Messaging

7.1 DIP Message pattern

7.2 Message Architecture

7.3 DIP IDs

1. DIP ID and DIP Role will be used to identify DIP Users within the DIP

2. DIP Addressing/Routing utilises DIP ID & DIP Role code

3. ISD (entity #M16) enables a 1-2-1 mapping between each DIP ID/ Role combination and MPID

7.4 Message/event channels

1. a Send Message API for incoming messages (https:/api.{environmment}. energydataintegrationplatform.co.uk/ {version}/ dip-channel/ {IF-xxx}); and

2. a Receive Messages Webhook for outgoing messages.

1. Level 1 - initial synchronous validation by the DIP;

2. Level 2 - secondary asynchronous validation by the DIP;

3. Level 3 - initial synchronous validation by the DIP User;

4. Level 4 - secondary asynchronous validation by the DIP User.

1. Level 5 – Response through a new message channel where the original recipient becomes the new sender e.g. IF-006 provides a response to IF-005;

2. Level 6 – where the response is via a third-party, e.g. Service Appointment by the Supplier (IF-031) via the Registration Service - the outcome and if appropriate error reason code will be contained within the body of a subsequent message (IF-035).

1. Recipient registers Webhook for receipt of publication; sender registers Webhook for the receipt of status messages;

2. Sender sends message/events to the DIP via the post /DIP-Channel/{IF-xxx} API;

3. Initial receipt of events/messages by the DIP is accompanied with an auditable acknowledgement. This will include the Level 1 validation HTTP response;

4. The message traverses through the DIP where it undertakes internal processing where Level 2 validation also occurs;

5. Message is relayed to the intended recipients;

6. Onward delivery of message/events from the DIP to the recipient occurs via registered Webhook with an auditable acknowledgement and Level 3 validation;

7. The recipient will undertake further validation checks (Level 4);

8. If an error condition is found these are reported back to DIP via the Status Message API;

9. Both the Level 3 synchronous and Level 4 asynchronous response reported back to Sender;

10. DIP relays all Level 2/3/4 responses to the Sender;

11. Sender receives all Level 2/3/4 responses via the Status Messages Webhook.

1. a synchronous check at the API that is communicated directly back to the Sender, and

2. further asynchronous checks and addressing that are carried out once the message has been initially received.

7.5 MPAN Address Maintenance Service

1. Each Registration Service must present an API that the DIP can query to initialise its permanent data store;

2. Data will need to be kept in two places: a permanent and a data cache;

3. The cache is populated from the permanent store and provides a fast lookup and the permanent store provides the permanent store;

4. The MAMS receives an input MPAN and message channel, and returns a list of downstream recipients;

5. After initialisation, the DIP is responsible for maintaining MAMs data in real time;

6. All message channels pertinent to the registration data flow will send a message to an internal queue within the DIP – This queue provides the MAMS function information;

7. Both permanent and cache are updated with the new details; and

8. MAMS can recognise a change of details and retain data for the historical lookup as 'old' messages may arrive that need to be sent to a previously responsible party;

1. For example, late consumption data (INTF-011) would need to be sent to the set of secondary parties that are pertinent for the specific day for the meter read – in this scenario, the header would define an applicable date/time field used for addressing.

7.6 Message structure

1. all data items properties are marked as required, where data items are optional then they will be marked as nullable.

2. The underlying datatype for each data item will reflect the nullable property, i.e., nullable – true/false.

7.7 Message routing

1. Always – This option does not require specific addressing action by the Sender. Configuration on the DIP will ensure that certain Messages are always delivered to certain recipients. For a specific message channel a message is either always sent to a named DIP User or all DIP Users assigned to a designated role (role is assigned to the message channel)

2. Secondary – The DIP holds Routing Table of 'Recognised Parties' e.g. Registration Service, LDSO, Current Supplier, Current Service Providers, etc. The DIP uses this routing table to address certain messages. The MPAN core in the M0 block and optionally the Event Code and/or other fields (to determine the targeted roles) provides the lookup to the MPAN lookup address function.

3. Primary – Set by the Sender of the message. The purpose of this is allow a message to be addressed to a specific DIP User or more commonly where only the sender is aware of the correct recipient. DIP uses the ‘Primary Recipient List’ in the A0 block in the message header.

7.8 Message/event obfuscation

1. String values - multiple ‘x’ for the maximum length of the field

2. Date values – the date ‘1900-01-01 00:00’ is used

3. Numeric values – no requirement at present

4. Boolean values – true

5. DIP IDs – ‘9999999999’ (to ensure it remains a numeric string for regex validation)

7.9 Message Compression Handling

1. REP-002 Supplier report for DUoS – aggregated data;

2. REP-002A Embedded Network report for DUoS – aggregated data;

3. REP-002B LDSO report for DUoS – aggregated data;

4. REP-003 BM Unit Allocated Demand Volumes to Market Participants;

5. REP-003A Aggregated BM Unit Allocated Demand Volumes to Supplier;

6. REP-004 Supplier Deemed Take Report;

7. REP-006 Aggregated Uncorrected volumes by CCC to Market Participants;

8. REP-007 VAS Exception Report to Suppliers and BSCCo;

9. REP-008 MDS Exception Report to LDSOs;

10. REP-009 EMRS Report for Suppliers;

11. REP-900 LDSO – DUoS E-Bill;

12. REP-901 LDSO – Aggregated DUoS Charges;

1. P03 property – to be used for the compressed payload following the instructions above; and

2. Schema property – this is the uncompressed message to be used for validation and will not be populated/ contain any data.

7.10 Message De-compression Handling

1. The receiver BASE64 decodes the contents of the CustomBlock (P03 property as per previous section);

2. The receiver takes this BASE 64 data and decompresses using GZIP;

3. The receiver will now have the decompressed payload for validation and onward processing.

7.11 Message configuration requirements

7.12 Message Egress

7.13 Message Ingress

7.14 Capacity Management

7.15 Message Security

1. Public Data – digitally signed;

2. MPAN – digitally signed;

3. MPAN + PII – digitally signed;

4. MPAN + Consumption data – digitally signed.

7.16 Message Alarms

8 Message choreography

8.1 Message choreography

8.2 Simple Message Exchange

8.3 Level 1 validation - DIP Synchronous rejection

8.4 Level 2 validation – DIP asynchronous rejection

8.5 Level 3 response - Recipient Synchronous Error (with retry)

8.6 Level 3 validation - Recipient Synchronous Response (no-retry)

8.7 Level 4 validation - Recipient validation response (asynchronous)

8.8 Consumption Replay

8.9 Recipient timeout and ‘Dead-Letter Queue’ handling

8.10 Error Handling & Message Distribution Patterns

8.11 Batch Message Handling

8.12 Workflow message handling

1. For IF-005, a Metering Service Smart (MSS) or Metering Service Advanced (MSA) will send a message to the Registration Service, using the DIP;

2. As part of Level 1 validation, the DIP will respond synchronously back to the MSS or MSA and this will contain the Correlation ID (DT-011). The MSS/ MSA will need to store this ‘Correlation ID’;

3. The DIP will (also) add the Correlation ID to block D0 and send PUB-005 to the Registration Service with the added information (in the common blocks), as well as the information in the original messages (in the common and custom blocks). The Registration Service (in this example) will also need to store this ‘CorrelationID’; and

4. At this point in time, both the MSS / MSA and Registration Service will have / store this "Correlation ID".

5. At a future point in time, as per the process, the Registration Service will send a message, using IF-006.

6. For IF-006, the Registration Service will populate the S1 block with the sender ‘Correlation ID’ (as this is a message that requires correlation). This is populated from Step 3 above (the one the Registration Service stored from IF-005).

7. The DIP will not change the data in the S1 block, it will pass this information through as per the design. The DIP will not populate the D0 block with a new Correlation ID. This way we have one Correlation ID that is consistent across all related messages.

8. The DIP will send this message to the MSS/ MSA (in this example) using the PUB-006 message and this will contain the ‘Correlation ID’. The MSS/ MSA can look this up in their systems against the ‘Correlation ID’ stored previously (as described in Step 2 above) to continue processing as per the design.

9 Use of APIs and Webhooks

9.1 Send Messages API

1. Message Construction – On sending a message the Sender is responsible for construction the message header and payload as described in the section below. The use of Sender Unique Reference is described below.

2. Message Addressing – When a message is sent, the DIP User will need to be aware of the type of addressing that needs to be applied and address the message accordingly. The type of addressing for each message channel is defined in the EMDS. The Sender is only responsible for primary addressing, the DIP is responsible for secondary and always addressing.

3. Message Signing – Once the message has been constructed it needs a digital signature written to the message header

4. API Call – The sender has the ability to send multiple event/messages in a single API. Messages can only be sent to a single message channel within the same API call.

5. API Response – The message sender will need to process the API response. The HTTP return code will provide the return for the whole transaction and needs to be considered with the response body as it will provide the details for the acceptance or otherwise for each message sent and will provide Transaction IDs for each message referenced by the Sender Unique Reference.

6. Back Off and Retry – DIP Users are expected to adopt a retry with exponential back off (up to a configurable maximum wait time) if there is a failure in connecting to the DIP.

7. Multiple Connections – DIP User may have multiple connections to the API endpoint. DIP Users may wish to logically separate their own interfaces and have a single connection per interface, and the DIP will support this pattern.

9.2 Send message DIP Processing (Level 1 Validation)

1. Validation of API key and establish Sender;

2. Ensure Interface Id is correct;

3. Schema validation checks are undertaken via the definitions described in the DIP Swagger. If schema validation fails on a single message within a transaction then all messages within the transaction are rejected with a ‘400’ code and a MSG1001 message in the response. Some of the checks are undertaken here include:

1. Check message structure, i.e. mandatory blocks are present for the corresponding interface and event code;

2. Check the message validation checks for the common blocks:

1. Check the environment is set correctly;

2. Check Interface Id/Event Code pair is an allowable combination;

3. Check the message Schema version aligns (API enumeration defines allowable values);

7. The following are undertaken outside the schema validation checks:

1. Check the Sender Id (“logical” sender) is entitled to send message for the Event Channel for the given role;

2. Check the DCP is entitled to send message on behalf of the “logical” sender for the given role/channel;

3. Check Sender Unique Reference is formatted correctly and unique;

8. Generate a Transaction ID (see below);

9. Generate a Transaction timestamp, i.e. record the current date/time (UTC);

10. If the message channel is configured to generate a Workflow Correlation Id;

11. Generate service desk/audit links;

12. Write the Transaction ID, Transaction timestamp, Workflow correlation Id, status code, status message and audit/service desk links to the response body;

13. Generate return code based on the results of the above checks for the all the messages received in the single transaction.

9.3 Receive Message Webhooks

1. The message channel ID that the corresponding Webhook needs to service, eg. IF-001

2. The DIP ID of the message recipient, eg. 1001012345

9.4 Receive Message Call back Request structure

1. the URL for the call back should not specify a port number as all communications are through standard HTTP ports, i.e. 443 for HTTPS;

2. the maximum number of messages the recipient can receive in a single transaction; and

3. the maximum payload size - the payload size sent in the call back (from the DIP to the DIP User) will not exceed this limit. This limit is ignored (by the DIP) if a single message cannot be transmitted.

9.5 Recipient responsibilities

1. Synchronous (Level 3); and

2. Asynchronous (Level 4).

9.6 Replay events

1. Replay API; and

2. Re-queue API.

1. Message Channel

2. Message Recipient DIP ID

1. Event Code (optional)

2. MPAN

3. Transaction IDs – array of message Transaction IDs to replay, or

4. Date/time from – transaction time from which the first message needs to be replayed.

5. Date/time to (optional) – transaction time message to be replayed

6. Version – message version (optional)

1. Through the UI interface on the DIP

2. Through an API Call

9.7 Message return codes and response body

9.8 DIP API Actions

1. Message Construction – The DIP will augment the original message with the added information detailed below in section 4.6.4.

2. API Call – The DIP will call the API declared in the call back; only messages on the same messages channels will be sent in the same API call. The payload size will not exceed the size specified in the call back request; the DIP will collate all outstanding messages on the channel and if there is a message that exceeds the limit requested, then that message will be sent in the next transaction.

3. API Response – The HTTP return code will provide the return for the whole transaction and needs to be considered with the response body as it will provide the details for the acceptance or otherwise for each message (see below for response code).

4. Back Off and Retry – The DIP has a retry with exponential backoff approach (up to a maximum wait time) and additional “circuit breakers” to ensure efficient handling of broken connections to DIP Users.

9.9 API Version Control

1. MAJOR version – when you make incompatible API changes;

2. MINOR version – when you add functionality in a backwards compatible manner; and

3. PATCH version – when you make backwards compatible bug fixes.

9.10 Message channel versioning

1. MAJOR version – a new major version of the API will be created when a new piece of functionality is required, or a change is required that will completely break the existing API. Under this scenario all interfaces will only have forwards looking data definitions and not support previous versions of data.

2. MINOR version – Where a new message channel is created, then a new minor version of the API will be created. This will not require a new API to be created, however, all the corresponding interface and publications data definitions will be need to be updated to accommodate the new channel.

3. PATCH version – Where the data definition of a single interface changes, then that will necessitate a patch version of the current version of the interface. Under ‘normal’ circumstances the incoming interfaces will only support the new version of the data definitions. If there is a requirement to support multiple version of the same interfaces (incoming) then a new minor version is required. In this scenario the publication of the messages will also support just the new current version. All data must have already traversed through the DIP.

9.11 Multi-version support – API and message channels

9.12 API Keys

9.13 API Key rotation

9.14 API Key(s) and DCPs

10 Signatures

10.1 Digital signatures

2. the first is covered by the establishment of a secure communications channel provided by mTLS, as described above;

3. the second is the authentication of individual Messages which is achieved through the application of a digital signature.

10.2 Digital signature formats

10.3 Message signing

1. generate the content of the Message, this is the Message body;

2. create a SHA256 hash of the Message body. BASE64 encode the output from the SHA256 hash (this is the SHA256 content hash);

3. generate an ISO8601 timestamp in the format: YYYY-MMDDTHH:MM:SS.XXX:Z. This is the Signature Date;

4. concatenate the Signature String in the following format:

1. Verb;Destination;Signature Date; SHA256 Content Hash;

2. Verb = Uppercase only, Destination = Lowercase only;

5. create a SHA256 hash of the Signature String (this is the SHA256 Signature string);

6. sign the SHA256 signature string using the Private Key of the signing certificate. This is the signed signature. Utilise RSA PKCS1 signature padding;

7. create BASE64 encoded string of the signed signature;

8. add to the request the header: X-DIP-Signature with the value of the BASE64 encoded signed signature;

9. add to the request the header: X-DIP-Signature-Date the value of the Signature date;

10. add to the request the header: X-DIP-Signature-Certificate with the BASE64 encoded signing certificate (this will contain only the Public Key);

11. add to the request the header: X-DIP-Content-Hash with the SHA256 Content Hash.

10.4 Verifying signatures

1. verify the Digital Certificate received (X-DIP-Signature-Certificate) is trusted and valid;

2. take the value of the header: X-DIP-Signature and decode using BASE64. This is the BASE64 Decoded Signature;

3. build a comparison string:

1. Verb - from HTTP Request received;

2. Destination - from HTTP Request (the url of the endpoint the Message has been received on);

3. Signature Date - ISO8601 timestamp in the format: YYYY-MM-DDTHH:MM:SS.XXX:Z (retrieved from the X-DIP-Signature-Date);

4. Content Hash - Create a SHA 256 hash of the JSON Message body received. Base 64 encode the output from the SHA 256 hash. For any request with an empty body (GET, DELETE requests) assume an empty JSON body – {}

4. create a SHA256 hash of the comparison string. This is the SHA256 hashed comparison string;

5. verify the SHA256 hashed comparison string with the BASE64 Decoded Signature, using the Public Key of the Digital Certificate received (X-DIP-Signature-Certificate).

10.5 Signature key generation and Certificate Signing Requests

11 DIP capabilities

11.1 Number of channels and DIP Users

1. Fifty message channels (five to ten channel annual increase)

2. Two hundred senders (twenty senders annual increase)

3. Two hundred recipients (twenty recipient annual increase)

11.2 Transactional volumes

11.3 Message Latency

1. up to average hourly volume, 90th percentile time of 3s or less

2. up to average hourly volume, 99th percentile response time of 10s or less

3. from average hourly to peak hourly volume, 90th percentile response time of 6s or less

4. from average hourly to peak hourly volume, 99th percentile response time of 30s or less

11.4 System availability

1. Percentage of uptime – 99.95% (unplanned)

2. Mean time to recovery (MTTR) – 60 minutes

3. Mean time between failures (MTBR) – n/a

4. Recovery time objective (RTO) – 60 minutes

5. Recovery point objective (RPO) – 0

Amendment Record