...

Table of Contents

Introduction

This document describes a certification program for EdgeX Foundry. This document was developed by the Certification Working Group to

...

Note: The term “certification authority” is used in this document to describe the organization that will be conducting the certification testing/evaluation. It is to be determined if this will be done within the EdgeX community or by a third party.

What is EdgeX certification?

EdgeX certification represents a process where software components that interact with EdgeX are evaluated and confirmed to meet a set of technical requirements and standards that will help insure compatibility. This document describes the framework that will be used by the EdgeX Foundation to verify that software seeking compliance conforms to specified characteristics.

...

For the purposes of this document, we are focused on core services and those services that interact with core services (e.g., device services and application services).

Why is Certification needed?

EdgeX Foundry is based on loosely-coupled micro-services bound by common APIs. Entire subsections can be replaced, combined, etc., with proprietary, differentiated “EdgeX-compliant” offerings. The community and users want assurance that EdgeX will continue to function as designed even when components from other sources are added or replaced.

Certification is critical for multi-vendor ecosystems. By ensuring the services can be easily and reliably integrated to drive an outcome, users have reduced risk and greater confidence in adopting EdgeX and third party components.

Benefits to Stakeholders

There are three primary stakeholders for EdgeX certification. These include the EdgeX developer community, third party partners that use and build products around EdgeX, and the end users that are employing EdgeX to implement projects.

...

• Confirmed interoperability as of a given EdgeX release
• Faster implementation
• Reduced integration risks
• Higher reliability

Certification Levels

The EdgeX certification process will consist of two levels: self-assessment and formal certification.

Self-Assessment

The self-assessment process enables an organization to conduct their own evaluation using EdgeX provided tools to confirm meeting basic certification requirements. This assessment will be a prerequisite for submitting a component to EdgeX for formal certification. Self-assessment alone may be useful for organizations to confirm their code even if they have no interest in pursuing formal certification.

...

Self-assessment is not formally recognized by the EdgeX Foundry. I.e., it does not convey the use of “EdgeX certified” in marketing materials or listing in the EdgeX certified products list.

Formal Certification

Formal EdgeX certification testing will be conducted by a certification authority designated by the EdgeX TSC. Components that successfully complete the certification process will be announced on the EdgeX web site and those organizations will be able to promote their component using the certification logo.

...

Submitted code will not be made public or stored beyond the period needed for testing. All test results will be retained.

Technical Requirements

This section describes the technical requirements for self-assessment and certification.

Assumptions

This section states the assumptions and beliefs that were used during development of the technical requirements.

• All certification submissions and work will be conducted in the English language.
• The certification evaluation process does not require submitters to provide source code.
• Services submitted for certification are expected to be packaged in Docker/Docker Compose.
• EdgeX is agnostic with regard to platform hardware and OS, but that platform independence is not tested.  Submitters are expected to state exceptions to platform independence.
• Certification is not dependent on the programming language or technology used in the service.
• EdgeX is orchestration agnostic.  Services should be designed that way and should not make any assumption about orchestration/ QoS
• EdgeX has versions. The testing tools have versions. Certification will be to an EdgeX release. We assume the testing tools are appropriate for the EdgeX release.

Submission Requirements

Prior to submitting a service for certification, submitters should …

...

• Guidance on service throughput, performance or scale limitations
• Source code, makefiles, or other development artifacts
• Additional runtime artifacts (e.g., containers that run in alternate platform or environments)
• Alternate configurations

Core Service

Submitters, when providing a core service for certification, must also provide…

• Information on any operations, configuration, additional functionality, etc. that differ from the reference implementation (example – providing configuration defaults and options that differ from the reference implementation service).
• No documentation is required if the EdgeX reference implementation documentation applies to the service under review.

Device Service

Submitters, when providing a device service for certification, must also provide…

• The device (hardware, drivers, etc.) or device simulator required to test the device service.
• Indications if the device service was built with (and therefore compliant with) an EdgeX SDK.  If not, the submitter must provide an indication of which features of the SDK are provided and not provided by the device service under consideration.
• Documentation explaining how devices are registered and provisioned with an example.
• Example applicable device profiles
• Guidance/expectations on the amounts and rates of data collection typically seen by the device service with regard to its typical devices.

Database Service

Submitters, when submitting a database using service for certification (Core Data, Metadata, Notifications, Logging, Export Client), must also provide…

• Database initialization code and how is it executed
• Persistence store details (if different from the reference implementation) including installation, cleanup, startup/shutdown procedures, etc.
• Guidance/expectation on data throughput, and scale

Client Application Service

Submitters, when submitting a user interface or any client user of EdgeX APIs must also provide…

• Information on the ability to handle single or multiple EdgeX instances
• Information on the ability to operate with or without the API Gateway running

Testing Requirements

The certification authority will …

...

• Explore and assess any code, configuration, or other artifacts for programming best practices, standardization, language idioms, etc.
• Explore and assess any code, configuration, or other artifacts for quality and depth of comments.
• Explore and assess any code, configuration, or other artifacts for compliance with copyright and license agreement indications.

Needed Tools

Execution of the self-assessment and certification process will require some tools and infrastructure to be assembled by various EdgeX working groups.

Product and Development

The following product and infrastructure items that will be needed:

...

• Consider using existing device services to confirm process and that they can be certified (test our own dog food)
• Assumption that some submitters will want to provide their code from a private repository. Need to investigate how to script pulling the code (private certs?) and running tests.

Process

The following items that will be needed to execute the certification process:

...

• The certification testing process and results are private between submitter and EdgeX Foundry until the process is completed.

Marketing and Promotion

In order to achieve the full benefits of the certification program it must be seen as having value and momentum. The following items will be helpful in that effort:

...

• Duties and requirements
• Selection criteria and evaluation process
• Cost and budget (handling payments)
• Service quality expectations

Certification Process

When an organization is ready to submit their service or product for certification, they will follow a published process available from the EdgeX web site. The beginning of this process is an application.

...

• The Certification WG and TSC will be notified
• Once approved, the submitter will be notified
• Announcement of the certification will be coordinated with the submitter
• Press announcement
• Appearance on EdgeX website

Commercial Issues

This section covers various commercial issues that will be involved in the operation of the certification program.

Eligibility

Formal certification is available to any organization or developer that is willing and able to submit the required information and successfully complete the testing. There is no requirement for a submitter to be a member of EdgeX Foundry or Linux Foundation. This is to encourage maximum participation in the certification program.

Fees

Certification is a process that will take some effort by both the submitter and the certification authority. Organizations that will most benefit from certification will be those that intend to make their components available in the market. These organizations should be expected to help cover the cost of conducting the certification program.

...

Q: Should there be a reduced fee for developers or organizations that are members of EdgeX Foundry?

Version Control

The certification process will be executed against the current release of EdgeX. The certification results and certificate will specify the version of EdgeX that was used for testing. There is no expiration of certification provided: 1) the certified component has not been modified, and 2) the component will be used with that release of EdgeX.

EdgeX Changes

EdgeX will continue to evolve with regular releases. Any list of EdgeX certified services will include the version information of the EdgeX release that was used for the certification. It will be up to users to evaluate if the changes to associated APIs are significant enough to invalidate the certification for the EdgeX release they plan to use.  It will be up to the developer of a component to decide if they want to certify their component on a new EdgeX release. 

Component Changes

It is expected that certified components will change as developers fix bugs and add new functionality. We want to encourage a process of continuous improvement. However, it is easy to imagine scenarios where the changes could cause a component to no longer meet the certification tests.

...

A process for recertifying component changes will be needed. If the changes are for bug fixes, it could be that self-assessment and request for review is sufficient to recertify the component. For enhancements, it may be that a full certification process will be required. This will be determined as we have a better understanding of the certification process as well as the volume and extent of changes to certified components.

Revoking Certification

There may be situations where EdgeX Foundry determines it is necessary to revoke certification of a component. This could be due to

...

We do not expect this to happen often, but it is necessary to protect the integrity of the certification program.

Road Map

As can be seen from the list of support tools required to support certification presented above, there is a lot of work required to get to the point of issuing meaningful certification of EdgeX components. We want to balance market value, time to market, and effort.

Setting Priorities

One of the key decisions is around which type of service should be the starting point. We have ranked them as follows:

...

It is our belief that Device Services offer the greatest value to the success of EdgeX. We are recommending that the EdgeX community dedicate itself to producing the necessary tools and infrastructure to accomplish certification of Device Services within two releases.

Phases

Given the complexity and effort to create a meaningful certification program, we are suggesting a phased approach in the crawl, walk, run model. Release targets for the phases will be defined in collaboration with the TSC and other WG.

Crawl

The goal of the Crawl phase is to be able to evaluate and declare an EdgeX Device Service component as being certified. The focus is on making sure the process and tests exist and are sufficient, even if highly manual for execution.

...

• A minimum of 5 EdgeX device services are certified

Walk

With a working process in hand, the priority of the Walk phase will be to make setting up tests and recording results more automated. A second goal will be to make a self-assessment test package available. We should gain sufficient knowledge during this phase to make a decision on the what resources will fulfill the certification authority role.

...

• Ability to execute certification tests of device services and gather results in a repeatable manner by someone who may not be an EdgeX expert or developer
• First release of self-assessment test package for Device Service
• Identify the requirements and selection process for a third party to act as certification authority

Run

With a working and automated process, the Run phase will focus on expanding tests to cover the remaining service types.

...

• Certification of all EdgeX service types (Device, Application, Core)
• Establishment of a certification authority

Open Issues

This section contains questions that are still open and under discussion by the Certification WG.

...

Q: Any types of organizations that should have reduced or no fees?

Glossary

Validation: The assurance that a product, service, or system meets the needs of the customer and other identified stakeholders. Is the project designed to meet the desired objectives/success criteria?

...

• MUST is equivalent to REQUIRED and SHALL indicating that the definition is an absolute requirement.
• MUST NOT is equivalent to SHALL NOT and indicates that it is an absolute prohibition of the specs.
• SHOULD is equivalent to RECOMMENDED means that there are valid reasons to ignore a particular requirement, but the implications need to be weighed.
• SHOULD NOT and NOT RECOMMENDED means that a particular behavior may be acceptable or useful, but again, the implications need to be weighed.
• MAY means OPTIONAL and that the requirement is truly optional. Interoperability with different systems that may or may not implement an optional requirement must be done.

Appendix 1. Device Service Certification Notes

This appendix is provided to give some insight into the level of detail that may be required for conducting certification of each service type.

...