April 25Revised May 2, 2019

Table of Contents

...

• Presentation of a letter of certification
• Use of certification logo
• Joint press announcement by LF Edge
• Being listed on EdgeX web site as being a certified component

...

EdgeX Foundry will make available a set of software tests and procedures that developers can download and use to conduct their own certification readiness assessment.

There is no registration process required for self-assessment. The tools for the assessment will be publicly available for download.

Self-assessment is not formally recognized by the EdgeX Foundry. I.e., it does not convey the use of “EdgeX certified” in marketing materials or listing in the EdgeX certified products list. However, it may be used as a qualifier for inclusion on lists of "EdgeX Ready" components.

Formal Certification

Formal EdgeX certification testing will be conducted by a certification authority designated by the EdgeX TSC. Components that successfully complete the certification process will be announced on the EdgeX web site and those organizations will be able to promote their component using the certification logo.

...

• The Certification WG and TSC will be notified
• Once approved, the submitter will be notified
• Announcement of the certification will be coordinated with the submitter
• Press announcement
• Appearance on EdgeX website

Commercial Issues

This section covers various commercial issues that will be involved in the operation of the certification program.

Eligibility

Formal certification is available to any organization or developer that is willing and able to submit the required information and successfully complete the testing. There is no requirement for a submitter to be a member of EdgeX Foundry or Linux Foundation. This is to encourage maximum participation in the certification program.

Fees

Certification is a process that will take some effort by both the submitter and the certification authority. Organizations that will most benefit from certification will be those that intend to make their components available in the market. These organizations should be expected to help cover the cost of conducting the certification program.

Certification is not intended as a “profit” generating revenue stream. Any fees are intended solely to cover the cost of administration.

During early phases of the certification program, while testing is being conducted by the community, there should be a period where the fees are waived. This will encourage developers to submit their components for certification and allow EdgeX Foundry to learn and improve the process. Regardless of fees or certification authority, any component that is certified will have met the appropriate technical requirements. Once self-assessment is available, and/or if a third party certification authority is involved, the fee waiver will end.

The fee schedule is to be determined once the typical level of effort has been identified for each type of service.

If the effort needed to retest a component is found to be low, there may the opportunity to have a reduced fee for recertifying a component on a new EdgeX release.  This is to be determined once we have more experience with the testing process and potential costs of the certification authority.

Q: Should there be a reduced fee for developers or organizations that are members of EdgeX Foundry?

Version Control

The certification process will be executed against the current release of EdgeX. The certification results and certificate will specify the version of EdgeX that was used for testing. There is no expiration of certification provided: 1) the certified component has not been modified, and 2) the component will be used with that release of EdgeX.

EdgeX Changes

EdgeX will continue to evolve with regular releases. Any list of EdgeX certified services will include the version information of the EdgeX release that was used for the certification. It will be up to users to evaluate if the changes to associated APIs are significant enough to invalidate the certification for the EdgeX release they plan to use.  It will be up to the developer of a component to decide if they want to certify their component on a new EdgeX release. 

Component Changes

It is expected that certified components will change as developers fix bugs and add new functionality. We want to encourage a process of continuous improvement. However, it is easy to imagine scenarios where the changes could cause a component to no longer meet the certification tests.

In these cases, the developer should continue to make the version of the certified component available. That assures users they have access to what was evaluated and certified. At the same time, the developer may make the revised version of their component available. This new component will no longer be considered as certified.

A process for recertifying component changes will be needed. If the changes are for bug fixes, it could be that self-assessment and request for review is sufficient to recertify the component. For enhancements, it may be that a full certification process will be required. This will be determined as we have a better understanding of the certification process as well as the volume and extent of changes to certified components.

Revoking Certification

There may be situations where EdgeX Foundry determines it is necessary to revoke certification of a component. This could be due to

• Errors discovered in certification testing
• Component changed after certification without informing users
• Complaints about component functionality or quality from the EdgeX user community
• Improper use or claims of EdgeX certification

Any decision to revoke certification will be made by the EdgeX TSC. The decision will be made following a report of the reasons prepared by the certification WG and a response from the component developer/organization.

We do not expect this to happen often, but it is necessary to protect the integrity of the certification program.

Road Map

As can be seen from the list of support tools required to support certification presented above, there is a lot of work required to get to the point of issuing meaningful certification of EdgeX components. We want to balance market value, time to market, and effort.

Setting Priorities

One of the key decisions is around which type of service should be the starting point. We have ranked them as follows:

...

Service Type

...

Value to Market

...

Level of Effort

...

Availability of Tests

...

Core

...

Low

...

Low

...

High

...

Device

...

High

...

High

...

Low

...

Application

...

Medium

...

Medium

...

Low

Road Map

As can be seen from the list of support tools required to support certification presented above, there is a lot of work required to get to the point of issuing meaningful certification of EdgeX components. We want to balance market value, time to market, and effort.

Decision 1 - Where to start?

Device Services offer the most commercial value.  Core services have the most tests available that could be used to more quickly establish and prove out the process. Application services are currently in development and may have the most resources available to provide necessary test artifacts. The Certification WG recommended Device Services as having the greatest short term commercial value.

It was decided that the EdgeX community focus on certification of Device Servers as its first priority.

Decision 2 - Self-Assessment or Certification?

There was some debate over the best way to start. Advocates of certification liked the formality and gravitas it would provide to commercial offerings.  Advocates of self-assessment pointed out the resource challenges and the extra burden of handling submitter code, setting up and executing tests, etc.  There was also a concern that only offering certification may be a burden that would reduce participation by developers in creating EdgeX components.

The decision was made to start with self-assessment.  This will enable the EdgeX community to achieve its goals for interoperability of components, while avoiding the burden of creating formal processes until the community and market demand has been established.  

Phases

Given the complexity and effort to create a meaningful certification program, we are suggesting a phased approach in the crawl, walk, run model. Release targets for the phases will be defined in collaboration with the TSC and other WG.

Crawl - "Assess and Learn"

The objective of the Crawl phase is to establish a self-assessment package for Device Services and make it available by Q4 2019. 

This will require the following:

• General test plan for Device Services that covers APIs, SMA support, operation, and basic performance metrics.
• Black box tests that can be run against any device service and produce results based on the test plan
• Instructions for how a user can deploy and execute the tests, and what metrics to record to help validate the tests were performed.
• Process for organizations to submit their test results for review and, if approved, inclusion on the EdgeX web site and other promotion.

Success will be having a self-assessment package available and being used by developers to confirm readiness of their components. The learnings from this experience will be used to inform the development of the formal certification process.

Walk - "Certification"

The objective of the Walk phase is to be able to evaluate and declare an EdgeX Device Service component as being certified. The focus is on making sure the process and tests exist and are sufficient, even if highly manual for execution.

MVP for certification will include:

• Submission process

• Evaluation process (manual)
  • Infrastructure
  • Black box tests
• Exercise the service against appropriate APIs
• Results reporting process
• Promotion of components that have achieved certification

It is likely that the initial certification candidates will be done using a very manual process. There will be gaps discovered, lessons learned, and need for more documentation. Once the process and tools are proven, we can plan future improvements.

Success criteria:

• A minimum of 5 EdgeX device services are certified

Jog - Automate and Formalize

With a working process in hand, the priority of the Walk phase will be to make setting up tests and recording results more automated. A second goal will be to make a self-assessment test package available. We should gain sufficient knowledge during this phase to make a decision on the what resources will fulfill the certification authority role.

Success criteria:

• Ability to execute certification tests of device services and gather results in a repeatable manner by someone who may not be an EdgeX expert or developer
• dentify the requirements and selection process for a third party to act as certification authority

Run - Deeper and Wider

With a working and automated process, the Run phase will focus on expanding tests to cover the remaining service types.

Success criteria:

• Certification of all EdgeX service types (Device, Application, Core)
• Establishment of a certification authority

Commercial Issues

This section covers various commercial issues that will be involved in the operation of the certification program.

Eligibility

Formal certification is available to any organization or developer that is willing and able to submit the required information and successfully complete the testing. There is no requirement for a submitter to be a member of EdgeX Foundry or Linux Foundation. This is to encourage maximum participation in the certification program.

Fees

Certification is a process that will take some effort by both the submitter and the certification authority. Organizations that will most benefit from certification will be those that intend to make their components available in the market. These organizations should be expected to help cover the cost of conducting the certification program.

Certification is not intended as a “profit” generating revenue stream. Any fees are intended solely to cover the cost of administration.

During early phases of the certification program, while testing is being conducted by the community, there should be a period where the fees are waived. This will encourage developers to submit their components for certification and allow EdgeX Foundry to learn and improve the process. Regardless of fees or certification authority, any component that is certified will have met the appropriate technical requirements. Once self-assessment is available, and/or if a third party certification authority is involved, the fee waiver will end.

The fee schedule is to be determined once the typical level of effort has been identified for each type of service.

If the effort needed to retest a component is found to be low, there may the opportunity to have a reduced fee for recertifying a component on a new EdgeX release.  This is to be determined once we have more experience with the testing process and potential costs of the certification authority.

Q: Should there be a reduced fee for developers or organizations that are members of EdgeX Foundry?

Version Control

The certification process will be executed against the current release of EdgeX. The certification results and certificate will specify the version of EdgeX that was used for testing. There is no expiration of certification provided: 1) the certified component has not been modified, and 2) the component will be used with that release of EdgeX.

EdgeX Changes

EdgeX will continue to evolve with regular releases. Any list of EdgeX certified services will include the version information of the EdgeX release that was used for the certification. It will be up to users to evaluate if the changes to associated APIs are significant enough to invalidate the certification for the EdgeX release they plan to use.  It will be up to the developer of a component to decide if they want to certify their component on a new EdgeX release. 

Component Changes

It is expected that certified components will change as developers fix bugs and add new functionality. We want to encourage a process of continuous improvement. However, it is easy to imagine scenarios where the changes could cause a component to no longer meet the certification tests.

In these cases, the developer should continue to make the version of the certified component available. That assures users they have access to what was evaluated and certified. At the same time, the developer may make the revised version of their component available. This new component will no longer be considered as certified.

A process for recertifying component changes will be needed. If the changes are for bug fixes, it could be that self-assessment and request for review is sufficient to recertify the component. For enhancements, it may be that a full certification process will be required. This will be determined as we have a better understanding of the certification process as well as the volume and extent of changes to certified components.

Revoking Certification

There may be situations where EdgeX Foundry determines it is necessary to revoke certification of a component. This could be due to

• Errors discovered in certification testing
• Component changed after certification without informing users
• Complaints about component functionality or quality from the EdgeX user community
• Improper use or claims of EdgeX certification

Any decision to revoke certification will be made by the EdgeX TSC. The decision will be made following a report of the reasons prepared by the certification WG and a response from the component developer/organization.

We do not expect this to happen often, but it is necessary to protect the integrity of the certification program.

Level of effort = apparatus required to execute tests

Device Services offer the most commercial value.  Core services have the most tests available that could be used to more quickly establish and prove out the process. Application services are currently in development and may have the most resources available to provide necessary test artifacts.

It is our belief that Device Services offer the greatest value to the success of EdgeX. We are recommending that the EdgeX community dedicate itself to producing the necessary tools and infrastructure to accomplish certification of Device Services within two releases.

Phases

Given the complexity and effort to create a meaningful certification program, we are suggesting a phased approach in the crawl, walk, run model. Release targets for the phases will be defined in collaboration with the TSC and other WG.

Crawl

The goal of the Crawl phase is to be able to evaluate and declare an EdgeX Device Service component as being certified. The focus is on making sure the process and tests exist and are sufficient, even if highly manual for execution.

MVP for certification will include:

• Submission process

• Evaluation process (manual)
  • Infrastructure
  • Black box tests
• Exercise the service against appropriate APIs
• Results reporting process
• Promotion of components that have achieved certification

It is likely that the initial certification candidates will be done using a very manual process. There will be gaps discovered, lessons learned, and need for more documentation. Once the process and tools are proven, we can plan future improvements.

Success criteria:

• A minimum of 5 EdgeX device services are certified

Walk

With a working process in hand, the priority of the Walk phase will be to make setting up tests and recording results more automated. A second goal will be to make a self-assessment test package available. We should gain sufficient knowledge during this phase to make a decision on the what resources will fulfill the certification authority role.

Success criteria:

• Ability to execute certification tests of device services and gather results in a repeatable manner by someone who may not be an EdgeX expert or developer
• First release of self-assessment test package for Device Service
• Identify the requirements and selection process for a third party to act as certification authority

Run

With a working and automated process, the Run phase will focus on expanding tests to cover the remaining service types.

Success criteria:

...

Open Issues

This section contains questions that are still open and under discussion by the Certification WG.

...