...

April 25, 2019

Introduction

...

Note: The term “certification authority” is used in this document to describe the organization that will be conducting the certification testing/evaluation. It is yet to be determined if this will be done within the EdgeX community or by a third party.

...

EdgeX certification represents a process where software components that interact with EdgeX are evaluated and confirmed to meet a set of technical requirements and standards that will help insure compatibility. This document describes the framework that will be used by the EdgeX Foundation to verify that software seeking compliance conforms to specified characteristics.

...

EdgeX Certification is initially intended for components that replace core services or interact with core services. As the certification program matures, there may be reason to certify accessories or applications that are outside of that scope. For example, third party applications that interface with application services.

...

Formal certification will involve a submission, evaluation, and results reporting process.

Developer follows documented process for to request certification preparation

• Conducts self-assessment
• Prepares documentation, sample code, and certification request

Certification authority receives processes certification request

• Acknowledges receipt of request
• Review submission for completeness and assigns to scheduleProcess process payment (if necessary)
• Assigns to schedule

Certification authority performs certification testing

• Receive Obtain service code (source, executable, container) and any dependency code to be tested
• Configure according to documentation
• Conducts evaluation testing
• Records test results and produces report

...

• Pass – issue certification
• Fail – reply with notescorrespond with submitter with details of failure, wait for resubmission
  

Submitted code will not be made public or stored beyond the period needed for testing. All test results will be retained.

...

• The service made available in a Docker container that is accessible to the certification authority.  The containers must be able to run on a Linux Ubuntu Server or Desktop 18.04 environment.
• An example Docker Compose file using the service under review with all the Docker configuration necessary to replicate the submitters expected runtime network.
• Statement on platform independence and platform requirements (such as minimum memory or other resources).  EdgeX assumes platform independence, but does not require independence if stated up front on the limitations and platform requirements.
• Statement on deployment/orchestration dependencies (such as an inability to run outside of Docker).  EdgeX is orchestration agnostic and assumes deployment/orchestration independence, but does not require it.  Services should be designed in a way that should not make any assumption about orchestration/deployment capability, but where they are not, this should be documented.
• License and usage agreements associated with the service.  Certification is available for open to open source as well as proprietary solutions, but requires the submitter to state the intended license.
• Any licenses necessary to conduct the testing.
• Documentation on self-assessment test results including the version string or fingerprints of the test tools used. If any test failures, provide an explanation on why the failure should be allowed under certification (request an exception).
• Documentation on any API extensions/changes - this document may not be made public but may be needed by the certification authority to understand how to test and provide certification.  Services should provide samples of use with documentation where they differ from the reference implementation.
• An application form and legal agreement as required by the EdgeX Certification process.

...

• Run the submitter’s Dockerized service in the blackbox Ubuntu 18.04 Linux/Intel platform test environment using Docker Compose to start the services (both the EdgeX and submitted service) and altering the Docker Compose file for the test environment per the submitter’s Compose file.
• Exercise the submitted service using the automated black box tests for that service for the version of certification requested on an Ubuntu 18.04 Linux/Intel platform.  Certification requires the submitted service to pass all the black box tests. Making a best effort to account for and accommodate any configuration changes or other aspects that would alter the test runs.
• Run the service for a minimum of 24 hours and measure that resource use and performance are not negatively impacted over time.
• Exercise the system management black box API tests to ensure that the service complies with the system management service needs.
• Manually inspect that the service API is a superset of the EdgeX service API.  Not every aspect of the service API may be tested in the EdgeX black box tests.
• Manually inspecting the service complies with EdgeX requirements of
• Providing appropriate logging and offering log statements that comply with EdgeX conventions
• Use and pass the correlation ID when calling on other services to aid in troubleshooting
• Comply with getting configuration from the registry service or local file system when not available (on bootstrapping)
• Retry to connect to dependent services when they are not available
• Manually inspect the service’s compliance with system management requirements. For example, checking that the service’s metrics requests are responding as expected and not just returning hardcoded values. [do we need a more complete list here]
• Exercise the security black box API tests to ensure that the service complies with the security service needs.
• Manually inspect the service’s compliance with security requirements – for example that the service can be configured to operate behind the API gateway and that no application secrets are stored in the clear. [do we need a more complete list here]
• Collect and review test results – allowing for black box test failures when the black box exceptions are requested and deemed satisfactory.
• Manually inspect and test that the services properly address exception-path use cases (for example, the service continues to try to reconnect with a dependent service when it is not available, or reconnects to a database when one is required and it drops).  [do we need a more complete list here]
• Manually review documentation provided by the submitter for completeness, accuracy, language issues, etc.
• Manually check that configuration of the service is in compliance with EdgeX configuration seeding.
• Manually check that database population and use with regard to schemas, seeding, and cleanup is handled in compliance with EdgeX standards.
• Check the performance metrics (CPU, memory, response time, etc.) of the other services for any negative impact.  Check that the service does not block or reduce performance of other services. Services causing negative impacting will not be certified.
• Produce a report on the performance of the service.  This report may contain information on resource (CPU, memory, storage, etc.) utilization, start time, response and latency times, or any other characteristic that may be of interest to a user.  The certification authority will not decline certification on the basis of performance results but will publish the results for user awareness.

...

...

• Consider using existing device services to confirm process and that they can be certified (test our own dog food)
• Assumption that some submitters will want to provide their code from a private repository. Need to investigate how to script pulling the code (private certs?) and running tests.

Process

The following items that will be needed to execute the certification process:

Other notes:

• Testing process is The certification testing process and results are private between submitter and EdgeX Foundry until it reaches conclusionthe process is completed.

Marketing and Promotion

In order to achieve the full benefits of the certification program it must be seen as having value and momentum. The following items will be helpful in that effort:

External Certification Authority

It may be necessary for the EdgeX working groups to be very hands on during the initial certification testing in order to quickly respond to any issues found in the process or tools. However, it may not be the best use of community members’ time to execute the certification process once it the process has been established and the volume of requests begins to grow. This is especially true if a decision is made to charge fees for certification or there is a need to have independence independent testing for requesters submitting sensitive code.

We believe it will be necessary for a third party to be contracted to manage the certification process. The third party, operating under the review of the TSC and the Certification WG, will take on administration and execution of the certification processing and testing. In addition to reducing the load on the community, it will also provide a repeatable and independent process that may be more respected by the marketplace.

If a third party is to be used, the Certification WG will create a document to guide the evaluation and selection process. This will include:

• Duties and requirements
• Selection criteria and evaluation process
• Cost and budget (handling payments)
• Service quality expectations

...

When an organization is ready to submit their service or product for certification, they will follow a standard processpublished process available from the EdgeX web site. The beginning of this process is an application.

...

Formal certification is available to any organization or developer that is willing and able to submit the required information and successfully pass through complete the testing. There is no requirement for a submitter to be a member of EdgeX Foundry or Linux Foundation. This is to encourage maximum participation in the certification program.

...

During early phases of the certification program, while testing is being conducted by the community, there should be a period where the fees are waived. This will encourage developers to submit their components for certification and allow EdgeX Foundry to better manage expectationslearn and improve the process. Regardless of fees or certification authority, any component that is certified will have met the appropriate technical requirements. Once self-assessment is available, and/or if a third party certification authority is involved, it will become necessary to charge fees for the certification processthe fee waiver will end.

The fee schedule is to be determined once the typical level of effort has been identified for each type of service.

If the effort needed to retest a component is found to be low, there may the opportunity to have a reduced fee for recertifying a component on a new EdgeX release.  This is to be determined once we have more experience with the testing process and potential costs of the certification authority.

Q: Should there be a reduced fee for developers or organizations that are members of EdgeX Foundry?

Version Control

The certification process will be executed against the current release of EdgeX. The certification results and certificate will specify the version of EdgeX that was used for testing. There is no expiration of certification provided: 1) the certified component has not been modified, and 2) the component will be used with that release of EdgeX.

...

EdgeX will continue to evolve with regular releases. Any list of EdgeX certified services will include the version information of the EdgeX release that was used for the certification. It will be up to users to evaluate if the changes to associated APIs are significant enough to invalidate the certification for the EdgeX release they plan to use.  It will be up to the developer of a component to decide if they want to certify their component on a new EdgeX release. 

Component Changes

It is expected that certified components will change as developers fix bugs and add new functionality. We want to encourage a process of continuous improvement. However, it is easy to imagine scenarios where the changes could make cause a component to no longer meet the certification goals and requirementstests.

In these cases, the developer should continue to make the version of the certified component available. That assures users they have access to what was evaluated and certified. At the same time, the developer may make the revised version of their component available. This new component will no longer be considered as certified.

A process for recertifying component changes will be needed. If the changes are for bug fixes, it could be that self-assessment and request for review is sufficient to recertify the component. For enhancements, it may be that a full certification process will be required. This will be determined as we get have a better understanding of the certification process as well as the volume and extent of changes to certified components.

...

There may be situations where EdgeX Foundry determines it is necessary to revoke certification of a component. This could be due to

• Errors discovered in compliance certification testing
• Code Component changed after certification without informing users
• Complaints about component functionality or quality from the EdgeX user community
• Improper use or claims of EdgeX certification

Any decision to revoke certification will be made by the EdgeX TSC. The decision will be made following a report of the reasons prepared by the certification WG and a response from the component developer/organization.

We do not expect this to happen often, but it is necessary to protect the integrity of the certification program.

...

Road Map

As can be seen from the list of support tools required to support certification presented above, there is a lot of work required to get to the point of issuing meaningful certification of EdgeX components. We want to balance market value, time to market, and effort.

...

Level of effort = availability of tools apparatus required to execute tests

Device Services offer the most commercial value.  Core services have the most tests available that could be used to more quickly establish and prove out the process. Application services are currently in development and may have the most resources available to provide necessary test artifacts.

...

The goal of the Crawl phase is to be able to evaluate and declare an EdgeX Device Service component as being certified. The focus is on making sure the process and tests exist and are sufficient, even if highly manual for execution.

MVP for certification will include:

• Submission process

• Evaluation process (manual)
  • Infrastructure
  • Black box tests
• Exercise the service against appropriate APIs
• Results reporting process
• Record Promotion of certified components
• Promotion of certified services
• that have achieved certification

It is likely that the initial certification candidates will be done using a very manual process. There will be gaps discovered, lessons learned, and need for more documentation. Once the process and tools are proven, we can issue certifications and plan future improvements.

...

Walk

With a working process in hand, the priority of the Walk phase will be to automate the process and make setting up tests and recording results more automated. A second goal will be to make a self-assessment test package available. We should gain sufficient knowledge during this phase to make a decision on the requirements for what resources will fulfill the certification authority role.

...

With a working and automated process, the Run phase will focus on expanding tests to cover the remaining service types in addition to continuous improvement of the process.

Success criteria:

• Certification of all EdgeX service types (Device, Application, Core)
• Establishment of a third party certification authority

Open Issues

...

Q: Do we need some sort of repo for the requestor to requester to submit their code?

Q: What is our liability for protecting submitted code?

...

Certification Authority: The organization that manages and executes the certification process on behalf of EdgeX Foundry.

self-assessment: A process that enables users to do their own informal testing prior to submission for certification.

Conformance Properties

• Consistency: do the different (parts of) software artifacts conform to each other?
• Functional: does input to the system produce the expected output?
• Behavioral: does the system meet general safety and progress properties like absence of deadlocks or are constraints on the specific states of the system met?
• Quality: do the artifacts fulfill nonfunctional requirements in the areas of for example performance, security, and usability?
• Compliance: do the artifacts conform to standards, guidelines, or legislation?
  
  

...

• Is this for new or existing (replacement) device service? Is it commercial vs open source?
• We would assume the process remains the same but there may be some points of departure.  See below.
• Are the tests only testing what is common to the DS SDK?  Are there other things that are to be tested – like how data is deposited to Core Data?
• We would assume that it does include how it impacts and behaves with other services, but this may come in phases (crawl, walk, run).
• Do we need to modify the current DS SDK black box tests to be more parameter driven – especially with regard to Command services?
• Yes, again, over time.
• How do we test for and explore “unusual behavior” – like how a DS might send data to Export or something like Security that we did not expect (or maybe want)?
• We won’t be able to do much at first (other than manual inspection) but over time this is something we’ll want to add more tests for.

...

1. Self-assessment is done by the submitter.
2. Device Service submitter sends the Certification WG the following:
   1. Self-assessment results/report
   2. Device service (containerized),
   3. Device (or simulator) used to test the device service
   4. Device profile(s) – and any documentation necessary to explain the profile(s)
   5. Documentation as per below.
      1. Specifically, the protocol details should be included in the documentation – for example in a Modbus device service, the submitter would indicate whether using Modbus RTU vs TCP/IP
   6. Guidance on how much data is going to be captured by default
3. Perform black box tests that we already have (API check)
   1. DS SDK black box/API tests
   2. Device Service specific tests (for device services that we already have – if any are written, which to date they are not)
4. Inspect or test that the service behaves well with regard to rest of EdgeX (some of this could be done via running existing other black box tests; most will have to be done by manual inspection today)
   1. Appropriately populates Core Data with readings/events
   2. Appropriately populates Value Descriptors
   3. Appropriately populates metadata (Devices, Device Services, Profiles,...)
   4. Appropriately receives and interacts with Command service (responding as expected)
   5. Appropriately gets its configuration through the config/registry service
   6. Uses the Logging microservice micro service to log errors, warnings, information and debug statements.
5. Perform “stability” exercise and report results. Leave the device service running for a certain extended period of time (time period to be determine) and examine:
   1. How much event/reading data did it report (did it seem to match expectations provided by the submitter?)
   2. How long does the service stay up – did it run out of resources or otherwise crash?
   3. Did it impact other services negatively (like causing any of them to fail due to data volume or size issues)?
6. Performance evaluation and metric collection
   1. Report on CPU, Memory and other resource usage
   2. Report on time to come up
   3. Report on storage usage
7. Documentation review (subjective review)
   1. Information on any operations, configuration, additional functionality, etc. that differ from the reference implementation (example – providing configuration defaults and options that differ from the reference implementation service).
8. After each phase of above, any failure, test observation or concern can be addressed with the submitter – allowing the submitter to address issues and re-iterate the entire process or a section as required
   1. Are the failures in line with submitters self-assessment documentation and exceptions noted? Are these reasonable and does EdgeX need to revamp tests to accommodate or at least grant an exception on the basis of what is discussed.
   2. Cert WG Privately reviews and makes a determination of the results and failures – allowing the submitter reiterate through the process
      

...