Delhi Release

Delivery:  ~ October 2018

The Delhi release is slated to provide the first system management capabilities while also implementing additional features in the security.  Delhi will continue to show improvements in performance by offering Device Service SDKs in Go and C.  Improved testing at all levels (unit, blackbox, performance, etc.) to ensure the quality of the system going forward while also providing a means to check backward compatibility.  Finally, this release will include design and implementation plans (scheduled for Edinburgh release and beyond) for the replacement of MongoDB as the reference implementation database, and a replacement to the export services to allow for better scale and flexibility at the northbound layer.  the scope of this release is smaller given the development cycle for the prior California release was made longer to accommodate all the Go Lang refactoring.

Release Themes and Objectives

• Deliver system management APIs in all the microservices and provide a system management agent for orchestrating multi-service system management commands and communications to 3rd party system management tools/platforms.
• Go and C device service SDKs with at least one device service implementation for example sake.
• The next wave of security features to include access control lists to grant access to appropriate services, and improved security service bootstrapping.
• Refactored and improved Go Lang microservices
• Options and implementation plan for database replacement
• Design and implementation plans for export service replacement with application services
• Provide an EdgeX UI suitable for use in exploring several instances of EdgeX

General Tasks and Notes

• Implement a performance test harness

  To establish a baseline of performance characteristics that can be used to understand EdgeX resource utilization, data throughput, etc.

• Support binary data

  Allow the EdgeX system (from device services through export services) to support binary data such as images, chunks of video or audio.  This will require device services, core data, and the export distribution to appropriately handle this type of data.  Local edge analytics may be fed binary data and create intelligence from it to allow for actuation at the edge based on the binary data (example: examine an image an detect the presence of a person of interest).

• Explore the use of Swagger to document the EdgeX REST API
  EdgeX currently uses RAML to disseminate information on microservice APIs.  Swagger offers more fidelity while also offering better code generation going forward.
• Provide an initial EdgeX user interface
  This Javascript-based user interface will allow users to explore the sensor data collected by EdgeX as well as provide some provisioning and management capability.  Used for prototyping, demonstration and management of small clusters of EdgeX instances.
• Update all services to return proper HTTP status codes on HTTP requrests

Core & Supporting Services Tasks and Notes

• Refactoring

  Refactor the services to implement data drive design, improves structure and organization of the code, improves ability to abstract infrastructure needs (database, messaging, etc.) to allow for replacements later, and allow services to bootstrap without artificial sleep mechanisms

• Implemented more structured / formatted logging
  Allows for better query and inspection of logs going forward.

• Implement security & system management API hooks

  Depending on the implementation needs of security and system management, there may be a need to make changes to the micro service code in support of these facets.  It is likely that each micro service will implement some sort of base service to provide data and system management functionality to the system management agent.  Access control to the service APIs will also be managed by the security services and code added to the service to require authentication before accessing the services will be needed.

Device Services and SDK Tasks and Notes

• Implement device service SDKs in Go Lang and C.
  The device service SDKs and resulting device services will allow the EdgeX community to retire the last of the Java micro services and complete the dramatic performance improvement of EdgeX.  

• A stretch goal includes providing the existing device services in Modbus, BACNet, BLE, etc. in either Go or C.  

  Many of the original device services where created with driver stacks that are suit for purpose on all platforms, are no longer supported, are not homogeneous in their make up (i.e. having some parts Java while other elements in Python), or are using stacks that are not consider the best option for the protocol today.  The BLE and BACnet device services fall under this categorization.

  Several of the device services were created to prove, conceptually, how to connect to a device using that protocol, but it may not be a full implementation of the protocol API.  For example, the SNMP device service implements enough to drive a Patlite and a few other devices, but does not understand all of SNMP.

• Implement security & system management API hooks

  Depending on the implementation needs of security and system management, there may be a need to make changes to the SDKs and existing micro service code in support of these facets.  It is likely that each micro service will implement some sort of base service to provide data and system management functionality to the system management agent.  As device services are typically created from an SDK, the same boilerplate code for the base service needs to be added to the SDK(s).  Access control to the service APIs will also be managed by the security services and code added to the service to require authentication before accessing the services will be needed.

Security Tasks and Notes

• 
  
• California release was focused on initially securing the EdgeX “gateway” and remains the priority
  • This work will be extended / augmented in this release (looking at alternative authentication, adding MQTT and other protocol reverse proxy, etc.)
    
• Code signing – how to certify integrity of the system
• The release will start to explore, potentially, securing EdgeX’s devices

• Explore potential use of hyperledger

System Management Tasks and Notes

• System management API (action, alerts, metric) as discussed and outlined here

• System management Agent (see same document above outlining the agent functionality and duties)

• Potentially explore "Gateway management" to include:
  
  • Demonstrate basic management of EdgeX via select 3rd party console (ex: VMware Pulse IoT Center, System Center, …)
  • Demonstrate EdgeX software updates
  • Updates to non-EdgeX components (drivers, end-devices)

Testing/QA

• Performance tests on startup time, request/response times on all APIs, latency to actuation from device service collection, through core data, to rules engine, command back to a device service.
  • Performance metric testing will include CPU and memory usage statistics
• Improved unit and integration testing across all EdgeX microservices
  • Returning unit test coverage to ~80%
  • Requires some Go code refactoring and decoupling among resources (like the database, logging, etc.)
  • Use of mock objects to isolate functions under test.

Target Performance

• The target is to run all of EdgeX (all the services to include the virtual device service) on a Raspberry Pi 3 (1 GB RAM, 64bit CPU, at least 32GB storage space, running Raspbian Linux distro)

  • This is not an endorsement of the platform as our favorite or otherwise endorsed platform

  • It only suggests the general characteristics of a “developer community” target platform

  • This may not be entirely feasible without all Go replacements, but is the target and the development community will report back when/where this is not possible

  • For example, it is unlikely the target security implementation will fit on this size platform

• Additional “developer community” targets

• Startup in 1 minute or less (post OS boot – from micro service start to all services up and available)

• Throughput for one piece of data (with one IP device connected by hard wire – versus WiFi) from data ingestion at the Device Service layer, through Core Data, to Rules Engine and back down through Command Service finally to Device Service for actuation will be < 1 second

Stretch Goals for the Release

• Facilitate East/West capability
  • Micro service load balancing, failover, scale-over, ...
  • Device from EdgeX A triggers action on device on EdgeX B
• Address data privacy concerns
  • EU laws and affirmation about data use/storage/etc.
  • HIP-A
• Explore streaming analytic needs/implementations
• Address multi-tenancy needs

Items for consideration

These items are suggestion for Delhi or later releases that have not yet been fully vetted or voted on by the TSC, but will be reviewed as part of an upcoming meeting.  Items in this list may be moved to the roadmap for other releases or discarded entirely.

• The EdgeX configuration today is flat - meaning a single layer of key/value pairs with no groups or other structure.  Configuration should, in the future, be categorized and grouped under a sub-structure to simplify the exploration and updating the config.  For example, all properies related to MongoDB connection (host, username, password, etc) would be grouped under a MongoDB configuration structure. 
• Downsampling: It is mentioned that the device service may receive from the device new unattended readings (e.g. in a pub/sub type of scenario). In this case, there should be a setting to specify whether we accept all readings or we decide to downsample because the source is pumping data too fast. This is actually a very common scenario when you deal with high frequency sensor packages.

• Command: in order to protect the device from harmful commands, there should be the possibility to set a Min and Max limit for the value that is accepted on every single command.  in fact the command service today is rather a hollow simple proxy, but in the future we very much envisioned adding additional security, caching to avoid having to hit the DS when unnecessary, and even grouping command requests for better resource conservation (especially for devices like BLE that get woken up when you hit them).

• Data Transformation: This is something we have always considered as a potential in EdgeX – that of a filter, even a small transformation engine between device services and core data. Not a full blown export, but something that serves in a similar fashion and was common across services. We even thought about making it some type of quick CEP/rules engine feeder for those decisions that can’t wait to go through the rest of the layers.

• Revisited and improved security threat model.
• Implementation of message bus alternative for intercommunication between microservices as an alternative to REST.

  While REST will not go away (a REST API will still exist around each micro service), there may be a need to implement point-to-point messaging between select services or to adopt some type of message bus unilaterally across all of EdgeX to support messaging among services.

  Messaging provides for more asynchronous communications, typically lower latency communications, and better (or more finely tuned) communication quality of service (QoS).  Are there places where messaging might be more appropriate (like between core data and export distro today).  Would a use case dictate the use of an alternate messaging infrastructure be used among all services with an underlying message bus to support it?  Would alternate protocols (SNMP, WebSockets, etc.) be desired in some use cases?  For the Delhi release, some alternate communication requirements, design and early message implementation experimentation is likely to occur.

• Should we allow configuration properties be overriden by command line provided properties (like Java allows/provides) for our Go services?