QA/Test Roadmap

'Barcelona': October 2017

'California': ~June 2018

'Delhi': ~October 2018

'Edinburgh': ~ April 2019

'Fuji': ~ October 2019

'Geneva': ~ April 2020

'Hanoi': ~ October 2020

Edinburgh Release (Committed)

With automated blackbox tests in place, the Edinburgh release will include better visualization/dashboard of test results (including look up and display of historical test results). Candidate tools include Telegraf + Grafana + InfluxDB, DataDog, Prometheus. https://github.com/edgexfoundry/blackbox-testing/issues/105
The Edinburgh release will include the capture of resource metrics to monitor performance. Metrics monitored will include memory and CPU consumption. https://github.com/edgexfoundry/edgex-go/issues/115
As a stretch goal, the Edinburgh release will include performance/load testing using a tool like Bender, Jmeter, Load Impact or other tool selected by the work group.
Security tests will be automated as mentioned above in the Security Group tasks. https://github.com/edgexfoundry/security-api-gateway/issues/44 & https://github.com/edgexfoundry/security-secret-store/issues/51
For the Edinburgh release, code contributors will now be required to supply additional or updated blackbox tests to cover changes made to the code base with any PR. Working group leads and project committers are head accountable for this change in procedures.
As a stretch goal, the project will look to produce Swagger documentation to better support testing and API documentation in future releases. RAML documentation will remain the officially API standard for Edinburgh, but additional Swagger documentation will be provided and weighed for use in future releases.

Improved unit tests across services (improvements should be b driven by specific WG dev teams)
Improve blackbox test structure including reorganization of the tests and better test case documentation (source ramya.ranganathan@intel.com)
Dockerize blackbox testing infrastructure for deployment simplification/flexibility
System integration tests – currently we don’t have any end to end tests e.g. Device Service read data -> Core Data -> Rules Engine or Export Service
Blackbox tests for new Application Services microservices
Blackbox test for Device Services - Virtual, Modbus, MQTT, BACNet, OPC UA. Develop common set of tests that can be used against all Device Services.
Configuration testing – currently all blackbox tests run with single static configuration
Automated performance testing

API Load testing (measure response time) and metrics (CPU, memory) collection for all EdgeX microservices (this work was started during the Edinburgh iteration)
EdgeX microservice startup times
Automated system level latency and throughout testing (e.g. device read to export or device read to analytics to device actuation)
Baseline performance of service binaries no container
Additional performance test runs against other container technologies supported (e.g. snaps)
The ability to create summary reports/dashboards of key EdgeX performance indicators with alerts if thresholds have been exceeded

Test coverage analysis e.g. using tools such as Codecov.io
Static code analysis e.g. using tools such as SonarQube or Coverity to identify badly written code, memory leaks and security vulnerabilities
Tracing during testing e.g. based on Open Tracing standard such as Zipkin, Jaeger

Improved unit tests across services (improvements should be b driven by specific WG dev teams)
Improve blackbox test structure including reorganization of the tests and better test case documentation (source ramya.ranganathan@intel.com)
Blackbox tests for new Application Services microservices
Blackbox test for Device Services - Virtual, Modbus, MQTT, BACNet, OPC UA. Develop common set of tests that can be used against all Device Services.
Configuration testing – currently all blackbox tests run with single static configuration
Automated performance testing

API Load testing (measure response time) and metrics (CPU, memory) collection for all EdgeX microservices (this work was started during the Edinburgh iteration)
EdgeX microservice startup times
Automated system level latency and throughout testing (e.g. device read to export or device read to analytics to device actuation) - STRTECH GOAL
The ability to create summary reports/dashboards of key EdgeX performance indicators with alerts if thresholds have been exceeded