Skip to main content

Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

Difference between revisions of "IoT/IoTConnectorProposal"

< IoT
(Description)
m (improve formatting)
Line 1: Line 1:
This is the '''draft''' version of a proposal to start an project under the Eclipse IoT umbrella for creating the IoT Connector component.
+
This is the '''draft''' version of a proposal to start a project under the Eclipse IoT umbrella for creating the IoT Connector component.
Feel free to add comments or add your name as an interested party if you would like to join the project or collaborate in any other way.
+
Feel free to add comments or add your name to the _Interested Parties_ section if you would like to join the project or collaborate in any other way.
  
 
= Proposal =
 
= Proposal =
Line 8: Line 8:
 
=== Background ===
 
=== Background ===
 
   
 
   
The open source community has produced a lot of excellent technology, frameworks and products that help with implementing IoT applications. A developer usually selects appropriate set of technology and components and incorporates them into an application covering all relevant aspects of an IoT solution ranging from device connectivity, management, monitoring, business logic and last but not least enforcing security at all levels. In an enterprise context a typical solution is about connecting a given number of homogenous devices to a particular application hosted on dedicated server infrastructure. Over time this results in a set of independent silo applications each managing their own (limited) set of devices. As of today, most of the technology created in Eclipse IoT projects specifically supports the development of IoT applications in this way.
+
The open source community has produced a lot of excellent technology, frameworks and products that help with implementing IoT applications. A developer usually selects an appropriate set of technology and components and incorporates them into an application. The chosen components need to suport the implementation of all relevant aspects of an IoT solution including device connectivity, management, monitoring, business logic and last but not least security enforcement at all levels. In an enterprise context a typical solution is about connecting a given number of homogenous devices to a particular application hosted on dedicated server infrastructure. Over time this results in a set of independent silo applications each managing their own (limited) set of devices. As of today, most of the technology created in Eclipse IoT projects specifically supports the development of IoT applications in this way.
  
Bosch is a major manufacturer of all kinds of electronic devices, most (if not all) of which will have connectivity built into them in the near future. Integrating these devices with individual IoT applications as descibed above has several drawbacks:
+
Bosch is a major manufacturer of all kinds of electronic devices, most of which (if not all) will have connectivity built into them in the near future. Integrating these devices with individual IoT applications as descibed above has several drawbacks:
 
* The repetitive implementation of common functionality like device communication and management is error prone and inefficient regarding costs, development time and runtime resources.
 
* The repetitive implementation of common functionality like device communication and management is error prone and inefficient regarding costs, development time and runtime resources.
* The tight integration of devices with a specific application effectively partitions the set of things into classes determined by the application the devices have been intially integrated with. This makes it hard to create new cross-domain solutions and business models leveraging devices coming from different application domains.
+
* The tight integration of devices with a specific application de facto partitions the set of things into classes determined by the application the devices have been intially integrated with. This makes it hard to create new cross-domain solutions and business models leveraging devices coming from different application domains.
 
* The applications implemented this way are often designed to interact with a limited number of devices in an enterprise environment only. Scaling out such applications with an increasing number of devices therefore often requires massive refactoring (if not re-architecting) of the device integration layer in order to support horizontal scalability as required in most cloud-based use cases.
 
* The applications implemented this way are often designed to interact with a limited number of devices in an enterprise environment only. Scaling out such applications with an increasing number of devices therefore often requires massive refactoring (if not re-architecting) of the device integration layer in order to support horizontal scalability as required in most cloud-based use cases.
  
 
=== Description ===
 
=== Description ===
  
Connectivity is at the heart of IoT solutions. Devices (things) need to be connected to a back end component where the data and functionality of the devices is leveraged to provide some higher level business value. IoT solution developers can pick from a wide array of existing (open source) technology to implement a device connectivity & management layer for the particular type of devices at hand. While this is often fun for the developers to do, the resulting solutions are often silo applications that lack the ability to scale horizontally with the number of devices connected and the number of back end components consuming the device data and functionality.
+
Connectivity is at the heart of IoT solutions. Devices (things) need to be connected to a back end component where the data and functionality of the devices is leveraged to provide some higher level business value. IoT solution developers can pick from a wide array of existing (open source) technology to implement a device connectivity & management layer for the particular type of devices at hand. While this is often fun for the developers to do, the resulting solutions are often silo applications lacking the ability to scale horizontally with the number of devices connected and the number of back end components consuming the device data and functionality.
  
The Eclipse IoT Working Group has therefore discussed  a more generic, cloud based IoT platform architecture that better supports the implementation of IoT solutions without requiring developers to solve some of the recurring (technical) challenges over and over again. The diagram below provides an overview of the IoT Server Platform as discussed in the working group.
+
The Eclipse IoT Working Group has therefore discussed  a more generic, cloud-based IoT platform architecture which better supports the implementation of IoT solutions without requiring developers to solve some of the recurring (technical) challenges over and over again. The diagram below provides an overview of the IoT Server Platform as discussed in the working group.
  
 
[[File:IoT_Server_Building_Blocks.jpg|1400px]]
 
[[File:IoT_Server_Building_Blocks.jpg|1400px]]
  
The diagram shows how devices in the field are connected to a cloud based back end either via a Field Gateway (e.g. something like Eclipse Kura) or directly to so called Protocol Adapters. The Protocol Adapters' responsibility is abstraction of communication protocol as well as providing location transparency of devices to the other back end components. The devices upload (sensor) data to the back end while the functions/services they expose can be invoked from the back end. These two directions of information flow can be characterized as follows:
+
The diagram shows how devices in the field are connected to a cloud-based back end either via a Field Gateway (e.g. something like Eclipse Kura) or directly to so-called ''Protocol Adapters''. The Protocol Adapters' responsibility is abstracting communication protocols as well as providing location transparency of devices to the other back end components. The devices upload (sensor) data to the back end while the functions/services they expose can be invoked from the back end. These two directions of information flow can be characterized as follows:
* Telemetry<br/>Data flowing upstream (left to right) from devices to the back end to a consumer like a Business Application or the Device Management component usually consists of a small set of discrete values like sensor readings or status property values. In most cases these messages are one-way only, i.e. devices sending this kind of data usually do not expect a reply from the back end.
+
* Telemetry<br/>'''Data''' flowing upstream (left to right) from devices to the back end to a consumer like a Business Application or the Device Management component usually '''consists of''' a small set of discrete values like '''sensor readings''' or status property values. In most cases these messages are one-way only, i.e. devices sending this kind of data usually do not expect a reply from the back end.
* Command & Control<br/>Messages flowing downstream (right to left) from back end components like Business Applications often represent invocations of services or functionality provided by connected devices, e.g. instructions to download and apply a firmware update, setting configuration parameters or querying the current reading of a sensor. In most cases a reply to the sent message is expected by the back end component.
+
* Command & Control<br/>'''Messages''' flowing downstream (right to left) from back end components like Business Applications often '''represent invocations of services''' or functionality provided by connected devices, e.g. instructions to download and apply a firmware update, setting configuration parameters or querying the current reading of a sensor. In most cases a reply to the sent message is expected by the back end component.
  
 
It seems reasonable to assume that the number of messages flowing upstream (Telemetry) will be orders of magnitude larger than the number of messages flowing downstream (Command & Control). The aggregated overall number of messages flowing upstream is expected to be in the range of several hundred thousand to millions per second. Note that in this architecture the same (cloud-based) infrastructure is shared by multiple solutions.
 
It seems reasonable to assume that the number of messages flowing upstream (Telemetry) will be orders of magnitude larger than the number of messages flowing downstream (Command & Control). The aggregated overall number of messages flowing upstream is expected to be in the range of several hundred thousand to millions per second. Note that in this architecture the same (cloud-based) infrastructure is shared by multiple solutions.
  
The IoT Connector provides the central link between the device-facing Protocol Adapters, additional re-usable back end components, e.g. Device Management or Software Provisioning, and last but not least the IoT solutions leveraging the devices' data and services. Solution developers can use the IoT Connector to uniformly and transparently interact with all kinds of devices without the need for caring about the particular communication protocol(s) the devices use. Multiple solutions can use the same IoT Connector instance running in a shared cloud environment in order to share the data and functionality of all connected devices. The IoT Connector ensures that only those components can consume data and control devices that have been granted authorization by the device owner. In this regard the IoT Connector can be considered an IoT specific message broker targeted at cloud deployment scenarios.
+
The IoT Connector provides the central link between the device-facing Protocol Adapters, additional re-usable back end components, e.g. Device Management or Software Provisioning, and last but not least the IoT solutions leveraging the devices' data and services. Solution developers can use the IoT Connector to uniformly and transparently interact with all kinds of devices without the need for caring about the particular communication protocol(s) the devices use. Multiple solutions can use the same IoT Connector instance '''running in a shared cloud environment''' in order to '''share the data and functionality of all connected devices'''. The IoT Connector ensures that only those components can consume data and control devices that have been granted authorization by the device owner. In this regard the IoT Connector can be considered an IoT specific message broker targeted at cloud deployment scenarios.
  
 
'''Features at a glance'''
 
'''Features at a glance'''
Line 35: Line 35:
 
* Support for different message exchange patterns
 
* Support for different message exchange patterns
 
* Used for cloud service federation
 
* Used for cloud service federation
* Provides Interfaces to support implementation of protocol adaptors that allow:
+
* Provides interfaces to support implementation of protocol adaptors which allow:
 
** Sending telemetry data
 
** Sending telemetry data
 
** Receiving device control messages (from applications/solutions)
 
** Receiving device control messages (from applications/solutions)
** Registering authorized upstream consumers of telemetry data received from connected devices
+
** Registering authorized consumers of telemetry data received from connected devices
  
 
=== Scope ===
 
=== Scope ===
  
The IoT Connector is at the heart of the IoT Server Platform as illustrated above in the Description section. It provides the link between the downstream Protocol Adapters and the upstream IoT solutions and other re-usable components leveraging the devices' data and services. The IoT Connector needs to fulfill a set of non-fucntional requirements, in particular regarding horizontal scalability, that are specific to both the deployment environment (cloud) and the intended architectural platform characteristics (as opposed to embedding a connectivity layer into applications individually). However, these requirements are not specific to any particular application domain. From a technical point of view it makes no difference if a sensor reading received via a LW-M2M protocol adapter represents a temperature or the relative humidity. In both cases the IoT Connector's responsibility is to forward the messages containing the values to (potentially multiple) authorized consumers without introducing a lot of latency.
+
The IoT Connector is at the heart of the IoT Server Platform as illustrated in the ''Description'' section. It provides the link between the downstream Protocol Adapters and the upstream IoT solutions and other re-usable components leveraging the devices' data and services.
  
Acknowledging the fact that there already exist many message brokers both in the open source world as well as by means of commercial offerings either to be deployed on premises or provided as cloud services, it is not the project's intention to create an additional message broker implementation. Instead, the goal is to create a uniform (remote) service interface that supports both the Telemetry as well as Command & Control message exchange pattern requirements. In order to do so, the IoT Connector also introduces a standard service interface for managing the identity and access restrictions.
+
The IoT Connector needs to fulfill a set of '''non-functional requirements''', in particular regarding '''horizontal scalability''', that are specific to both the deployment environment (cloud) and the intended architectural platform characteristics (as opposed to embedding a connectivity layer into applications individually). However, these requirements are not specific to any particular application domain. From a technical point of view it makes no difference if a sensor reading received via a LWM2M protocol adapter represents a temperature or the relative humidity. In both cases the IoT Connector's responsibility is to forward the messages containing the values to (potentially multiple) authorized consumers without introducing too much latency.
  
The IoT Connector project should provide an initial set of implementations of the service interfaces described above, leveraging existing messaging infrastructure components. As a starting point we would like to provide an implementation based on RabbitMQ because of its easy availability both as a service in existing Cloud Foundry based environments as well as in the form of pre-built Docker images. In order to also provide an implementation supporting horizontal scale-out, we would also like to create an implementation based on Apache Kafka. Future versions may also support using other cloud based offerings (e.g. Microsoft's Azure Message Bus or Amazon's Simple Queue Service).
+
Acknowledging the fact that there already exist many message brokers both in the open source world as well as by means of commercial offerings either to be deployed on premises or provided as cloud services, '''it is not the project's intention to create an additional message broker''' implementation. Instead, the goal is to create a '''uniform (remote) service interface''' that supports both the Telemetry as well as Command & Control message exchange pattern requirements. In order to do so, the IoT Connector '''also''' introduces a '''standard service interface for managing the identity and access restrictions'''.
 +
 
 +
The IoT Connector project should provide an initial set of implementations of the service interfaces described above, leveraging existing messaging infrastructure components. As a starting point we would like to provide an implementation based on ''RabbitMQ'' because of its easy availability both as a service in existing Cloud Foundry based environments as well as in the form of pre-built Docker images. In order to also provide an implementation supporting horizontal scale-out, we would also like to create an implementation based on ''Apache Kafka''. Future versions may also support using other cloud-based offerings (e.g. Microsoft's ''Azure Message Bus'' or Amazon's ''Simple Queue Service'').
  
 
=== People ===
 
=== People ===

Revision as of 11:30, 1 December 2015

This is the draft version of a proposal to start a project under the Eclipse IoT umbrella for creating the IoT Connector component. Feel free to add comments or add your name to the _Interested Parties_ section if you would like to join the project or collaborate in any other way.

Proposal

The following sections describe the intent and scope of the IoT Connector project we would like to start at Eclipse. The layout follows the Eclipse Create Project page.

Background

The open source community has produced a lot of excellent technology, frameworks and products that help with implementing IoT applications. A developer usually selects an appropriate set of technology and components and incorporates them into an application. The chosen components need to suport the implementation of all relevant aspects of an IoT solution including device connectivity, management, monitoring, business logic and last but not least security enforcement at all levels. In an enterprise context a typical solution is about connecting a given number of homogenous devices to a particular application hosted on dedicated server infrastructure. Over time this results in a set of independent silo applications each managing their own (limited) set of devices. As of today, most of the technology created in Eclipse IoT projects specifically supports the development of IoT applications in this way.

Bosch is a major manufacturer of all kinds of electronic devices, most of which (if not all) will have connectivity built into them in the near future. Integrating these devices with individual IoT applications as descibed above has several drawbacks:

  • The repetitive implementation of common functionality like device communication and management is error prone and inefficient regarding costs, development time and runtime resources.
  • The tight integration of devices with a specific application de facto partitions the set of things into classes determined by the application the devices have been intially integrated with. This makes it hard to create new cross-domain solutions and business models leveraging devices coming from different application domains.
  • The applications implemented this way are often designed to interact with a limited number of devices in an enterprise environment only. Scaling out such applications with an increasing number of devices therefore often requires massive refactoring (if not re-architecting) of the device integration layer in order to support horizontal scalability as required in most cloud-based use cases.

Description

Connectivity is at the heart of IoT solutions. Devices (things) need to be connected to a back end component where the data and functionality of the devices is leveraged to provide some higher level business value. IoT solution developers can pick from a wide array of existing (open source) technology to implement a device connectivity & management layer for the particular type of devices at hand. While this is often fun for the developers to do, the resulting solutions are often silo applications lacking the ability to scale horizontally with the number of devices connected and the number of back end components consuming the device data and functionality.

The Eclipse IoT Working Group has therefore discussed a more generic, cloud-based IoT platform architecture which better supports the implementation of IoT solutions without requiring developers to solve some of the recurring (technical) challenges over and over again. The diagram below provides an overview of the IoT Server Platform as discussed in the working group.

IoT Server Building Blocks.jpg

The diagram shows how devices in the field are connected to a cloud-based back end either via a Field Gateway (e.g. something like Eclipse Kura) or directly to so-called Protocol Adapters. The Protocol Adapters' responsibility is abstracting communication protocols as well as providing location transparency of devices to the other back end components. The devices upload (sensor) data to the back end while the functions/services they expose can be invoked from the back end. These two directions of information flow can be characterized as follows:

  • Telemetry
    Data flowing upstream (left to right) from devices to the back end to a consumer like a Business Application or the Device Management component usually consists of a small set of discrete values like sensor readings or status property values. In most cases these messages are one-way only, i.e. devices sending this kind of data usually do not expect a reply from the back end.
  • Command & Control
    Messages flowing downstream (right to left) from back end components like Business Applications often represent invocations of services or functionality provided by connected devices, e.g. instructions to download and apply a firmware update, setting configuration parameters or querying the current reading of a sensor. In most cases a reply to the sent message is expected by the back end component.

It seems reasonable to assume that the number of messages flowing upstream (Telemetry) will be orders of magnitude larger than the number of messages flowing downstream (Command & Control). The aggregated overall number of messages flowing upstream is expected to be in the range of several hundred thousand to millions per second. Note that in this architecture the same (cloud-based) infrastructure is shared by multiple solutions.

The IoT Connector provides the central link between the device-facing Protocol Adapters, additional re-usable back end components, e.g. Device Management or Software Provisioning, and last but not least the IoT solutions leveraging the devices' data and services. Solution developers can use the IoT Connector to uniformly and transparently interact with all kinds of devices without the need for caring about the particular communication protocol(s) the devices use. Multiple solutions can use the same IoT Connector instance running in a shared cloud environment in order to share the data and functionality of all connected devices. The IoT Connector ensures that only those components can consume data and control devices that have been granted authorization by the device owner. In this regard the IoT Connector can be considered an IoT specific message broker targeted at cloud deployment scenarios.

Features at a glance

  • Secure message dispatching
  • Support for different message exchange patterns
  • Used for cloud service federation
  • Provides interfaces to support implementation of protocol adaptors which allow:
    • Sending telemetry data
    • Receiving device control messages (from applications/solutions)
    • Registering authorized consumers of telemetry data received from connected devices

Scope

The IoT Connector is at the heart of the IoT Server Platform as illustrated in the Description section. It provides the link between the downstream Protocol Adapters and the upstream IoT solutions and other re-usable components leveraging the devices' data and services.

The IoT Connector needs to fulfill a set of non-functional requirements, in particular regarding horizontal scalability, that are specific to both the deployment environment (cloud) and the intended architectural platform characteristics (as opposed to embedding a connectivity layer into applications individually). However, these requirements are not specific to any particular application domain. From a technical point of view it makes no difference if a sensor reading received via a LWM2M protocol adapter represents a temperature or the relative humidity. In both cases the IoT Connector's responsibility is to forward the messages containing the values to (potentially multiple) authorized consumers without introducing too much latency.

Acknowledging the fact that there already exist many message brokers both in the open source world as well as by means of commercial offerings either to be deployed on premises or provided as cloud services, it is not the project's intention to create an additional message broker implementation. Instead, the goal is to create a uniform (remote) service interface that supports both the Telemetry as well as Command & Control message exchange pattern requirements. In order to do so, the IoT Connector also introduces a standard service interface for managing the identity and access restrictions.

The IoT Connector project should provide an initial set of implementations of the service interfaces described above, leveraging existing messaging infrastructure components. As a starting point we would like to provide an implementation based on RabbitMQ because of its easy availability both as a service in existing Cloud Foundry based environments as well as in the form of pre-built Docker images. In order to also provide an implementation supporting horizontal scale-out, we would also like to create an implementation based on Apache Kafka. Future versions may also support using other cloud-based offerings (e.g. Microsoft's Azure Message Bus or Amazon's Simple Queue Service).

People

Project Lead(s)

  •  ???

Committers

  •  ???

Mentors

  •  ???

Interested Parties

  • Bosch Software Innovations GmbH
  • Red Hat, Inc.
  • innoQ Deutschland GmbH (Thomas Eichstädt-Engelen)

Why Here?

TBD

Source Code

Back to the top