Skip to main content

Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

Difference between revisions of "OpenGENESIS WG"

(Abstract for released Paper added)
(41 intermediate revisions by 5 users not shown)
Line 1: Line 1:
== Mission ==
+
[[File:OpenGENESIS_Logo_%28horizontal%29.png|460px]]
  
The mission of openGENESIS is to provide knowledge, methods and tools for the assessment of artificial intelligence (AI) used for autonomous driving. Learning algorithms should be proved to be safe and roadworthy before deployed to public road. From today’s point of view, we are not able to completely understand and explain the detailed functionality of learned algorithms due to the high complexity. openGENESIS will supply public and regulation authorities with necessary approaches to deal with the challenges of approval and certification of AI.
+
'''The openGENESIS collaboration platform leverages knowledge among its members, enabling them to cooperate efficiently and share research results in an open access domain. This will establish a strong global exchange between industry, research and regulators to develop common criteria for the quality of AI. '''
Providing a “TÜV for AI”.
+
  
== Added value ==
+
== Charter and Participation Agreement ==
 +
The [https://www.eclipse.org/org/workinggroups/opengenesis_charter.php openGENESIS Charter] is available for review.
 +
The [https://www.eclipse.org/org/workinggroups/wpga/opengenesis_working_group_participation_agreement.pdf openGENESIS Participation Agreement] is also available.
  
openGENESIS provides a collaboration platform for leveraging knowledge among members. The members cooperate with each other efficiently and provide the results in an open access domain. A strong global exchange between industry, research and regulators will be established. Arrive at a common understanding of quality aspects for artificial intelligence.  
+
Please contact ralph dot mueller at eclipse minus foundation dot org for more information.
 +
 
 +
== Mission ==
 +
 
 +
openGENESIS is a collaborative platform with the mission to provide knowledge, methods and tools for the assessment of artificial intelligence (AI) that is used within autonomous driving applications. Before deployment onto public roads, learning algorithms must be proven safe and roadworthy. However, our current understanding of AI’s complex functionality is limited, especially in relation to machine learning algorithms. openGENESIS will provide both public and regulatory authorities with approaches to help them deal with the challenges of AI approval and certification. Providing a “TÜV for AI”.
  
 
== Scope ==
 
== Scope ==
  
openGENESIS drives and supports the investigations of a verifiable, understandable and not least certifiable AI. This includes activities of adopting functional safety approaches to symbolic and subsymbolic artificial intelligence as well as the development of new methods and metrics for a formal verification of neuronal networks. Approaches regarding a better understandability as well as reproducibility of the decisions of AI will take place. An open data set will be established and research should be enabled with direct access to practice and safety relevant challenges. Finally, also social concerns should be considered in aspects for a certification of learned algorithms.
+
openGENESIS drives and supports the investigation of an understandable, verifiable and certifiable AI. This includes the adoption of functional safety approaches to symbolic and sub-symbolic AI, as well as the development of new methods and metrics for the verification of neuronal networks. Approaches regarding the assessment of performance, robustness and comprehensibility will be compiled. In addition, an open data set is planned to be shared to enable research based on realistic cases and safety relevant challenges. Finally, social concerns should also be considered relating to the certification of learned algorithms.
 +
 
 +
[[File:openGENESIS_scope_overview.png|720px]]
  
 
== Structure ==
 
== Structure ==
  
openGENESIS provides a convenient and legal framework with lean and efficient structures. This is ensured by the Eclipse Foundation, which hosts the working group of openGENESIS.
+
openGENESIS provides a convenient and legal framework with lean and efficient structures. It is ensured by the Eclipse Foundation, which hosts the working group of openGENESIS. The collaborative platform also provides a flexible setting for openGENESIS spotlight projects (GSLP) to solve specific challenges in the community.
  
[[File:20181017_openGENESIS_structure_overview.png]]
+
[[File:openGENESIS_structure_overview.png|920px]]
  
The working group enables the flexible setting of different – so called – openGENESIS spotlight pro-jects (“GSLP”) among partners to solve concrete challenges that are to be defined by the spotlight projects. Such a spotlight project can be formed by one or more participants with a new aspect or im-proved approach that contributes to advances in the scope of openGENESIS. The partners retain their intellectual property, but have to share the results and methodologies with the community. Spotlight projects can provide data to other projects or to the open domain, if the spotlight partners agree togeth-er.
+
These projects can be initiated by one or more participant, taking a new aspect or improving approaches within the scope of openGENESIS. Partners retain their intellectual property, but are required to share their results and methodologies within the community. If the spotlight partners agree, spotlight projects can provide data to other projects or to the public. A steering committee manages the openGENESIS platform, defining and maintaining the scope and road map. It also decides on the acceptance and integration of new spotlight projects, and acts as a first contact.
To organize the working group a steering committee is established. This committee manages openGENESIS by defining and maintaining the scope as well as the road map, deciding on the acceptance and the inte-gration of new spotlight projects and act as a first contact.
+
 
 +
== Steering Committee ==
 +
=== Steering Committee Chair ===
 +
Matthis Eicher, TÜV Süd
 +
=== Members of the Steering Committee ===
 +
* TÜV SÜD [https://www.tuv-sud.com/home_com [TÜV SÜD]]
 +
 
 +
== Members ==
 +
* DFKI (German Research Center for Artificial Intelligence) [https://www.dfki.de/web/ [DFKI]]
 +
* Incenda AI GmbH [https://incenda.de/ [Incenda AI]]
 +
* TÜV SÜD [https://www.tuv-sud.com/home_com [TÜV SÜD]]
 +
 
 +
== Interested parties ==
 +
* Mathworks [https://www.mathworks.com [Mathworks]]
 +
* Phantasma Labs [https://phantasma.global [Phantasma Labs]]
 +
* Bosch [https://www.bosch.de [Bosch]]
 +
* Eteration [https://www.opencanvas.ai [Eteration]]
 +
* iMotion [http://www.imotion.ai [iMotion]]
 +
* Intel Labs [https://www.intel.com/content/www/us/en/research/overview.html [Intel Labs]]
  
 
== Participation ==
 
== Participation ==
  
To generally participate in the openGENESIS working group a minimum membership in the Eclipse Founda-tion as a “Solution Member” is necessary.
+
The openGENESIS working group is open at any time to all new members who meet these conditions.
Within the working group of openGENESIS two classes of memberships are available – Development Member and Driver Member.
+
There are different classes of openGENESIS working group membership - Driver Member, Development Member and Guest Member. Each of these classes is described in detail below:
A detailed description of corresponding rights, duties and fees can be found in the openGENESIS Charta.
+
 
 +
'''Driver Members'''
 +
 
 +
Driver Members are organizations that view the openGENESIS activities as strategic to their organization. These members will invest resources (see openGENESIS Charta for more details) to sustain and shape the activities of this working group as well as maintain the scope of openGENESIS. An openGENESIS WG Driver Member must be at least an Solutions Member of the Eclipse Foundation ([https://www.eclipse.org/membership/become_a_member/]) and must execute the openGENESIS Participation Agreement. They can also initiate and participate in spotlight projects, and are additionally part of the steering committee.
 +
 
 +
'''Development Members'''
 +
 
 +
Development Members are organizations that are interested to benefit from the openGENESIS community. They will lead or participate in "openGENESIS Spotlight Projects" ("GSLP") and investigate the technical questions. Development membership is usually without an additional fee. Development members will invest efforts to the openGENESIS Spotlight Projects ("GSLP") as defined in the proposal. An openGENESIS WG Driver Member must be at least an Solutions Member of the Eclipse Foundation ([https://www.eclipse.org/membership/become_a_member/]) and must execute the openGENESIS Participation Agreement. They are also able to initiate and to participate in spotlight projects under the umbrella of openGENESIS.
 +
 
 +
''' Guest Members'''
 +
 
 +
Guest members are organizations who have been invited for one year by the Steering Committee of openGENESIS WG. Typical Guest Members include academic entities or non-profit organizations, who want to have a closer look before deciding on their strategy. Guest Members are not allowed to become a Steering Committee Member and they have no right to vote concerning the matters of the working group. Guest Members must be at least an Associate Member of the Eclipse Foundation and are required to sign the openGENESIS Participation Agreement. Guest Members are allowed to participate in the openGENESIS WG within the scope of their invitation.
 +
 
 +
== Start a new openGENESIS spotlight project (GSLP) ==
 +
 
 +
Note: the planning and publication of a GSLP proposal does not necessarily require the execution. It can also be used to share the idea and find cooperation partners.
 +
 
 +
The steering committee is pleased to discuss possible ideas for new GSLP's, to assist in the search for a cooperation partner and to provide assistance in the process of initiating the spotlight project. Depending on the project type, different approaches are possible.
 +
 
 +
===Generally conceivable project types===
 +
The necessary procedure is agreed for each project individually together with the Eclipse Foundation and the openGENSIS Steering Committee.
 +
 
 +
'''Classical Software development projects'''
 +
 
 +
This kind of project is marked by a long lifecycle, maintanance activities, updates and adjustments to new environments. Eclipse Foundation is requested to provide necessary infrastructure like GIT, an building-environment, etc.. Complete Eclipse Foundation community is invited to contribute to the project. Examples for this kind of projects are a Neuronal Network Debugger, an AI Metrics calculation tool, or a simulation environment to provide training and testing data for AI.
 +
 
 +
'''Knowledge building projects'''
 +
 
 +
This kind of project is marked by a relatively short lifecycle, self-contained project and no need for maintanance or updates. The Eclipse Foundation is only requested to provide lilltle resources like an Wiki or Tuleap or similar. Examples for this kind of projects are the descriptions of a data labelling process for machine learning, or the elicitation of requirements regarding an AI evaluation process, or fundamental mathematical considerations regarding properties of AI.
 +
 
 +
'''Provide data-sets'''
 +
 
 +
This kind of project is marked by high demands on storage capacities and traffic. In addition, concerns with regard to GDPR and similar must often also be taken into account.
 +
 
 +
===General process for establishing a new spotlight project===
 +
 
 +
* Informal description* of the proposed GSLP is available
 +
* Synchronization about the GSLP proposal with the steering committee
 +
* Formal description (dependent on the project type) of the proposed GSLP
 +
* Official steering committee approval of the GSLP
 +
* Depending on the project type, an EMO approval is passed by the Eclipse Foundation.
 +
* Start of the new openGENESIS spotlight project
 +
 
 +
[[File:OpenGENESIS_GSLP_proposal_process.png|720px]]
 +
 
 +
For an informal description of the proposed GSLP, the following aspects should be considered:
 +
* (Tentative) title of spotlight project
 +
* Your contact information
 +
* Participating entities and project leader
 +
* Scope of the spotlight project proposal
 +
* Description of the spotlight project
 +
* Planned deliverable
 +
* Planned licensing
 +
* Known legal issues
 +
* Project schedule and resources
 +
* Future work
 +
* Necessary infrastructure/tools
 +
* Further interested parties
 +
 
 +
=== Examples for potential openGENESIS spotlight projects (GSLP) ===
 +
 
 +
The participants of a spotlight project define their own development or research object. In order to give a better idea of possible goals within openGENESIS, the following examples of presumably relevant considerations are listed:
 +
* Tool to check the datasets for suitability and representation of the ODD and compliance with the labeling specification
 +
* Establishment of an open and public available training and validation data set
 +
* Methods for goal-oriented generation of synthetic data of behavior of vulnerable road users
 +
* Identification of the necessary properties of synthetic data for an adequate real data representation
 +
* Advances in understandable and explainable AI
 +
* Find and define metrics for quantitative statements about performance and robustness
 +
* Discuss possible adjustments of safety standards to fit for machine learning
 +
* Platform based AI abstraction for better algorithm verification
 +
 
 +
 
 +
== Established openGENESIS spotlight projects (GSLP) ==
 +
===A reliable AI data labeling process===
 +
'''Scope:'''
 +
A process for annotating data and generating corresponding ground-truth information to train and test Artificial Intelligence (especially for Machine Learning) shall be described, investigated and potential weaknesses identified.
 +
 
 +
'''Description:'''
 +
Machine learning is based on training networks from data and also evaluating them on such data. This requires both the data itself and the associated ground truth information of this data. This ground-truth information is created in an "annotation/labeling process" partly automatically, partly manually and basically has the potential to be erroneous.
 +
If the data contain erroneous ground truth information, the function of the network can be learned only to a limited extent or even incorrectly, and secondly, an evaluation of the AI function on this data cannot be trusted.
 +
Therefore, it is absolutely necessary that the ground truth information is correct. This is already anchored in the annotation process, which is why this should be the focus of consideration of this project.
 +
 
 +
As deliverable a whitepaper is planned, which describes the process, the potential weaknesses and possible mitigations.
 +
 
 +
To reach the deliverable, the following steps were performed:
 +
* creation of a ML meta development process diagram
 +
* creation of an labeling process diagram
 +
* description of labeling process
 +
* analysis of labeling process and weakness identification
 +
* discussion of potential mitigations
 +
* creation of the whitepaper
 +
 
 +
 
 +
'''Participants:'''
 +
* Incenda AI GmbH [https://incenda.de/ [Incenda AI]]
 +
* TÜV SÜD [https://www.tuv-sud.com/home_com [TÜV SÜD]]
 +
 
 +
 
 +
'''Launch:''' April 2019
 +
 
 +
'''Links & Publications:'''
 +
 
 +
[[https://wiki.eclipse.org/images/0/0e/WhitePaper_Process_considerations-A_reliable_AI_data_labeling_process.pdf WhitePaper - Process considerations: A reliable AI data labeling process]] (V 1.3 - August 2020 / CC-BY 4.0)
 +
 
 +
Abstract:
 +
 
 +
Due to its wide success in recent years, Artificial Intelligence (AI) is being used in more and more systems. As established Software Engineering practices, including development processes, fail to capture the complexity and additional challenges of developing AI systems, many Software Developers struggle using AI, especially in safety critical areas like healthcare or automotive.
 +
 
 +
One of the AI methods with the highest impact so far is supervised Machine Learning (ML). The performance of supervised ML is determined to a large extent by the data used to train and evaluate the developed models and the application of established Software Engineering practices. Common issues include data and label quality, immature frameworks and processes for supervised ML development, a lack of traceability of requirements to implementation and limited transparency of some models.
 +
 
 +
The contribution of this whitepaper is the discussion and establishment of a sound supervised ML lifecycle, with a focus on data quality, from the intent of developing a system using supervised ML to the decommissioning of the developed system. The different steps of the lifecycle are detailed and a deep-dive into the labeling steps is provided by defining a labeling process. The discussion includes activities that are recommended to be performed in order to create high quality labels and raises typical issues during labeling.
 +
 
 +
== Regular Meetings ==
 +
'''Schedule'''
 +
* Face to face meetings - twice a year
 +
* Online meetings - once a month
 +
 
 +
 
 +
'''Communication'''
 +
* Steering committee meetings are open for openGENESIS members
 +
* Meeting minutes are shared over [https://accounts.eclipse.org/mailing-list/opengenesis-wg [mailing list]] and this wiki
 +
 
 +
== Past Events ==
 +
=== Steering committee meeting of the openGENESIS Working Group, online ===
 +
'''Date:''' Tue, July 08, 2020, 14:00 PM - 15:00 PM CEST
 +
 
 +
Steering committee meeting of the openGENESIS Working group.
 +
 
 +
More information:
 +
* [https://wiki.eclipse.org/images/a/a4/OpenGENESIS_steering_committee_meeting_minutes_20200708.pdf Steering Committee Meeting Minutes]
 +
 
 +
=== Steering committee meeting of the openGENESIS Working Group, online ===
 +
'''Date:''' Tue, Nov 05, 2019, 10:00 AM - 11:00 AM CET
 +
 
 +
Regular steering committee meeting of the openGENESIS Working group.
 +
 
 +
More information:
 +
* [https://wiki.eclipse.org/images/4/4a/OpenGENESIS_steering_committee_meeting_minutes_20191105.pdf Steering Committee Meeting Minutes]
 +
 
 +
=== Steering committee meeting of the openGENESIS Working Group, online ===
 +
'''Date:''' Wed, Sep 25, 2019, 09:00 AM - 10:00 AM CEST
 +
 
 +
Regular steering committee meeting of the openGENESIS Working group.
 +
 
 +
More information:
 +
* [https://wiki.eclipse.org/images/7/7b/OpenGENESIS_steering_committee_meeting_minutes_20190925.pdf Steering Committee Meeting Minutes]
 +
 
 +
=== Steering committee meeting of the openGENESIS Working Group, online ===
 +
'''Date:''' Thu, Aug 20, 2019, 09:00 AM - 10:00 AM CEST
 +
 
 +
Regular steering committee meeting of the openGENESIS Working group.
 +
 
 +
More information:
 +
* [https://wiki.eclipse.org/images/0/05/20190820_openGENESIS_steering_committee_meeting_minutes.pdf Steering Committee Meeting Minutes]
 +
 
 +
=== Foundation meeting of the openGENESIS Working Group, online ===
 +
'''Date:''' Thu, July 18, 2019, 09:00 AM - 11:00 AM CEST
 +
 
 +
This meeting was the official foundation of the openGENSIS working group.
 +
Various bodies were set up and elected, a working mode for the steering committee was set up and requirements for the approval process of spotlight projects were discussed.
 +
 
 +
More information:
 +
* [https://wiki.eclipse.org/images/d/d8/20190722_openGENESIS_foundation_meeting_minutes.pdf Foundation Meeting Minutes]
 +
 
 +
=== First meeting of the openGENESIS Working Group, TÜV Süd, Munich  ===
 +
 
 +
'''Date:''' Tue, April 30, 2019, 9:00 AM – 4:30 PM CEST
 +
 
 +
From the invitation letter:
 +
 
 +
This event will bring all interested parties together for information about the Eclipse Working Group idea.
 +
We will also discuss technical aspects which seem interesting for closer examination and start an open exchange.
 +
 
 +
More information:
 +
* [https://wiki.eclipse.org/images/4/4a/Agenda_openGENESIS_KickOff_20190430.pdf Agenda]
 +
 
 +
Presentations from the event:
 +
 
 +
* [https://cloud.dfki.de/owncloud/index.php/s/zm7RdJBCfPaHHz7#pdfviewer The GAP Between AI Development and Automotive Safety Criteria, C. Müller (DFKI)]
 +
* [https://wiki.eclipse.org/images/4/4c/OpenGENESIS_Meeting_Munich.pdf Eclipse Working Groups, R. Mueller (EF)]
 +
* [https://wiki.eclipse.org/images/3/34/SUMO-Overview.pdf Eclipse SUMO, R. Hilbrich (DLR)]
 +
* [https://wiki.eclipse.org/images/1/1d/OpenGenesis_iMotion_20190430.pdf Proposal Spotlight Projects, Lucy Song (Imotion)]
 +
 
 +
== Interfaces to other initiatives ==
 +
 
 +
=== Lernende Systeme – Plattform für Künstliche Intelligenz ===
 +
'''Link:''' [https://www.plattform-lernende-systeme.de/home.html [https://www.plattform-lernende-systeme.de/home.htm]]
 +
 
 +
'''Project type:''' German national funded project - by BMBF
 +
 
 +
'''Coordinator:''' acatech - Deutsche Akademie der Technikwissenschaften e.V.
 +
 
 +
'''Scope:'''
 +
Scope is to leverage knowledge and to point out perspectives in the area of artificial intelligence to position Germany internationally as a technology leader for learning systems. This should be reached by combination of the expertise from science, business and society.
 +
 
 +
'''Interface to openGENESIS:'''
 +
 
 +
Safety aspects – e.g. in the following groups
 +
* AG 3 (IT Security, Privacy, Law and Ethic)
 +
* AG 5 (Mobility and intelligent Infrastructure)
 +
 
 +
=== KI-Plattform - Konzipierung einer KI-Datenplattform zum Entwickeln und Testen autonomer Fahrzeuge ===
 +
 
 +
'''Link:''' [https://www.elektronikforschung.de/projekte/ki-plattform [https://www.elektronikforschung.de/projekte/ki-plattform]]
 +
 
 +
'''Project type:''' German national funded project - by BMBF
 +
 
 +
'''Coordinator:''' Volkswagen AG
 +
 
 +
'''Scope:'''
 +
 
 +
KI-Plattform will be used to design a common database for learning and test data, which will be available to industrial companies, research institutes and testing organisations as well.
 +
The project will design uniform and flexible data structures that can be used for current and future sensors. Beside real data, synthetic generated or modified data will also be considered. In addition, legal framework conditions and data protection aspects are examined and suitable operator concepts are developed.
 +
 
 +
'''Interface to openGENESIS:'''
  
'''Development Members''' – which is usually without additional fee – need to execute the openGENESIS Participation Agreement. These members are able to initiate or participate in spotlight projects under the umbrella of openGENESIS. To start a spotlight project, a corresponding proposal with a project description should be sent to the steering committee. These proposals will be considered regularly. The description should contain at least the following information:
+
AI Training and Testing Database is one of the major aspects of openGENESIS. Therefore, a close cooperation and coordination should be established and maintained.
  
* (tentative) title of spotlight project
+
=== VDA Leitinitiative - KI Absicherung ===
* your contact information
+
* participating entities
+
* objective of spotlight proposal
+
* description of approach to reach the objec-tive
+
* definition of planned data privacy
+
  
 +
'''Link:''' tbd
  
'''Driver Members''' – which contribute by annual fees to the working group – need to execute the openGENESIS Participation Agreement. Like the Develop-ment Members, they can initiate and participate in spotlight projects. They are additionally part of the steering committee. There they will sustain and shape the activities of the working group as well as maintain the scope of openGENESIS.
+
'''Project type:''' German national funded
  
== Founding members ==
+
'''Coordinator:''' tbd.
* DFKI (German Research Center for Artificial Intelligence)
+
* TÜV SÜD
+
  
== Current Status==
+
'''Scope:''' tbd.
  
The [https://www.eclipse.org/org/workinggroups/opengenesis_charter.php openGENESIS Charter] is drafted and available for review .
+
'''Interface to openGENESIS:'''
 +
Yes - tbd.
  
Please contact ralph dot mueller at eclipse minus foundation dot org for more information.
 
  
 
[[Category:Eclipse IWG]]
 
[[Category:Eclipse IWG]]
 
[[Category:Super Community]]
 
[[Category:Super Community]]

Revision as of 06:17, 4 September 2020

OpenGENESIS Logo (horizontal).png

The openGENESIS collaboration platform leverages knowledge among its members, enabling them to cooperate efficiently and share research results in an open access domain. This will establish a strong global exchange between industry, research and regulators to develop common criteria for the quality of AI.

Charter and Participation Agreement

The openGENESIS Charter is available for review. The openGENESIS Participation Agreement is also available.

Please contact ralph dot mueller at eclipse minus foundation dot org for more information.

Mission

openGENESIS is a collaborative platform with the mission to provide knowledge, methods and tools for the assessment of artificial intelligence (AI) that is used within autonomous driving applications. Before deployment onto public roads, learning algorithms must be proven safe and roadworthy. However, our current understanding of AI’s complex functionality is limited, especially in relation to machine learning algorithms. openGENESIS will provide both public and regulatory authorities with approaches to help them deal with the challenges of AI approval and certification. Providing a “TÜV for AI”.

Scope

openGENESIS drives and supports the investigation of an understandable, verifiable and certifiable AI. This includes the adoption of functional safety approaches to symbolic and sub-symbolic AI, as well as the development of new methods and metrics for the verification of neuronal networks. Approaches regarding the assessment of performance, robustness and comprehensibility will be compiled. In addition, an open data set is planned to be shared to enable research based on realistic cases and safety relevant challenges. Finally, social concerns should also be considered relating to the certification of learned algorithms.

OpenGENESIS scope overview.png

Structure

openGENESIS provides a convenient and legal framework with lean and efficient structures. It is ensured by the Eclipse Foundation, which hosts the working group of openGENESIS. The collaborative platform also provides a flexible setting for openGENESIS spotlight projects (GSLP) to solve specific challenges in the community.

OpenGENESIS structure overview.png

These projects can be initiated by one or more participant, taking a new aspect or improving approaches within the scope of openGENESIS. Partners retain their intellectual property, but are required to share their results and methodologies within the community. If the spotlight partners agree, spotlight projects can provide data to other projects or to the public. A steering committee manages the openGENESIS platform, defining and maintaining the scope and road map. It also decides on the acceptance and integration of new spotlight projects, and acts as a first contact.

Steering Committee

Steering Committee Chair

Matthis Eicher, TÜV Süd

Members of the Steering Committee

Members

Interested parties

Participation

The openGENESIS working group is open at any time to all new members who meet these conditions. There are different classes of openGENESIS working group membership - Driver Member, Development Member and Guest Member. Each of these classes is described in detail below:

Driver Members

Driver Members are organizations that view the openGENESIS activities as strategic to their organization. These members will invest resources (see openGENESIS Charta for more details) to sustain and shape the activities of this working group as well as maintain the scope of openGENESIS. An openGENESIS WG Driver Member must be at least an Solutions Member of the Eclipse Foundation ([1]) and must execute the openGENESIS Participation Agreement. They can also initiate and participate in spotlight projects, and are additionally part of the steering committee.

Development Members

Development Members are organizations that are interested to benefit from the openGENESIS community. They will lead or participate in "openGENESIS Spotlight Projects" ("GSLP") and investigate the technical questions. Development membership is usually without an additional fee. Development members will invest efforts to the openGENESIS Spotlight Projects ("GSLP") as defined in the proposal. An openGENESIS WG Driver Member must be at least an Solutions Member of the Eclipse Foundation ([2]) and must execute the openGENESIS Participation Agreement. They are also able to initiate and to participate in spotlight projects under the umbrella of openGENESIS.

Guest Members

Guest members are organizations who have been invited for one year by the Steering Committee of openGENESIS WG. Typical Guest Members include academic entities or non-profit organizations, who want to have a closer look before deciding on their strategy. Guest Members are not allowed to become a Steering Committee Member and they have no right to vote concerning the matters of the working group. Guest Members must be at least an Associate Member of the Eclipse Foundation and are required to sign the openGENESIS Participation Agreement. Guest Members are allowed to participate in the openGENESIS WG within the scope of their invitation.

Start a new openGENESIS spotlight project (GSLP)

Note: the planning and publication of a GSLP proposal does not necessarily require the execution. It can also be used to share the idea and find cooperation partners.

The steering committee is pleased to discuss possible ideas for new GSLP's, to assist in the search for a cooperation partner and to provide assistance in the process of initiating the spotlight project. Depending on the project type, different approaches are possible.

Generally conceivable project types

The necessary procedure is agreed for each project individually together with the Eclipse Foundation and the openGENSIS Steering Committee.

Classical Software development projects

This kind of project is marked by a long lifecycle, maintanance activities, updates and adjustments to new environments. Eclipse Foundation is requested to provide necessary infrastructure like GIT, an building-environment, etc.. Complete Eclipse Foundation community is invited to contribute to the project. Examples for this kind of projects are a Neuronal Network Debugger, an AI Metrics calculation tool, or a simulation environment to provide training and testing data for AI.

Knowledge building projects

This kind of project is marked by a relatively short lifecycle, self-contained project and no need for maintanance or updates. The Eclipse Foundation is only requested to provide lilltle resources like an Wiki or Tuleap or similar. Examples for this kind of projects are the descriptions of a data labelling process for machine learning, or the elicitation of requirements regarding an AI evaluation process, or fundamental mathematical considerations regarding properties of AI.

Provide data-sets

This kind of project is marked by high demands on storage capacities and traffic. In addition, concerns with regard to GDPR and similar must often also be taken into account.

General process for establishing a new spotlight project

  • Informal description* of the proposed GSLP is available
  • Synchronization about the GSLP proposal with the steering committee
  • Formal description (dependent on the project type) of the proposed GSLP
  • Official steering committee approval of the GSLP
  • Depending on the project type, an EMO approval is passed by the Eclipse Foundation.
  • Start of the new openGENESIS spotlight project

OpenGENESIS GSLP proposal process.png

For an informal description of the proposed GSLP, the following aspects should be considered:

  • (Tentative) title of spotlight project
  • Your contact information
  • Participating entities and project leader
  • Scope of the spotlight project proposal
  • Description of the spotlight project
  • Planned deliverable
  • Planned licensing
  • Known legal issues
  • Project schedule and resources
  • Future work
  • Necessary infrastructure/tools
  • Further interested parties

Examples for potential openGENESIS spotlight projects (GSLP)

The participants of a spotlight project define their own development or research object. In order to give a better idea of possible goals within openGENESIS, the following examples of presumably relevant considerations are listed:

  • Tool to check the datasets for suitability and representation of the ODD and compliance with the labeling specification
  • Establishment of an open and public available training and validation data set
  • Methods for goal-oriented generation of synthetic data of behavior of vulnerable road users
  • Identification of the necessary properties of synthetic data for an adequate real data representation
  • Advances in understandable and explainable AI
  • Find and define metrics for quantitative statements about performance and robustness
  • Discuss possible adjustments of safety standards to fit for machine learning
  • Platform based AI abstraction for better algorithm verification


Established openGENESIS spotlight projects (GSLP)

A reliable AI data labeling process

Scope: A process for annotating data and generating corresponding ground-truth information to train and test Artificial Intelligence (especially for Machine Learning) shall be described, investigated and potential weaknesses identified.

Description: Machine learning is based on training networks from data and also evaluating them on such data. This requires both the data itself and the associated ground truth information of this data. This ground-truth information is created in an "annotation/labeling process" partly automatically, partly manually and basically has the potential to be erroneous. If the data contain erroneous ground truth information, the function of the network can be learned only to a limited extent or even incorrectly, and secondly, an evaluation of the AI function on this data cannot be trusted. Therefore, it is absolutely necessary that the ground truth information is correct. This is already anchored in the annotation process, which is why this should be the focus of consideration of this project.

As deliverable a whitepaper is planned, which describes the process, the potential weaknesses and possible mitigations.

To reach the deliverable, the following steps were performed:

  • creation of a ML meta development process diagram
  • creation of an labeling process diagram
  • description of labeling process
  • analysis of labeling process and weakness identification
  • discussion of potential mitigations
  • creation of the whitepaper


Participants:


Launch: April 2019

Links & Publications:

[WhitePaper - Process considerations: A reliable AI data labeling process] (V 1.3 - August 2020 / CC-BY 4.0)

Abstract:

Due to its wide success in recent years, Artificial Intelligence (AI) is being used in more and more systems. As established Software Engineering practices, including development processes, fail to capture the complexity and additional challenges of developing AI systems, many Software Developers struggle using AI, especially in safety critical areas like healthcare or automotive.

One of the AI methods with the highest impact so far is supervised Machine Learning (ML). The performance of supervised ML is determined to a large extent by the data used to train and evaluate the developed models and the application of established Software Engineering practices. Common issues include data and label quality, immature frameworks and processes for supervised ML development, a lack of traceability of requirements to implementation and limited transparency of some models.

The contribution of this whitepaper is the discussion and establishment of a sound supervised ML lifecycle, with a focus on data quality, from the intent of developing a system using supervised ML to the decommissioning of the developed system. The different steps of the lifecycle are detailed and a deep-dive into the labeling steps is provided by defining a labeling process. The discussion includes activities that are recommended to be performed in order to create high quality labels and raises typical issues during labeling.

Regular Meetings

Schedule

  • Face to face meetings - twice a year
  • Online meetings - once a month


Communication

  • Steering committee meetings are open for openGENESIS members
  • Meeting minutes are shared over [mailing list] and this wiki

Past Events

Steering committee meeting of the openGENESIS Working Group, online

Date: Tue, July 08, 2020, 14:00 PM - 15:00 PM CEST

Steering committee meeting of the openGENESIS Working group.

More information:

Steering committee meeting of the openGENESIS Working Group, online

Date: Tue, Nov 05, 2019, 10:00 AM - 11:00 AM CET

Regular steering committee meeting of the openGENESIS Working group.

More information:

Steering committee meeting of the openGENESIS Working Group, online

Date: Wed, Sep 25, 2019, 09:00 AM - 10:00 AM CEST

Regular steering committee meeting of the openGENESIS Working group.

More information:

Steering committee meeting of the openGENESIS Working Group, online

Date: Thu, Aug 20, 2019, 09:00 AM - 10:00 AM CEST

Regular steering committee meeting of the openGENESIS Working group.

More information:

Foundation meeting of the openGENESIS Working Group, online

Date: Thu, July 18, 2019, 09:00 AM - 11:00 AM CEST

This meeting was the official foundation of the openGENSIS working group. Various bodies were set up and elected, a working mode for the steering committee was set up and requirements for the approval process of spotlight projects were discussed.

More information:

First meeting of the openGENESIS Working Group, TÜV Süd, Munich

Date: Tue, April 30, 2019, 9:00 AM – 4:30 PM CEST

From the invitation letter:

This event will bring all interested parties together for information about the Eclipse Working Group idea. We will also discuss technical aspects which seem interesting for closer examination and start an open exchange.

More information:

Presentations from the event:

Interfaces to other initiatives

Lernende Systeme – Plattform für Künstliche Intelligenz

Link: [https://www.plattform-lernende-systeme.de/home.htm]

Project type: German national funded project - by BMBF

Coordinator: acatech - Deutsche Akademie der Technikwissenschaften e.V.

Scope: Scope is to leverage knowledge and to point out perspectives in the area of artificial intelligence to position Germany internationally as a technology leader for learning systems. This should be reached by combination of the expertise from science, business and society.

Interface to openGENESIS:

Safety aspects – e.g. in the following groups

  • AG 3 (IT Security, Privacy, Law and Ethic)
  • AG 5 (Mobility and intelligent Infrastructure)

KI-Plattform - Konzipierung einer KI-Datenplattform zum Entwickeln und Testen autonomer Fahrzeuge

Link: [https://www.elektronikforschung.de/projekte/ki-plattform]

Project type: German national funded project - by BMBF

Coordinator: Volkswagen AG

Scope:

KI-Plattform will be used to design a common database for learning and test data, which will be available to industrial companies, research institutes and testing organisations as well. The project will design uniform and flexible data structures that can be used for current and future sensors. Beside real data, synthetic generated or modified data will also be considered. In addition, legal framework conditions and data protection aspects are examined and suitable operator concepts are developed.

Interface to openGENESIS:

AI Training and Testing Database is one of the major aspects of openGENESIS. Therefore, a close cooperation and coordination should be established and maintained.

VDA Leitinitiative - KI Absicherung

Link: tbd

Project type: German national funded

Coordinator: tbd.

Scope: tbd.

Interface to openGENESIS: Yes - tbd.

Back to the top