Jump to: navigation, search

G-Eclipse-Amazon-Integration

Basic

Here we want to discuss how an integration of Amazon Web Services (AWS) into the g-Eclipse framework can be realized. First we want to integrate the services S3 and EC2 from AWS. At the very beginning g-Eclipse started as a tool for Grid infrastructures, but as the differences between Grid and Cloud computing are not that big, the g-Eclipse model seems to be able to "host" the integration for AWS too.

Goals

  • The tool should be able to manage the lifecycle of EC2 instances
  • The tool should enable the ssh login into running instance
  • The tool should be able to configure existing instance from the g-Eclipse framework.
  • The tool should provide information of all running instances from one user to enable a efficient control of the resources.
  • In the future, we want to provide a dedicated AMI (Amazon Machine Image), which
    • contains Java and a dedicated service e.g. a tomcat server or a OSGi kernel
  • In the future we want to access dedicated AMI instances to enable the remote deployment of OSGi bundles

EC2 Specific Ideas

The integration of EC2 into the g-Eclipse framework must be done with the Grid Project view of the g-Eclipse framework. Therefore a Virtual Organisation for the AWS is needed. If the AWS VO exists, a new Grid Project can be created. In the Grid Project view, the resources of EC2 can be presented as following:

  • The Computing node in the Grid Project view should the list of running EC2 images which can be used the user.
  • The Service node in the Grid Project view lists the potential available AMI images for a user.
  • The Storage node in the Grid Project view lists the S3 file system created by the user.

Currently, there seems to be no aquivalent entity for jobs in the AWS world. For the Grid job concept within AWS further discussion is needed.

EC2 simple Use Cases

The following Use Cases should be supported by the g-Eclipse framework

1. The User creates his own Amazon Virtual Organisation as he defines VO for Gria or gLite. There he defines somehow his authentication and authorization properties.

  • Open preferences --> g-Eclipse --> Virtual Organization --> New...
  • select AWS VO
  • enter the requested parametes (i.e. AWS key, ...)

2. The User creates a Grid Project for the AWS VO

  • New... --> Grid Project
  • select one of the AWS VOs the user created
  • The Grid Project view presents the running instances as Computing resources and the available AMI as services. (The integration of the AWS S3 needs further discussions.)

3. The User starts a new instance

  • Select the node "Computing" under the AWS VO in the Grid Project View and right click.
  • Select from the pop up menu the item "Start new EC2 instance..."
  • A wizard is started to select from available AMI images and to start the instance. The new instance is presented immediately under Computing in the Grid Project view.

4. The User stops an running instance

  • Select the instance in the Grid Project view and right click.
  • Select from the pop up menu the item "Delete"
  • The instance is stopped and removed from the Grid project view.

5. The User connect to a running resource via ssh

  • Select the instance in the Grid Project view and right click.
  • Select from the pop up menu the item "SSH login..."
  • The ssh connection is established with the selected resource.

(* Using a created keypair for each instantiation of an image, a direct login (rightclick on instance in CE list) via ssh should be made possible. Because of the used new cert no additional login is required and the user is directly faced with the blinking shell in the console view.)

6. The User connects to file system of the running resource

(Further discussion needed!)

  • The running instance is shown in the Grid Project under "Computing"
  • Select a "Computing" instance in Grid Project view and right click.
  • Select from the pop up menu the item "Mount as Grid connection" (by using a sftp implementation of EFS??)

7. The User creates a new Amazon Machine Image After starting a dedicated image and after performing some changes in the image,

  • Select the changed instance and right click
  • Select from the pop up menu the item "Create new AMI..."
  • A wizard starts which asks you for more input to create the new AMI image.

EC2 specifc Use Cases

(Further discussion after we finished with the first implemenation of the simple use cases!)

  • start a preconfigured AMI including a OSGi kernel
  • Use a OSGi bundle repository to deploy application in the EC2 resources.

Architecture

The ultimate goal is to provide a flexible architecture for all available Amazon Webservices. Thereby individual services like EC2, S3 or SimpleDB should act as providers to a common AWS middleware (container). The core aws plugin should manage the user credentials and provide the InformationService.

To accomplish this, the aws plugin should expose an extension point to which an aws provider could contribute. An aws provider is hence asked to expose its own service offerings to the aws plugin, which fills the InformationService with the accumulated services from the various providers. Additionally the aws provider provides custom wizard elements when a new VO is created and also plugs itself into the geclipse workbench as it sees fit.

Open Issues

  • Is there something like a group in S3/EC2? This could be used like a VO to browse the resources of a user/member.
  • Deployment: Launch configuration or is there a concept like deployment e.g. in Webtools?
  • Currently the S3 authentication implementation != x.509
  • Is it possible (how?) to get the information about other S3 buckets that belong to a certain group?
  • X.509 libraries, clarify IP issues, bouncycastle
  • Management of running AMI instances
  • Should work with proxies requiring authentication (cockpit application works fine, but AWS in g-Eclipse still has issues)

Dictionary

  • S3/EC2 UIDs == (in a first approach) VO (Virtual Organization)
  • S3 Storage == SE (Storage Element)
  • AMI (Amazon Machine Image) == CE (Compute Element)
  • Installing, starting bundles in a running AMI == Application Deployment

Important Interfaces

  • IGridElement
  • IGridElementCreator
  • IVirtualOrganization (AbstractVirtualOrganization)
  • IGridInfoService
  • IAuthenticationTokenDescription, IAuthenticationToken (e.g. the Globus (x.509) implementation)