G-Eclipse-Amazon-Integration

Basic

Here we want to discuss how an integration of Amazon Web Services (AWS) into the g-Eclipse framework can be realized. First we want to integrate the services S3 and EC2 from AWS. At the very beginning g-Eclipse started as a tool for Grid infrastructures, but as the differences between Grid and Cloud computing are not that big, the g-Eclipse model seems to be able to "host" the integration for AWS too.

Goals

• The tool should be able to manage the lifecycle of EC2 instances
• The tool should enable the ssh login into running instance
• The tool should be able to configure existing instance from the g-Eclipse framework.
• The tool should provide information of all running instances from one user to enable a efficient control of the resources.

• In the future, we want to provide a dedicated AMI (Amazon Machine Image), which
  • contains Java and a dedicated service e.g. a tomcat server or a OSGi kernel
• In the future we want to access dedicated AMI instances to enable the remote deployment of OSGi bundles

EC2 Specific Ideas

The integration of EC2 into the g-Eclipse framework must be done with the Grid Project view of the g-Eclipse framework. Therefore a Virtual Organisation for the AWS is needed. If the AWS VO exists, a new Grid Project can be created. In the Grid Project view, the resources of EC2 can be presented as following:

• The Computing node in the Grid Project view should the list of running EC2 images which can be used the user.
• The Service node in the Grid Project view lists the potential available AMI images for a user.
• The Storage node in the Grid Project view lists the S3 file system created by the user.

Currently, there seems to be no aquivalent entity for jobs in the AWS world. For the Grid job concept within AWS further discussion is needed.

EC2 Specific Use Cases

The following Use Cases should be supported by the g-Eclipse framework

1. The User creates his own Amazon Virtual Organisation as he defines VO for Gria or gLite. There he defines somehow his authentication and authorization properties.

• Open preferences --> g-Eclipse --> Virtual Organization --> New...
• select AWS VO
• enter the requested parametes (i.e. AWS key, ...)

2. The User creates a Grid Project for the AWS VO

• New... --> Grid Project
• select one of the AWS VOs the user created
• The Grid Project view presents the running instances as Computing resources and the available AMI as services. (The integration of the AWS S3 needs further discussions.)

3. The User starts a new instance

• Select the node "Computing" under the AWS VO in the Grid Project View and right click.
• Select from the pop up menu the item "Start new EC2 instance..."
• A wizard is started to select from available AMI images and to start the instance. The new instance is presented immediately under Computing in the Grid Project view.

4. The User stops an running instance

• Select the instance in the Grid Project view and right click.
• Select from the pop up menu the item "Delete"
• The instance is stopped and removed from the Grid project view.

5. The User connect to a running resource via ssh

With a running image up and running the following interfacing technologies provide a good entry into the amazon virtual server:

• Using a created keypair for each instantiation of an image, a direct login (rightclick on instance in CE list) via ssh should be made possible. Because of the used new cert no additional login is required and the user is directly faced with the blinking shell in the console view.
• Similiary to the above, the user should be able to browse the new host system by selecting the node and opening it in the connection view via scp or sftp etc.

Open Issues

• Is there something like a group in S3/EC2? This could be used like a VO to browse the resources of a user/member.
• Deployment: Launch configuration or is there a concept like deployment e.g. in Webtools?
• Currently the S3 authentication implementation != x.509
• Is it possible (how?) to get the information about other S3 buckets that belong to a certain group?
• X.509 libraries, clarify IP issues, bouncycastle
• Management of running AMI instances
• Should work with proxies requiring authentication (cockpit application works fine, but AWS in g-Eclipse still has issues)

Dictionary

• S3/EC2 UIDs == (in a first approach) VO (Virtual Organization)
• S3 Storage == SE (Storage Element)
• AMI (Amazon Machine Image) == CE (Compute Element)
• Installing, starting bundles in a running AMI == Application Deployment

Important Interfaces

• IGridElement
• IGridElementCreator
• IVirtualOrganization (AbstractVirtualOrganization)
• IGridInfoService
• IAuthenticationTokenDescription, IAuthenticationToken (e.g. the Globus (x.509) implementation)