Skip to main content

Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

Difference between revisions of "Babel / Server Tool Development Process"

(currently Gerrit is used)
 
(5 intermediate revisions by one other user not shown)
Line 19: Line 19:
 
===Install Git, PHP, Apache, MySQL if needed===
 
===Install Git, PHP, Apache, MySQL if needed===
 
On Fedora 9, for example (as root):
 
On Fedora 9, for example (as root):
     # yum install mysql-server mysql php php-mysql httpd
+
     # yum install mysql-server mysql php php-mysql httpd git
You can choose to install cvs and/or svn depending on what repositories you will be working with:
+
     # /etc/init.d/httpd  start
    # yum install git
+
     # yum install svn
+
 
     # /etc/init.d/mysqld start
 
     # /etc/init.d/mysqld start
    # /etc/init.d/httpd start
 
  
 +
And you need to set password on MySQL like as following :
 +
    # mysql -u root
 +
    mysql> SELECT Host, User FROM mysql.user;
 +
    mysql> SET PASSWORD FOR 'root'@'127.0.0.1' = PASSWORD('password you want to set');
 +
    mysql> SET PASSWORD FOR 'root'@'localhost' = PASSWORD('password you want to set');
 +
    mysql> SET PASSWORD FOR 'root'@'other shown hosts' = PASSWORD('password you want to set');
 +
 +
(option) You can drop unnecessary database named as test;
 +
    mysql> drop database test;
 +
 +
(option) You can choose to install cvs and/or svn depending on what repositories you will be working with :
 +
    # yum install cvs
 +
    # yum install subversion
 +
 +
===Requirements for PHP===
 +
PHP must have the ability to encode data in the [http://www.json.org/ JSON] format.  As of PHP 5.2 'json_encode' and 'json_decode' are included by default.  If you are using an earlier version you will need to install the JSON PECL extension, find another json_encode library or write your own json_encode function.  See [http://www.php.net/manual/en/json.setup.php this page] to find out more information on JSON and PHP.
 +
And you must set short_open_tag as On in php.ini .
 +
 +
Edit (eg) <tt>/etc/php.ini</tt>
 +
    short_open_tag = On
  
 
===Install Java===
 
===Install Java===
 
Install your favorite Java distro, and pay particular attention to have the jar command to be accessible in the path, since it will be used by the background jobs to create the language packs.
 
Install your favorite Java distro, and pay particular attention to have the jar command to be accessible in the path, since it will be used by the background jobs to create the language packs.
 +
 +
=== Obtaining the code from Git ===
 +
    cd /path/to/babel/
 +
    git clone git://git.eclipse.org/gitroot/babel/server.git
 +
    cd /path/to/babel/server/html/callback
 +
    ln -s ../login.php login.php
  
 
===Create a directory alias in Apache===
 
===Create a directory alias in Apache===
Line 39: Line 62:
 
     </Directory>
 
     </Directory>
  
Apache may need to be reloaded or restarted for the alias to take effect. (eg Fedora 9: <tt>/etc/init.d/httpd restart</tt>)
+
Apache needs to be restarted for the alias to take effect.
 
+
On Fedora 9, for example:
=== Obtaining the code from Git ===
+
    # /etc/init.d/httpd restart
    cd /path/to/babel/
+
    git clone git://git.eclipse.org/gitroot/babel/server.git
+
  
 
=== Write or reuse the addon ===
 
=== Write or reuse the addon ===
Line 65: Line 86:
 
Configure the addons/babel.eclipse.org/base.conf or addons/babel.eclipse.org/staging/base.conf (depending what you did above) file with the MySQL database information.
 
Configure the addons/babel.eclipse.org/base.conf or addons/babel.eclipse.org/staging/base.conf (depending what you did above) file with the MySQL database information.
 
     db_read_host=localhost
 
     db_read_host=localhost
     db_read_user=babel
+
     db_read_user=babel (or as you create)
     db_read_pass=password
+
     db_read_pass=password (as you set)
 
     db_read_name=babel
 
     db_read_name=babel
 
     context=dev
 
     context=dev
  
 
Using your favourite MySQL administration tool, create a database for Babel. It is also recommended to create a MySQL user for Babel.
 
Using your favourite MySQL administration tool, create a database for Babel. It is also recommended to create a MySQL user for Babel.
 +
 
     mysql -u root -p
 
     mysql -u root -p
 
     mysql> create database babel;
 
     mysql> create database babel;
 
     mysql> grant SELECT, INSERT, UPDATE, DELETE on babel.* to babel@'localhost' identified by 'password';
 
     mysql> grant SELECT, INSERT, UPDATE, DELETE on babel.* to babel@'localhost' identified by 'password';
 
     mysql> flush privileges;
 
     mysql> flush privileges;
   
+
 
 
Replace the usernames, hostname and password according to your specific environment. Load the babel tables and sample data (the root account is used here, as the 'babel' account doesn't have CREATE/DROP privileges):
 
Replace the usernames, hostname and password according to your specific environment. Load the babel tables and sample data (the root account is used here, as the 'babel' account doesn't have CREATE/DROP privileges):
  
Line 88: Line 110:
 
by: DROP TRIGGER IF EXISTS `ins_version`;
 
by: DROP TRIGGER IF EXISTS `ins_version`;
  
===Requirements for PHP===
+
At last, set encoding. if you do not know location mysql setting, run following and open the file in left edge from result.
PHP must have the ability to encode data in the [http://www.json.org/ JSON] format. As of PHP 5.2 'json_encode' and 'json_decode' are included by default. If you are using an earlier version you will need to install the JSON PECL extension, find another json_encode library or write your own json_encode function.  See [http://www.php.net/manual/en/json.setup.php this page] to find out more information on JSON and PHP.
+
    use mysql --help | grep my.cnf
 +
 
 +
Edit (eg) <tt>/etc/my.cnf</tt>
 +
    [mysqld]
 +
    character-set-server = utf8
 +
    collation-server = utf8_general_ci
 +
    skip-character-set-client-handshake
 +
    ...
 +
    [mysql]
 +
    default-character-set = utf8
 +
 
 +
And restart MySQL
 +
    /etc/init.d/mysql restart
  
 
=== Phoenix is no longer required for Babel Server===
 
=== Phoenix is no longer required for Babel Server===
Line 95: Line 129:
  
 
=== Logging in ===
 
=== Logging in ===
 
 
Log into Babel at http://localhost/babel/login.php.
 
Log into Babel at http://localhost/babel/login.php.
  
Line 150: Line 183:
  
  
2. Push the new tag: Right click the repo > Team > Remote... > Push...
+
2. Push the new tag: Right click the repo > Team > Remote... > Push to Gerrit...
[[Image:Tagging2 - push.png]]
+
[[Image:Pushtogerrit.png]]
 
+
 
+
 
+
Click the Add All Tags Spec button, then continue the wizard accepting the defaults.
+
 
+
[[Image:Tagging3 - add all tags.png]]
+
 
+
 
+
 
+
  
  
3. After a few moments, the new tag should appear in the cGit web interface, with handy links to download the tag-specific snapshots:
 
[[Image:Tagging4 - cgit view.png]]
 
  
 +
And select 'refs/for/' option and type 'master' as Gerrit Branch.
  
 +
[[Image:Pushtogerrit2.png]]
  
  
  
4.  Send an email to babel-dev@eclipse.org to announce the request for a deployment, specifying the release tag.
 
  
 +
3. After a few moments, you'll notice a URL to the code review Change.
  
 +
4. Committers will review your change. If accepted, your change will be merged.
  
  

Latest revision as of 08:34, 14 February 2016

Babel Team Development Strategy

Prioritizing Strategy

The Babel team's strategy to processing bugs and feature requests is (in order):

  1. Bugs that cause incorrect translations to be saved or prevent people from enter translations
  2. Finishing the version zero issues (deployment, documentation)
  3. Bugs preventing project teams from adding their projects to Babel
  4. Server performance issues
  5. Bugs preventing users from consuming the output of Babel
  6. New features


Setting up a Development Environment

The Babel server environment is a typical LAMP environment. Apache 2.x, PHP 5.x, MySQL 5.x, Java and a Git client are likely the only prerequisites.

Steps to setting up a Babel Development Environment on Linux. For Windows users, please refer to the Running the server on Windows section. If you want to attempt an install of Babel on a SourceForge web server, see Babel / Server Tool SourceForge Installation .

Install Git, PHP, Apache, MySQL if needed

On Fedora 9, for example (as root):

   # yum install mysql-server mysql php php-mysql httpd git
   # /etc/init.d/httpd  start
   # /etc/init.d/mysqld start

And you need to set password on MySQL like as following :

   # mysql -u root
   mysql> SELECT Host, User FROM mysql.user;
   mysql> SET PASSWORD FOR 'root'@'127.0.0.1' = PASSWORD('password you want to set');
   mysql> SET PASSWORD FOR 'root'@'localhost' = PASSWORD('password you want to set');
   mysql> SET PASSWORD FOR 'root'@'other shown hosts' = PASSWORD('password you want to set');

(option) You can drop unnecessary database named as test;

   mysql> drop database test;

(option) You can choose to install cvs and/or svn depending on what repositories you will be working with :

   # yum install cvs
   # yum install subversion

Requirements for PHP

PHP must have the ability to encode data in the JSON format. As of PHP 5.2 'json_encode' and 'json_decode' are included by default. If you are using an earlier version you will need to install the JSON PECL extension, find another json_encode library or write your own json_encode function. See this page to find out more information on JSON and PHP. And you must set short_open_tag as On in php.ini .

Edit (eg) /etc/php.ini

   short_open_tag = On

Install Java

Install your favorite Java distro, and pay particular attention to have the jar command to be accessible in the path, since it will be used by the background jobs to create the language packs.

Obtaining the code from Git

   cd /path/to/babel/
   git clone git://git.eclipse.org/gitroot/babel/server.git
   cd /path/to/babel/server/html/callback
   ln -s ../login.php login.php

Create a directory alias in Apache

Edit (eg) /etc/httpd/conf/httpd.conf to add:

   Alias /babel "/path/to/babel/server/html/"
   <Directory "/path/to/babel/server/html/">
       AllowOverride None
       Order allow,deny
       Allow from all
   </Directory>

Apache needs to be restarted for the alias to take effect. On Fedora 9, for example:

   # /etc/init.d/httpd restart

Write or reuse the addon

You need to select an addon in the addons folder to work with. You can either:

  • Create a addons.conf file in the addons folder, and add this line in the file:

addons=babel.eclipse.org/staging

  • Create an environment variable that will be used by the www-data user (or whatever user will be executing the php files):

export BABEL_ADDONS=babel.eclipse.org/staging

If you want to develop using the Babel look and feel, you should be using the value "babel.eclipse.org". If you want to reuse the style of the staging instance, use "babel.eclipse.org/staging".

You can write your own addon, making it independent or reusing part of the existing addons.

Set up the MySQL database

Configure the addons/babel.eclipse.org/base.conf or addons/babel.eclipse.org/staging/base.conf (depending what you did above) file with the MySQL database information.

   db_read_host=localhost
   db_read_user=babel (or as you create)
   db_read_pass=password (as you set)
   db_read_name=babel
   context=dev

Using your favourite MySQL administration tool, create a database for Babel. It is also recommended to create a MySQL user for Babel.

   mysql -u root -p
   mysql> create database babel;
   mysql> grant SELECT, INSERT, UPDATE, DELETE on babel.* to babel@'localhost' identified by 'password';
   mysql> flush privileges;

Replace the usernames, hostname and password according to your specific environment. Load the babel tables and sample data (the root account is used here, as the 'babel' account doesn't have CREATE/DROP privileges):

   mysql -u root -p babel < /path/to/babel/server/babel-setup.sql

The first time you'll run the Babel setup script, you will get an error from occurring for a trigger that does not exist.

To avoid this error, install MySQL version >= 5.0.32 or version >= 5.1.14 to be able to use IF EXISTS clause for DROP TRIGGER statement and modify the Babel sql script:

Replace: DROP TRIGGER `ins_version`;

by: DROP TRIGGER IF EXISTS `ins_version`;

At last, set encoding. if you do not know location mysql setting, run following and open the file in left edge from result.

   use mysql --help | grep my.cnf

Edit (eg) /etc/my.cnf

   [mysqld]
   character-set-server = utf8
   collation-server = utf8_general_ci
   skip-character-set-client-handshake
   ...
   [mysql]
   default-character-set = utf8

And restart MySQL

   /etc/init.d/mysql restart

Phoenix is no longer required for Babel Server

Previously, Eclipse Phoenix code was required for Babel Server to run. This requirement has been removed (bug 217488).

Logging in

Log into Babel at http://localhost/babel/login.php.

   username: babel@eclipse.org
   password: password

Launch the maps processor: http://localhost/babel/process_map_files.php

Creating (building) a language pack

To create a language pack, simply run php5 path/to/generate1.php from a command line (or from cron). This requires many Linux commands to be found in the PATH environment variable; notably, the jar command that is included with most JDKs.

Background jobs

The background jobs have evolved to a separate page.

Running the server on Windows

The core Babel team uses Linux, but we've had reports of the Babel server running in a Windows environment. Here are some notes:

  • Don't use the PHP crypt() function for authentication: bug 242011

Submitting code patches

Server code patches must be attached to Bugzilla bugs and enhancement requests. Please ensure patches are submitted against a recently updated HEAD stream. See Obtaining the code from Git, above.

Committer Development Process

Babel committers develop new code, and work on bug fixes on their local development environment. Only when a code set is complete can it be committed to the server for staging.

Before doing any work on the server code, please follow these simple guidelines:

  1. Make sure there is a bug that defines the scope of your work
  2. Implement the change on your local development environment
  3. Submit a patch for peer review
  4. Commit your code. Use the following syntax for the first line of your commit comment:
   [123456] Description of the bug
  1. Test the code on the staging area. Git pushes are published to the staging area within a minute.
  2. Request a release (see below) when it's appropriate.

Submitting database changes

Database versioning is something we are working on for the next Milestone release (Milestone 1). Currently if you want to make changes to the live database you need to put the SQL queries for alter/insert/update/create into the schema_change.sql file. When the next tagged version of the code is released the web master will process the schema_change.sql file by hand and make the appropriate changes to the database and test files.


Release process

Babel Server code in Git is published to http://babel.eclipse.org/staging automatically every minute. To deploy the code to the live site, you must tag your local repo, push the tag, then deploy the tagged code to the live site.


1. Tag the Git code with a release tag: R_x_YYYYMMDDHHMM, where x is the release number and YYYYMMDDHHMM is the local time of the tag. Right click the repo > Team > Advanced > Tag. Remember, tagging is a local repository operation. New tags must be pushed upstream. Tagging.png



2. Push the new tag: Right click the repo > Team > Remote... > Push to Gerrit... Pushtogerrit.png


And select 'refs/for/' option and type 'master' as Gerrit Branch.

Pushtogerrit2.png



3. After a few moments, you'll notice a URL to the code review Change.

4. Committers will review your change. If accepted, your change will be merged.


The Designated Deployment Drone (Kit, Denis) then establishes an SSH connection to the Babel server, and as the deployment user 'genie':

  1. Execute any MySQL database changes in the schema_changes.sql file at the Babel Git root, then commit an empty file
  2. run the ./deployBabelLive.sh <tag> to deploy the release, paying close attention to merges, conflicts or other potential errors.

Data Bridges

The Babel server has two data bridges:

  • Eclipse Bugzilla: this bridge is used to maintain a mapping of the Bugzilla user accounts on the Babel server. A process on the Bugzilla server and on the Babel server monitors this URL for changes, triggering an export (and an import) of the data.
  • Eclipse Project/Version info: When the Bugzilla bridge is run, the Project/Version info is pulled from this URL.

Back to the top