Skip to main content

Notice: this Wiki will be going read only early in 2024 and edits will no longer be possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

EMF Build Server Setup

Revision as of 03:49, 15 January 2008 by Codeslave.ca.ibm.com (Talk | contribs) (→‎Install via apt-get)

EMF Build Server Setup

You will need to be root for most of these tasks.

Install OS

Installation began with Ubuntu Server 6.06LTS, for x86. Very simple and straight forward. I partitioned the drive 36.4G drive like this: 

sdb1   Boot, NC   Primary   Linux ext3   10000.01M
sdb2              Primary   Linux swap    1000.00M
sdb3              Primary   Linux ext3   25391.44M

Install via apt-get

apt-get install apache2 perl5 curl cvs sun-java6-jdk
apt-get install php5 php5-cgi php5-cli php5-curl 
apt-get install zip unzip tar bzip2 sudo screen sendmail
apt-get install openssh-server ccrypt tofrodos
apt-get install libcvsservice0 cvs tkcvs
apt-get install vim vim-common vim-scripts vim-gtk
apt-get install libgtk2.0-common libgtk2.0-0 libgtk2.0-bin

apt-get install xserver-xorg-core xvfb xserver-xorg-driver-vesa xrgb \
  xserver-xorg xserver-xorg-input-mouse xfs xkill xlsfonts xlsclients \
  xinit xinitrc xkeyboard-config kbd xserver-xorg-input-kbd xkbutils \
  xset xsetroot libxp6 libxpm4 libxp-java xnest menu twm metacity \
  xkbutils xxkb xkbset xkbsel libxkbsel-dev

Install & symlink

Set default editor to vim

  • If you prefer vim to nano, you'll have to do this:
export EDITOR=vim
  - or - 
cd /etc/alternative; rm -f editor; ln -s /usr/bin/vim editor

Recreate ssh keys

  • If upgrading from an existing server, you can copy the old server's ssh keys so that the server will "look" the same to users connecting via ssh. 
cp /mnt/sda1/etc/ssh/* /etc/ssh/

Set a static IP address

/etc/network/interfaces

# The loopback interface
auto lo
iface lo inet loopback

# The first network card - this entry was created during the Debian installation
# (network, broadcast and gateway are optional)
auto eth0
iface eth0 inet static
address 9.xx.yyy.zzz
netmask 255.255.254.0
network 9.xx.yyy.0
broadcast 9.xx.yyy.255
gateway 9.xx.yyy.1

Add bootable partitions to grub loader

/boot/grub/menu.lst

default         0
timeout         5

title           Ubuntu, kernel 2.6.15-26-server
root            (hd1,0)
kernel          /boot/vmlinuz-2.6.15-26-server root=/dev/sdb1 ro quiet splash
initrd          /boot/initrd.img-2.6.15-26-server
savedefault
boot

title           Ye Old Debian on sda1, kernel 2.4.25, no initrd
root            (hd0,0)
kernel          /boot/vmlinuz-2.4.25 root=/dev/sda1
boot

Fix web user (www-data)

  • Add group 'www':
addgroup www
  • Edit /etc/group. Add www group:

/etc/group

www:x:1008:www-data,user1,user2,user3,...
  • Edit /etc/passwd. Change users' home directories and shell:

/etc/passwd

www-data:x:33:33:www-data:/home/www-data:/bin/sh
  • Edit /etc/sudoers to let you run commands and switch to the web user w/o needing a password. Add the following lines:

/etc/sudoers

# Members of the admin group may gain root privileges
%admin ALL=(ALL) ALL

# Members of the www group can sudo to www-data
%www            ALL = (www-data) NOPASSWD: ALL
  • Switch to the web user. You should NOT be prompted for a password.
sudo -u www-data bash
  • Create an ssh key, WITH NO PASSPHRASE. Store in ~/.ssh/id_rsa and ~/.ssh/id_rsa.pub:
ssh-keygen -b 2048 -t rsa
  • Copy contents of ~/.ssh/id_rsa.pub into ~/.ssh/authorized_keys file for user on dev.eclipse.org who will be running builds. This is so that the web user can commit changes to cvs (tagging, updating map files) for I, M, S & R builds. This is only required for EMF builds.
  • Test by ssh'ing to dev.eclipse.org, where _username_ should be replaced with your actual username:
ssh _username_@dev.eclipse.org
  • Run newgrp www so that when new files are created, they will use the group id www instead of www-data:
newgrp www
  • Set umask 022 so that files will be created with group write perms 664 (see #/etc/bashrc):
umask 022
  • Set the remote shell connection method for CVS to be ssh instead of the default rsh (see #/etc/bashrc):
export CVS_RSH=/usr/bin/ssh
  • Set an ANT_HOME and JAVA_HOME, and add ant to the PATH (see #/etc/bashrc):
export ANT_HOME=/opt/apache-ant-1.6
export JAVA_HOME=/opt/sun-java2-5.0
export PATH=${PATH}:${ANT_HOME}/bin
  • Switch to the root user.
  • Replace /etc/bashrc with this:

/etc/bashrc

[ -f ~/.alias ] && . ~/.alias

# set PATH so it includes user's private bin if it exists
if [ -d ~/bin ] ; then
    PATH=~/bin:"${PATH}"
fi

umask 022
export GREP_OPTIONS='--color=auto'
export GTK2_RC_FILES=$HOME/.gtkrc-2.0
export KDEDIR=/usr
export QTDIR=/usr/share/qt3
export EDITOR=vi
export LC_ALL=C; #for perl

norm="\033[0;39m";grey="\033[1;30m";green="\033[1;32m";brown="\033[0;33m";
yellow="\033[1;33m";blue="\033[1;34m";cyan="\033[1;36m";red="\033[1;31m";

ucol=$grey;
hcol=$yellow;
rcol=$red;

if [ $UID -eq 0 ]; then # root
  prompt="\! $rcol\]\u@$hcol\h$norm:\$(pwd)\\[$norm\]\n$rcol#$norm ";  fcol=$brown;
else
  prompt="\! $ucol\]\u@$hcol\h$norm:\$(pwd)\\[$norm\]\n$ucol\$$norm "; fcol=$ucol;
fi

# If this is an xterm set the title to user@host:dir
case $TERM in
 xterm*)
  export PS1=$prompt"\[\e]30;\u@\H\a\]"
  export PROMPT_COMMAND='echo -ne "\033]0;${USER}@${HOSTNAME}: ${PWD}\007"'
  ;;
 *)
  export PS1=$prompt
  ;;
esac
  • Add the following to .bashrc and .bash_profile files:

~/.bashrc

[ -f /etc/bashrc ] && . /etc/bashrc

~/.bash_profile

[ -f ~/.bashrc ] && . ~/.bashrc
  • Copy both .bashrc and .bash_profile into /etc/skel, so that new users will get these files in their home dirs.

Change Apache's default web dir

  • For consistency w/ other servers, edit /etc/apache2/sites-enabled/000-default and set Apache's default DocumentRoot to /var/www/html instead of /var/www.

Set up web content

  • Copy content from an existing server partition or else extract from CVS:
mkdir -p /home/www-data/build/modeling; cd /home/www-data/build/modeling;
cvs -d :pserver:anonymous@dev.eclipse.org:/cvsroot/modeling -q co -d scripts releng-common/tools/scripts
mkdir -p /var/www/html; cd /var/www/html; 
cvs -d :pserver:anonymous@dev.eclipse.org:/cvsroot/org.eclipse -q co -d modeling www/modeling
cvs -d :pserver:anonymous@dev.eclipse.org:/cvsroot/org.eclipse -q co -d eclipse.org-common www/eclipse.org-common

Fix permissions & ownership

  • Create a script called /usr/local/bin/wwwup, which does this:
#!/bin/bash

# update cvs
cd /var/www/html/modeling; cvs -q up -Pd;
cd /var/www/html/eclipse.org-common; cvs -q up -Pd;

# fix perms/groups
cd /var/www/html; find . -type f -exec chmod 664 {} \;
cd /var/www/html; find . -type d -exec chmod 2775 {} \;
cd /var/www/html; find . -exec chown www-data:www {} \;

# fix perms/groups
cd /home/www-data/build; find . -type f -exec chmod 664 {} \;
cd /home/www-data/build; find . -type d -exec chmod 2775 {} \;
cd /home/www-data/build; find . -exec chown www-data:www {} \;

# make scripts executable
cd /home/www-data/build/modeling/scripts; find . -type f -name "*.sh" -exec chmod 755 {} \;
  • Make the script executable, then run it:
chmod 775 /usr/local/bin/wwwup;
wwwup

Fix hostname

  • Ensure your /etc/hosts file contains something a line such as this, so that the server knows its own name:
127.0.0.1     localhost
9.xx.yyy.zzz  emf.torolab.ibm.com emf

Set up Build's Web UI

  • Clone existing web UI, then fix server name, version, branch, JDK requirement, email address & username:
cd /var/www/html/modeling/emft/
cp -r mwe newComponent;
vi newComponent/build/_common.php 

$ cat _common.php
<?php
require_once ("../../../includes/buildServer-common.php");

$options = array (
       "BaseBuilderBranch" => "v20070614",

       "BranchAndJDK" => array (
               "emft.eclipse.org=------------,------------",
               "0.7.0=HEAD,/opt/sun-java2-5.0",

               "build.eclipse.org=------------,------------",
               "0.7.0=HEAD,/opt/public/common/ibm-java2-ppc-50"
       ),

       "Mapfile_Rule_Default" => 0, // 0: "Use Map, No Tagging=use-false" or 1:"Generate Map, No Tagging=gen-false"

       "EmailDefault" => "someone@company.net", // prefil email contact box with comma-sep'd list

       "Users" => array("username", "username", null) /* build user, eclipse cvs user, IES cvs user */
);
?>
  • Create folder where builds will be produced
mkdir /home/www-data/build/modeling/emft/newComponent/downloads/drops
  • If the above new /build/ folder was created outside the web root and committed to CVS, update web UI in web path:
cd /var/www/html/modeling/emft/;
cvs -q up -Pd newComponent;

$isBuildServer = $_SERVER["SERVER_NAME"] = "127.0.0.1" || (preg_match("/^(emft|build)\.eclipse\.org$/", $_SERVER["SERVER_NAME"])) || $isEMFserver;
  • Check the website, http://servername/modeling/emft/newComponent/build/

Password Protection w/ .htaccess

  • If you require .htaccess security, put an .htaccess file in the /build/ folder like this one:
AuthName "Build Server Password Required"
AuthUserFile /path/to/password/file
AuthGroupFile /dev/null
AuthType Basic
require user authorizedUsername
  • Then create a password file in /path/to/password/file. You can use htpasswd to generate the password.
authorizedUsername:lXfv8s4tra.qI
  • Set permissions on the password file:
 chmod 444 /path/to/password/file
 chown www-data:www-data /path/to/password/file
  • Check the website, http://servername/modeling/emft/newComponent/build/ -- this time, you should be prompted for a password. Depending on your browser, you may have to close & reopen it.

Run a build

From the build page, http://servername/modeling/emft/newComponent/build/, fire a build.

Check the build log while it's running or after it completes. If you see any messages such as permission denied or the following, something is amiss.

Debugging tips

execvp: Permission denied
  • Shell scripts must contain unix line endings. Run dos2unix to make sure, if copying them from a non-unix filesystem.
  • Shell scripts /home/www-data/build/emft/scripts/*.sh must be executable. See #Fix permissions & ownership.



  • Directories, eg. /home/www-data/build/emft/jet/downloads/drops/1.0.0 must be writable by the web user (or group www). See #Fix permissions & ownership.



eclipse.org: Connection refused
cvs [checkout aborted]: end of file from server (consult above messages if any)
  • CVS connection refusals are the result of not being able to automatically ssh as the web user to _username_@dev.eclipse.org. This

could be a problem with ssh keys (see above - #Fix web user (www-data)) or the method CVS uses to connect (rsh instead of ssh - see #/etc/bashrc).



  • For any compilation-related problems, missing file problems, or other issues not touched upon in this document, see EMFT_Procedures.

Display build logs, details & test results

Builds - including unpublished Nightly builds - are listed on the downloads page here: http://emf.torolab.ibm.com/modeling/emf/downloads/.

Published builds are located here: http://downloads.eclipse.org/modeling/emf/downloads/.

Add additional users

  • As root, run:
adduser -p newuserpassword newuser
  • Then copy .bashrc, .bash_profile, and .alias from an existing user to the new user's home dir (if /etc/skel is not used).
 cp /home/user/.bashrc /home/user/.bash_profile /home/user/.alias /home/newuser
  • Fix permissions on copied files
 chown newuser:newuser /home/newuser/.bashrc /home/newuser/.bash_profile /home/newuser/.alias

Install & configure mysql server 5

  • Install
 apt-get install mysql-server-5.0 mysql-client-5.0

CVS configuration

  • Install
apt-get install cvs
  • Create cvs group
vi /etc/group
  • Add the following line (or similar), listing all the users to have CVS access, then exit and save (ESC, :wq)
cvs:x:115:user1,user2,user3,...
  • Create CVSROOT
cd /home; ln -s /var/lib/cvs
  • Set permissions
cd /home/cvs
chgrp -R cvs .
chmod -R 2775 . CVSROOT
  • Edit /home/cvs/CVSROOT/config; add or change LockDir to be:
LockDir=/var/lock/cvs
  • Create lock dir and set permissions
mkdir /var/lock/cvs
chgrp cvs /var/lock/cvs
chmod g+s /var/lock/cvs

Verify X server

  • Ensure X server is running. You will have to have installed all the x* packages listed above in the apt-get section. Type:
# ps aux --forest | grep root | grep -i x
  • You should see something like this:
root      3820  0.0  0.0   4456  2428 ?        Ss   Jan11   0:00 /usr/bin/xfs -daemon
root     12128  0.0  0.2  14128  7304 tty2     Ss+  Jan14   0:20 X
root     12264  0.0  0.0   2408  1160 ?        S    Jan14   0:00 /bin/sh /usr/bin/x-session-manager
root     12289  0.0  0.0   4336   692 ?        Ss   Jan14   0:00  \_ /usr/bin/ssh-agent x-session-manager
  • If not running, start X by setting runlevel to 5 in /etc/inittab:
# The default runlevel.
id:5:initdefault:
  • Or, to change runlevels without rebooting, type:
telinit 5
  • Then restart the server or manually start up X (as root) with one or more of these:
X &                                start the X server
startx &                           start the X server
startkde &                         start the K Desktop Environment
kdm &                              start the K Display Manager

/etc/init.d/xfs start &            start the X font server
/etc/init.d/x11-common start &     start the X server and ICE socket dirs
/etc/init.d/kdm start &            start the K display manager
  • To connect headlessly to an x server, try this. Ignore any font errors.
Xvfb :42 -screen 0 1024x768x24 -ac &
Xnest :43 -display :42 -depth 24 &
twm -display localhost:43.0 &
export DISPLAY=localhost:43.0; ulimit -c unlimited &
xterm &
  • If the x server allowed you to connect to it, you can run this:
ps ax --forest | grep xterm -B5
  • You should see something like this -- note that the second bash is running on a different pts:
5792 ?        S      0:00      \_ sshd: nickb@pts/1
5793 pts/1    Ss     0:00          \_ -bash
5795 pts/1    S      0:00              \_ Xvfb :42 -screen 0 1024x768x24 -ac
5799 pts/1    S      0:00              \_ Xnest :43 -display :42 -depth 24
5801 pts/1    S      0:00              \_ twm -display localhost:43.0
5807 pts/1    S      0:00              \_ xterm
5809 pts/2    Ss+    0:00              |   \_ bash
5842 pts/1    R+     0:00              \_ ps ax --forest
5843 pts/1    S+     0:00              \_ grep xterm -B5

See Also

Additional info and configuration steps are listed in EMFT Build Server Setup.

Retrieved from "https://wiki.eclipse.org/index.php?title=EMF_Build_Server_Setup&oldid=74391"

Back to the top