Skip to main content

Notice: This Wiki is now read only and edits are no longer possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

EMFT Build Server Setup

Revision as of 18:00, 28 July 2006 by Unnamed Poltroon (Talk) (→‎Secure filesystem access with tripwire)

EMFT Build Server Setup

You will need to be root for most of these tasks.

Set up web content

FROM (emf.torolab.ibm.com)TO (emft.eclipse.org)
/home/www-data/build/emft-->/home/www-data/build/emft
/var/www/emft-->/var/www/html/emft
/var/www/technology/emft-->/var/www/html/technology/emft

Fix permissions & ownership

cd /var/www/html; find . -type f -exec chmod 664 {} \;
cd /var/www/html; find . -type d -exec chmod 775 {} \;
cd /var/www/html; find . -exec chown apache:www {} \;

cd /home/www-data/build; find . -type f -exec chmod 664 {} \;
cd /home/www-data/build; find . -type d -exec chmod 775 {} \;
cd /home/www-data/build; find . -exec chown apache:www {} \;
cd /home/www-data/build/scripts; find . -type f -name "*.sh" -exec chmod 755 {} \;

Apache 2 w/ PHP 5 was already installed. Only a few additional programs needed to be installed as well.

Install via yum using yum-xen.conf file

  • Get a list of the available updates
yum -c http://people.redhat.com/~katzj/yum-xen.conf list 2>&1 | tee /tmp/yum-list.txt
  • Install required groups of installs - Java Development & Development Tools
yum -c http://people.redhat.com/~katzj/yum-xen.conf groupinstall "Java Development"
yum -c http://people.redhat.com/~katzj/yum-xen.conf groupinstall "Development Tools"
  • Install gtk- and x11-related packages (in order to do Eclipse UI JUnit tests)
yum -c http://people.redhat.com/~katzj/yum-xen.conf install gtk2.i386

=============================================================================
 Package                 Arch       Version          Repository        Size
=============================================================================
Installing:
 gtk2                    i386       2.6.10-2.fc4.4   updates-released  4.8 M
Installing for dependencies:
 atk                     i386       1.9.1-1          base              178 k
 pango                   i386       1.8.1-2          base              272 k

yum -c http://people.redhat.com/~katzj/yum-xen.conf install xorg-x11.i386 xorg-x11-libs.i386 xorg-x11-xdm.i386 

=============================================================================
 Package                 Arch       Version          Repository        Size
=============================================================================
Installing:
 xorg-x11                i386       6.8.2-37.FC4.49.2  updates-released   14 M
 xorg-x11-xdm            i386       6.8.2-37.FC4.49.2  updates-released  170 k
Updating:
 xorg-x11-libs           i386       6.8.2-37.FC4.49.2  updates-released  2.5 M
Installing for dependencies:
 chkfontpath             i386       1.10.0-4         base               13 k
 fonts-xorg-base         noarch     6.8.2-1          base              7.3 M
 ttmkfdir                i386       3.0.9-16.1       updates-released   49 k
 xinitrc                 noarch     4.0.18.1-1       updates-released   28 k
 xorg-x11-font-utils     i386       6.8.2-37.FC4.49.2  updates-released  124 k
 xorg-x11-xauth          i386       6.8.2-37.FC4.49.2  updates-released  102 k
 xorg-x11-xfs            i386       6.8.2-37.FC4.49.2  updates-released  141 k
 xterm                   i386       208-4.FC4        updates-released  191 k

Install & symlink

Fix web user (apache)

  • Edit /etc/group. Add www group:

/etc/group

www:x:501:user,apache
  • Edit /etc/passwd. Change user's home directory and shell:

/etc/passwd

apache:!!:48:48:Apache:/home/www-data:/bin/bash
  • Edit /etc/sudoers to let you run commands and switch to the web user w/o needing a password. Add the following lines:

/etc/sudoers

%www  ALL = (apache) NOPASSWD: ALL, (root) /usr/bin/su apache
  • Switch to the web user. You should NOT be prompted for a password.
sudo -u apache bash
  • Create an ssh key, WITH NO PASSPHRASE. Store in ~/.ssh/id_rsa and ~/.ssh/id_rsa.pub:
ssh-keygen -b 2048 -t rsa
  • Copy contents of ~/.ssh/id_rsa.pub into ~/.ssh/authorized_keys file for user on dev.eclipse.org who

will be running builds. This is so that the web user can commit changes to cvs (tagging, updating map files) for I, M, S & R builds.

  • Test by ssh'ing to dev.eclipse.org, where _username_ should be replaced with your actual username:
ssh _username_@dev.eclipse.org
  • Run newgrp www so that when new files are created, they will use the group id www instead of apache:
newgrp www
  • Set umask 022 so that files will be created with group write perms 664 (see #/etc/bashrc):
umask 022
  • Set the remote shell connection method for CVS to be ssh instead of the default rsh (see #/etc/bashrc):
export CVS_RSH=/usr/bin/ssh
  • Set an ANT_HOME and JAVA_HOME, and add ant to the PATH (see #/etc/bashrc):
export ANT_HOME=/opt/apache-ant-1.6
export JAVA_HOME=/opt/sun-java2-5.0
export PATH=${PATH}:${ANT_HOME}/bin
  • Switch to the root user.
  • Append the following into /etc/bashrc, where _username_ should be replaced with your actual dev.eclipse.org username:

/etc/bashrc

umask 022
export ANT_HOME=/opt/apache-ant-1.6
export JAVA_HOME=/opt/sun-java2-5.0
export PATH=${PATH}:${ANT_HOME}/bin
export CVSROOT=:ext:_username_@dev.eclipse.org:/cvsroot/technology
export CVS_RSH=/usr/bin/ssh
  • Add the following to .bashrc and .bash_profile files:

~/.bashrc

if [ -f /etc/bashrc ]; then
  . /etc/bashrc
fi

if [ "$PS1" ]; then
  # enable color support of ls and also add handy aliases
  eval `dircolors -b`
  alias ls='ls --color=auto'
  alias ll='ls -l --color=auto'

  # set a fancy prompt
  # 1;30 - grey, 1;31 - red, 1;32 - green, 1;33 - yellow, 1;34 - blue, etc.
  PS1="\[\033[<b>1;30</b>m\]\u@\h:\w\\[\033[0;39m\] \$ "
  export PS1=$PS1"\[\e]30;\u@\H:\w\a\]"
  PATH=$PATH:/opt/ibm-java2-1.4/bin:/opt/apache-ant-1.6/bin
  export PATH
fi

source ~/.alias
cat ~/.alias

~/.bash_profile

if [ -f ~/.bashrc ]; then
  . ~/.bashrc
fi

PATH=$PATH:$HOME/bin
export PATH
unset USERNAME

Secure build script

  • Since the webserver is public but builds should only be run by authorized users, we must secure access to the build.php script.
  • Edit Apache config file to allow .htaccess rule changes to take affect. Change None to All:

/etc/httpd/conf/httpd.conf

# AllowOverride controls what directives may be placed in .htaccess files.
# It can be "All", "None", or any combination of the keywords:
#   Options FileInfo AuthConfig Limit
#
  AllowOverride All
  • Create password file, where _username_ should be replaced with an actual username and /path/to/password/file should be replaced with an actual path:
httpasswd -c /path/to/password/file _username_
  • You can add additional users to the password file like this:
httpasswd /path/to/password/file _username2_
  • Create .htaccess file, replacing /path/to/password/file with the same path used in the previous step:

/var/www/html/emft/.htaccess

AuthType Basic
AuthName "EMFT Build Server @ Eclipse.org"
AuthUserFile /path/to/password/file
Require valid-user
  • Restart apache, eg.:
/usr/sbin/httpd -k restart
 - or -
apache2ctl restart

Run a build

Using the username and password set up in the previous step, go here: http://emft.eclipse.org/emft/build.php.

Check the build log while it's running or after it completes. If you see any messages such as permission denied or the following, something is amiss.

Debugging tips

execvp: Permission denied
  • Shell scripts must contain unix line endings. Run dos2unix to make sure, if copying them from a non-unix filesystem.
  • Shell scripts /home/www-data/build/emft/scripts/*.sh must be executable. See #Fix permissions & ownership.

--

  • Directories, eg. /home/www-data/build/emft/jet/downloads/drops/1.0.0 must be writable by the web user

(or group www). See #Fix permissions & ownership.

-- 

eclipse.org: Connection refused
cvs [checkout aborted]: end of file from server (consult above messages if any)
  • CVS connection refusals are the result of not being able to automatically ssh as the web user to _username_@dev.eclipse.org. This

could be a problem with ssh keys (see above - #Fix web user (apache)) or the method CVS uses to connect (rsh instead of ssh - see #/etc/bashrc).

--

  • For any compilation-related problems, missing file problems, or other issues not touched upon in this document, see EMFT_Procedures.

Display build logs, details & test results

Builds - including unpublished Nightly builds - are listed on the downloads page here: http://emft.eclipse.org/technology/emft/downloads/.

Published builds are located here: http://downloads.eclipse.org/technology/emft/downloads/.

Add additional users

  • As root, run:
adduser -p newuserpassword newuser
  • Then copy .bashrc, .bash_profile, and .alias from an existing user to the new user's home dir.
 cp /home/user/.bashrc /home/user/.bash_profile /home/user/.alias /home/newuser
  • Fix permissions on copied files
 chown newuser:newuser /home/newuser/.bashrc /home/newuser/.bash_profile /home/newuser/.alias

Install & configure mysql server

Install via yum

 yum update
 yum install mysql

Set mysql root user password

Per instructions posted here http://dev.mysql.com/doc/refman/4.1/en/resetting-permissions.html:

  • Stop running daemon
 /etc/init.d/mysqld stop
  • Start mysqld server
 /usr/bin/mysqld_safe --skip-grant-tables --user=root &

 mysql -u root
  • In mysql, run the following commands:
 UPDATE mysql.user SET Password=PASSWORD('newpwd') WHERE User='root';
 FLUSH PRIVILEGES;
 QUIT;
  • Stop server
 kill `cat /var/run/mysqld/host_name.pid`;
  • Wait until process is actually killed
 ps ax | grep mysqld
  • Start mysqld daemon
 /etc/init.d/mysqld start
  • To verify password set correctly:
 mysql -u root -p
  • Enter new assigned root password when prompted

Add a user

  • Start up mysql if not already connected.
 mysql -u root -p
  • In mysql, run the following commands:
 GRANT ALL PRIVILEGES ON *.* TO newuser@"%" IDENTIFIED BY 'newuser' WITH GRANT OPTION;
 GRANT ALL PRIVILEGES ON *.* TO newuser@"localhost" IDENTIFIED BY 'newuser' WITH GRANT OPTION;
 QUIT;

Firewall configuration

  • Create the following script as /root/firewall
#!/bin/sh
# suggested by webmaster@eclipse.org (Denis)

# Flush the INPUT chain
/sbin/iptables -F INPUT
/sbin/iptables -P INPUT ACCEPT

# Flush FORWARD chain
/sbin/iptables -F FORWARD
/sbin/iptables -P FORWARD DROP

# Drop all SSH connections
/sbin/iptables -I INPUT -p tcp --dport 22 -j REJECT
/sbin/iptables -I INPUT -p tcp --dport 22 -j LOG

# Accept from *.eclipse.org
/sbin/iptables -I INPUT -p tcp -s 206.191.52.32/27 --dport 22 -j ACCEPT

# Allow one IP
# /sbin/iptables -I INPUT -p tcp -s 200.31.39.228/32 --dport 22 -j ACCEPT

# Allow an entire subnet
# /sbin/iptables -I INPUT -p tcp -s 201.89.34.0/24 --dport 22 -j ACCEPT
  • Then link to it from init.d and rc.d to make it run on vserver startup
ln -s /root/firewall /etc/init.d/firewall
ln -s /etc/init.d/firewall /etc/rc.d/rc3.d/S56firewall
ln -s /etc/init.d/firewall /etc/rc.d/rc5.d/S56firewall

CVS configuration

(With thanks to http://michael-amorose.com/articles/computers/cvs/6_1.html)

  • Install CVS
yum install cvs
  • Create cvs group
vi /etc/group
  • Add the following line (or similar), listing all the users to have CVS access, then exit and save (ESC, :wq)
cvs:x:115:user1,user2,user3,...
  • Create CVSROOT
mkdir -p /cvsroot/modeling
cvs -d /cvsroot/modeling init
  • Set permissions
cd /cvsroot/modeling
chgrp -R cvs .
chmod -R 2775 . CVSROOT

CVS Mirroring

  • In order to create a clone of the dev.eclipse.org cvs repositories so as to be able to massage them in a sandbox environment, the following script can be used:
#!/bin/bash
user=nickb;

# to use these colour escapes, must do `echo -e`
red="\033[1;31m"; green="\033[1;32m"; yellow="\033[1;33m"; blue="\033[1;34m"; norm="\033[0;39m";

getFromRemote () {
  root=$1; proj=$2;
  echo -e $green"Synch /cvsroot/$root/$proj ..."$norm;
  rsync -Phzogtr $user@dev.eclipse.org:/cvsroot/$root/$proj .;
}

mkdir -p ./cvsroot/modeling; cd ./cvsroot/modeling;

# tech homes, source
for d in gmt-home; do getFromRemote technology $d; done
for d in org.eclipse.gmt org.eclipse.gmf org.eclipse.emft org.eclipse.mddi; do getFromRemote technology $d; done

# tools homes, source
for d in gef-home emf-home uml2-home; do getFromRemote tools $d; done
for d in org.eclipse.gef.source-feature org.eclipse.gef.examples-feature org.eclipse.gef-feature \
  org.eclipse.gef.examples.ediagram  org.eclipse.releng.gefbuilder org.eclipse.gef.source \
  org.eclipse.draw2d.examples  org.eclipse.draw2d  org.eclipse.gef.examples.text org.eclipse.gef.doc.isv \
  org.eclipse.gef.test org.eclipse.gef.examples.logic org.eclipse.gef.examples.shapes org.eclipse.gef.examples.flow \
  org.eclipse.gef org.eclipse.draw2d.test org.eclipse.draw2d.doc.isv org.eclipse.gef.releng org.eclipse.uml2 \
  org.eclipse.emf org.eclipse.emf.ecore.sdo org.eclipse.xsd org.eclipse.emf.releng.build org.eclipse.uml2.releng ; do \
  getFromRemote tools $d; done

echo "";
du --max-depth=1 -h;
cd -;
  • Then copy or merge these files into /cvsroot/modeling

Secure filesystem access with tripwire

  • Install tripwire
yum install tripwire

tripwire                i386       2.3.1-22         extras            1.8 M
  • Configure tripwire
tripwire-setup-keyfiles
  • Review/edit policy file in /etc/tripwire/twpol.txt
  • Initialize tripwire database. Log problems to a textfile, then filter them to extract just a list of missing files
tripwire --init 2>&1 | tee problems.txt; \
grep Filename problems.txt > problems.txt1; \
cat problems.txt1 | awk -F: '{ print $2 }' > problems.txt; \
rm -fr problems.txt1;
  • Remove any warnings about missing files (ie., things not actually installed)
cp twpol.txt twpol.txt.bak
for f in `cat problems.txt`; do 
  f=${f//\//\\\/}; # echo $f; # escape slashes
  cat twpol.txt | sed -e "s/ $f /#REMOVE# $f /" > twpol.txt1; diff twpol.txt1 twpol.txt; mv twpol.txt1 twpol.txt; 
done; 
diff twpol.txt twpol.txt.bak
  • Remove the database and start over using the new text policy file.
rm -fr /var/lib/tripwire/emft.eclipse.org.twd;
mkdir old; mv emft.eclipse.org-local.key site.key problems.txt* tw.* *.bak old/
tripwire-setup-keyfiles
tripwire --init  2>&1 | tee problems.txt
  • Perform integrity check
tripwire --check

rpm -i ccrypt-1.7-1.i386.rpm
  • Encrypt policy file in /etc/tripwire/twpol.txt
ccrypt -e /etc/tripwire/twpol.txt

Thanks to the following Red Hat manuals for assistance in setup:

http://www.redhat.com/docs/manuals/linux/RHL-7.3-Manual/ref-guide/ch-tripwire.html http://www.redhat.com/docs/manuals/linux/RHL-7.3-Manual/ref-guide/s1-tripwire-install.html#S2-TRIPWIRE-INSTALL-POST

--Codeslave.ca.ibm.com 16:54, 28 July 2006 (EDT)

Retrieved from "https://wiki.eclipse.org/index.php?title=EMFT_Build_Server_Setup&oldid=8197"

Back to the top