Faculty of Information Technology
Software Engineering Group

Building Smalltalk/X jv-branch from sources

This text describes how to compile Smalltalk/X jv-branch from sources. The same process and tools is used on SWING CI server to build Smalltalk/X jv-branch packages.

In a Nutshell

A quick guide for the impatient...

  1. Make sure you have set up a build environment as instructed
  2. Open a shell (on Linux) or command prompt (on Windows) and execute:
    hg clone https://bitbucket.org/janvrany/stx-goodies-builder-rake stx-jv
    cd stx-jv
    rake
    
  3. Go out for a short or long walk, depending on your internet connection and CPU power.

Once rake finishes (without an error, indeed) everthing should be compiled. You can then run Smalltalk/X by:

projects/smalltalk/smalltalk

or on Windows:

projects\smalltalk\smalltalk.bat

If compilation fails, please read the rest of this document, especially section on compiling, updating and hacking. If it is still failing, feel free to ask.

Have Fun!

Background

A little background information on what buildscripts actually do...

Building Smalltalk/X jv-branch is a little complex and requires number of steps:

  • First, you need to fetch all the sources from various repositories. As of now, from Jan Vrany's BitBucket and from eXept's public CVS. All sources has to be put into proper directory structure.
  • Second, as source of stc and librun is not publicly available, pre-build binaries has to be downloaded. As of now, they're downloaded from (former) SWING Research Group CI Server.
  • Various patches and tweaks has to be applied to sources in order to compile.
  • Finally compile all packages in dependency order

Build scripts utomate all necessary steps so it all happens automagically - ideally it all should be as easy as typing

rake

Not always it is like that, but we aim to make it just like this :-)

Setting up a Build Environment

To build a Smalltalk/X you need to have a number of tools installed and set up, such as C compiler toolchain. These tools are not needed to run Smalltalk/X once compiled.

The rest of this section gives you a detailed information on how to set up your system. Generally speaking, you need following:

  • A GCC compiler - 4.7.x, 5.x or 6.x
  • GNU Make
  • Mercurial and CVS clients
  • Ruby 2.1 or newer
  • rake
  • JDK 7
  • Apache Ant

Windows

Tested on Window 7 Professional, 64bit.

  • Install MSYS2 from https://msys2.github.io/. Make sure to install it to C:\MSYS64, otherwise Smalltalk/X build scripts won't find it!. Once installer finishes, open a command prompt and execute following commands:
    c:\msys64\usr\bin\pacman.exe -Sy pacman
    c:\msys64\usr\bin\pacman.exe -Syu
    c:\msys64\usr\bin\pacman.exe -Su
    
  • Note: During pacman -Syu you maybe asked to close command prompt window. Don't forget to close it otherwise you will have to start from beginning.
  • Note2: When you want to start use MSYS2 shell use msys2_shell.cmd (installer installs incorrectly .bat files).
  • Install GNU Make and CVS client. To do so, execute following in a command prompt:
    c:\msys64\usr\bin\pacman.exe -S make cvs openssh ssh-pageant-git zip unzip
    
  • Add c:\msys64\usr\bin to your PATH:
    setx PATH "%PATH%;c:\msys64\usr\bin"
    
    Close the command prompt an open it again before continuing''' This is important so the new PATH become effective.
  • Install GCC toolchain. For building 64bit Smalltalk/X, install:
    c:\msys64\usr\bin\pacman.exe -S  mingw-w64-x86_64-gcc mingw-w64-x86_64-SDL
    
    For 32bit builds, install:
    c:\msys64\usr\bin\pacman.exe -S  mingw-w64-i686-gcc mingw-w64-i686-SDL
    
    You may install both if you want to build both, 64bit and 32bit versions.
  • Install Java JDK 7 or newer. Though Oracle Java would do, we recommend using Zulu from http://www.azul.com/downloads/zulu/zulu-windows/ az Zulu 7 is recommended for use with STX:LIBJAVA on Windows. Make sure java.exe and javac.exe are in your PATH. To do so, you may use following commands:
    setx PATH "%PATH%;C:\Program Files\Zulu\zulu-7\bin"
    setx JAVA_HOME "C:\Program Files\Zulu\zulu-7"
    
    Close the command prompt an open it again before continuing''' This is important so the new PATH become effective.
  • Install Apache Ant - https://wiki.apache.org/ant/AntOnWindows. Make sure ant command is in your PATH.

This should be everything you need to compile Smalltalk/X jv-branch as described above

Linux

Following guides are for building 64bit Smalltalk/X on 64bit Linux (or 32bit Smalltalk/X on 32bit Linux). It is possible - of course - to build ac 32bit Smalltalk/X on 64bit Linux. In that case you need to install 32bit libraries and headers. How to do that is left as an exercise to dear reader/. Hint: on DEB-based distros you need to install pretty much same packages with :i386 suffix, on RPM-based with .i686 plus something like gcc-multilib.

Debian (and other derivatives such as Ubuntu or Mint)

Tested on Debian Stretch, both 64bit and 32bit, standard installations.

Install following packages:

sudo aptitude install ruby mercurial cvs build-essential bison flex libc6-dev gcc-multilib libx11-dev default-jdk ant libglib2.0-dev libwebkitgtk-dev libsoup2.4-dev libxext-dev unixodbc-dev libgl1-mesa-dev libfl-dev

This should be everything you need to compile Smalltalk/X jv-branch as described above

Building 32bit Smalltalk/X on 64bit system

To build 32bit Smalltalk/X on 64bit system, you need to install 32bit versions of some libraries in addition to above:

dpkg --add-architecture i386
apt-get update 
apt-get install libx11-dev:i386 libxext-dev:i386 libxft2:i386 libxinerama-dev:i386 libodbc1:i386 odbcinst1debian2:i386 libgl1-mesa-dev:i386 libfl-dev:i386

CentOS 7

Tested on CentOS 7.2 (Minimal). Thanks to Patrik Svestka for this guide!

First couple steps only install X environment. If you already have it installed or don't want it, you may skip them.

  • First install XServer (with xfce)
    sudo yum groupinstall "Development Tools" "X Window system" xfce Fonts
    systemctl set-default graphical.target
    
  • Optional: For the system to have "nice" fonts
    sudo yum install xorg-x11-fonts-Type1 xorg-x11-fonts-misc
    

Following is required to compile Smalltalk/X jv-branch

  • Install rvm (management for ruby - system ruby is not a good idea):
    rvm requirements
    rvm install ruby-2.3
    rvm docs generate-ri # documentation
    
  • Install java (open-jdk 8)
    sudo yum install java
    
  • unixODBC needed to start StX (installs handy ODBC connector)
    sudo yum install unixODBC 
    
  • Versioning systems (for getting the sources from)
    sudo yum install mercurial cvs
    
  • Devel packages needed for compilation
    sudo yum install glib2 libX11-devel libXext-devel qtwebkit libsoup libXft-devel libXinerama-devel ant bison flex
    

This should be everything you need to compile Smalltalk/X jv-branch as described above

  • Optional: Install XSLT and DocBook? Stylesheets. This is only required if you want to create deployment artifacts, i.e., run rake install or rake artifacts
    sudo yum install libxslt-devel docbook5-style-xsl docbook-style-xsl
    
    Since paths on CentOS differ from those on Debian, workaround this by creating symlinks:
    cd /usr/share/xml/
    mkdir -p docbook/stylesheet/
    cd /usr/share/xml/docbook/stylesheet
    ln -s /usr/share/sgml/docbook/xsl-stylesheets-1.78.1/ docbook-xsl
    

Fedora 23

Tested on Fedora 23 Workstation, 64bit, standard installation. Thanks to Martin Kobetic and Patrik Svestka for hints.

  • Install following packages:
    sudo dnf install gcc java ant ruby rake mercurial cvs bison flex unixODBC glib2 libX11-devel libXext-devel qtwebkit libsoup libXft-devel libXinerama-devel ant 
    

This should be everything you need to compile Smalltalk/X jv-branch as described above

Compiling, recompiling, updating and hacking

The whole compilation is driven by rake, a Ruby build tool. All daily tasks are implemented as rake targets and therefore can be invoked by simply running

rake <target>

in a shell (command prompt).

Remember, build scripts are also versioned and evolve (slowly, but they do), so you'd better update them regularly.

Task: checkout fresh sources and compile

To fetch all the source code use checkout target, i.e,

rake checkout

This will check all necessary sources from corresponding repositories into build subdirectory.

To compile, use - surprise - compiler target.

Task: update previously checked out sources to latest revisions

rake update

will update every repository to the most recent revision (aka HEAD or master). Then use

rake compile

to recompile. Following a custom, this would recompile only what changed (and what depends on changed sources).

Task: run compiled Smalltalk/X

To run compiled Smalltalk/X, go to build/stx/projects/smalltalk directory and run smalltalk (smalltalk.bat on Windows):

cd build/stx/projects/smalltalk
./smalltalk -I --quick

(-I means no snapshot restart, --quick means no search for autoloaded classes)

Task: run tests to make sure everything is fine

Once compiled (or hacked) you may want to make sure that everything is fine by running tests. As you may have guessed, use rake to run test task:

rake test

Beware that it may take a quite some time to run all the tests. Make sure you're not running on battery as it may go flat rather quickly. Ideally, all tests should pass. Once all run, a short summary is printed. You may find detailed XML report in reports directory. If not all of them pass, please report an issue!

Task: TODO

To be written...

Building Smalltalk/X on Jenkins CI

Work in progress. This section may be incomplete needs to be polished.

Setting up Jenkins Server

Smalltalk/X pipeline script uses a number of API which Pipeline sandbox plugin does not allow. To allow all of them at once, edit $JENKINS_HOME/scriptApproval.xml and add following:

<approvedSignatures>
    <string>method com.cloudbees.plugins.credentials.common.IdCredentials getId</string>
    <string>method com.cloudbees.plugins.credentials.common.StandardCredentials getDescription</string>
    <string>method hudson.plugins.mercurial.MercurialSCM getBranch</string>
    <string>method hudson.plugins.mercurial.MercurialSCM getCredentialsId</string>
    <string>method hudson.plugins.mercurial.MercurialSCM getSource</string>
    <string>method java.lang.Class getName</string>
    <string>method java.lang.Object hashCode</string>
    <string>method java.lang.Class isInstance java.lang.Object</string>
    <string>method java.util.Collection remove java.lang.Object</string>
    <string>method java.util.Collection toArray</string>
    <string>new java.util.ArrayList java.util.Collection</string>
    <string>new java.util.HashMap</string>
    <string>new java.util.HashMap java.util.Map</string>
    <string>staticMethod com.cloudbees.plugins.credentials.CredentialsProvider lookupCredentials java.lang.Class</string>
    <string>staticMethod java.lang.System identityHashCode java.lang.Object</string>
    <string>staticMethod org.codehaus.groovy.runtime.DefaultGroovyMethods contains java.lang.Object[] java.lang.Object</string>
    <string>staticMethod org.codehaus.groovy.runtime.DefaultGroovyMethods plus java.util.List java.lang.Iterable</string>
    <string>staticMethod org.codehaus.groovy.runtime.DefaultGroovyMethods putAt java.lang.Object java.lang.String java.lang.Object</string>
  </approvedSignatures>

Setting up Build Slaves

In addition to setup described above, additional packages / setup is needed when running a full CI pipeline on a build slave.

Debian (and alike)

Install following extra packages:

aptitude install xsltproc docbook-xsl

Windows

Install following extra packages:

pacman -Su libxslt docbook-xsl

IMPORTANT: make sure you run c:\msys64\msys2.exe as the jenkins user. This creates MSYS2 home directory for the user - otherwise SSH won't be able to store known_hosts file there and checkout would fail. See MSYS ssh error: Could not create directory '/home/<username>/.ssh'

NOTE: If using Windows 10, you may want to enable long paths by default. To so so, set registry key:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem\LongPathsEnabled

to 1 (as 32-bit DWORD).

Troubleshooting

When things go wrong, check this troubleshooting checklist for common problems (and solutions).

Last modified 6 days ago Last modified on Nov 15, 2017, 12:37:15 AM