wiki:Documentation/BuildingStXWithRakefiles

Context Navigation

Version 69 (modified by jan vrany, 5 years ago) ( diff )
--

Building Smalltalk/X jv-branch from sources

Building Smalltalk/X jv-branch from sources

This text describes how to compile Smalltalk/X jv-branch from sources. The same process and tools is used on SWING CI server to build Smalltalk/X jv-branch packages.

In a Nutshell

A quick guide for the impatient...

Make sure you have set up a build environment as instructed

Open a shell (on Linux) or command prompt (on Windows) and execute:

hg clone https://bitbucket.org/janvrany/stx-goodies-builder-rake stx-jv
cd stx-jv
rake

Go out for a short or long walk, depending on your internet connection and CPU power.

Once rake finishes (without an error, indeed) everthing should be compiled. You can then run Smalltalk/X by:

projects/smalltalk/smalltalk

or on Windows:

projects\smalltalk\smalltalk.bat

If compilation fails, please read the rest of this document, especially section on compiling, updating and hacking. If it is still failing, feel free to ask.

Have Fun!

Background

A little background information on what buildscripts actually do...

Building Smalltalk/X jv-branch is a little complex and requires number of steps:

First, you need to fetch all the sources from various repositories. As of now, from Jan Vrany's BitBucket and from eXept's public CVS. All sources has to be put into proper directory structure.
Second, as source of stc and librun is not publicly available, pre-build binaries have to be downloaded. As of now, they're downloaded from (former) SWING Research Group CI Server.
Various patches and tweaks has to be applied to sources in order to compile.
Finally compile all packages in dependency order

Build scripts utomate all necessary steps so it all happens automagically - ideally it all should be as easy as typing

rake

Not always it is like that, but we aim to make it just like this :-)

Setting up a Build Environment

To build a Smalltalk/X you need to have a number of tools installed and set up, such as C compiler toolchain. These tools are not needed to run Smalltalk/X once compiled.

The rest of this section gives you a detailed information on how to set up your system. Generally speaking, you need following:

A GCC compiler - 4.7.x, 5.x or 6.x
GNU Make
Mercurial and CVS clients
Ruby 2.1 or newer
rake
JDK (Java Development Kit)
Apache Ant

Windows

Windows 7

Tested on Window 7 Professional, 64bit.

Install Mercurial from https://www.mercurial-scm.org/. Make sure the path to hg.exe is in your PATH.
Install Ruby 2.2.x from http://rubyinstaller.org/downloads/. Make sure Ruby interpreter is in your PATH (by checking corresponding checkbox when installing it) - the best way to install ruby is to use ruby manager like uru (https://bitbucket.org/jonforums/uru)

Install MSYS2 from https://msys2.github.io/. Make sure to install it to C:\MSYS64, otherwise Smalltalk/X build scripts won't find it!. Once installer finishes, open a command prompt and execute following commands:
```
c:\msys64\usr\bin\pacman.exe -Sy pacman
c:\msys64\usr\bin\pacman.exe -Syu
c:\msys64\usr\bin\pacman.exe -Su
```
Note: during pacman -Syu you maybe asked to close command prompt window. Don't forget to close it otherwise you will have to start from beginning.

Note: when you want to start use MSYS2 shell use msys2_shell.cmd (installer installs incorrectly .bat files).

Install GNU Make and CVS client. To do so, execute following in a command prompt:
```
c:\msys64\usr\bin\pacman.exe -S make cvs openssh ssh-pageant-git zip unzip
```
Add c:\msys64\usr\bin to your PATH:
```
setx PATH "%PATH%;c:\msys64\usr\bin"
```
Close the command prompt an open it again before continuing''' This is important so the new PATH become effective.
Install GCC toolchain. For building 64bit Smalltalk/X, install:
```
c:\msys64\usr\bin\pacman.exe -S  mingw-w64-x86_64-gcc mingw-w64-x86_64-SDL
```
For 32bit builds, install:
```
c:\msys64\usr\bin\pacman.exe -S  mingw-w64-i686-gcc mingw-w64-i686-SDL
```
You may install both if you want to build both, 64bit and 32bit versions.
Install Java JDK 7 or newer. Though Oracle Java would do, we recommend using Zulu from http://www.azul.com/downloads/zulu/zulu-windows/ az Zulu 7 is recommended for use with STX:LIBJAVA on Windows. Make sure java.exe and javac.exe are in your PATH. To do so, you may use following commands:
```
setx PATH "%PATH%;C:\Program Files\Zulu\zulu-7\bin"
setx JAVA_HOME "C:\Program Files\Zulu\zulu-7"
```
Close the command prompt an open it again before continuing''' This is important so the new PATH become effective.
Install Apache Ant - https://wiki.apache.org/ant/AntOnWindows. Make sure ant command is in your PATH.

This should be everything you need to compile Smalltalk/X jv-branch as described above

Windows 10 (using scoop)

Tested on Windows 10 x64 (Version 10.0.15063). Thanks to Patrik Svestka for testing and this guide.

This guide uses scoop installer as it makes the installation process much easier. See scoop crashcourse for quick reference on scoop.

Install scoop. Execute the following in powershell n a custom directory c:\app_scoop:

[environment]::setEnvironmentVariable('SCOOP','c:\app_scoop','User')
$env:SCOOP='C:\app_scoop'
iex (new-object net.webclient).downloadstring('https://get.scoop.sh')

Install 7zip, git and MSYS2:
```
scoop install 7zip
scoop install git
scoop install msys2
```
Note: Don't forget to check your PATH if it was updated correctly with msys2. In case you are missing it update your user PATH with (use your own path): C:\app_scoop\apps\msys2\current\mingw64\bin.

Installing ruby

Select option 2 (update msys2 installation), it will install Dev-kit including MINGW/MSYS gcc. In my case it is gcc.exe (Rev1, Built by MSYS2 project) 8.2.0 (when invoked in powershell via gcc --version)

scoop install ruby

In case you want to switch rubies you need a rubies switcher. One of the best for windows is uru. Since the package is not offered in the official buckets , due to some special needs, the uru developer offers a link for easy install:

scoop install https://bitbucket.org/jonforums/uru/downloads/uru.json

Note: You can also switch among rubies also via scoop itself. The uru application is probably easier to use for switching.

Important note on gcc versions: For Smalltalk/X compilation you need to use mingw-64-gcc not the gcc for msys2.

Correct gcc:

To check you need to run pacman -Ss gcc from within msys2 shell. You should have these installed (will install automatically when you start from scratch):

mingw64/mingw-w64-x86_64-gcc 8.2.0-1 (mingw-w64-x86_64-toolchain) [installed]
mingw64/mingw-w64-x86_64-gcc-libs 8.2.0-1 (mingw-w64-x86_64-toolchain) [installed]                               d]

You will get this information at your powershell when running gcc --version:

gcc.exe (Rev1, Built by MSYS2 project) 8.2.0

Wrong gcc:

Do not install msys gcc! It will break the Smalltalk/X compilation. This gcc packages SHOULD NOT be installed:

msys/gcc 7.3.0-3 (msys2-devel)
    The GNU Compiler Collection - C and C++ frontends

Otherwise you will get similar errors like these when compiling Smalltalk/X:

"jerror.c:112: undefined reference to `__getreent'"

In case you already installed MSYS2 gcc you need to remove it via pacman -R gcc. If you already started a compilation process for Smalltalk/X you need to remove the whole smalltalk/X build directory.

To install needed applications further (some maybe installed as dependencies, which is fine):
```
scoop install openssh innounp
scoop install mercurial
```

If you inted to use integrated Smalltalk/X Java which is build into Smalltalk/X VM (Virtual Machine) you need to:

Adding a java bucket

scoop bucket add java

Installing zulu7 (Tested and certified OpenJDK Java builds), which translates into Java 1.7 implementation:

scoop install zulu7

Adding a versions bucket (contains previous versions of programs - we need it for Ant in our case):

scoop bucket add versions

To install Ant 1.9 needed to work with Java 1.7

scoop install ant19

Since there is a c:\MSYS2 path hardcoded in Smalltalk/X, which will be fixed in future versions, you need to create a symbolic link to point you to wherever you installed the msys2 application:

You can do that in powershell 5 the following way:

New-Item -Path C:\MSYS2 -ItemType SymbolicLink -Value C:\app_scoop\apps\msys2\current

To install Evolve mercurial plugin:

Install Python 2.7 (Note: you need to have the bucket versions added):

scoop install python27

Then upgrate the pip python installer:

python -m pip install --upgrade pip

Installing Evolve:

pip install --user hg-evolve

To add it to TortoiseHG find your mercurial.ini (usually at c:\users\login) and update it with:

evolve = C:\Users\<login>\AppData\Roaming\Python\Python27\site-packages\hgext3rd\evolve

This should be everything you need to compile Smalltalk/X jv-branch as described above.

Linux

Following guides are for building 64bit Smalltalk/X on 64bit Linux (or 32bit Smalltalk/X on 32bit Linux). It is possible - of course - to build ac 32bit Smalltalk/X on 64bit Linux. In that case you need to install 32bit libraries and headers. How to do that is left as an exercise to dear reader/. Hint: on DEB-based distros you need to install pretty much same packages with :i386 suffix, on RPM-based with .i686 plus something like gcc-multilib.

Debian (and other derivatives such as Ubuntu or Mint)

Tested on Debian Stretch, both 64bit and 32bit, standard installations.

Install following packages:

sudo aptitude install ruby mercurial cvs build-essential bison flex libc6-dev gcc-multilib libx11-dev default-jdk ant libglib2.0-dev libwebkitgtk-dev libsoup2.4-dev libxext-dev unixodbc-dev libgl1-mesa-dev libfl-dev

Install and setup JDK 7 for use with stx:libjava (optional):

Smalltalk/X Java integration (stx:libjava) requires OpenJDK 6 or (better) 7. Java 8 and newer are not yet supported. However, you do not need OpenJDK 7 installed in order to build the system - Open JDK is required only when you want to run Java code within Smalltalk/X.

To install OpenJDK 7, the best is to install Zulu 7 (Azul provides .deb packages). The only issue is that Zulu packages set themselves as default for java and javac command which will cause problems since other Java packages these days requires newer Java (ant, for example). To fix that, use Debian's update-alternatives to switch back to your desired default Java version. In short:
wget https://cdn.azul.com/zulu/bin/zulu7.24.0.1-jdk7.0.191-linux_amd64.deb
sudo dpkg -i zulu7.24.0.1-jdk7.0.191-linux_amd64.deb
sudo update-alternatives --config java
sudo update-alternatives --config javac

This should be everything you need to compile Smalltalk/X jv-branch as described above

To build 32bit Smalltalk/X on 64bit system, you need to install 32bit versions of some libraries in addition to above:

dpkg --add-architecture i386
apt-get update 
apt-get install libx11-dev:i386 libxext-dev:i386 libxft2:i386 libxinerama-dev:i386 libodbc1:i386 odbcinst1debian2:i386 libgl1-mesa-dev:i386 libfl-dev:i386

CentOS 7

Tested on CentOS 7.2 (Minimal). Thanks to Patrik Svestka for this guide!

First couple steps only install X environment. If you already have it installed or don't want it, you may skip them.

First install XServer (with xfce)

sudo yum groupinstall "Development Tools" "X Window system" xfce Fonts
systemctl set-default graphical.target

Optional: For the system to have "nice" fonts

sudo yum install xorg-x11-fonts-Type1 xorg-x11-fonts-misc

Following is required to compile Smalltalk/X jv-branch

Install rvm (management for ruby - system ruby is not a good idea):

rvm requirements
rvm install ruby-2.3
rvm docs generate-ri # documentation

Install java (open-jdk 8)
```
sudo yum install java
```

unixODBC needed to start StX (installs handy ODBC connector)
```
sudo yum install unixODBC 
```

Versioning systems (for getting the sources from)
```
sudo yum install mercurial cvs
```

Devel packages needed for compilation

sudo yum install glib2 libX11-devel libXext-devel qtwebkit libsoup libXft-devel libXinerama-devel ant bison flex

This should be everything you need to compile Smalltalk/X jv-branch as described above

Optional: Install XSLT and DocBook Stylesheets. This is only required if you want to create deployment artifacts, i.e., run rake install or rake artifacts
```
sudo yum install libxslt-devel docbook5-style-xsl docbook-style-xsl
```
Since paths on CentOS differ from those on Debian, workaround this by creating symlinks:
```
cd /usr/share/xml/
mkdir -p docbook/stylesheet/
cd /usr/share/xml/docbook/stylesheet
ln -s /usr/share/sgml/docbook/xsl-stylesheets-1.78.1/ docbook-xsl
```

Fedora 23

Tested on Fedora 23 Workstation, 64bit, standard installation. Thanks to Martin Kobetic and Patrik Svestka for hints.

Install following packages:

sudo dnf install gcc java ant ruby rake mercurial cvs bison flex unixODBC glib2 libX11-devel libXext-devel qtwebkit libsoup libXft-devel libXinerama-devel ant

This should be everything you need to compile Smalltalk/X jv-branch as described above

openSUSE Tumbleweed

Tested on openSUSE Tumbleweed on June 29th, 2018. Thanks to Patrik Svestka for testing and this guide.

Install ruby via rvm:
```
rvm install ruby
```

Needed packages:

sudo zypper install mercurial fontconfig fontconfig-devel bison flex unixODBC unixODBC-devel glib2 libX11-devel libXext-devel qtwebkit libsoup libXft-devel libXinerama-devel ant

Deal with java:

The stx:libjava currently supports only java-openjdk 1.7 so you have to install it on your system to be able to work with it. This java is no longer available in the default repositories. You need to add the Java bootstrap repository in order to install Java openjdk 1.7.0.

First you need to add a Java bootstrap repository:

Java bootstrap repostitory via command (see the link for other mirrors):

sudo zypper ar -f http://mirror.karneval.cz/pub/linux/opensuse/repositories/Java:/bootstrap/openSUSE_Tumbleweed/Java:bootstrap.repo

Then you will be asked to verify the key fingerprint:

New repository or package signing key received:

  Repository:       Attic project for packages that are needed to bootstrap java without java (openSUSE_Tumbleweed)
  Key Name:         Java OBS Project <Java@build.opensuse.org>
  Key Fingerprint:  97119219 72E27C87 BBC1BA89 E38C29BC 4276E0B9
  Key Created:      Mon 18 Jun 2018 03:42:22 PM CEST
  Key Expires:      Wed 26 Aug 2020 03:42:22 PM CEST
  Rpm Name:         gpg-pubkey-4276e0b9-5b27b6be

After you have verified the fingerprint you can press t to trust it and you will add the needed repository.

Then check all the available java 1.7.0 openjdk versions with:

xxx@localhost:~/build/java7> sudo zypper search java-1_7
Loading repository data...
Reading installed packages...

S  | Name                                  | Summary                                                          | Type
---+---------------------------------------+------------------------------------------------------------------+-----------
i+ | java-1_7_0-openjdk                    | SUSE's implementation of the OpenJDK 7 runtime environment       | package
   | java-1_7_0-openjdk                    | SUSE's implementation of the OpenJDK 7 runtime environment       | srcpackage
   | java-1_7_0-openjdk-accessibility      | Accessibility connector for OpenJDK 7                            | package
   | java-1_7_0-openjdk-debuginfo          | Debug information for package java-1_7_0-openjdk                 | package
   | java-1_7_0-openjdk-debugsource        | Debug sources for package java-1_7_0-openjdk                     | package
   | java-1_7_0-openjdk-demo               | OpenJDK 7 Demos                                                  | package
   | java-1_7_0-openjdk-demo-debuginfo     | Debug information for package java-1_7_0-openjdk-demo            | package
i+ | java-1_7_0-openjdk-devel              | SUSE's implementation of the OpenJDK 7 Development Environment   | package
   | java-1_7_0-openjdk-devel-debuginfo    | Debug information for package java-1_7_0-openjdk-devel           | package
i+ | java-1_7_0-openjdk-headless           | OpenJDK 7 runtime environment without X, audio and video support | package
   | java-1_7_0-openjdk-headless-debuginfo | Debug information for package java-1_7_0-openjdk-headless        | package
   | java-1_7_0-openjdk-javadoc            | OpenJDK 7 API Documentation                                      | package
   | java-1_7_0-openjdk-src                | OpenJDK 7 Source Bundle                                          | package

You can then install the Java openjdk 1.7.0 with

sudo zypper install java-1_7_0-openjdk java-1_7_0-openjdk-devel java-1_7_0-openjdk-headless

Note: When installing these rpm(s) you will probably see a warning like this one:

java-1_7_0-openjdk-devel-1.7.0.181-2.5.x86_64.rpm:
    Header V3 DSA/SHA1 Signature, key ID 4276e0b9: NOKEY
    V3 DSA/SHA1 Signature, key ID 4276e0b9: NOKEY

Which means that you don't have the signature key with which the packages were signed installed. That is why it is important to check the Key Fingerprint when adding the repository. (If you can find the key you can add it too).

To check currently installed and activated java:

sudo /usr/sbin/update-alternatives --config java

Note: update-alternatives - maintain symbolic links determining default commands

You could get the following output:

There are 4 choices for the alternative java (providing /usr/bin/java).

Selection	Path	Priority	Status
0	/usr/lib64/jvm/jre-1.8.0-oracle/bin/java	2700	auto mode
* 1	/usr/lib64/jvm/jre-1.7.0-openjdk/bin/java	1705	manual mode
2	/usr/lib64/jvm/jre-1.8.0-openjdk/bin/java	1805	manual mode
3	/usr/lib64/jvm/jre-1.8.0-oracle/bin/java	2700	manual mode
4	/usr/lib64/jvm/jre-11-openjdk/bin/java	2105	manual mode

Press <enter> to keep the current choice[*], or type selection number:

When you activate it you will see a message:

Press <enter> to keep the current choice[*], or type selection number: 1
update-alternatives: using /usr/lib64/jvm/jre-1.7.0-openjdk/bin/java to provide /usr/bin/java (java) in manual mode

Note: An excerpt from *update-alternatives* man pages to explain the above table.

Each link group is, at any given time, in one of two modes: automatic or manual. When a group is in automatic mode, the alternatives system will automatically decide, as packages are installed and removed, whether and how to update the links. In manual mode, the alternatives system will retain the choice of the administrator and avoid changing the links (except when something is broken).

Link groups are in automatic mode when they are first introduced to the system. If the system administrator makes changes to the system's automatic settings, this will be noticed the next time update-alternatives is run on the changed link's group, and the group will automatically be switched to manual mode.

Each alternative has a priority associated with it. When a link group is in automatic mode, the alternatives pointed to by members of the group will be those which have the highest priority.

When using the --config option, update-alternatives will list all of the choices for the link group of which given name is the master alternative name. The current choice is marked with a ‘*’. You will then be prompted for your choice regarding this link group. Depending on the choice made, the link group might no longer be in auto mode. You will need to use the --auto option in order to return to the automatic mode (or you can rerun --config and select the entry marked as automatic).

After all these steps you can check if the correct java version is now active:

xxx@localhost:~>java -version

java version "1.7.0_181"
OpenJDK Runtime Environment (IcedTea 2.6.14) (build 1.7.0_181 suse-2.5-x86_64)
OpenJDK 64-Bit Server VM (build 24.181-b01, mixed mode)

xxx@localhost:~> env | grep JAVA

JAVA_ROOT=/usr/lib64/jvm/java-1.7.0
JAVA_HOME=/usr/lib64/jvm/java-1.7.0
JAVA_BINDIR=/usr/lib64/jvm/java-1.7.0/bin

If you see similar results as above then you have installed your Java OpenJDK 1.7.0 successfully.

FreeBSD

Note: FreeBSD port is still under development. As of now (2018-08-01) it runs, the IDE opens but there are still some problems with Java. It does not pass standard test suite, not yet. Stay tuned.

Install following packages using pkg:

sudo pkg install bash cvs mercurial ruby rubygem-rake flex bison openjdk-7.111.01 unixODBC apache-ant-1.9.7 pkgconf

Compiling, recompiling, updating and hacking

The whole compilation is driven by rake, a Ruby build tool. All daily tasks are implemented as rake targets and therefore can be invoked by simply running

rake <target>

in a shell (command prompt).

Remember, build scripts are also versioned and evolve (slowly, but they do), so you'd better update them regularly.

Task: checkout fresh sources and compile

To fetch all the source code use checkout target, i.e,

rake checkout

This will check all necessary sources from corresponding repositories into build subdirectory.

To compile, use - surprise - compiler target.

Task: update previously checked out sources to latest revisions

rake update

will update every repository to the most recent revision (aka HEAD or master). Then use

rake compile

to recompile. Following a custom, this would recompile only what changed (and what depends on changed sources).

To compile only one package, for example, smallsense you have to do it the following way:

rake stx:goodies/smallsense

Task: run compiled Smalltalk/X

To run compiled Smalltalk/X, go to build/stx/projects/smalltalk directory and run smalltalk (smalltalk.bat on Windows):

cd build/stx/projects/smalltalk
./smalltalk -I --quick

(-I means no snapshot restart, --quick means no search for autoloaded classes)

Task: run tests to make sure everything is fine

Once compiled (or hacked) you may want to make sure that everything is fine by running tests. As you may have guessed, use rake to run test task:

rake test

Beware that it may take a quite some time to run all the tests. Make sure you're not running on battery as it may go flat rather quickly. Ideally, all tests should pass. Once all run, a short summary is printed. You may find detailed XML report in reports directory. If not all of them pass, please report an issue!

You can also test only one package. For example:

To run only mercurial tests:

rake test:package:stx:libscm/mercurial

To run only regressiontests:

rake test:package:stx:goodies/regression

Task: TODO

To be written...

Building Smalltalk/X on Jenkins CI

Work in progress. This section may be incomplete needs to be polished.

Setting up Jenkins Server

Smalltalk/X pipeline script uses a number of API which Pipeline sandbox plugin does not allow. To allow all of them at once, edit $JENKINS_HOME/scriptApproval.xml and add following:

<approvedSignatures>
    <string>method com.cloudbees.plugins.credentials.common.IdCredentials getId</string>
    <string>method com.cloudbees.plugins.credentials.common.StandardCredentials getDescription</string>
    <string>method hudson.plugins.mercurial.MercurialSCM getBranch</string>
    <string>method hudson.plugins.mercurial.MercurialSCM getCredentialsId</string>
    <string>method hudson.plugins.mercurial.MercurialSCM getSource</string>
    <string>method java.lang.Class getName</string>
    <string>method java.lang.Object hashCode</string>
    <string>method java.lang.Class isInstance java.lang.Object</string>
    <string>method java.util.Collection remove java.lang.Object</string>
    <string>method java.util.Collection toArray</string>
    <string>new java.util.ArrayList java.util.Collection</string>
    <string>new java.util.HashMap</string>
    <string>new java.util.HashMap java.util.Map</string>
    <string>staticMethod com.cloudbees.plugins.credentials.CredentialsProvider lookupCredentials java.lang.Class</string>
    <string>staticMethod java.lang.System identityHashCode java.lang.Object</string>
    <string>staticMethod org.codehaus.groovy.runtime.DefaultGroovyMethods contains java.lang.Object[] java.lang.Object</string>
    <string>staticMethod org.codehaus.groovy.runtime.DefaultGroovyMethods plus java.util.List java.lang.Iterable</string>
    <string>staticMethod org.codehaus.groovy.runtime.DefaultGroovyMethods putAt java.lang.Object java.lang.String java.lang.Object</string>
    <string>method java.util.Calendar add int int</string>
    <string>method java.util.Calendar getTimeInMillis</string>
    <string>method java.util.Calendar set int int</string>
  </approvedSignatures>

Setting up Build Slaves

In addition to setup described above, additional packages / setup is needed when running a full CI pipeline on a build slave.

Debian (and alike)

Install following extra packages:

aptitude install xsltproc docbook-xsl

Windows

Install following extra packages:

pacman -Su libxslt docbook-xsl

IMPORTANT: make sure you run c:\msys64\msys2.exe as the jenkins user. This creates MSYS2 home directory for the user - otherwise SSH won't be able to store known_hosts file there and checkout would fail. See MSYS ssh error: Could not create directory '/home/<username>/.ssh'

NOTE: If using Windows 10, you may want to enable long paths by default. To so so, set registry key:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem\LongPathsEnabled

to 1 (as 32-bit DWORD).

Troubleshooting

When things go wrong, check this troubleshooting checklist for common problems (and solutions).

Note: See TracWiki for help on using the wiki.

Download in other formats:

Plain Text