Posssible memory overwrite when extending oldspace

[jan vrany]

When an oldspace is extended (i.e., made larger), it may overflow to a memory area used for other housekeeping structures (such as global list, shadow lists, remsets and so on). Naturally, this causes all sorts of crashes due to memory corruption (housekeeping structures are overwritten by object data).

---

The problem is that oldspace (and few other spaces) are mmap()-ed and some housekeeping structures are malloc()-ed. Smalltalk/X VM does not support segmented oldspace (too bad!) so the only way to make it larger is to mmap() couple more pages at the end of current oldspace. To do so, the current (broken) code uses MAP_FIXED. This is very bad, because (from mmap() documentation):

Don't interpret addr as a hint: place the mapping at exactly that address. addr must be a multiple of the page size. If the memory region specified by addr and len overlaps pages of any existing mapping(s), then the overlapped part of the existing mapping(s) will be discarded. If the specified address cannot be used, mmap() will fail. Because requiring a fixed address for a mapping is less portable, the use of this option is discouraged.

So, if by chance happens that housekeeping structures are malloc()-ed in the area behind current old end, extending mmap() succeeds and oldspace would overlap with malloc()-ed data (and Smalltalk will eventually crash). Never, ever, use MAP_FIXED!

I wonder how it could ever work.

A proper fix would require a major rewrite of Smalltalk/X memory manager support segmented oldspace. Several workarounds are possible, each with its own issues. Sigh.

This is less of a problem on 64bit as there's lot more space than actual memory, but still...

[jan vrany]

To address this, at least partially, various measures have been implemented:

• As a prerequisite, a completely new set of functions to allocate memory and manage from OS have been introduced (see hmm.h and hmm.c). A code in memory manager has been cleaned up and now uses functions provided. This in itself resulted in deletion of some 3000 (!) lines of code.
• The oldspace compaction now uses 2-phase, in-place compaction - the old baker-style compaction has been removed. The problem with it was that it required twice as much memory in code (imagine compacting 32GB heap) and - once allocated - hindered oldspace growth.
• The general problem with oldspace growth is that some other data might be mapped just beyond the current oldspace end so oldspace cannot be extended. To make sure that we can grow, VM now reserves "enough" memory, but does not commit it. The reserved memory get commited only if there's need for it, i.e., when there are objects to put there. "Enough memory" means 32GB on 64bit VM and 1GB on 32bit (except on Windows that - due to high fragmentation of address space - gives away as much as 384MB).

While this strictly speaking does not make sure an oldspace can grow when there's free memory, it's make it more likely that it can grow (to certain limit).

However, the new code makes sure no memory is overwritten (no MAP_FIXED).

Support for proper segmented old space is still desirable but it's left as future work.