id,summary,reporter,owner,description,type,status,priority,milestone,component,resolution,keywords,cc,cvshead 107,Posssible memory overwrite when extending oldspace,jan vrany,,"When an oldspace is extended (i.e., made larger), it may overflow to a memory area used for other housekeeping structures (such as global list, shadow lists, remsets and so on). Naturally, this causes all sorts of crashes due to memory corruption (housekeeping structures are overwritten by object data). ---- The problem is that oldspace (and few other spaces) are `mmap()`-ed and some housekeeping structures are `malloc()`-ed. Smalltalk/X VM does not support segmented oldspace (too bad!) so the only way to make it larger is to `mmap()` couple more pages at the end of current oldspace. To do so, the current (broken) code uses `MAP_FIXED`. This is very bad, because (from `mmap()` documentation): > Don't interpret addr as a hint: place the mapping at exactly that address. addr must be a multiple of the page size. If the memory region specified by addr and len overlaps pages of any existing mapping(s), then the overlapped part of the existing mapping(s) will be discarded. If the specified address cannot be used, mmap() will fail. Because requiring a fixed address for a mapping is less portable, the use of this option is discouraged. So, if by chance happens that housekeeping structures are `malloc()`-ed in the area behind current old end, extending `mmap()` succeeds and oldspace would overlap with `malloc()`-ed data (and Smalltalk will eventually crash). Never, ever, use `MAP_FIXED`! I wonder how it could ever work. A proper fix would require a major rewrite of Smalltalk/X memory manager support segmented oldspace. Several workarounds are possible, each with its own issues. Sigh. This is less of a problem on 64bit as there's lot more space than actual memory, but still...",defect,closed,critical,8.0.0,virtual machine,fixed,,,1