StX 8.0.0 crash when using workspace

[Patrik Svestka]

Don't know how to reproduce but I did the following steps:

1) from the main window open workspace
2) type Smalltalk version (with the auto-raise feature on)
3) with arrow keys scroll- and -down

I'm attaching the complete log file for the session.

[Patrik Svestka]

Log file for crash session

[Patrik Svestka]

My settings file

[Jan Vrany]

It seems that endless recursion causes VM to terminate, A minimal code that causes this is:

| b | 
b := [ b value ]. 
b value.

This is kind of understandable - if a thread exhausts its stack and does not handle the exception properly, OS has no other option than kill the offending process. However, Smalltalk/X should handle this - terminating the VM this way without notice is not acceptable.

Indeed why Smallsense enters an endless recursion is another matter (and will be addressed later after VM survives it)

[Jan Vrany]

Two particular problems I found when chasing this one are connected to the fact than when a debugger is opened because of recursion interrupt, opening a debugger requires a lot of stack,
something that's very limited in this very situation (remember, recursion is "detected" by the fast the stack is almost exhausted).

The above two patches address two cases where too much stack is used. Please, try them and let me know if it makes things any better. On my system, this helps with the "minimal" testcase mentioned above.

[Patrik Svestka]

Unfortunatelly the patches did not help much. The VM still crashes hard.

[Jan Vrany]

So, it seems that the problem is (was) that the debugger (which sits on the top of debugee's stack) has enough space. The "yellow" zone is not 256k on 64bit system, quite a lot...

Please, revert both previous patches and test it with new librun from test ​build #158. What you should see is a debugger showing a recursion interrupt. But the VM should no longer crash (does not on my system).

If it does not crash, then apply this SmallSense patch​ which should prevent the endless recursion.

Please, let me know this helps.

[Patrik Svestka]

I have applied both. It works! Great job!

[Jan Vrany]

The test fails on Linux, i386.

[Jan Vrany]

To sum it up, the problem was two-fold:

1. The yellow stack zone was too small on 64bit Windows to accommodate the debugger's code (since St/X debugger sits on top of the debugee). So, the debugger itself stepped into an "orange" zone, causing the process to terminate. This has been addressed in commits:
   • 90d2931db6cd/stx.librun - Win32: initialize process's stackTop based in data in Window's TEB
   • 2777c921a299/stx.librun - Issue #186: refactored, unified and documented stack safe area sizes.

2. Second problem was that the inlined version of Block >> value did not check of (recursion) interrupts so the process continued to recurse until it stepped onto "orange" zone and, again, has been terminated. This happened only in synthetic testcases. This has been addressed in commits:
   • 277cc0b28c2a/stx.librun - Issue #186: check for interrupts in inlined version of Block >> value
   • 139960138901/stx.stc - Issue #186: check for interrupts in inlined version of Block >> value

[Jan Vrany]

Actually, there was third problem:

3. The __ySTACK_HANDLER__() routine on Windows/x86_64 broke the ABI requirement of stack alignment to 16bytes. This caused later a protection violation MCVCRT's _setjmp() because it saves XMM registers using movdqa %xmm1, 0x50(%rcx) which requires memory operand to be 16bytes aligned (which it was not, due to a misalign in __ySTACK_HANDLER__(). Fixed in commit af2e53e3ada5/stx.librun.