December 23, 2009

Unbreaking Mono on FreeBSD 6.4

Categories: FreeBSD, Mono.

A few days ago ports/140916: lang/mono ( installation fails was opened. The reported problem had already been reported a few time but with insufficient feedback so far and I was not able to diagnose the problem, not mentioning providing a fix. But this time, the reporter could spot that the problem was happening for him only with FreeBSD-6.4. I did the test on a fresh FreeBSD-6.4 install and could trigger the error:

arthur# portsnap fetch
arthur# portsnap extract
arthur# cd /usr/ports/lang/mono
arthur# make -V PKGNAME
arthur# make
if test -w ../mcs; then :; else chmod -R +w ../mcs; fi
cd ../mcs && gmake NO_DIR_CHECK=1 PROFILES='net_1_1 net_2_0 net_3_5 net_2_1' CC='cc' all-profiles
gmake[3]: Entering directory `/usr/ports/lang/mono/work/mono-'
gmake profile-do--net_1_1--all profile-do--net_2_0--all profile-do--net_3_5--all profile-do--net_2_1--all
gmake[4]: Entering directory `/usr/ports/lang/mono/work/mono-'
gmake PROFILE=basic all
gmake[5]: Entering directory `/usr/ports/lang/mono/work/mono-'
gmake[6]: *** [build/deps/basic-profile-check.exe] Error 1
gmake[6]: Entering directory `/usr/ports/lang/mono/work/mono-'
*** The compiler 'false' doesn't appear to be usable.
*** Trying the 'monolite' directory.
gmake[7]: Entering directory `/usr/ports/lang/mono/work/mono-'
gmake[8]: *** [build/deps/basic-profile-check.exe] Error 138
gmake[8]: Entering directory `/usr/ports/lang/mono/work/mono-'
*** The contents of your 'monolite' directory may be out-of-date
*** You may want to try 'make get-monolite-latest'
gmake[8]: *** [do-profile-check-monolite] Error 1
gmake[8]: Leaving directory `/usr/ports/lang/mono/work/mono-'
gmake[7]: *** [do-profile-check] Error 2
gmake[7]: Leaving directory `/usr/ports/lang/mono/work/mono-'
gmake[6]: *** [do-profile-check-monolite] Error 2
gmake[6]: Leaving directory `/usr/ports/lang/mono/work/mono-'
gmake[5]: *** [do-profile-check] Error 2
gmake[5]: Leaving directory `/usr/ports/lang/mono/work/mono-'
gmake[4]: *** [profile-do--basic--all] Error 2
gmake[4]: Leaving directory `/usr/ports/lang/mono/work/mono-'
gmake[3]: *** [profiles-do--all] Error 2
gmake[3]: Leaving directory `/usr/ports/lang/mono/work/mono-'
gmake[2]: *** [all-local] Error 2
gmake[2]: Leaving directory `/usr/ports/lang/mono/work/mono-'
gmake[1]: *** [all-recursive] Error 1
gmake[1]: Leaving directory `/usr/ports/lang/mono/work/mono-'
gmake: *** [all] Error 2
*** Error code 1

Stop in /usr/ports/lang/mono.
*** Error code 1

Stop in /usr/ports/lang/mono.

A first look leads us on a wrong way (lane?): *** The compiler 'false' doesn't appear to be usable. This message is actually not an error message: the mono C# compiler (mcs(1)) is written in C# so some bootstrapping may be required if no version of mono is present on the system. The mono port adds EXTERNAL_MCS=false to MAKE_ARGS so that the mono port will be build exactly the same way regardless of an available mcs(1) in the path.

Hopefully, I tested on an old machine and could see that something was running for a couple seconds before failing. Ctrl+T in tcsh(1) told me more:

load: 0.08  cmd: mono 10910 [ksesigwait] 0.13u 0.10s 0% 24492k
load: 0.08  cmd: mono 10910 [ksesigwait] 0.13u 0.10s 0% 24964k

The good news is that some mono application is running. But wait, what is this ksesigwait state? A first look at the FreeBSD Kernel Cross Reference reveals that this state is defined in kern/kern_kse.c; the man page is kse(2); Wikipedia states KSEs were mandatory at introduction; made optional at kernel build time in the 7.0 release and removed from the 8.0 release with a compatibility library.

The second good news is that a 22MB mono.core file is written in the mcs directory.

arthur# gdb -q /usr/ports/lang/mono/work/mono- mono.core
Core was generated by `mono'.
Program terminated with signal 10, Bus error.
Reading symbols from /usr/local/lib/
Loaded symbols for /usr/local/lib/
Reading symbols from /usr/local/lib/
Loaded symbols for /usr/local/lib/
Reading symbols from /usr/local/lib/
Loaded symbols for /usr/local/lib/
Reading symbols from /usr/local/lib/
Loaded symbols for /usr/local/lib/
Reading symbols from /usr/local/lib/
Loaded symbols for /usr/local/lib/
Reading symbols from /lib/
Loaded symbols for /lib/
Reading symbols from /lib/
Loaded symbols for /lib/
Reading symbols from /lib/
Loaded symbols for /lib/
Reading symbols from /libexec/
Loaded symbols for /libexec/
#0  0x2856bf2a in signalcontext () from /lib/
[New Thread 0x8319c00 (sleeping)]
[New Thread 0x8319800 (sleeping)]
[New Thread 0x8319600 (LWP 100098)]
[New Thread 0x82fd000 (runnable)]
[New LWP 100058]
(gdb) thread apply all bt

Thread 5 (LWP 100058):
#0  0x2856bf2a in signalcontext () from /lib/
#1  0x28527fc4 in pthread_mutexattr_init () from /lib/
#2  0x28303480 in ?? ()

Thread 4 (Thread 0x82fd000 (runnable)):
#0  0x285eb0ac in __vfprintf () from /lib/
#1  0x2855ff90 in vasprintf () from /lib/
#33814 0x081464b8 in mono_runtime_invoke (method=0x82fd81c, obj=0x0, params=0xbfbfe510, exc=0x0) at object.c:2401
#33815 0x0814785b in mono_runtime_exec_main (method=0x82fd81c, args=0x28657f60, exc=0x0) at object.c:3301
#33816 0x0814736c in mono_runtime_run_main (method=0x82fd81c, argc=2, argv=0xbfbfe74c, exc=0x0) at object.c:3089
#33817 0x080cdf7d in mono_jit_exec (domain=0x28653ee0, assembly=0x833c880, argc=3, argv=0xbfbfe748) at driver.c:924
#33818 0x080ce0e8 in main_thread_handler (user_data=0xbfbfe690) at driver.c:972
#33819 0x080cfc3a in mono_main (argc=6, argv=0xbfbfe73c) at driver.c:1647
#33820 0x08058f90 in main (argc=6, argv=0xbfbfe73c) at main.c:34

Thread 3 (Thread 0x8319600 (LWP 100098)):
#0  0x2852f5f3 in pthread_testcancel () from /lib/
#1  0x28527fc4 in pthread_mutexattr_init () from /lib/
#2  0x28303480 in ?? ()

Thread 2 (Thread 0x8319800 (sleeping)):
#0  0x28528097 in pthread_mutexattr_init () from /lib/
#1  0x28521ade in _nanosleep () from /lib/
#2  0x28521c42 in nanosleep () from /lib/
#3  0x081e32ca in collection_thread (unused=0x0) at collection.c:34
#4  0x28520449 in pthread_create () from /lib/
#5  0x285deecb in _ctx_start () from /lib/

Thread 1 (Thread 0x8319c00 (sleeping)):
#0  0x28528097 in pthread_mutexattr_init () from /lib/
#1  0x2852822b in pthread_mutexattr_init () from /lib/
#2  0x2852c839 in _pthread_cond_wait () from /lib/
#3  0x2852cd82 in pthread_cond_wait () from /lib/
#4  0x081e86d8 in _wapi_handle_timedwait_signal_handle (handle=0x1d05, timeout=0x0, alertable=0, poll=0) at handles.c:1605
#5  0x081e846e in _wapi_handle_wait_signal_handle (handle=0x1d05, alertable=0) at handles.c:1548
#6  0x08205f2e in WaitForSingleObjectEx (handle=0x1d05, timeout=4294967295, alertable=0) at wait.c:205
#7  0x081608a0 in finalizer_thread (unused=0x0) at gc.c:1061
#8  0x081806f0 in start_wrapper (data=0x8326380) at threads.c:623
#9  0x08200367 in thread_start_routine (args=0x832c230) at threads.c:286
#10 0x08223adf in GC_start_routine (arg=0x2865aec0) at pthread_support.c:1382
#11 0x28520449 in pthread_create () from /lib/
#12 0x285deecb in _ctx_start () from /lib/
#0  0x2852f5f3 in pthread_testcancel () from /lib/

Yes, you read it well, the thread 4 backtrace is 33820 frame long! Looks like we have a stack overflow... Since the problem occurs on FreeBSD 6 (where KSE is enabled by default) and not on FreeBSD 7 (where KSE is disabled by default) nor FreeBSD 8 (No KSE at all), it is likely to be related to the thread implementation.

Let's launched one more time gmake(1) and hit Ctrl+Z to suspend all when mono was filling-in it's stack in order to have some hints about the exact command being executed. Reducing it to a simple test is trivial, e.g.:

arthur# pwd
arthur# env MONO_PATH="$PWD/class/lib/monolite" ../mono/mini/mono --config ../runtime/etc/mono/config ./class/lib/monolite/mcs.exe
Bus error (core dumped)

Now we know what is being run. Let's get some info about this binary file:

arthur# ldd ../mono/mini/mono
../mono/mini/mono: => /usr/local/lib/ (0x28304000) => /usr/local/lib/ (0x28309000) => /usr/local/lib/ (0x283c9000) => /usr/local/lib/ (0x283d2000) => /usr/local/lib/ (0x284c7000) => /lib/ (0x284fc000) => /lib/ (0x28512000) => /lib/ (0x28537000)

pthread(3) list all thread libraries available on the system:

On my FreeBSD 8.0-STABLE machine, ldd(1) reports (mono-2.6 instead of 2.4):

marvin# ldd `which mono`
/usr/local/bin/mono: => /usr/local/lib/ (0x8008b4000) => /usr/local/lib/ (0x8009b8000) => /usr/local/lib/ (0x800b74000) => /usr/local/lib/ (0x800dca000) => /usr/local/lib/ (0x800ed3000) => /usr/local/lib/ (0x8010cd000) => /lib/ (0x8011fd000) => /lib/ (0x80131c000) => /lib/ (0x801434000) => /usr/local/lib/ (0x80166e000) => /usr/local/lib/ (0x80189f000) => /usr/lib/ (0x80248e000) => /lib/ (0x802699000)

So let's try to use libthr on FreeBSD 6. This can be achieved using libmap.conf(5): instead of /lib/ we want to use /usr/lib/

arthur# cat >> /etc/libmap.conf << EOT


arthur# make
arthur# make tests
363 test(s) passed. 0 test(s) did not pass.
arthur# echo $?


No Comments Yet

Comments RSS feed | Leave a Reply…