musl - RISC-V musl port and toolchain builder

Follow @Openwall on Twitter for new release announcements and other news
[<prev] [next>] [day] [month] [year] [list]
Message-id: <8B64E4E2-F2DA-431A-9F9E-474772A99073@mac.com>
Date: Sun, 29 Oct 2017 18:14:39 +1300
From: Michael Clark <michaeljclark@....com>
To: musl@...ts.openwall.com
Subject: RISC-V musl port and toolchain builder

Hi Folks,

A quick status update on the RISC-V musl port and building musl toolchains for RISC-V.

I posted parts of this update earlier to the RISC-V software development mailing list but have since tagged a new toolchain builder with an undocumented “native-cross” feature that allows building native toolchains which are dynamically linked against the target’s musl libc using the target’s cross compiler:

- https://github.com/rv8-io/musl-riscv-toolchain/
- https://github.com/rv8-io/musl-riscv/

Some background: I picked up maintenance of a branch of musl-riscv from a GSoC project. I did this to use musl libc in a toolchain for cross-platform emulator benchmarking across 6 architectures: riscv32, riscv64, i386, x86_64, arm and aarch64 [1] [2]. The bulk of the musl-riscv porting was completed during the GSoC project however the port needed some testing and a few updates to match ABI and toolchain changes done during binutils and gcc upstreaming earlier in the year. The port is now in a mostly usable state, with the exception of ELF thread local storage (which I could use some help with). Shared libraries are working and have been tested both in a full system emulator running riscv-linux and in a user mode simulator.

I’ve tagged a pre-release of musl-riscv along with a toolchain build script that uses a snapshot of riscv-linux for installation of linux headers:

	• https://github.com/rv8-io/musl-riscv/releases/tag/v1.1.17-riscv-a5
	• https://github.com/rv8-io/musl-riscv-toolchain/releases/tag/v7.2.0-6
	• https://github.com/rv8-io/riscv-linux/releases/tag/linux-riscv-4.12-v7_0

We could switch to musl-cross-make however at the time I needed to use gcc-7.1, latest binutils 2.28 for riscv support along with snapshots of musl-riscv and linux-riscv and none of these were supported by musl-cross-make at that time and I happened to have a toolchain build script I was using privately. The other 4 architectures were added so that I could compare code gen in compiler explorer [3]. The x86 32-bit and 64-bit compilers are configured to tune for core2. The arm target is armv7-a and the aarch64 target is armv8-a. I recently added a “native-cross” feature to test native riscv gcc in user-mode simulators such as qemu-riscv and rv8. The native cross build needs some fine tuning as it currently shares the sysroot with the cross toolchain but installs untransformed native tools into the SYSROOT/usr/bin directory (where SYSROOT=PREFIX/TUPLE). I’d like to figure out how to build the native toolchains with / as the sysroot so the sysroot can be used as a chroot for use with a user-mode simulator. I haven’t yet figured out how to make the run-time sysroot differ from the build-time sysroot?

Installation

	# tested on Debian Stretch - /opt/riscv is writable
        $ sudo apt-get install -y build-essential bison flex
	$ git clone https://github.com/rv8-io/musl-riscv-toolchain.git
	$ cd musl-riscv-toolchain
	$ sh bootstrap.sh riscv64
	$ export PATH=$PATH:/opt/riscv/musl-riscv-toolchain-7.2.0-6/bin
	$ riscv64-linux-musl-gcc --version
	riscv64-linux-musl-gcc (GCC) 7.2.0
	Copyright (C) 2017 Free Software Foundation, Inc.
	This is free software; see the source for copying conditions. There is NO
	warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. 

Alliteratively you can build all supported architectures like so:

	$ time ( for i in riscv32 riscv64 i386 x86_64 arm aarch64; do sh bootstrap.sh $i ; done ) | tee build.log

Note: the bootstrap script default to -j8 and building the 6 supported architectures requires 17GB of space and will take about a couple hours on an 8 core machine.

Issues fixed since picking up GSoC musl-riscv branch:

	• gcc patch to set the musl dynamic linker name (ELF interp)
		• patch is applied automatically by the toolchain build script
		• /lib/ld-musl-riscv32.so.1 (-mabi=ilp32d, default, hard float)
		• /lib/ld-musl-riscv64.so.1 (-mabi=lp64d, default, hard float)
		• /lib/ld-musl-riscv32-sf.so.1 (-mabi=ilp32, soft float)
		• /lib/ld-musl-riscv64-sf.so.1 (-mabi=lp64, soft float)
		• /lib/ld-musl-riscv32-sp.so.1 (-mabi=ilp32f, single precision)
		• /lib/ld-musl-riscv64-sp.so.1 (-mabi=lp64f, single precision)
	• fixed failing pthread tests.
		• a_cas was deadlocking (updated a_cas in atomic_a.h, fixed missing inline asm constraint)
		• defined the minimal set of atomics required by the musl library
	• fixed failing sigaltstack tests (update sigaltstack and ucontext in signal.h)
	• fixed failing ipc_sem tests (added struct semid_ds in sem.h)
	• fixed failing stat tests (defined blksize_t and nlink_t in alltypes.h.in)
	• rename sigcontext __regs to gregs so that gcc would compile
	• rename _gp to __global_pointer$ in the crt to work with current binutils
	• change definition of long double to quadruple precision
	• update syscalls.h.in to use asm-generic syscall definitions
	• update stat.h to use asm-generic stat definition

Remaining issues:

	• rebase to current musl-libc
	• check results of tests that are expected to fail (compare with other architectures)
	• ELF thread local variables are not being initialised
		• tls_init test is failing
	• lower priority - atomics require fallback to syscall if A extension is not present - need to check latest kernel ABI

It looks like RISC-V might make it into linux 4.15 [4] I suspect we should wait until we are in mainline linux before considering adding the riscv arch directories to musl?

I would also like to know how a new port should be divided up into patches for submission. The history will need squashing as there is some churn in the history. I don’t believe there are any changes outside of arch/riscv32 and arch/riscv64 so its pretty low risk.

We’ll need to address the ELF TLS issue. I have debugged it to the point of confirming that it appears to be finding the TLS ELF segments (i see calls to __copy_tls, memcpy) but new threads are not having their thread locals initialised. tp is being set correctly otherwise pthread_self and the other thread tests would not be working. I have an additional patch to set DTP_OFFSET to use the same value as glibc but that doesn’t appear to help. It needs more time with a debugger to find out why the thread initialisation code is not copying initial state for thread locals from .tdata

--- a/arch/riscv64/pthread_arch.h
+++ b/arch/riscv64/pthread_arch.h
@@ -8,4 +8,6 @@ static inline struct pthread *__pthread_self()
 #define TLS_ABOVE_TP
 #define TP_ADJ(p) ((char *)p + sizeof(struct pthread) - 16)
 
+#define DTP_OFFSET 0x800
+
 #define MC_PC gregs[0]


Michael.

[1] https://carrv.github.io/papers/clark-rv8-carrv2017.pdf
[2] https://rv8.io/bench
[3] https://cx.rv8.io/
[4] https://www.phoronix.com/scan.php?page=news_item&px=RISC-V-Linux-V9
Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.