Pthreads

Dank's Sagelike Wisdom Concerning POSIX Threads

• The static initializers are unsung heroes. Learn them.
• If you are using asynchronous cancellation, you have a bug.
• If you are using pthread_attr_setstackaddr(), and you don't know why that's funny, you have a bug.
• Use of sched_yield() is almost certainly a bug, and furthermore it is womanly in nature.
  • This logic is trivially extended to pthread_yield().
• The Triforce of POSIX Thread stacks:
  • If you aren't allocating a stack fit to your thread, you will one day smash that stack.
  • If you're using your stack, you should probably heap-allocate or use TLS.
  • Recursion's all fun and games until your recurse on through to the other side.
  • Yes, this all applies to an unthreaded program (especially an embedded one), with a critical difference -- memory management is process-based, causing noisy failures in the unthreaded case. Meanwhile, a POSIX thread will happily push a frame down right over his buddy's DTV/TLS in NPTL (see NPTL, "Memory Allocation"). Have a good time debugging that, champ!
• Failure to check every return, especially pthread_mutex_destroy(), is cause for many many lashes of electric wire.
• Those who fail to designate an alternate signal stack will one day wish they had.
• Solaris 2.5's lack of timeslicing among compute-bound user threads can be addressed in two ways:
  • Setting the concurrency level to the number of compute-bound user threads + 1.
    • Surely this is not the Tao of Programming.
  • Spawning no more than one compute-bound user thread per CPU, and providing preemption.
    • Surely this is the Tao of Programming.
• Those who say, "I need never spawn thread one on Solaris 2.5" would trade their birthright for cold pottage.
• Before trying to solve an issue with POSIX thread scheduling parameters, see if you can't just drink some hydrochloric acid.

Compiling code with pthreads

First off, POSIX requires _REENTRANT be defined in the scope of all code run in a multithreaded manner. This is most typically achieved via -D_REENTRANT on the gcc command line. Some gcc versions / target architectures provide a -pthread option which sets -D_REENTRANT (as well as setting up pthread linking), as evidenced below:

Date: Sun, 8 Apr 2007 23:33:23 -0400
From: nick black <nick_black@securecomputing.com>
To: Sven Krasser <skrasser@securecomputing.com>
Cc: Nick Black <nblack@securecomputing.com>
Subject: Re: [repper-304] REENTRANT is required by POSIX for any code using pthreads

Sven Krasser rigorously showed:
> This is on Linux:
> skrasser@ctapd01:~$ cat test.c
> #ifdef _REENTRANT
> #error xxx
> #endif
> skrasser@ctapd01:~$ gcc -pthread test.c
> test.c:2:2: error: #error xxx

interesting. from the gcc pinfo page, "Index of Options":

* pthread <1>:                           SPARC Options.      (line  240)
* pthread <2>:                           RS/6000 and PowerPC Options.
                                                             (line  653)
* pthread:                               IA-64 Options.      (line  106)
* pthreads:                              SPARC Options.      (line  234)

how odd. i must investigate. in any case, yes, -pthread is definitely
defining _REEENTRANT here:

[diaconicon](0) $ touch t.c
[diaconicon](0) $ cpp -dM -pthread t.c > e 2>&1
[diaconicon](1) $ cpp -dM t.c > f 2>&1
[diaconicon](1) $ diff -ur f e
--- f   2007-04-08 23:31:12.000000000 -0400
+++ e   2007-04-08 23:31:05.000000000 -0400
@@ -26,6 +27,7 @@
 #define __DECIMAL_DIG__ 21
 #define __gnu_linux__ 1
 #define __LDBL_HAS_QUIET_NAN__ 1
+#define _REENTRANT 1
 #define __GNUC__ 4
 #define __DBL_MAX__ 1.7976931348623157e+308
 #define __DBL_HAS_INFINITY__ 1
[diaconicon](1) $

interesting. no matter what, you definitely need to be declaring
-D_THREAD_SAFE, though, as my previous example asserted.

off to dig through gcc source....

Nonetheless, this is a poorly-documented gcc-specific method; best to always provide -D_REENTRANT explicitly. -D_REENTRANT and other preprocessor directives which affect code selection should always precede -include options to gcc, ie:

gcc -D_REENTRANT -I/usr/local -include pthread.h ...

On FreeBSD's libc (and also possibly with regard to other third-part libraries), it's also necessary to define _THREAD_SAFE, as evidenced below (furthermore, you need _POSIX_PTHREAD_SEMANTICS and _P1003_1B_VISIBLE for reasons I determined long ago and have since forgotten, see the make snippet below --dank):

Newsgroups: sys.research.subversion.repper
From: Nick Black <nblack@securecomputing.com>
Subject: Re: [repper-304] REENTRANT is required by POSIX for any code using pthr
On 2007-04-08, Sven Krasser <skrasser@securecomputing.com> wrote:
> Hmm, doesn't -pthread do that?
> -pthread
>    Adds support for multithreading with the pthreads library.  This
>    option sets flags for both the preprocessor and linker.

where do you see this? from the gcc docs on freebsd 4.10-p24 (source:
/usr/src/contrib/gcc/gcc.1):

-pthread Link a user-threaded process against libc_r instead of libc.
         Objects linked into user-threaded processes  should  be
         compiled with -D_THREAD_SAFE.

[newdhcpbox](0) $ cat > g.c
#include <stdio.h>

feof(stderr)
[newdhcpbox](0) $ gcc -pthread -D_THREAD_SAFE -E g.c | grep stderr
extern FILE *__stdinp, *__stdoutp, *__stderrp;
feof((__stderrp) )
[newdhcpbox](0) $ gcc -pthread -E g.c | grep stderr
extern FILE *__stdinp, *__stdoutp, *__stderrp;
(((   (__stderrp)    )->_flags & 0x0020 ) != 0)
[newdhcpbox](0) $ grep -C3 feof /usr/include/stdio.h
__BEGIN_DECLS
void     clearerr __P((FILE *));
int      fclose __P((FILE *));
int      feof __P((FILE *));
int      ferror __P((FILE *));
int      fflush __P((FILE *));
int      fgetc __P((FILE *));
--
                (*(p)->_p = (c), (int)*(p)->_p++))
#endif

#define __sfeof(p)      (((p)->_flags & __SEOF) != 0)
#define __sferror(p)    (((p)->_flags & __SERR) != 0)
#define __sclearerr(p)  ((void)((p)->_flags &= ~(__SERR|__SEOF)))
#define __sfileno(p)    ((p)->_file)
#define __sfileno(p)    ((p)->_file)
--
 * See ISO/IEC 9945-1 ANSI/IEEE Std 1003.1 Second Edition 1996-07-12
 * B.8.2.7 for the rationale behind the *_unlocked() macros.
 */
#define feof_unlocked(p)        __sfeof(p)
#define ferror_unlocked(p)      __sferror(p)
#define clearerr_unlocked(p)    __sclearerr(p)

--
#endif

#ifndef  _THREAD_SAFE
#define feof(p)         feof_unlocked(p)
#define ferror(p)       ferror_unlocked(p)
#define clearerr(p)     clearerr_unlocked(p)

[newdhcpbox](0) $

--
nick black          "np:  the class of dashed hopes and idle dreams."

Failure to properly define _REENTRANT (and, where applicable, _THREAD_SAFE) will result in silent inclusions of unsafe, improperly-locked code and global variables in the place of thread-local data (ala errno(3)).

Linking objects with pthreads

On linux, this is as simple as things get: add -lpthread to the linker options. There's only two major pthread libraries on Linux, both of them affiliated with glibc -- LinuxThreads and then NPTL. NPTL is used on all recent distributions. -lpthread will link the primary pthreads implementation on your build machine.

On FreeBSD, the situation is a bit more complex, due to multiple pthread implementations, some of which are only available on recent FreeBSD versions. FreeBSD 4 provides the simplest case, with only two implementations:

• native BSD threads
• the LinuxThreads port (devel/linuxthreads; this does NOT require emulators/linux-base*)

all threaded code muse use the reentrant version of the gcc builtin libs; this is done via -lgcc_r (-pthread should set this up). In addition, use -llthread to link against LinuxThreads.