From 5afdca0087dad2994ad4fcdfe7f489f4dbcab7b3 Mon Sep 17 00:00:00 2001 From: Ulrich Drepper Date: Wed, 11 Mar 1998 12:42:25 +0000 Subject: LinuxThreads library. 1998-03-11 00:42 Wolfram Gloger * linuxthreads/manager.c: Enable resetting of the thread scheduling policy to SCHED_OTHER when the parent thread has a different one. 1998-02-01 13:51 Ulrich Drepper * sysdeps/unix/sysv/linux/bits/posix_opt.h: Define _POSIX_ASYNCHRONOUS_IO. * sysdeps/pthread/pthread.h: Define bits for Unix98 variants of mutexes. * mutex.c: Implement new mutex types. * internals.h: Include . * libpthread.map: Add __erno_location and __h_errno_location. * errno.c: Return pointer to variable actually in use. This might not be the one in the thread structure. * internals.h (struct _pthread_descr_struct): Add new fields p_errnop and p_h_errnop. * manager.c (__pthread_manager): Set p_errnop and p_h_errnop member of manager thread structure. (pthread_handle_create): Set p_errnop and p_h_errnop members for new thread. * pthread.c: Adapt initializer for thread structures. (__pthread_initial_thread): Set p_errnop and p_h_errnop member. (__pthread_reset_main_thread): Reset p_errnop and p_h_errnop of current thread to global variables. 1998-01-31 17:27 Ulrich Drepper * rwlock.c: New file. * Makefile (libpthread-routines): Add rwlock. * sysdeps/pthread/pthread.h: Define data structures and declare functions. * libpthread.map: Add new functions. 1997-12-18 13:50 Philip Blundell * sysdeps/arm/pt-machine.h: New file; add ARM support. * sysdeps/arm/Implies: likewise. * README: Document it. 1997-12-13 Andreas Schwab * signals.c: Remove unneeded initializer for sigwaited, saving a 1997-04-11 01:18 Andreas Schwab * semaphore.c (sem_init): Set sem_spinlock only if available. 1997-12-04 01:48 Ulrich Drepper * mutex.c: Implement PTHREAD_MUTEX_CHECKERROR. * sysdeps/pthread/pthread.h: Define PTHREAD_MUTEX_CHECKERROR. * Makefile: Update from LinuxThreads 0.7. * internals.h. Likewise. * manager.c: Likewise. * mutex.c: Likewise. * pthread.c: Likewise. * signals.c: Likewise. * specific.c: Likewise. * Examples/ex3.c: Likewise. 1997-11-20 18:13 Ulrich Drepper * pthread.c (__pthread_reset_main_thread): Close pipe only if still open. 1997-10-29 05:38 Ulrich Drepper * wrapsyscall.c: Add socket functions which are also cancelation points. 1997-10-19 21:40 Wolfram Gloger * specific.c (__libc_internal_tsd_set, __libc_internal_tsd_get): New functions for fast thread specific data within libc. * internals.h: Add new array p_libc_specific to struct _pthread_descr_struct. * sysdeps/pthread/bits/libc-lock.h: Declare new functions. 1997-10-13 05:39 Ulrich Drepper * semaphore.h: Add __BEGIN_DECLS/__END_DECLS. Reported by Ralf Corsepius . 1997-08-29 03:05 Ulrich Drepper * internals.h (struct _pthread_descr_struct): Add definitions for two-level specific key handling. * manager.c (pthread_handle_create): Initialize specific memory array. * specific.c: Implement two-level key handling. * weaks.c: Don't provide dummy key handling. * sysdeps/pthread/bits/libc-lock.h: Typedef __libc_lock_t (no #define). Add definition of __libc_key_t. * sysdeps/unix/sysv/linux/bits/local_lim.h: Define PTHREAD_KEYS_MAX as 1024. Add definition of _POSIX_THREAD_DESTRUCTOR_ITERATIONS and PTHREAD_DESTRUCTOR_ITERATIONS. * manager.c (pthread_handle_create): Compare mmap result with MAP_FAILED. * ptfork.c: Rename to __pthread_atfork and make old name a weak alias. * sysdeps/pthread/bits/pthread.h: Add prototype for __pthread_atfork. 1997-08-22 19:04 Richard Henderson sysdeps/sparc -> sysdeps/sparc/sparc32 sysdeps/sparc64 -> sysdeps/sparc/sparc64 * internals.h: Change definition of THREAD_SELF to be an expression, not a statement that did a return. * sysdeps/alpha/pt-machine.h (THREAD_SELF): Update accordingly. * sysdeps/sparc/sparc32/pt-machine.h (THREAD_SELF, INIT_THREAD_SELF): Follow Solaris and use a "system reserved" register (%g6) to hold the thread descriptor. * sysdeps/sparc/sparc64/pt-machine.h: Likewise. 1997-08-03 00:09 Ulrich Drepper * mutex.c: Correct pthread_once. Patch by Xavier Leroy. * sysdeps/pthread/pthread.h: Add prototype for __pthread_once. * sysdeps/pthread/bits/pthread.h: Add macros for __libc_once. * semaphore.c: Include spinlock.h only when needed. * specific.c (__pthread_setsepcific, __pthread_getspecific): Reject keys for entries not in use. * weaks.c: Implement key handling functions for real. 1997-06-29 01:04 Richard Henderson Initial sparc64-linux support: * linuxthreads/sysdeps/sparc64/Implies: New file. * linuxthreads/sysdeps/sparc64/pt-machine.h: Likewise. 1997-06-29 00:48 Ulrich Drepper * semaphore.c: Include spinlock.h at correct place. Patch by HJ Lu. 1997-06-13 10:06 Richard Henderson The Great Bit File Move: * sysdeps/alpha/semaphorebits.h: -> .../bits/semaphore.h. * sysdeps/powerpc/semaphorebits.h: Likewise. * sysdeps/pthread/cmpxchg/semaphorebits.h: Likewise. * sysdeps/pthread/no-cmpxchg/semaphorebits.h: Likewise. * sysdeps/pthread/libc-lock.h: -> bits/ * sysdeps/pthread/stdio-lock.h: Likewise. * sysdeps/unix/sysv/linux/local_lim.h: Likewise. * sysdeps/unix/sysv/linux/posix_opt.h: Likewise. * semaphore.h: Likewise. * sysdeps/pthread/pthread.h: Likewise. * lockfile.c: -> . * semaphore.h: Likewise. * Makefile: (headers): foo.h -> bits/foo.h. * sysdeps/pthread/Makefile: Likewise. 1997-04-11 01:18 Andreas Schwab * semaphore.c (sem_init): Set sem_spinlock only if available. * sysdeps/m68k/pt-machine.h (testandset, __compare_and_swap): Fix asm constraints. 1997-04-09 03:00 Ulrich Drepper Update from LinuxThreads 0.6. * attr.c (pthread_attr_getdetachstate): Use __sched_get_priority_max and __sched_get_priority_min instead of names without `__'. * manager.c: Rewrite large parts to implement opaque pthread_t. * cancel.c: Adapt for opaque pthread_t type. * condvar.c: Likewise. * errno.c: Likewise. * join.c: Likewise. * mutex.c: Likewise. * pthread.c: Likewise. * signals.c: Likewise. * specific.c: Likewise. * restart.h: Likewise. * queue.h: Likewise. * Examples/ex3.c: Likewise. * Examples/ex4.c: Likewise. * sysdeps/pthread/pthread.h: Likewise. * pthread.c: Accumulate time for all threads in thread manager. * semaphore.c: Implement fallback implementation for architectures sometimes missing compare-exchange operations. * cancel.c (pthread_cancel): Validate handle argument. * join.c (pthread_join): Likewise. (pthread_detach): Likewise. * signals.c (pthread_kill): Likewise. * spinlock.h (acquire): Use __sched_yield not sched_yield. * queue.h (enqueue): Enqueue thread according to priority. * internals.c (struct pthread_start_args): New struct for passing args to cloning function. (struct _pthread): Rename to _pthread_descr_struct and adapt for opaque pthread_t. * Examples/Makefile (clean): Pass -f option to rm. * sysdeps/i386/pt-machine.h: Add check for compare-exchange instruction and define TEST_FOR_COMPARE_AND_SWAP. * sysdeps/i386/i486/pt-machine.h: Removed. * sysdeps/unix/sysv/linux/local_lim.h (PTHREAD_THREADS_MAX): Increase to 1024. 1997-04-04 16:38 Ulrich Drepper * restart.h (suspend): Clear p_signal before suspending. (suspend_with_cancellation): Likewise. Patch by Xavier Leroy . * weaks.c: Make __pthread_key_create return 1. * sysdeps/pthread/libc-lock.h: Define __libc_key_create, __libc_getspecific, __libc_setspecific, and __libc_key_t. * sysdeps/pthread/stdio-lock.h: Don't care for implementation not using libio. 1997-03-19 15:13 Miguel de Icaza * sysdeps/sparc/pt-machine (RELEASE): Fix. 1997-03-01 07:55 Geoff Keating * sysdeps/powerpc/Implies: Added. * sysdeps/powerpc/pt-machine.h: Added. * sysdeps/powerpc/semaphorebits.h: Added. 1997-01-22 01:22 Ulrich Drepper * linuxtheads/pthread.c (__pthread_initial_thread): Correct initializer. (__pthread_manager_thread): Likewise. Reported by Andreas Jaeger. 1997-01-18 22:15 Richard Henderson Since sigset_t no longer fits in a register, we can't pass in the thread's initial mask so easily. Take this opportunity to simplify the clone implementation by only accepting a single void* argument. * linuxthreads/manager.c (__pthread_manager): Put thread vitals in the thread struct instead of as arguments through clone. (pthread_start_thread): Look for them there. * linuxthreads/internals.h (struct _pthread): Add p_initial_fn, p_initial_fn_arg, p_initial_mask. Fix __pthread_manager proto. * linuxthreads/pthread.c (pthread_initialize_manager): Revise clone invocation. --- linuxthreads/FAQ.html | 986 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 986 insertions(+) create mode 100644 linuxthreads/FAQ.html (limited to 'linuxthreads/FAQ.html') diff --git a/linuxthreads/FAQ.html b/linuxthreads/FAQ.html new file mode 100644 index 0000000000..45d2387db2 --- /dev/null +++ b/linuxthreads/FAQ.html @@ -0,0 +1,986 @@ + + +LinuxThreads Frequently Asked Questions + + +

LinuxThreads Frequently Asked Questions
+ (with answers)

+ +

+ +A. The big picture
+B. Getting more information
+C. Issues related to the C library
+D. Problems, weird behaviors, potential bugs
+E. Missing functions, wrong types, etc
+F. C++ issues
+G. Debugging LinuxThreads programs
+H. Compiling multithreaded code; errno madness
+I. X-Windows and other libraries
+J. Signals and threads
+K. Internals of LinuxThreads

+ +

+

+ +

A. The big picture

+ +

A.1: What is LinuxThreads?

+ +LinuxThreads is a Linux library for multi-threaded programming. +It implements the Posix 1003.1c API (Application Programming +Interface) for threads. It runs on any Linux system with kernel 2.0.0 +or more recent, and a suitable C library (see section B). +

+ +

A.2: What are threads?

+ +A thread is a sequential flow of control through a program. +Multi-threaded programming is, thus, a form of parallel programming +where several threads of control are executing concurrently in the +program. All threads execute in the same memory space, and can +therefore work concurrently on shared data.

+ +Multi-threaded programming differs from Unix-style multi-processing in +that all threads share the same memory space (and a few other system +resources, such as file descriptors), instead of running in their own +memory space as is the case with Unix processes.

+ +Threads are useful for two reasons. First, they allow a program to +exploit multi-processor machines: the threads can run in parallel on +several processors, allowing a single program to divide its work +between several processors, thus running faster than a single-threaded +program, which runs on only one processor at a time. Second, some +programs are best expressed as several threads of control that +communicate together, rather than as one big monolithic sequential +program. Examples include server programs, overlapping asynchronous +I/O, and graphical user interfaces.

+ +

A.3: What is POSIX 1003.1c?

+ +It's an API for multi-threaded programming standardized by IEEE as +part of the POSIX standards. Most Unix vendors have endorsed the +POSIX 1003.1c standard. Implementations of the 1003.1c API are +already available under Sun Solaris 2.5, Digital Unix 4.0, +Silicon Graphics IRIX 6, and should soon be available from other +vendors such as IBM and HP. More generally, the 1003.1c API is +replacing relatively quickly the proprietary threads library that were +developed previously under Unix, such as Mach cthreads, Solaris +threads, and IRIX sprocs. Thus, multithreaded programs using the +1003.1c API are likely to run unchanged on a wide variety of Unix +platforms.

+ +

A.4: What is the status of LinuxThreads?

+ +In short, it's not completely finished (hence the version numbers in +0.x), but what is done is pretty mature. +LinuxThreads implements almost all of Posix 1003.1c, as well as a few +extensions. The only part of LinuxThreads that does not conform yet +to Posix is signal handling (see section J). Apart +from the signal stuff, all the Posix 1003.1c base functionality is +provided and conforms to the standard (to the best of my knowledge). +The signal stuff is hard to get right, at least without special kernel +support, and while I'm definitely looking at ways to implement the +Posix behavior for signals, this might take a long time before it's +completed.

+ +

A.5: How stable is LinuxThreads?

+ +The basic functionality (thread creation and termination, mutexes, +conditions, semaphores) is very stable. Several industrial-strength +programs, such as the AOL multithreaded Web server, use LinuxThreads +and seem quite happy about it. There are some rough edges in +the LinuxThreads / C library interface, at least with libc 5, but most +of these rough edges are fixed in glibc 2, which should soon become +the standard C library for Linux distributions (see section C).

+ +

+

+ +

B. Getting more information

+ +

B.1: What are good books and other sources of +information on POSIX threads?

+ +The FAQ for comp.programming.threads lists several books: +http://www.serpentine.com/~bos/threads-faq/.

+ +There are also some online tutorials. Follow the links from the +LinuxThreads web page: +http://pauillac.inria.fr/~xleroy/linuxthreads.

+ +

B.2: I'd like to be informed of future developments on +LinuxThreads. Is there a mailing list for this purpose?

+ +I post LinuxThreads-related announcements on the newsgroup +comp.os.linux.announce, +and also on the mailing list +linux-threads@magenet.com. +You can subscribe to the latter by writing +majordomo@magenet.com.

+ +

B.3: What are good places for discussing +LinuxThreads?

+ +For questions about programming with POSIX threads in general, use +the newsgroup +comp.programming.threads. +Be sure you read the +FAQ +for this group before you post.

+ +For Linux-specific questions, use +comp.os.linux.development.apps +and comp.os.linux.development.kernel. +The latter is especially appropriate for questions relative to the +interface between the kernel and LinuxThreads.

+ +Very specific LinuxThreads questions, and in particular everything +that looks like a potential bug in LinuxThreads, should be mailed +directly to me (Xavier.Leroy@inria.fr). Before mailing +me, make sure that your question is not answered in this FAQ.

+ +

B.4: I'd like to read the POSIX 1003.1c standard. Is +it available online?

+ +Unfortunately, no. POSIX standards are copyrighted by IEEE, and +IEEE does not distribute them freely. You can buy paper copies from +IEEE, but the price is fairly high ($120 or so). If you disagree with +this policy and you're an IEEE member, be sure to let them know.

+ +On the other hand, you probably don't want to read the standard. It's +very hard to read, written in standard-ese, and targeted to +implementors who already know threads inside-out. A good book on +POSIX threads provides the same information in a much more readable form. +I can personally recommend Dave Butenhof's book, Programming +with POSIX threads (Addison-Wesley). Butenhof was part of the +POSIX committee and also designed the Digital Unix implementations of +POSIX threads, and it shows.

+ +Another good source of information is the X/Open Group Single Unix +specification which is available both +on-line +and as a +book and CD/ROM. +That specification includes pretty much all the POSIX standards, +including 1003.1c, with some extensions and clarifications.

+ +

+

+ +

C. Issues related to the C library

+ +

C.1: Which version of the C library should I use +with LinuxThreads?

+ +Most current Linux distributions come with libc version 5, maintained +by H.J.Lu. For LinuxThreads to work properly, you must use either +libc 5.2.18 or libc 5.4.12 or later. Avoid 5.3.12 and 5.4.7: these +have problems with the per-thread errno variable. +

+ +Unfortunately, many popular Linux distributions (e.g. RedHat 4.2) come +with libc 5.3.12 preinstalled -- the one that does not work with +LinuxThreads. Fortunately, you can often find pre-packaged binaries +of more recent versions of libc for these distributions. In the case +of RedHat 4, there is a RPM package for libc-5.4 in the "contrib" +area of RedHat FTP sites. +

+ +

C.2: What about glibc 2, a.k.a. libc 6?

+ +It's the next generation libc for Linux, developed by Ulrich +Drepper and other FSF collaborators. glibc 2 offers much better +support for threads than libc 5. Indeed, thread support was +planned from the very early stages of glibc 2, while it's a +last-minute addition to libc 5. glibc 2 actually comes with a +specially adapted version of LinuxThreads, which you can drop in the +glibc 2 sources as an add-on package. + +

C.3: So, should I switch to glibc 2, or stay with a +recent libc 5?

+ +Depends how you plan to do it. Switching an already installed +system from libc 5 to glibc 2 is not completely straightforward. +See the Glibc2 +HOWTO for more information. +But (re-)installing a Linux distribution based on glibc 2 is easy. +One such distribution available now is RedHat 5.0. Debian and other +Linux distributors will also provide glibc 2-based distributions in the +near future. +

+ +

C.4: Where can I find glibc 2 and the version of +LinuxThreads that goes with it?

+ +On prep.ai.mit.edu and its many, many mirrors around the world. +See http://www.gnu.org/order/ftp.html +for a list of mirrors.

+ +

+

+ +

D. Problems, weird behaviors, potential bugs

+ +

D.1: When I compile LinuxThreads, I run into problems in +file libc_r/dirent.c

+ +You probably mean: +
 
+        libc_r/dirent.c:94: structure has no member named `dd_lock'
+
+I haven't actually seen this problem, but several users reported it. +My understanding is that something is wrong in the include files of +your Linux installation (/usr/include/*). Make sure +you're using a supported version of the C library. (See section B).

+ +

D.2: When I compile LinuxThreads, I run into problems with +/usr/include/sched.h: there are several occurrences of +_p that the C compiler does not understand

+ +Yes, /usr/include/sched.h that comes with libc 5.3.12 is broken. +Replace it with the sched.h file contained in the +LinuxThreads distribution. But really you should not be using libc +5.3.12 with LinuxThreads! (See question C.1.)

+ +

D.3: My program does fdopen() on a file +descriptor opened on a pipe. When I link it with LinuxThreads, +fdopen() always returns NULL!

+ +You're using one of the buggy versions of libc (5.3.12, 5.4.7., etc). +See question C.1 above.

+ +

D.4: My program crashes the first time it calls +pthread_create() !

+ +You wouldn't be using glibc 2.0, by any chance? That's a known bug +with glibc 2.0. Please upgrade to 2.0.1 or later.

+ +

D.5: When I'm running a program that creates N +threads, top or ps +display N+2 processes that are running my program. What do all these +processes correspond to?

+ +Due to the general "one process per thread" model, there's one process +for the initial thread and N processes for the threads it created +using pthread_create. That leaves one process +unaccounted for. That extra process corresponds to the "thread +manager" thread, a thread created internally by LinuxThreads to handle +thread creation and thread termination. This extra thread is asleep +most of the time. + +

D.6: Scheduling seems to be very unfair when there +is strong contention on a mutex: instead of giving the mutex to each +thread in turn, it seems that it's almost always the same thread that +gets the mutex. Isn't this completely broken behavior?

+ +What happens is the following: when a thread unlocks a mutex, all +other threads that were waiting on the mutex are sent a signal which +makes them runnable. However, the kernel scheduler may or may not +restart them immediately. If the thread that unlocked the mutex +tries to lock it again immediately afterwards, it is likely that it +will succeed, because the threads haven't yet restarted. This results +in an apparently very unfair behavior, when the same thread repeatedly +locks and unlocks the mutex, while other threads can't lock the mutex.

+ +This is perfectly acceptable behavior with respect to the POSIX +standard: for the default scheduling policy, POSIX makes no guarantees +of fairness, such as "the thread waiting for the mutex for the longest +time always acquires it first". This allows implementations of +mutexes to remain simple and efficient. Properly written +multithreaded code avoids that kind of heavy contention on mutexes, +and does not run into fairness problems. If you need scheduling +guarantees, you should consider using the real-time scheduling +policies SCHED_RR and SCHED_FIFO, which have +precisely defined scheduling behaviors.

+ +

D.7: I have a simple test program with two threads +that do nothing but printf() in tight loops, and from the +printout it seems that only one thread is running, the other doesn't +print anything!

+ +If you wait long enough, you should see the second thread kick in. +But still, you're right, one thread prevents the other one from +running for long periods of time. The reason is explained in +question D.6 above: printf() performs +locking on stdout, and thus your two threads contend very +heavily for the mutex associated with stdout. But if you +do some real work between two calls to printf(), you'll +see that scheduling becomes much smoother.

+ +

D.8: I've looked at <pthread.h> +and there seems to be a gross error in the pthread_cleanup_push +macro: it opens a block with { but does not close it! +Surely you forgot a } at the end of the macro, right? +

+ +Nope. That's the way it should be. The closing brace is provided by +the pthread_cleanup_pop macro. The POSIX standard +requires pthread_cleanup_push and +pthread_cleanup_pop to be used in matching pairs, at the +same level of brace nesting. This allows +pthread_cleanup_push to open a block in order to +stack-allocate some data structure, and +pthread_cleanup_pop to close that block. It's ugly, but +it's the standard way of implementing cleanup handlers.

+ +

+

+ +

E. Missing functions, wrong types, etc

+ +

E.1: Where is pthread_yield() ? How +comes LinuxThreads does not implement it?

+ +Because it's not part of the (final) POSIX 1003.1c standard. +Several drafts of the standard contained pthread_yield(), +but then the POSIX guys discovered it was redundant with +sched_yield() and dropped it. So, just use +sched_yield() instead. + +

E.2: I've found some type errors in +<pthread.h>. +For instance, the second argument to pthread_create() +should be a pthread_attr_t, not a +pthread_attr_t *. Also, didn't you forget to declare +pthread_attr_default?

+ +No, I didn't. What you're describing is draft 4 of the POSIX +standard, which is used in OSF DCE threads. LinuxThreads conforms to the +final standard. Even though the functions have the same names as in +draft 4 and DCE, their calling conventions are slightly different. In +particular, attributes are passed by reference, not by value, and +default attributes are denoted by the NULL pointer. Since draft 4/DCE +will eventually disappear, you'd better port your program to use the +standard interface.

+ +

E.3: I'm porting an application from Solaris and I +have to rename all thread functions from thr_blah to +pthread_blah. This is very annoying. Why did you change +all the function names?

+ +POSIX did it. The thr_* functions correspond to Solaris +threads, an older thread interface that you'll find only under +Solaris. The pthread_* functions correspond to POSIX +threads, an international standard available for many, many platforms. +Even Solaris 2.5 and later support the POSIX threads interface. So, +do yourself a favor and rewrite your code to use POSIX threads: this +way, it will run unchanged under Linux, Solaris, and quite a lot of +other platforms.

+ +

E.4: How can I suspend and resume a thread from +another thread? Solaris has the thr_suspend() and +thr_resume() functions to do that; why don't you?

+ +The POSIX standard provides no mechanism by which a thread A can +suspend the execution of another thread B, without cooperation from B. +The only way to implement a suspend/restart mechanism is to have B +check periodically some global variable for a suspend request +and then suspend itself on a condition variable, which another thread +can signal later to restart B.

+ +Notice that thr_suspend() is inherently dangerous and +prone to race conditions. For one thing, there is no control on where +the target thread stops: it can very well be stopped in the middle of +a critical section, while holding mutexes. Also, there is no +guarantee on when the target thread will actually stop. For these +reasons, you'd be much better off using mutexes and conditions +instead. The only situations that really require the ability to +suspend a thread are debuggers and some kind of garbage collectors.

+ +If you really must suspend a thread in LinuxThreads, you can send it a +SIGSTOP signal with pthread_kill. Send +SIGCONT for restarting it. +Beware, this is specific to LinuxThreads and entirely non-portable. +Indeed, a truly conforming POSIX threads implementation will stop all +threads when one thread receives the SIGSTOP signal! +One day, LinuxThreads will implement that behavior, and the +non-portable hack with SIGSTOP won't work anymore.

+ +

E.5: LinuxThreads does not implement +pthread_attr_setstacksize() nor +pthread_attr_setstackaddr(). Why?

+ +These two functions are part of optional components of the POSIX +standard, meaning that portable applications should test for the +"feature test" macros _POSIX_THREAD_ATTR_STACKSIZE and +_POSIX_THREAD_ATTR_STACKADDR (respectively) before using these +functions.

+ +pthread_attr_setstacksize() lets the programmer specify +the maximum stack size for a thread. In LinuxThreads, stacks start +small (4k) and grow on demand to a fairly large limit (2M), which +cannot be modified on a per-thread basis for architectural reasons. +Hence there is really no need to specify any stack size yourself: the +system does the right thing all by itself. Besides, there is no +portable way to estimate the stack requirements of a thread, so +setting the stack size is pretty useless anyway.

+ +pthread_attr_setstackaddr() is even more questionable: it +lets users specify the stack location for a thread. Again, +LinuxThreads takes care of that for you. Why you would ever need to +set the stack address escapes me.

+ +

E.6: LinuxThreads does not support the +PTHREAD_SCOPE_PROCESS value of the "contentionscope" +attribute. Why?

+ +With a "one-to-one" model, as in LinuxThreads (one kernel execution +context per thread), there is only one scheduler for all processes and +all threads on the system. So, there is no way to obtain the behavior of +PTHREAD_SCOPE_PROCESS. + +

E.7: LinuxThreads does not implement process-shared +mutexes, conditions, and semaphores. Why?

+ +This is another optional component of the POSIX standard. Portable +applications should test _POSIX_THREAD_PROCESS_SHARED +before using this facility. +

+The goal of this extension is to allow different processes (with +different address spaces) to synchronize through mutexes, conditions +or semaphores allocated in shared memory (either SVR4 shared memory +segments or mmap()ed files). +

+The reason why this does not work in LinuxThreads is that mutexes, +conditions, and semaphores are not self-contained: their waiting +queues contain pointers to linked lists of thread descriptors, and +these pointers are meaningful only in one address space. +

+Matt Messier and I spent a significant amount of time trying to design a +suitable mechanism for sharing waiting queues between processes. We +came up with several solutions that combined two of the following +three desirable features, but none that combines all three: +

    +
  • allow sharing between processes having different UIDs +
  • supports cancellation +
  • supports pthread_cond_timedwait +
+We concluded that kernel support is required to share mutexes, +conditions and semaphores between processes. That's one place where +Linus Torvalds's intuition that "all we need in the kernel is +clone()" fails. +

+Until suitable kernel support is available, you'd better use +traditional interprocess communications to synchronize different +processes: System V semaphores and message queues, or pipes, or sockets. +

+ +

+

+ +

F. C++ issues

+ +

F.1: Are there C++ wrappers for LinuxThreads?

+ +Douglas Schmidt's ACE library contains, among a lot of other +things, C++ wrappers for LinuxThreads and quite a number of other +thread libraries. Check out +http://www.cs.wustl.edu/~schmidt/ACE.html

+ +

F.2: I'm trying to use LinuxThreads from a C++ +program, and the compiler complains about the third argument to +pthread_create() !

+ +You're probably trying to pass a class member function or some +other C++ thing as third argument to pthread_create(). +Recall that pthread_create() is a C function, and it must +be passed a C function as third argument.

+ +

F.3: I'm trying to use LinuxThreads in conjunction +with libg++, and I'm having all sorts of trouble.

+ +From what I understand, thread support in libg++ is completely broken, +especially with respect to locking of iostreams. H.J.Lu wrote: +
+If you want to use thread, I can only suggest egcs and glibc. You +can find egcs at +http://www.cygnus.com/egcs. +egcs has libsdtc++, which is MT safe under glibc 2. If you really +want to use the libg++, I have a libg++ add-on for egcs. +
+
+

+ +

G. Debugging LinuxThreads programs

+ +

G.1: Can I debug LinuxThreads program using gdb?

+ +Essentially, no. gdb is basically not aware of the threads. It +will let you debug the main thread, and also inspect the global state, +but you won't have any control over the other threads. Worse, you +can't put any breakpoint anywhere in the code: if a thread other than +the main thread hits the breakpoint, it will just crash!

+ +For running gdb on the main thread, you need to instruct gdb to ignore +the signals used by LinuxThreads. Just do: +

+        handle SIGUSR1 nostop pass noprint
+        handle SIGUSR2 nostop pass noprint
+
+
+ +

G.2: What about attaching to a running thread using +the attach command of gdb?

+ +For reasons I don't fully understand, this does not work.

+ +

G.3: But I know gdb supports threads on some +platforms! Why not on Linux?

+ +You're correct that gdb has some built-in support for threads, in +particular the IRIX "sprocs" model, which is a "one thread = one +process" model fairly close to LinuxThreads. But gdb under IRIX uses +ioctls on /proc to control debugged processes, while +under Linux it uses the traditional ptrace(). The support +for threads is built in the /proc interface, but some +work remains to be done to have it in the ptrace() +interface. In summary, it should not be impossible to get gdb to work +with LinuxThreads, but it's definitely not trivial. + +

G.4: OK, I'll do post-mortem debugging, then. But +gdb cannot read core files generated by a multithreaded program! Or, +the core file is readable from gcc, but does not correspond to the +thread that crashed! What happens?

+ +Some versions of gdb do indeed have problems with post-mortem +debugging in general, but this is not specific to LinuxThreads. +Recent Linux distributions seem to have corrected this problem, +though.

+ +Regarding the fact that the core file does not correspond to the +thread that crashed, the reason is that the kernel will not dump core +for a process that shares its memory with other processes, such as the +other threads of your program. So, the thread that crashes silently +disappears without generating a core file. Then, all other threads of +your program die on the same signal that killed the crashing thread. +(This is required behavior according to the POSIX standard.) The last +one that dies is no longer sharing its memory with anyone else, so the +kernel generates a core file for that thread. Unfortunately, that's +not the thread you are interested in. + +

G.5: How can I debug multithreaded programs, then?

+ +Assertions and printf() are your best friends. Try to debug +sequential parts in a single-threaded program first. Then, put +printf() statements all over the place to get execution traces. +Also, check invariants often with the assert() macro. In truth, +there is no other effective way (save for a full formal proof of your +program) to track down concurrency bugs. Debuggers are not really +effective for concurrency problems, because they disrupt program +execution too much.

+ +

+

+ +

H. Compiling multithreaded code; errno madness

+ +

H.1: You say all multithreaded code must be compiled +with _REENTRANT defined. What difference does it make?

+ +It affects include files in three ways: +
    +
  • The include files define prototypes for the reentrant variants of +some of the standard library functions, +e.g. gethostbyname_r() as a reentrant equivalent to +gethostbyname().

    + +

  • If _REENTRANT is defined, some +<stdio.h> functions are no longer defined as macros, +e.g. getc() and putc(). In a multithreaded +program, stdio functions require additional locking, which the macros +don't perform, so we must call functions instead.

    + +

  • More importantly, <errno.h> redefines errno when +_REENTRANT is +defined, so that errno refers to the thread-specific errno location +rather than the global errno variable. This is achieved by the +following #define in <errno.h>: +
    +        #define errno (*(__errno_location()))
+
    +which causes each reference to errno to call the +__errno_location() function for obtaining the location +where error codes are stored. libc provides a default definition of +__errno_location() that always returns +&errno (the address of the global errno variable). Thus, +for programs not linked with LinuxThreads, defining +_REENTRANT makes no difference w.r.t. errno processing. +But LinuxThreads redefines __errno_location() to return a +location in the thread descriptor reserved for holding the current +value of errno for the calling thread. Thus, each thread operates on +a different errno location. +
+

+ +

H.2: Why is it so important that each thread has its +own errno variable?

+ +If all threads were to store error codes in the same, global errno +variable, then the value of errno after a system call or library +function returns would be unpredictable: between the time a system +call stores its error code in the global errno and your code inspects +errno to see which error occurred, another thread might have stored +another error code in the same errno location.

+ +

H.3: What happens if I link LinuxThreads with code +not compiled with -D_REENTRANT?

+ +Lots of trouble. If the code uses getc() or +putc(), it will perform I/O without proper interlocking +of the stdio buffers; this can cause lost output, duplicate output, or +just crash other stdio functions. If the code consults errno, it will +get back the wrong error code. The following code fragment is a +typical example: +
+        do {
+          r = read(fd, buf, n);
+          if (r == -1) {
+            if (errno == EINTR)   /* an error we can handle */
+              continue;
+            else {                /* other errors are fatal */
+              perror("read failed");
+              exit(100);
+            }
+          }
+        } while (...);
+
+Assume this code is not compiled with -D_REENTRANT, and +linked with LinuxThreads. At run-time, read() is +interrupted. Since the C library was compiled with +-D_REENTRANT, read() stores its error code +in the location pointed to by __errno_location(), which +is the thread-local errno variable. Then, the code above sees that +read() returns -1 and looks up errno. Since +_REENTRANT is not defined, the reference to errno +accesses the global errno variable, which is most likely 0. Hence the +code concludes that it cannot handle the error and stops.

+ +

H.4: With LinuxThreads, I can no longer use the signals +SIGUSR1 and SIGUSR2 in my programs! Why?

+ +LinuxThreads needs two signals for its internal operation. +One is used to suspend and restart threads blocked on mutex, condition +or semaphore operations. The other is used for thread cancellation. +Since the only two signals not reserved for the Linux kernel are +SIGUSR1 and SIGUSR2, LinuxThreads has no +other choice than using them. I know this is unfortunate, and hope +this problem will be addressed in future Linux kernels, either by +freeing some of the regular signals (unlikely), or by providing more +than 32 signals (as per the POSIX 1003.1b realtime extensions).

+ +In the meantime, you can try to use kernel-reserved signals either in +your program or in LinuxThreads. For instance, +SIGSTKFLT and SIGUNUSED appear to be +unused in the current Linux kernels for the Intel x86 architecture. +To use these in LinuxThreads, the only file you need to change +is internals.h, more specifically the two lines: +

+        #define PTHREAD_SIG_RESTART SIGUSR1
+        #define PTHREAD_SIG_CANCEL SIGUSR2
+
+Replace them by e.g. +
+        #define PTHREAD_SIG_RESTART SIGSTKFLT
+        #define PTHREAD_SIG_CANCEL SIGUNUSED
+
+Warning: you're doing this at your own risks.

+ +

H.5: Is the stack of one thread visible from the +other threads? Can I pass a pointer into my stack to other threads? +

+ +Yes, you can -- if you're very careful. The stacks are indeed visible +from all threads in the system. Some non-POSIX thread libraries seem +to map the stacks for all threads at the same virtual addresses and +change the memory mapping when they switch from one thread to +another. But this is not the case for LinuxThreads, as it would make +context switching between threads more expensive, and at any rate +might not conform to the POSIX standard.

+ +So, you can take the address of an "auto" variable and pass it to +other threads via shared data structures. However, you need to make +absolutely sure that the function doing this will not return as long +as other threads need to access this address. It's the usual mistake +of returning the address of an "auto" variable, only made much worse +because of concurrency. It's much, much safer to systematically +heap-allocate all shared data structures.

+ +

+

+ +

I. X-Windows and other libraries

+ +

I.1: My program uses both Xlib and LinuxThreads. +It stops very early with an "Xlib: unknown 0 error" message. What +does this mean?

+ +That's a prime example of the errno problem described in question H.2. The binaries for Xlib you're using have not been +compiled with -D_REENTRANT. It happens Xlib contains a +piece of code very much like the one in question H.2. So, your Xlib fetches the error code from the +wrong errno location and concludes that an error it cannot handle +occurred.

+ +

I.2: So, what can I do to build a multithreaded X +Windows client?

+ +The best solution is to recompile the X libraries with multithreading +options set. They contain optional support for multithreading; it's +just that all binary distributions for Linux were built without this +support. See the file README.Xfree3.3 in the LinuxThreads +distribution for patches and info on how to compile thread-safe X +libraries from the Xfree3.3 distribution. The Xfree3.3 sources are +readily available in most Linux distributions, e.g. as a source RPM +for RedHat. Be warned, however, that X Windows is a huge system, and +recompiling even just the libraries takes a lot of time and disk +space.

+ +Another, less involving solution is to call X functions only from the +main thread of your program. Even if all threads have their own errno +location, the main thread uses the global errno variable for its errno +location. Thus, code not compiled with -D_REENTRANT +still "sees" the right error values if it executes in the main thread +only.

+ +

This is a lot of work. Don't you have precompiled +thread-safe X libraries that you could distribute?

+ +No, I don't. Sorry. But you could approach the maintainers of +your Linux distribution to see if they would be willing to provide +thread-safe X libraries.

+ +

I.3: Can I use library FOO in a multithreaded +program?

+ +Most libraries cannot be used "as is" in a multithreaded program. +For one thing, they are not necessarily thread-safe: calling +simultaneously two functions of the library from two threads might not +work, due to internal use of global variables and the like. Second, +the libraries must have been compiled with -D_REENTRANT to avoid +the errno problems explained in question H.2. +

+ +

I.4: What if I make sure that only one thread calls +functions in these libraries?

+ +This avoids problems with the library not being thread-safe. But +you're still vulnerable to errno problems. At the very least, a +recompile of the library with -D_REENTRANT is needed. +

+ +

I.5: What if I make sure that only the main thread +calls functions in these libraries?

+ +That might actually work. As explained in question I.1, +the main thread uses the global errno variable, and can therefore +execute code not compiled with -D_REENTRANT.

+ +

I.6: SVGAlib doesn't work with LinuxThreads. Why? +

+ +Because both LinuxThreads and SVGAlib use the signals +SIGUSR1 and SIGUSR2. One of the two should +be recompiled to use different signals. See question H.4. +

+ + +

+

+ +

J. Signals and threads

+ +

J.1: When it comes to signals, what is shared +between threads and what isn't?

+ +Signal handlers are shared between all threads: when a thread calls +sigaction(), it sets how the signal is handled not only +for itself, but for all other threads in the program as well.

+ +On the other hand, signal masks are per-thread: each thread chooses +which signals it blocks independently of others. At thread creation +time, the newly created thread inherits the signal mask of the thread +calling pthread_create(). But afterwards, the new thread +can modify its signal mask independently of its creator thread.

+ +

J.2: When I send a SIGKILL to a +particular thread using pthread_kill, all my threads are +killed!

+ +That's how it should be. The POSIX standard mandates that all threads +should terminate when the process (i.e. the collection of all threads +running the program) receives a signal whose effect is to +terminate the process (such as SIGKILL or SIGINT +when no handler is installed on that signal). This behavior makes a +lot of sense: when you type "ctrl-C" at the keyboard, or when a thread +crashes on a division by zero or a segmentation fault, you really want +all threads to stop immediately, not just the one that caused the +segmentation violation or that got the SIGINT signal. +(This assumes default behavior for those signals; see question +J.3 if you install handlers for those signals.)

+ +If you're trying to terminate a thread without bringing the whole +process down, use pthread_cancel().

+ +

J.3: I've installed a handler on a signal. Which +thread executes the handler when the signal is received?

+ +If the signal is generated by a thread during its execution (e.g. a +thread executes a division by zero and thus generates a +SIGFPE signal), then the handler is executed by that +thread. This also applies to signals generated by +raise().

+ +If the signal is sent to a particular thread using +pthread_kill(), then that thread executes the handler.

+ +If the signal is sent via kill() or the tty interface +(e.g. by pressing ctrl-C), then the POSIX specs say that the handler +is executed by any thread in the process that does not currently block +the signal. In other terms, POSIX considers that the signal is sent +to the process (the collection of all threads) as a whole, and any +thread that is not blocking this signal can then handle it.

+ +The latter case is where LinuxThreads departs from the POSIX specs. +In LinuxThreads, there is no real notion of ``the process as a whole'': +in the kernel, each thread is really a distinct process with a +distinct PID, and signals sent to the PID of a thread can only be +handled by that thread. As long as no thread is blocking the signal, +the behavior conforms to the standard: one (unspecified) thread of the +program handles the signal. But if the thread to which PID the signal +is sent blocks the signal, and some other thread does not block the +signal, then LinuxThreads will simply queue in +that thread and execute the handler only when that thread unblocks +the signal, instead of executing the handler immediately in the other +thread that does not block the signal.

+ +This is to be viewed as a LinuxThreads bug, but I currently don't see +any way to implement the POSIX behavior without kernel support.

+ +

J.3: How shall I go about mixing signals and threads +in my program?

+ +The less you mix them, the better. Notice that all +pthread_* functions are not async-signal safe, meaning +that you should not call them from signal handlers. This +recommendation is not to be taken lightly: your program can deadlock +if you call a pthread_* function from a signal handler! +

+ +The only sensible things you can do from a signal handler is set a +global flag, or call sem_post on a semaphore, to record +the delivery of the signal. The remainder of the program can then +either poll the global flag, or use sem_wait() and +sem_trywait() on the semaphore.

+ +Another option is to do nothing in the signal handler, and dedicate +one thread (preferably the initial thread) to wait synchronously for +signals, using sigwait(), and send messages to the other +threads accordingly. + +

J.4: When one thread is blocked in +sigwait(), other threads no longer receive the signals +sigwait() is waiting for! What happens?

+ +It's an unfortunate consequence of how LinuxThreads implements +sigwait(). Basically, it installs signal handlers on all +signals waited for, in order to record which signal was received. +Since signal handlers are shared with the other threads, this +temporarily deactivates any signal handlers you might have previously +installed on these signals.

+ +Though surprising, this behavior actually seems to conform to the +POSIX standard. According to POSIX, sigwait() is +guaranteed to work as expected only if all other threads in the +program block the signals waited for (otherwise, the signals could be +delivered to other threads than the one doing sigwait(), +which would make sigwait() useless). In this particular +case, the problem described in this question does not appear.

+ +One day, sigwait() will be implemented in the kernel, +along with others POSIX 1003.1b extensions, and sigwait() +will have a more natural behavior (as well as better performances).

+ +

+

+ +

K. Internals of LinuxThreads

+ +

K.1: What is the implementation model for +LinuxThreads?

+ +LinuxThreads follows the so-called "one-to-one" model: each thread is +actually a separate process in the kernel. The kernel scheduler takes +care of scheduling the threads, just like it schedules regular +processes. The threads are created with the Linux +clone() system call, which is a generalization of +fork() allowing the new process to share the memory +space, file descriptors, and signal handlers of the parent.

+ +Advantages of the "one-to-one" model include: +

    +
  • minimal overhead on CPU-intensive multiprocessing (with +about one thread per processor); +
  • minimal overhead on I/O operations; +
  • a simple and robust implementation (the kernel scheduler does +most of the hard work for us). +
+The main disadvantage is more expensive context switches on mutex and +condition operations, which must go through the kernel. This is +mitigated by the fact that context switches in the Linux kernel are +pretty efficient.

+ +

K.2: Have you considered other implementation +models?

+ +There are basically two other models. The "many-to-one" model +relies on a user-level scheduler that context-switches between the +threads entirely in user code; viewed from the kernel, there is only +one process running. This model is completely out of the question for +me, since it does not take advantage of multiprocessors, and require +unholy magic to handle blocking I/O operations properly. There are +several user-level thread libraries available for Linux, but I found +all of them deficient in functionality, performance, and/or robustness. +

+ +The "many-to-many" model combines both kernel-level and user-level +scheduling: several kernel-level threads run concurrently, each +executing a user-level scheduler that selects between user threads. +Most commercial Unix systems (Solaris, Digital Unix, IRIX) implement +POSIX threads this way. This model combines the advantages of both +the "many-to-one" and the "one-to-one" model, and is attractive +because it avoids the worst-case behaviors of both models -- +especially on kernels where context switches are expensive, such as +Digital Unix. Unfortunately, it is pretty complex to implement, and +requires kernel support which Linux does not provide. Linus Torvalds +and other Linux kernel developers have always been pushing the +"one-to-one" model in the name of overall simplicity, and are doing a +pretty good job of making kernel-level context switches between +threads efficient. LinuxThreads is just following the general +direction they set.

+ +

K.3: I looked at the LinuxThreads sources, and I saw +quite a lot of spinlocks and busy-waiting loops to acquire these +spinlocks. Isn't this a big waste of CPU time?

+ +Look more carefully. Spinlocks are used internally to protect +LinuxThreads's data structures, but these locks are held for very +short periods of time: 10 instructions or so. The probability that a +thread has to loop busy-waiting on a taken spinlock for more than, +say, 100 cycles is very, very low. When a thread needs to wait on a +mutex, condition, or semaphore, it actually puts itself on a waiting +queue, then suspends on a signal, consuming no CPU time at all. The +thread will later be restarted by sending it a signal when the state +of the mutex, condition, or semaphore changes.

+ +

+
Xavier.Leroy@inria.fr
+ + -- cgit v1.2.3-70-g09d2