Avoid late failure in dlopen in global scope update [BZ #25112]

The call to add_to_global in dl_open_worker happens after running ELF
constructors for new objects.  At this point, proper recovery from
malloc failure would be quite complicated: We would have to run the
ELF destructors and close all opened objects, something that we
currently do not do.

Instead, this change splits add_to_global into two phases,
add_to_global_prepare (which can raise an exception, called before ELF
constructors run), and add_to_global_finish (which cannot, called
after ELF constructors).  A complication arises due to recursive
dlopen: After the inner dlopen consumes some space, the pre-allocation
in the outer dlopen may no longer be sufficient.  A new member in the
namespace structure, _ns_global_scope_pending_adds keeps track of the
maximum number of objects that need to be added to the global scope.
This enables the inner add_to_global_prepare call to take into account
the needs of an outer dlopen.

Most code in the dynamic linker assumes that the number of global
scope entries fits into an unsigned int (matching the r_nlist member
of struct r_scop_elem).  Therefore, change the type of
_ns_global_scope_alloc to unsigned int (from size_t), and add overflow
checks.

Tested on x86_64-linux-gnu and i686-linux-gnu.

1 files changed, 8 insertions, 1 deletions

diff --git a/sysdeps/generic/ldsodefs.h b/sysdeps/generic/ldsodefs.h
index 07789022e0..ed0aed752f 100644
--- a/sysdeps/generic/ldsodefs.h
+++ b/sysdeps/generic/ldsodefs.h
@@ -328,7 +328,14 @@ struct rtld_global
     /* This is zero at program start to signal that the global scope map is
        allocated by rtld.  Later it keeps the size of the map.  It might be
        reset if in _dl_close if the last global object is removed.  */
-    size_t _ns_global_scope_alloc;
+    unsigned int _ns_global_scope_alloc;
+
+    /* During dlopen, this is the number of objects that still need to
+       be added to the global scope map.  It has to be taken into
+       account when resizing the map, for future map additions after
+       recursive dlopen calls from ELF constructors.  */
+    unsigned int _ns_global_scope_pending_adds;
+
     /* Search table for unique objects.  */
     struct unique_sym_table
     {