Update.

	* intl/gettextP.h (struct loaded_domain): Add conv element.
	* intl/dcgettext.c (find_msg): Rename to _nl_find_msg and make public.
	Instead of returning found message directly convert it using iconv
	if a conversion was found when opening the file.
	* intl/loadinfo.h: Protect against multiple inclusion.
	Declare _nl_find_msg.
	* intl/loadmsgcat.c (_nl_load_domain): Try to determine charset used
	in the message file and if necessary find approrpiate
	conversion to match currently selected charset.

8 files changed, 245 insertions, 36 deletions

diff --git a/ChangeLog b/ChangeLog
index d036936303..6956296a12 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -9,6 +9,16 @@
 
 	* elf/elf.h: Corrected SHT_ values from new draft.
 
+	* intl/gettextP.h (struct loaded_domain): Add conv element.
+	* intl/dcgettext.c (find_msg): Rename to _nl_find_msg and make public.
+	Instead of returning found message directly convert it using iconv
+	if a conversion was found when opening the file.
+	* intl/loadinfo.h: Protect against multiple inclusion.
+	Declare _nl_find_msg.
+	* intl/loadmsgcat.c (_nl_load_domain): Try to determine charset used
+	in the message file and if necessary find approrpiate
+	conversion to match currently selected charset.
+
 1999-08-20  Andreas Schwab  <schwab@suse.de>
 
 	* sysdeps/unix/sysv/linux/sys/sendfile.h: Fix typo.
diff --git a/NOTES b/NOTES
index b5c48b861d..5614f67059 100644
--- a/NOTES
+++ b/NOTES
@@ -18,18 +18,57 @@ comments.  You could also use the `-D' option to GCC, but it's better
 if you make the source files indicate their own meaning in a
 self-contained way.
 
+   This system exists to allow the library to conform to multiple
+standards.  Although the different standards are often described as
+supersets of each other, they are usually incompatible because larger
+standards require functions with names that smaller ones reserve to the
+user program.  This is not mere pedantry -- it has been a problem in
+practice.  For instance, some non-GNU programs define functions named
+`getline' that have nothing to do with this library's `getline'.  They
+would not be compilable if all features were enabled indescriminantly.
+
+   This should not be used to verify that a program conforms to a
+limited standard.  It is insufficent for this purpose, as it will not
+protect you from including header files outside the standard, or
+relying on semantics undefined within the standard.
+
  - Macro: _POSIX_SOURCE
      If you define this macro, then the functionality from the POSIX.1
      standard (IEEE Standard 1003.1) is available, as well as all of the
      ISO C facilities.
 
+     The state of `_POSIX_SOURCE' is irrelevant if you define the macro
+     `_POSIX_C_SOURCE' to a positive integer.
+
  - Macro: _POSIX_C_SOURCE
-     If you define this macro with a value of `1', then the
-     functionality from the POSIX.1 standard (IEEE Standard 1003.1) is
-     made available.  If you define this macro with a value of `2',
-     then both the functionality from the POSIX.1 standard and the
-     functionality from the POSIX.2 standard (IEEE Standard 1003.2) are
-     made available.  This is in addition to the ISO C facilities.
+     Define this macro to a positive integer to control which POSIX
+     functionality is made available.  The greater the value of this
+     macro, the more functionality is made available.
+
+     If you define this macro to a value greater than or equal to `1',
+     then the functionality from the 1990 edition of the POSIX.1
+     standard (IEEE Standard 1003.1-1990) is made available.
+
+     If you define this macro to a value greater than or equal to `2',
+     then the functionality from the 1992 edition of the POSIX.2
+     standard (IEEE Standard 1003.2-1992) is made available.
+
+     If you define this macro to a value greater than or equal to
+     `199309L', then the functionality from the 1993 edition of the
+     POSIX.1b standard (IEEE Standard 1003.1b-1993) is made available.
+
+     Greater values for `_POSIX_C_SOURCE' will enable future extensions.
+     The POSIX standards process will define these values as necessary,
+     and the GNU C Library should support them some time after they
+     become standardized.  The 1996 edition of POSIX.1 (ISO/IEC 9945-1:
+     1996) states that if you define `_POSIX_C_SOURCE' to a value
+     greater than or equal to `199506L', then the functionality from
+     the 1996 edition is made available.
+
+     The Single Unix Specification specify that setting this macro to
+     the value `199506L' selects all the values specified by the POSIX
+     standards plus those of the Single Unix Specification, i.e., is the
+     same as if `_XOPEN_SOURCE' is set to `500' (see below).
 
  - Macro: _BSD_SOURCE
      If you define this macro, functionality derived from 4.3 BSD Unix
@@ -72,12 +111,12 @@ self-contained way.
 
      If the macro `_XOPEN_SOURCE' has the value 500 this includes all
      functionality described so far plus some new definitions from the
-     Single Unix specification, version 2.
+     Single Unix Specification, version 2.
 
  - Macro: _LARGEFILE_SOURCE
      If this macro is defined some extra functions are available which
      rectify a few shortcomings in all previous standards.  More
-     concreten the functions `fseeko' and `ftello' are available.
+     concrete the functions `fseeko' and `ftello' are available.
      Without these functions the difference between the ISO C interface
      (`fseek', `ftell') and the low-level POSIX interface (`lseek')
      would lead to problems.
@@ -85,7 +124,7 @@ self-contained way.
      This macro was introduced as part of the Large File Support
      extension (LFS).
 
- - Variable: Macro _LARGEFILE64_SOURCE
+ - Macro: _LARGEFILE64_SOURCE
      If you define this macro an additional set of function gets
      available which enables to use on 32 bit systems to use files of
      sizes beyond the usual limit of 2GB.  This interface is not
@@ -103,7 +142,7 @@ self-contained way.
      extension (LFS).  It is a transition interface for the time 64 bit
      offsets are not generally used (see `_FILE_OFFSET_BITS'.
 
- - Variable: _FILE_OFFSET_BITS
+ - Macro: _FILE_OFFSET_BITS
      This macro lets decide which file system interface shall be used,
      one replacing the other.  While `_LARGEFILE64_SOURCE' makes the
      64 bit interface available as an additional interface
diff --git a/intl/dcgettext.c b/intl/dcgettext.c
index cc5299e460..0429cc904e 100644
--- a/intl/dcgettext.c
+++ b/intl/dcgettext.c
@@ -83,6 +83,10 @@ void free ();
 # include <locale.h>
 #endif
 
+#if defined HAVE_SYS_PARAM_H || defined _LIBC
+# include <sys/param.h>
+#endif
+
 #include "gettext.h"
 #include "gettextP.h"
 #ifdef _LIBC
@@ -92,6 +96,11 @@ void free ();
 #endif
 #include "hash-string.h"
 
+/* Thread safetyness.  */
+#ifdef _LIBC
+# include <bits/libc-lock.h>
+#endif
+
 /* @@ end of prolog @@ */
 
 #ifdef _LIBC
@@ -171,8 +180,6 @@ const char _nl_default_dirname[] = GNULOCALEDIR;
 struct binding *_nl_domain_bindings;
 
 /* Prototypes for local functions.  */
-static char *find_msg PARAMS ((struct loaded_l10nfile *domain_file,
-			       const char *msgid)) internal_function;
 static const char *category_to_name PARAMS ((int category)) internal_function;
 static const char *guess_category_value PARAMS ((int category,
 						 const char *categoryname))
@@ -396,7 +403,7 @@ DCGETTEXT (domainname, msgid, category)
 
       if (domain != NULL)
 	{
-	  retval = find_msg (domain, msgid);
+	  retval = _nl_find_msg (domain, msgid);
 
 	  if (retval == NULL)
 	    {
@@ -404,7 +411,7 @@ DCGETTEXT (domainname, msgid, category)
 
 	      for (cnt = 0; domain->successor[cnt] != NULL; ++cnt)
 		{
-		  retval = find_msg (domain->successor[cnt], msgid);
+		  retval = _nl_find_msg (domain->successor[cnt], msgid);
 
 		  if (retval != NULL)
 		    break;
@@ -428,9 +435,9 @@ weak_alias (__dcgettext, dcgettext);
 #endif
 
 
-static char *
+char *
 internal_function
-find_msg (domain_file, msgid)
+_nl_find_msg (domain_file, msgid)
      struct loaded_l10nfile *domain_file;
      const char *msgid;
 {
@@ -464,8 +471,88 @@ find_msg (domain_file, msgid)
 	  && strcmp (msgid,
 		     domain->data + W (domain->must_swap,
 				       domain->orig_tab[nstr - 1].offset)) == 0)
-	return (char *) domain->data + W (domain->must_swap,
-					  domain->trans_tab[nstr - 1].offset);
+	{
+	  /* We found an entry.  If we have to convert the string to use
+	     a different character set this is the time.  */
+	  char *result =
+	    (char *) domain->data + W (domain->must_swap,
+				       domain->trans_tab[nstr - 1].offset);
+
+	  if (
+#if HAVE_ICONV || defined _LIBC
+	      domain->conv != (iconv_t) -1
+#endif
+	      )
+	    {
+	      /* We are supposed to do a conversion.  First allocate an
+		 appropriate table with the same structure as the hash
+		 table in the file where we can put the pointers to the
+		 converted strings in.  */
+	      if (domain->conv_tab == NULL
+		  && ((domain->conv_tab = (char **) calloc (domain->hash_size,
+							    sizeof (char *)))
+		      == NULL))
+		/* Mark that we didn't succeed allocating a table.  */
+		domain->conv_tab = (char **) -1;
+
+	      if (domain->conv_tab == (char **) -1)
+		/* Nothing we can do, no more memory.  */
+		return NULL;
+
+	      if (domain->conv_tab[idx] == NULL)
+		{
+		  /* We haven't used this string so far, so it is not
+		     translated yet.  Do this now.  */
+#ifdef _LIBC
+		  /* For glibc we use a bit more efficient memory handling.
+		     We allocate always larger blocks which get used over
+		     time.  This is faster than many small allocations.   */
+		  __libc_lock_define_initialized (static, lock)
+		  static char *freemem;
+		  static size_t freemem_size;
+		  /* Note that we include the NUL byte.  */
+		  size_t resultlen = strlen (result) + 1;
+		  const char *inbuf = result;
+		  size_t inbytesleft = resultlen;
+		  char *outbuf = freemem;
+		  size_t outbytesleft = freemem_size;
+
+		  __libc_lock_lock (lock);
+
+		  while (iconv (domain->conv, &inbuf, &inbytesleft, &outbuf,
+				&outbytesleft) == (size_t) -1L)
+		    {
+		      if (errno != E2BIG)
+			goto out;
+
+		      /* We must resize the buffer.  */
+		      freemem_size = MAX (2 * freemem_size, 4064);
+		      freemem = (char *) malloc (freemem_size);
+		      if (freemem == NULL)
+			goto out;
+
+		      inbuf = result;
+		      inbytesleft = resultlen;
+		      outbuf = freemem;
+		      outbytesleft = freemem_size;
+		    }
+
+		  /* We have now in our buffer a converted string.  Put this
+		     in the hash table  */
+		  domain->conv_tab[idx] = freemem;
+		  freemem = outbuf;
+		  freemem_size = outbytesleft;
+
+		out:
+		  __libc_lock_unlock (lock);
+#endif
+		}
+
+	      result = domain->conv_tab[idx];
+	    }
+
+	  return result;
+	}
 
       while (1)
 	{
diff --git a/intl/gettextP.h b/intl/gettextP.h
index bcbe2720a7..bea4404167 100644
--- a/intl/gettextP.h
+++ b/intl/gettextP.h
@@ -1,6 +1,6 @@
 /* Header describing internals of gettext library
-   Copyright (C) 1995, 1996, 1997, 1998 Free Software Foundation, Inc.
-   Written by Ulrich Drepper <drepper@gnu.ai.mit.edu>, 1995.
+   Copyright (C) 1995, 1996, 1997, 1998, 1999 Free Software Foundation, Inc.
+   Written by Ulrich Drepper <drepper@cygnus.com>, 1995.
 
    The GNU C Library is free software; you can redistribute it and/or
    modify it under the terms of the GNU Library General Public License as
@@ -20,6 +20,10 @@
 #ifndef _GETTEXTP_H
 #define _GETTEXTP_H
 
+#if defined HAVE_ICONV || defined _LIBC
+# include <iconv.h>
+#endif
+
 #include "loadinfo.h"
 
 /* @@ end of prolog @@ */
@@ -67,6 +71,10 @@ struct loaded_domain
   struct string_desc *trans_tab;
   nls_uint32 hash_size;
   nls_uint32 *hash_tab;
+#if defined HAVE_ICONV || defined _LIBC
+  iconv_t conv;
+#endif
+  char **conv_tab;
 };
 
 struct binding
diff --git a/intl/loadinfo.h b/intl/loadinfo.h
index 35d98f0d9b..ea1bf05806 100644
--- a/intl/loadinfo.h
+++ b/intl/loadinfo.h
@@ -1,4 +1,4 @@
-/* Copyright (C) 1996, 1997, 1998 Free Software Foundation, Inc.
+/* Copyright (C) 1996, 1997, 1998, 1999 Free Software Foundation, Inc.
    This file is part of the GNU C Library.
    Contributed by Ulrich Drepper <drepper@cygnus.com>, 1996.
 
@@ -17,6 +17,9 @@
    write to the Free Software Foundation, Inc., 59 Temple Place - Suite 330,
    Boston, MA 02111-1307, USA.  */
 
+#ifndef _LOADINFO_H
+#define _LOADINFO_H	1
+
 #ifndef PARAMS
 # if __STDC__
 #  define PARAMS(args) args
@@ -77,3 +80,10 @@ extern int _nl_explode_name PARAMS ((char *name, const char **language,
 				     const char **revision));
 
 extern char *_nl_find_language PARAMS ((const char *name));
+
+
+extern char *_nl_find_msg PARAMS ((struct loaded_l10nfile *domain_file,
+				   const char *msgid))
+     internal_function;
+
+#endif	/* loadinfo.h */
diff --git a/intl/loadmsgcat.c b/intl/loadmsgcat.c
index 76887e8b4f..23d738882b 100644
--- a/intl/loadmsgcat.c
+++ b/intl/loadmsgcat.c
@@ -31,10 +31,23 @@
 # include <stdlib.h>
 #endif
 
+#if defined HAVE_STRING_H || defined _LIBC
+# ifndef _GNU_SOURCE
+#  define _GNU_SOURCE	1
+# endif
+# include <string.h>
+#else
+# include <strings.h>
+#endif
+
 #if defined HAVE_UNISTD_H || defined _LIBC
 # include <unistd.h>
 #endif
 
+#ifdef _LIBC
+# include <langinfo.h>
+#endif
+
 #if (defined HAVE_MMAP && defined HAVE_MUNMAP && !defined DISALLOW_MMAP) \
     || (defined _LIBC && defined _POSIX_MAPPED_FILES)
 # include <sys/mman.h>
@@ -47,6 +60,10 @@
 #include "gettext.h"
 #include "gettextP.h"
 
+#ifdef _LIBC
+# include "../locale/localeinfo.h"
+#endif
+
 /* @@ end of prolog @@ */
 
 #ifdef _LIBC
@@ -79,6 +96,7 @@ _nl_load_domain (domain_file)
   struct mo_file_header *data = (struct mo_file_header *) -1;
   int use_mmap = 0;
   struct loaded_domain *domain;
+  char *nullentry;
 
   domain_file->decided = 1;
   domain_file->data = NULL;
@@ -200,9 +218,40 @@ _nl_load_domain (domain_file)
       return;
     }
 
-  /* Show that one domain is changed.  This might make some cached
-     translations invalid.  */
-  ++_nl_msg_cat_cntr;
+  /* Now find out about the character set the file is encoded with.
+     This can be found (in textual form) in the entry "".  If this
+     entry does not exist or if this does not contain the `charset='
+     information, we will assume the charset matches the one the
+     current locale and we don't have to perform any conversion.  */
+#if HAVE_ICONV || defined _LIBC
+  domain->conv = (iconv_t) -1;
+#endif
+  nullentry = _nl_find_msg (domain_file, "");
+  if (nullentry != NULL)
+    {
+      char *charsetstr = strstr (nullentry, "charset=");
+
+      if (charsetstr != NULL)
+	{
+	  size_t len;
+	  char *charset;
+
+	  charsetstr += strlen ("charset=");
+	  len = strcspn (charsetstr, " \t\n");
+
+	  charset = (char *) alloca (len + 1);
+#if defined _LIBC || HAVE_MEMPCPY
+	  *((char *) mempcpy (charset, charsetstr, len)) = '\0';
+#else
+	  memcpy (charset, charsetstr, len);
+	  charset[len] = '\0';
+#endif
+
+#if HAVE_ICONV || defined _LIBC
+	  domain->conv = iconv_open ((*_nl_current[LC_CTYPE])->values[_NL_ITEM_INDEX (CODESET)].string, charset);
+#endif
+	}
+    }
 }
 
 
diff --git a/linuxthreads/ChangeLog b/linuxthreads/ChangeLog
index 14230c1c8d..165b5f70bd 100644
--- a/linuxthreads/ChangeLog
+++ b/linuxthreads/ChangeLog
@@ -1,3 +1,9 @@
+1999-08-20  Ulrich Drepper  <drepper@cygnus.com>
+
+	* pthread.c (__pthread_reset_main_thread): Undo last change.
+	(__pthread_kill_other_threads_np): Reset signal handlers for the
+	signals we used in the thread implementation here.
+
 1999-08-19  Ulrich Drepper  <drepper@cygnus.com>
 
 	* pthread.c (__pthread_reset_main_thread): Reset signal handlers
diff --git a/linuxthreads/pthread.c b/linuxthreads/pthread.c
index 7cc6b26866..86f24469de 100644
--- a/linuxthreads/pthread.c
+++ b/linuxthreads/pthread.c
@@ -635,7 +635,6 @@ static void pthread_handle_sigdebug(int sig)
 void __pthread_reset_main_thread()
 {
   pthread_descr self = thread_self();
-  struct sigaction sa;
 
   if (__pthread_manager_request != -1) {
     /* Free the thread manager stack */
@@ -647,17 +646,6 @@ void __pthread_reset_main_thread()
     __pthread_manager_request = __pthread_manager_reader = -1;
   }
 
-  /* Reset the signal handlers behaviour for the signals the
-     implementation uses since this would be passed to the new
-     process.  */
-  sigemptyset(&sa.sa_mask);
-  sa.sa_flags = 0;
-  sa.sa_handler = SIG_DFL;
-  __sigaction(__pthread_sig_restart, &sa, NULL);
-  __sigaction(__pthread_sig_cancel, &sa, NULL);
-  if (__pthread_sig_debug > 0)
-    __sigaction(__pthread_sig_debug, &sa, NULL);
-
   /* Update the pid of the main thread */
   THREAD_SETMEM(self, p_pid, __getpid());
   /* Make the forked thread the main thread */
@@ -673,11 +661,23 @@ void __pthread_reset_main_thread()
 
 void __pthread_kill_other_threads_np(void)
 {
+  struct sigaction sa;
   /* Terminate all other threads and thread manager */
   pthread_exit_process(0, NULL);
   /* Make current thread the main thread in case the calling thread
      changes its mind, does not exec(), and creates new threads instead. */
   __pthread_reset_main_thread();
+
+  /* Reset the signal handlers behaviour for the signals the
+     implementation uses since this would be passed to the new
+     process.  */
+  sigemptyset(&sa.sa_mask);
+  sa.sa_flags = 0;
+  sa.sa_handler = SIG_DFL;
+  __sigaction(__pthread_sig_restart, &sa, NULL);
+  __sigaction(__pthread_sig_cancel, &sa, NULL);
+  if (__pthread_sig_debug > 0)
+    __sigaction(__pthread_sig_debug, &sa, NULL);
 }
 weak_alias (__pthread_kill_other_threads_np, pthread_kill_other_threads_np)