diff options
Diffstat (limited to 'manual/charset.texi')
| -rw-r--r-- | manual/charset.texi | 5786 |
1 files changed, 2895 insertions, 2891 deletions
diff --git a/manual/charset.texi b/manual/charset.texi index b7b2f734a8..bae2910236 100644 --- a/manual/charset.texi +++ b/manual/charset.texi @@ -1,2892 +1,2896 @@ -@node Character Set Handling, Locales, String and Array Utilities, Top
-@c %MENU% Support for extended character sets
-@chapter Character Set Handling
-
-@ifnottex
-@macro cal{text}
-\text\
-@end macro
-@end ifnottex
-
-Character sets used in the early days of computing had only six, seven,
-or eight bits for each character: there was never a case where more than
-eight bits (one byte) were used to represent a single character. The
-limitations of this approach became more apparent as more people
-grappled with non-Roman character sets, where not all the characters
-that make up a language's character set can be represented by @math{2^8}
-choices. This chapter shows the functionality that was added to the C
-library to support multiple character sets.
-
-@menu
-* Extended Char Intro:: Introduction to Extended Characters.
-* Charset Function Overview:: Overview about Character Handling
- Functions.
-* Restartable multibyte conversion:: Restartable multibyte conversion
- Functions.
-* Non-reentrant Conversion:: Non-reentrant Conversion Function.
-* Generic Charset Conversion:: Generic Charset Conversion.
-@end menu
-
-
-@node Extended Char Intro
-@section Introduction to Extended Characters
-
-A variety of solutions is available to overcome the differences between
-character sets with a 1:1 relation between bytes and characters and
-character sets with ratios of 2:1 or 4:1. The remainder of this
-section gives a few examples to help understand the design decisions
-made while developing the functionality of the @w{C library}.
-
-@cindex internal representation
-A distinction we have to make right away is between internal and
-external representation. @dfn{Internal representation} means the
-representation used by a program while keeping the text in memory.
-External representations are used when text is stored or transmitted
-through some communication channel. Examples of external
-representations include files waiting in a directory to be
-read and parsed.
-
-Traditionally there has been no difference between the two representations.
-It was equally comfortable and useful to use the same single-byte
-representation internally and externally. This comfort level decreases
-with more and larger character sets.
-
-One of the problems to overcome with the internal representation is
-handling text that is externally encoded using different character
-sets. Assume a program that reads two texts and compares them using
-some metric. The comparison can be usefully done only if the texts are
-internally kept in a common format.
-
-@cindex wide character
-For such a common format (@math{=} character set) eight bits are certainly
-no longer enough. So the smallest entity will have to grow: @dfn{wide
-characters} will now be used. Instead of one byte per character, two or
-four will be used instead. (Three are not good to address in memory and
-more than four bytes seem not to be necessary).
-
-@cindex Unicode
-@cindex ISO 10646
-As shown in some other part of this manual,
-@c !!! Ahem, wide char string functions are not yet covered -- drepper
-a completely new family has been created of functions that can handle wide
-character texts in memory. The most commonly used character sets for such
-internal wide character representations are Unicode and @w{ISO 10646}
-(also known as UCS for Universal Character Set). Unicode was originally
-planned as a 16-bit character set; whereas, @w{ISO 10646} was designed to
-be a 31-bit large code space. The two standards are practically identical.
-They have the same character repertoire and code table, but Unicode specifies
-added semantics. At the moment, only characters in the first @code{0x10000}
-code positions (the so-called Basic Multilingual Plane, BMP) have been
-assigned, but the assignment of more specialized characters outside this
-16-bit space is already in progress. A number of encodings have been
-defined for Unicode and @w{ISO 10646} characters:
-@cindex UCS-2
-@cindex UCS-4
-@cindex UTF-8
-@cindex UTF-16
-UCS-2 is a 16-bit word that can only represent characters
-from the BMP, UCS-4 is a 32-bit word than can represent any Unicode
-and @w{ISO 10646} character, UTF-8 is an ASCII compatible encoding where
-ASCII characters are represented by ASCII bytes and non-ASCII characters
-by sequences of 2-6 non-ASCII bytes, and finally UTF-16 is an extension
-of UCS-2 in which pairs of certain UCS-2 words can be used to encode
-non-BMP characters up to @code{0x10ffff}.
-
-To represent wide characters the @code{char} type is not suitable. For
-this reason the @w{ISO C} standard introduces a new type that is
-designed to keep one character of a wide character string. To maintain
-the similarity there is also a type corresponding to @code{int} for
-those functions that take a single wide character.
-
-@comment stddef.h
-@comment ISO
-@deftp {Data type} wchar_t
-This data type is used as the base type for wide character strings.
-I.e., arrays of objects of this type are the equivalent of @code{char[]}
-for multibyte character strings. The type is defined in @file{stddef.h}.
-
-The @w{ISO C90} standard, where @code{wchar_t} was introduced, does not
-say anything specific about the representation. It only requires that
-this type is capable of storing all elements of the basic character set.
-Therefore it would be legitimate to define @code{wchar_t} as @code{char},
-which might make sense for embedded systems.
-
-But for GNU systems @code{wchar_t} is always 32 bits wide and, therefore,
-capable of representing all UCS-4 values and, therefore, covering all of
-@w{ISO 10646}. Some Unix systems define @code{wchar_t} as a 16-bit type
-and thereby follow Unicode very strictly. This definition is perfectly
-fine with the standard, but it also means that to represent all
-characters from Unicode and @w{ISO 10646} one has to use UTF-16 surrogate
-characters, which is in fact a multi-wide-character encoding. But
-resorting to multi-wide-character encoding contradicts the purpose of the
-@code{wchar_t} type.
-@end deftp
-
-@comment wchar.h
-@comment ISO
-@deftp {Data type} wint_t
-@code{wint_t} is a data type used for parameters and variables that
-contain a single wide character. As the name suggests this type is the
-equivalent of @code{int} when using the normal @code{char} strings. The
-types @code{wchar_t} and @code{wint_t} often have the same
-representation if their size is 32 bits wide but if @code{wchar_t} is
-defined as @code{char} the type @code{wint_t} must be defined as
-@code{int} due to the parameter promotion.
-
-@pindex wchar.h
-This type is defined in @file{wchar.h} and was introduced in
-@w{Amendment 1} to @w{ISO C90}.
-@end deftp
-
-As there are for the @code{char} data type macros are available for
-specifying the minimum and maximum value representable in an object of
-type @code{wchar_t}.
-
-@comment wchar.h
-@comment ISO
-@deftypevr Macro wint_t WCHAR_MIN
-The macro @code{WCHAR_MIN} evaluates to the minimum value representable
-by an object of type @code{wint_t}.
-
-This macro was introduced in @w{Amendment 1} to @w{ISO C90}.
-@end deftypevr
-
-@comment wchar.h
-@comment ISO
-@deftypevr Macro wint_t WCHAR_MAX
-The macro @code{WCHAR_MAX} evaluates to the maximum value representable
-by an object of type @code{wint_t}.
-
-This macro was introduced in @w{Amendment 1} to @w{ISO C90}.
-@end deftypevr
-
-Another special wide character value is the equivalent to @code{EOF}.
-
-@comment wchar.h
-@comment ISO
-@deftypevr Macro wint_t WEOF
-The macro @code{WEOF} evaluates to a constant expression of type
-@code{wint_t} whose value is different from any member of the extended
-character set.
-
-@code{WEOF} need not be the same value as @code{EOF} and unlike
-@code{EOF} it also need @emph{not} be negative. I.e., sloppy code like
-
-@smallexample
-@{
- int c;
- ...
- while ((c = getc (fp)) < 0)
- ...
-@}
-@end smallexample
-
-@noindent
-has to be rewritten to use @code{WEOF} explicitly when wide characters
-are used:
-
-@smallexample
-@{
- wint_t c;
- ...
- while ((c = wgetc (fp)) != WEOF)
- ...
-@}
-@end smallexample
-
-@pindex wchar.h
-This macro was introduced in @w{Amendment 1} to @w{ISO C90} and is
-defined in @file{wchar.h}.
-@end deftypevr
-
-
-These internal representations present problems when it comes to storing
-and transmittal. Because each single wide character consists of more
-than one byte, they are effected by byte-ordering. Thus, machines with
-different endianesses would see different values when accessing the same
-data. This byte ordering concern also applies for communication protocols
-that are all byte-based and, thereforet require that the sender has to
-decide about splitting the wide character in bytes. A last (but not least
-important) point is that wide characters often require more storage space
-than a customized byte-oriented character set.
-
-@cindex multibyte character
-@cindex EBCDIC
- For all the above reasons, an external encoding that is different
-from the internal encoding is often used if the latter is UCS-2 or UCS-4.
-The external encoding is byte-based and can be chosen appropriately for
-the environment and for the texts to be handled. A variety of different
-character sets can be used for this external encoding (information that
-will not be exhaustively presented here--instead, a description of the
-major groups will suffice). All of the ASCII-based character sets
-[_bkoz_: do you mean Roman character sets? If not, what do you mean
-here?] fulfill one requirement: they are "filesystem safe." This means
-that the character @code{'/'} is used in the encoding @emph{only} to
-represent itself. Things are a bit different for character sets like
-EBCDIC (Extended Binary Coded Decimal Interchange Code, a character set
-family used by IBM), but if the operation system does not understand
-EBCDIC directly the parameters-to-system calls have to be converted first
-anyhow.
-
-@itemize @bullet
-@item
-The simplest character sets are single-byte character sets. There can
-be only up to 256 characters (for @w{8 bit} character sets), which is
-not sufficient to cover all languages but might be sufficient to handle
-a specific text. Handling of a @w{8 bit} character sets is simple. This
-is not true for other kinds presented later, and therefore, the
-application one uses might require the use of @w{8 bit} character sets.
-
-@cindex ISO 2022
-@item
-The @w{ISO 2022} standard defines a mechanism for extended character
-sets where one character @emph{can} be represented by more than one
-byte. This is achieved by associating a state with the text.
-Characters that can be used to change the state can be embedded in the
-text. Each byte in the text might have a different interpretation in each
-state. The state might even influence whether a given byte stands for a
-character on its own or whether it has to be combined with some more
-bytes.
-
-@cindex EUC
-@cindex Shift_JIS
-@cindex SJIS
-In most uses of @w{ISO 2022} the defined character sets do not allow
-state changes which cover more than the next character. This has the
-big advantage that whenever one can identify the beginning of the byte
-sequence of a character one can interpret a text correctly. Examples of
-character sets using this policy are the various EUC character sets
-(used by Sun's operations systems, EUC-JP, EUC-KR, EUC-TW, and EUC-CN)
-or Shift_JIS (SJIS, a Japanese encoding).
-
-But there are also character sets using a state which is valid for more
-than one character and has to be changed by another byte sequence.
-Examples for this are ISO-2022-JP, ISO-2022-KR, and ISO-2022-CN.
-
-@item
-@cindex ISO 6937
-Early attempts to fix 8 bit character sets for other languages using the
-Roman alphabet lead to character sets like @w{ISO 6937}. Here bytes
-representing characters like the acute accent do not produce output
-themselves: one has to combine them with other characters to get the
-desired result. For example, the byte sequence @code{0xc2 0x61}
-(non-spacing acute accent, followed by lower-case `a') to get the ``small
-a with acute'' character. To get the acute accent character on its own,
-one has to write @code{0xc2 0x20} (the non-spacing acute followed by a
-space).
-
-Character sets like @w[ISO 6937] are used in some embedded systems such
-as teletex.
-
-@item
-@cindex UTF-8
-Instead of converting the Unicode or @w{ISO 10646} text used internally,
-it is often also sufficient to simply use an encoding different than
-UCS-2/UCS-4. The Unicode and @w{ISO 10646} standards even specify such an
-encoding: UTF-8. This encoding is able to represent all of @w{ISO
-10646} 31 bits in a byte string of length one to six.
-
-@cindex UTF-7
-There were a few other attempts to encode @w{ISO 10646} such as UTF-7,
-but UTF-8 is today the only encoding which should be used. In fact, with
-any luck UTF-8 will soon be the only external encoding that has to be
-supported. It proves to be universally usable and its only disadvantage
-is that it favors Roman languages by making the byte string
-representation of other scripts (Cyrillic, Greek, Asian scripts) longer
-than necessary if using a specific character set for these scripts.
-Methods like the Unicode compression scheme can alleviate these
-problems.
-@end itemize
-
-The question remaining is: how to select the character set or encoding
-to use. The answer: you cannot decide about it yourself, it is decided
-by the developers of the system or the majority of the users. Since the
-goal is interoperability one has to use whatever the other people one
-works with use. If there are no constraints, the selection is based on
-the requirements the expected circle of users will have. In other words,
-if a project is expected to be used in only, say, Russia it is fine to use
-KOI8-R or a similar character set. But if at the same time people from,
-say, Greece are participating one should use a character set which allows
-all people to collaborate.
-
-The most widely useful solution seems to be: go with the most general
-character set, namely @w{ISO 10646}. Use UTF-8 as the external encoding
-and problems about users not being able to use their own language
-adequately are a thing of the past.
-
-One final comment about the choice of the wide character representation
-is necessary at this point. We have said above that the natural choice
-is using Unicode or @w{ISO 10646}. This is not required, but at least
-encouraged, by the @w{ISO C} standard. The standard defines at least a
-macro @code{__STDC_ISO_10646__} that is only defined on systems where
-the @code{wchar_t} type encodes @w{ISO 10646} characters. If this
-symbol is not defined one should avoid making assumptions about the wide
-character representation. If the programmer uses only the functions
-provided by the C library to handle wide character strings there should
-be no compatibility problems with other systems.
-
-@node Charset Function Overview
-@section Overview about Character Handling Functions
-
-A Unix @w{C library} contains three different sets of functions in two
-families to handle character set conversion. One of the function families
-(the most commonly used) is specified in the @w{ISO C90} standard and,
-therefore, is portable even beyond the Unix world. Unfortunately this
-family is the least useful one. These functions should be avoided
-whenever possible, especially when developing libraries (as opposed to
-applications).
-
-The second family of functions got introduced in the early Unix standards
-(XPG2) and is still part of the latest and greatest Unix standard:
-@w{Unix 98}. It is also the most powerful and useful set of functions.
-But we will start with the functions defined in @w{Amendment 1} to
-@w{ISO C90}.
-
-@node Restartable multibyte conversion
-@section Restartable Multibyte Conversion Functions
-
-The @w{ISO C} standard defines functions to convert strings from a
-multibyte representation to wide character strings. There are a number
-of peculiarities:
-
-@itemize @bullet
-@item
-The character set assumed for the multibyte encoding is not specified
-as an argument to the functions. Instead the character set specified by
-the @code{LC_CTYPE} category of the current locale is used; see
-@ref{Locale Categories}.
-
-@item
-The functions handling more than one character at a time require NUL
-terminated strings as the argument. I.e., converting blocks of text
-does not work unless one can add a NUL byte at an appropriate place.
-The GNU C library contains some extensions to the standard that allow
-specifying a size, but basically they also expect terminated strings.
-@end itemize
-
-Despite these limitations the @w{ISO C} functions can be used in many
-contexts. In graphical user interfaces, for instance, it is not
-uncommon to have functions that require text to be displayed in a wide
-character string if the text is not simple ASCII. The text itself might come
-from a file with translations and the user should decide about the
-current locale which determines the translation and therefore also the
-external encoding used. In such a situation (and many others) the
-functions described here are perfect. If more freedom while performing
-the conversion is necessary take a look at the @code{iconv} functions
-(@pxref{Generic Charset Conversion}).
-
-@menu
-* Selecting the Conversion:: Selecting the conversion and its properties.
-* Keeping the state:: Representing the state of the conversion.
-* Converting a Character:: Converting Single Characters.
-* Converting Strings:: Converting Multibyte and Wide Character
- Strings.
-* Multibyte Conversion Example:: A Complete Multibyte Conversion Example.
-@end menu
-
-@node Selecting the Conversion
-@subsection Selecting the conversion and its properties
-
-We already said above that the currently selected locale for the
-@code{LC_CTYPE} category decides about the conversion which is performed
-by the functions we are about to describe. Each locale uses its own
-character set (given as an argument to @code{localedef}) and this is the
-one assumed as the external multibyte encoding. The wide character
-character set always is UCS-4, at least on GNU systems.
-
-A characteristic of each multibyte character set is the maximum number
-of bytes that can be necessary to represent one character. This
-information is quite important when writing code that uses the
-conversion functions (as shown in the examples below).
-The @w{ISO C} standard defines two macros which provide this information.
-
-
-@comment limits.h
-@comment ISO
-@deftypevr Macro int MB_LEN_MAX
-@code{MB_LEN_MAX} specifies the maximum number of bytes in the multibyte
-sequence for a single character in any of the supported locales. It is
-a compile-time constant and is defined in @file{limits.h}.
-@pindex limits.h
-@end deftypevr
-
-@comment stdlib.h
-@comment ISO
-@deftypevr Macro int MB_CUR_MAX
-@code{MB_CUR_MAX} expands into a positive integer expression that is the
-maximum number of bytes in a multibyte character in the current locale.
-The value is never greater than @code{MB_LEN_MAX}. Unlike
-@code{MB_LEN_MAX} this macro need not be a compile-time constant, and in
-the GNU C library it is not.
-
-@pindex stdlib.h
-@code{MB_CUR_MAX} is defined in @file{stdlib.h}.
-@end deftypevr
-
-Two different macros are necessary since strictly @w{ISO C90} compilers
-do not allow variable length array definitions, but still it is desirable
-to avoid dynamic allocation. This incomplete piece of code shows the
-problem:
-
-@smallexample
-@{
- char buf[MB_LEN_MAX];
- ssize_t len = 0;
-
- while (! feof (fp))
- @{
- fread (&buf[len], 1, MB_CUR_MAX - len, fp);
- /* @r{... process} buf */
- len -= used;
- @}
-@}
-@end smallexample
-
-The code in the inner loop is expected to have always enough bytes in
-the array @var{buf} to convert one multibyte character. The array
-@var{buf} has to be sized statically since many compilers do not allow a
-variable size. The @code{fread} call makes sure that @code{MB_CUR_MAX}
-bytes are always available in @var{buf}. Note that it isn't
-a problem if @code{MB_CUR_MAX} is not a compile-time constant.
-
-
-@node Keeping the state
-@subsection Representing the state of the conversion
-
-@cindex stateful
-In the introduction of this chapter it was said that certain character
-sets use a @dfn{stateful} encoding. That is, the encoded values depend
-in some way on the previous bytes in the text.
-
-Since the conversion functions allow converting a text in more than one
-step we must have a way to pass this information from one call of the
-functions to another.
-
-@comment wchar.h
-@comment ISO
-@deftp {Data type} mbstate_t
-@cindex shift state
-A variable of type @code{mbstate_t} can contain all the information
-about the @dfn{shift state} needed from one call to a conversion
-function to another.
-
-@pindex wchar.h
-@code{mbstate_t} is defined in @file{wchar.h}. It was introduced in
-@w{Amendment 1} to @w{ISO C90}.
-@end deftp
-
-To use objects of type @code{mbstate_t} the programmer has to define such
-objects (normally as local variables on the stack) and pass a pointer to
-the object to the conversion functions. This way the conversion function
-can update the object if the current multibyte character set is stateful.
-
-There is no specific function or initializer to put the state object in
-any specific state. The rules are that the object should always
-represent the initial state before the first use, and this is achieved by
-clearing the whole variable with code such as follows:
-
-@smallexample
-@{
- mbstate_t state;
- memset (&state, '\0', sizeof (state));
- /* @r{from now on @var{state} can be used.} */
- ...
-@}
-@end smallexample
-
-When using the conversion functions to generate output it is often
-necessary to test whether the current state corresponds to the initial
-state. This is necessary, for example, to decide whether to emit
-escape sequences to set the state to the initial state at certain
-sequence points. Communication protocols often require this.
-
-@comment wchar.h
-@comment ISO
-@deftypefun int mbsinit (const mbstate_t *@var{ps})
-The @code {mbsinit} function determines whether the state object pointed
-to by @var{ps} is in the initial state. If @var{ps} is a null pointer or
-the object is in the initial state the return value is nonzero. Otherwise
-it is zero.
-
-@pindex wchar.h
-@code {mbsinit} was introduced in @w{Amendment 1} to @w{ISO C90} and is
-declared in @file{wchar.h}.
-@end deftypefun
-
-Code using @code {mbsinit} often looks similar to this:
-
-@c Fix the example to explicitly say how to generate the escape sequence
-@c to restore the initial state.
-@smallexample
-@{
- mbstate_t state;
- memset (&state, '\0', sizeof (state));
- /* @r{Use @var{state}.} */
- ...
- if (! mbsinit (&state))
- @{
- /* @r{Emit code to return to initial state.} */
- const wchar_t empty[] = L"";
- const wchar_t *srcp = empty;
- wcsrtombs (outbuf, &srcp, outbuflen, &state);
- @}
- ...
-@}
-@end smallexample
-
-The code to emit the escape sequence to get back to the initial state is
-interesting. The @code{wcsrtombs} function can be used to determine the
-necessary output code (@pxref{Converting Strings}). Please note that on
-GNU systems it is not necessary to perform this extra action for the
-conversion from multibyte text to wide character text since the wide
-character encoding is not stateful. But there is nothing mentioned in
-any standard which prohibits making @code{wchar_t} using a stateful
-encoding.
-
-@node Converting a Character
-@subsection Converting Single Characters
-
-The most fundamental of the conversion functions are those dealing with
-single characters. Please note that this does not always mean single
-bytes. But since there is very often a subset of the multibyte
-character set which consists of single byte sequences there are
-functions to help with converting bytes. Frequently, ASCII is a subpart
-of the multibyte character set. In such a scenario, each ASCII character
-stands for itself, and all other characters have at least a first byte
-that is beyond the range @math{0} to @math{127}.
-
-@comment wchar.h
-@comment ISO
-@deftypefun wint_t btowc (int @var{c})
-The @code{btowc} function (``byte to wide character'') converts a valid
-single byte character @var{c} in the initial shift state into the wide
-character equivalent using the conversion rules from the currently
-selected locale of the @code{LC_CTYPE} category.
-
-If @code{(unsigned char) @var{c}} is no valid single byte multibyte
-character or if @var{c} is @code{EOF}, the function returns @code{WEOF}.
-
-Please note the restriction of @var{c} being tested for validity only in
-the initial shift state. No @code{mbstate_t} object is used from
-which the state information is taken, and the function also does not use
-any static state.
-
-@pindex wchar.h
-The @code{btowc} function was introduced in @w{Amendment 1} to @w{ISO C90}
-and is declared in @file{wchar.h}.
-@end deftypefun
-
-Despite the limitation that the single byte value always is interpreted
-in the initial state this function is actually useful most of the time.
-Most characters are either entirely single-byte character sets or they
-are extension to ASCII. But then it is possible to write code like this
-(not that this specific example is very useful):
-
-@smallexample
-wchar_t *
-itow (unsigned long int val)
-@{
- static wchar_t buf[30];
- wchar_t *wcp = &buf[29];
- *wcp = L'\0';
- while (val != 0)
- @{
- *--wcp = btowc ('0' + val % 10);
- val /= 10;
- @}
- if (wcp == &buf[29])
- *--wcp = L'0';
- return wcp;
-@}
-@end smallexample
-
-Why is it necessary to use such a complicated implementation and not
-simply cast @code{'0' + val % 10} to a wide character? The answer is
-that there is no guarantee that one can perform this kind of arithmetic
-on the character of the character set used for @code{wchar_t}
-representation. In other situations the bytes are not constant at
-compile time and so the compiler cannot do the work. In situations like
-this it is necessary @code{btowc}.
-
-@noindent
-There also is a function for the conversion in the other direction.
-
-@comment wchar.h
-@comment ISO
-@deftypefun int wctob (wint_t @var{c})
-The @code{wctob} function (``wide character to byte'') takes as the
-parameter a valid wide character. If the multibyte representation for
-this character in the initial state is exactly one byte long the return
-value of this function is this character. Otherwise the return value is
-@code{EOF}.
-
-@pindex wchar.h
-@code{wctob} was introduced in @w{Amendment 1} to @w{ISO C90} and
-is declared in @file{wchar.h}.
-@end deftypefun
-
-There are more general functions to convert single character from
-multibyte representation to wide characters and vice versa. These
-functions pose no limit on the length of the multibyte representation
-and they also do not require it to be in the initial state.
-
-@comment wchar.h
-@comment ISO
-@deftypefun size_t mbrtowc (wchar_t *restrict @var{pwc}, const char *restrict @var{s}, size_t @var{n}, mbstate_t *restrict @var{ps})
-@cindex stateful
-The @code{mbrtowc} function (``multibyte restartable to wide
-character'') converts the next multibyte character in the string pointed
-to by @var{s} into a wide character and stores it in the wide character
-string pointed to by @var{pwc}. The conversion is performed according
-to the locale currently selected for the @code{LC_CTYPE} category. If
-the conversion for the character set used in the locale requires a state,
-the multibyte string is interpreted in the state represented by the
-object pointed to by @var{ps}. If @var{ps} is a null pointer, a static,
-internal state variable used only by the @code{mbrtowc} function is
-used.
-
-If the next multibyte character corresponds to the NUL wide character,
-the return value of the function is @math{0} and the state object is
-afterwards in the initial state. If the next @var{n} or fewer bytes
-form a correct multibyte character, the return value is the number of
-bytes starting from @var{s} that form the multibyte character. The
-conversion state is updated according to the bytes consumed in the
-conversion. In both cases the wide character (either the @code{L'\0'}
-or the one found in the conversion) is stored in the string pointed to
-by @var{pwc} if @var{pwc} is not null.
-
-If the first @var{n} bytes of the multibyte string possibly form a valid
-multibyte character but there are more than @var{n} bytes needed to
-complete it, the return value of the function is @code{(size_t) -2} and
-no value is stored. Please note that this can happen even if @var{n}
-has a value greater than or equal to @code{MB_CUR_MAX} since the input
-might contain redundant shift sequences.
-
-If the first @code{n} bytes of the multibyte string cannot possibly form
-a valid multibyte character, no value is stored, the global variable
-@code{errno} is set to the value @code{EILSEQ}, and the function returns
-@code{(size_t) -1}. The conversion state is afterwards undefined.
-
-@pindex wchar.h
-@code{mbrtowc} was introduced in @w{Amendment 1} to @w{ISO C90} and
-is declared in @file{wchar.h}.
-@end deftypefun
-
-Use of @code{mbrtowc} is straightforward. A function which copies a
-multibyte string into a wide character string while at the same time
-converting all lowercase characters into uppercase could look like this
-(this is not the final version, just an example; it has no error
-checking, and sometimes leaks memory):
-
-@smallexample
-wchar_t *
-mbstouwcs (const char *s)
-@{
- size_t len = strlen (s);
- wchar_t *result = malloc ((len + 1) * sizeof (wchar_t));
- wchar_t *wcp = result;
- wchar_t tmp[1];
- mbstate_t state;
- size_t nbytes;
-
- memset (&state, '\0', sizeof (state));
- while ((nbytes = mbrtowc (tmp, s, len, &state)) > 0)
- @{
- if (nbytes >= (size_t) -2)
- /* Invalid input string. */
- return NULL;
- *result++ = towupper (tmp[0]);
- len -= nbytes;
- s += nbytes;
- @}
- return result;
-@}
-@end smallexample
-
-The use of @code{mbrtowc} should be clear. A single wide character is
-stored in @code{@var{tmp}[0]}, and the number of consumed bytes is stored
-in the variable @var{nbytes}. If the conversion is successful, the
-uppercase variant of the wide character is stored in the @var{result}
-array and the pointer to the input string and the number of available
-bytes is adjusted.
-
-The only non-obvious thing about @code{mbrtowc} might be the way memory
-is allocated for the result. The above code uses the fact that there
-can never be more wide characters in the converted results than there are
-bytes in the multibyte input string. This method yields a pessimistic
-guess about the size of the result, and if many wide character strings
-have to be constructed this way or if the strings are long, the extra
-memory required to be allocated because the input string contains
-multibyte characters might be significant. The allocated memory block can
-be resized to the correct size before returning it, but a better solution
-might be to allocate just the right amount of space for the result right
-away. Unfortunately there is no function to compute the length of the wide
-character string directly from the multibyte string. There is, however, a
-function which does part of the work.
-
-@comment wchar.h
-@comment ISO
-@deftypefun size_t mbrlen (const char *restrict @var{s}, size_t @var{n}, mbstate_t *@var{ps})
-The @code{mbrlen} function (``multibyte restartable length'') computes
-the number of at most @var{n} bytes starting at @var{s} which form the
-next valid and complete multibyte character.
-
-If the next multibyte character corresponds to the NUL wide character,
-the return value is @math{0}. If the next @var{n} bytes form a valid
-multibyte character, the number of bytes belonging to this multibyte
-character byte sequence is returned.
-
-If the the first @var{n} bytes possibly form a valid multibyte
-character but the character is incomplete, the return value is
-@code{(size_t) -2}. Otherwise the multibyte character sequence is invalid
-and the return value is @code{(size_t) -1}.
-
-The multibyte sequence is interpreted in the state represented by the
-object pointed to by @var{ps}. If @var{ps} is a null pointer, a state
-object local to @code{mbrlen} is used.
-
-@pindex wchar.h
-@code{mbrlen} was introduced in @w{Amendment 1} to @w{ISO C90} and
-is declared in @file{wchar.h}.
-@end deftypefun
-
-The attentive reader now will note that @code{mbrlen} can be implemented
-as
-
-@smallexample
-mbrtowc (NULL, s, n, ps != NULL ? ps : &internal)
-@end smallexample
-
-This is true and in fact is mentioned in the official specification.
-How can this function be used to determine the length of the wide
-character string created from a multibyte character string? It is not
-directly usable, but we can define a function @code{mbslen} using it:
-
-@smallexample
-size_t
-mbslen (const char *s)
-@{
- mbstate_t state;
- size_t result = 0;
- size_t nbytes;
- memset (&state, '\0', sizeof (state));
- while ((nbytes = mbrlen (s, MB_LEN_MAX, &state)) > 0)
- @{
- if (nbytes >= (size_t) -2)
- /* @r{Something is wrong.} */
- return (size_t) -1;
- s += nbytes;
- ++result;
- @}
- return result;
-@}
-@end smallexample
-
-This function simply calls @code{mbrlen} for each multibyte character
-in the string and counts the number of function calls. Please note that
-we here use @code{MB_LEN_MAX} as the size argument in the @code{mbrlen}
-call. This is acceptable since a) this value is larger then the length of
-the longest multibyte character sequence and b) we know that the string
-@var{s} ends with a NUL byte, which cannot be part of any other multibyte
-character sequence but the one representing the NUL wide character.
-Therefore, the @code{mbrlen} function will never read invalid memory.
-
-Now that this function is available (just to make this clear, this
-function is @emph{not} part of the GNU C library) we can compute the
-number of wide character required to store the converted multibyte
-character string @var{s} using
-
-@smallexample
-wcs_bytes = (mbslen (s) + 1) * sizeof (wchar_t);
-@end smallexample
-
-Please note that the @code{mbslen} function is quite inefficient. The
-implementation of @code{mbstouwcs} with @code{mbslen} would have to
-perform the conversion of the multibyte character input string twice, and
-this conversion might be quite expensive. So it is necessary to think
-about the consequences of using the easier but imprecise method before
-doing the work twice.
-
-@comment wchar.h
-@comment ISO
-@deftypefun size_t wcrtomb (char *restrict @var{s}, wchar_t @var{wc}, mbstate_t *restrict @var{ps})
-The @code{wcrtomb} function (``wide character restartable to
-multibyte'') converts a single wide character into a multibyte string
-corresponding to that wide character.
-
-If @var{s} is a null pointer, the function resets the state stored in
-the objects pointed to by @var{ps} (or the internal @code{mbstate_t}
-object) to the initial state. This can also be achieved by a call like
-this:
-
-@smallexample
-wcrtombs (temp_buf, L'\0', ps)
-@end smallexample
-
-@noindent
-since, if @var{s} is a null pointer, @code{wcrtomb} performs as if it
-writes into an internal buffer, which is guaranteed to be large enough.
-
-If @var{wc} is the NUL wide character, @code{wcrtomb} emits, if
-necessary, a shift sequence to get the state @var{ps} into the initial
-state followed by a single NUL byte, which is stored in the string
-@var{s}.
-
-Otherwise a byte sequence (possibly including shift sequences) is written
-into the string @var{s}. This only happens if @var{wc} is a valid wide
-character (i.e., it has a multibyte representation in the character set
-selected by locale of the @code{LC_CTYPE} category). If @var{wc} is no
-valid wide character, nothing is stored in the strings @var{s},
-@code{errno} is set to @code{EILSEQ}, the conversion state in @var{ps}
-is undefined and the return value is @code{(size_t) -1}.
-
-If no error occurred the function returns the number of bytes stored in
-the string @var{s}. This includes all bytes representing shift
-sequences.
-
-One word about the interface of the function: there is no parameter
-specifying the length of the array @var{s}. Instead the function
-assumes that there are at least @code{MB_CUR_MAX} bytes available since
-this is the maximum length of any byte sequence representing a single
-character. So the caller has to make sure that there is enough space
-available, otherwise buffer overruns can occur.
-
-@pindex wchar.h
-@code{wcrtomb} was introduced in @w{Amendment 1} to @w{ISO C90} and is
-declared in @file{wchar.h}.
-@end deftypefun
-
-Using @code{wcrtomb} is as easy as using @code{mbrtowc}. The following
-example appends a wide character string to a multibyte character string.
-Again, the code is not really useful (or correct), it is simply here to
-demonstrate the use and some problems.
-
-@smallexample
-char *
-mbscatwcs (char *s, size_t len, const wchar_t *ws)
-@{
- mbstate_t state;
- /* @r{Find the end of the existing string.} */
- char *wp = strchr (s, '\0');
- len -= wp - s;
- memset (&state, '\0', sizeof (state));
- do
- @{
- size_t nbytes;
- if (len < MB_CUR_LEN)
- @{
- /* @r{We cannot guarantee that the next}
- @r{character fits into the buffer, so}
- @r{return an error.} */
- errno = E2BIG;
- return NULL;
- @}
- nbytes = wcrtomb (wp, *ws, &state);
- if (nbytes == (size_t) -1)
- /* @r{Error in the conversion.} */
- return NULL;
- len -= nbytes;
- wp += nbytes;
- @}
- while (*ws++ != L'\0');
- return s;
-@}
-@end smallexample
-
-First the function has to find the end of the string currently in the
-array @var{s}. The @code{strchr} call does this very efficiently since a
-requirement for multibyte character representations is that the NUL byte
-is never used except to represent itself (and in this context, the end
-of the string).
-
-After initializing the state object the loop is entered where the first
-task is to make sure there is enough room in the array @var{s}. We
-abort if there are not at least @code{MB_CUR_LEN} bytes available. This
-is not always optimal but we have no other choice. We might have less
-than @code{MB_CUR_LEN} bytes available but the next multibyte character
-might also be only one byte long. At the time the @code{wcrtomb} call
-returns it is too late to decide whether the buffer was large enough. If
-this solution is unsuitable, there is a very slow but more accurate
-solution.
-
-@smallexample
- ...
- if (len < MB_CUR_LEN)
- @{
- mbstate_t temp_state;
- memcpy (&temp_state, &state, sizeof (state));
- if (wcrtomb (NULL, *ws, &temp_state) > len)
- @{
- /* @r{We cannot guarantee that the next}
- @r{character fits into the buffer, so}
- @r{return an error.} */
- errno = E2BIG;
- return NULL;
- @}
- @}
- ...
-@end smallexample
-
-Here we perform the conversion that might overflow the buffer so that
-we are afterwards in the position to make an exact decision about the
-buffer size. Please note the @code{NULL} argument for the destination
-buffer in the new @code{wcrtomb} call; since we are not interested in the
-converted text at this point, this is a nice way to express this. The
-most unusual thing about this piece of code certainly is the duplication
-of the conversion state object, but if a change of the state is necessary
-to emit the next multibyte character, we want to have the same shift state
-change performed in the real conversion. Therefore, we have to preserve
-the initial shift state information.
-
-There are certainly many more and even better solutions to this problem.
-This example is only provided for educational purposes.
-
-@node Converting Strings
-@subsection Converting Multibyte and Wide Character Strings
-
-The functions described in the previous section only convert a single
-character at a time. Most operations to be performed in real-world
-programs include strings and therefore the @w{ISO C} standard also
-defines conversions on entire strings. However, the defined set of
-functions is quite limited; therefore, the GNU C library contains a few
-extensions which can help in some important situations.
-
-@comment wchar.h
-@comment ISO
-@deftypefun size_t mbsrtowcs (wchar_t *restrict @var{dst}, const char **restrict @var{src}, size_t @var{len}, mbstate_t *restrict @var{ps})
-The @code{mbsrtowcs} function (``multibyte string restartable to wide
-character string'') converts an NUL-terminated multibyte character
-string at @code{*@var{src}} into an equivalent wide character string,
-including the NUL wide character at the end. The conversion is started
-using the state information from the object pointed to by @var{ps} or
-from an internal object of @code{mbsrtowcs} if @var{ps} is a null
-pointer. Before returning, the state object is updated to match the state
-after the last converted character. The state is the initial state if the
-terminating NUL byte is reached and converted.
-
-If @var{dst} is not a null pointer, the result is stored in the array
-pointed to by @var{dst}; otherwise, the conversion result is not
-available since it is stored in an internal buffer.
-
-If @var{len} wide characters are stored in the array @var{dst} before
-reaching the end of the input string, the conversion stops and @var{len}
-is returned. If @var{dst} is a null pointer, @var{len} is never checked.
-
-Another reason for a premature return from the function call is if the
-input string contains an invalid multibyte sequence. In this case the
-global variable @code{errno} is set to @code{EILSEQ} and the function
-returns @code{(size_t) -1}.
-
-@c XXX The ISO C9x draft seems to have a problem here. It says that PS
-@c is not updated if DST is NULL. This is not said straightforward and
-@c none of the other functions is described like this. It would make sense
-@c to define the function this way but I don't think it is meant like this.
-
-In all other cases the function returns the number of wide characters
-converted during this call. If @var{dst} is not null, @code{mbsrtowcs}
-stores in the pointer pointed to by @var{src} either a null pointer (if
-the NUL byte in the input string was reached) or the address of the byte
-following the last converted multibyte character.
-
-@pindex wchar.h
-@code{mbsrtowcs} was introduced in @w{Amendment 1} to @w{ISO C90} and is
-declared in @file{wchar.h}.
-@end deftypefun
-
-The definition of the @code{mbsrtowcs} function has one important
-limitation. The requirement that @var{dst} has to be a NUL-terminated
-string provides problems if one wants to convert buffers with text. A
-buffer is normally no collection of NUL-terminated strings but instead a
-continuous collection of lines, separated by newline characters. Now
-assume that a function to convert one line from a buffer is needed. Since
-the line is not NUL-terminated the source pointer cannot directly point
-into the unmodified text buffer. This means, either one inserts the NUL
-byte at the appropriate place for the time of the @code{mbsrtowcs}
-function call (which is not doable for a read-only buffer or in a
-multi-threaded application) or one copies the line in an extra buffer
-where it can be terminated by a NUL byte. Note that it is not in general
-possible to limit the number of characters to convert by setting the
-parameter @var{len} to any specific value. Since it is not known how
-many bytes each multibyte character sequence is in length, one can only
-guess.
-
-@cindex stateful
-There is still a problem with the method of NUL-terminating a line right
-after the newline character which could lead to very strange results.
-As said in the description of the @code{mbsrtowcs} function above the
-conversion state is guaranteed to be in the initial shift state after
-processing the NUL byte at the end of the input string. But this NUL
-byte is not really part of the text. I.e., the conversion state after
-the newline in the original text could be something different than the
-initial shift state and therefore the first character of the next line
-is encoded using this state. But the state in question is never
-accessible to the user since the conversion stops after the NUL byte
-(which resets the state). Most stateful character sets in use today
-require that the shift state after a newline be the initial state--but
-this is not a strict guarantee. Therefore, simply NUL-terminating a
-piece of a running text is not always an adequate solution and,
-therefore, should never be used in generally used code.
-
-The generic conversion interface (@pxref{Generic Charset Conversion})
-does not have this limitation (it simply works on buffers, not
-strings), and the GNU C library contains a set of functions which take
-additional parameters specifying the maximal number of bytes which are
-consumed from the input string. This way the problem of
-@code{mbsrtowcs}'s example above could be solved by determining the line
-length and passing this length to the function.
-
-@comment wchar.h
-@comment ISO
-@deftypefun size_t wcsrtombs (char *restrict @var{dst}, const wchar_t **restrict @var{src}, size_t @var{len}, mbstate_t *restrict @var{ps})
-The @code{wcsrtombs} function (``wide character string restartable to
-multibyte string'') converts the NUL-terminated wide character string at
-@code{*@var{src}} into an equivalent multibyte character string and
-stores the result in the array pointed to by @var{dst}. The NUL wide
-character is also converted. The conversion starts in the state
-described in the object pointed to by @var{ps} or by a state object
-locally to @code{wcsrtombs} in case @var{ps} is a null pointer. If
-@var{dst} is a null pointer, the conversion is performed as usual but the
-result is not available. If all characters of the input string were
-successfully converted and if @var{dst} is not a null pointer, the
-pointer pointed to by @var{src} gets assigned a null pointer.
-
-If one of the wide characters in the input string has no valid multibyte
-character equivalent, the conversion stops early, sets the global
-variable @code{errno} to @code{EILSEQ}, and returns @code{(size_t) -1}.
-
-Another reason for a premature stop is if @var{dst} is not a null
-pointer and the next converted character would require more than
-@var{len} bytes in total to the array @var{dst}. In this case (and if
-@var{dest} is not a null pointer) the pointer pointed to by @var{src} is
-assigned a value pointing to the wide character right after the last one
-successfully converted.
-
-Except in the case of an encoding error the return value of the
-@code{wcsrtombs} function is the number of bytes in all the multibyte
-character sequences stored in @var{dst}. Before returning the state in
-the object pointed to by @var{ps} (or the internal object in case
-@var{ps} is a null pointer) is updated to reflect the state after the
-last conversion. The state is the initial shift state in case the
-terminating NUL wide character was converted.
-
-@pindex wchar.h
-The @code{wcsrtombs} function was introduced in @w{Amendment 1} to
-@w{ISO C90} and is declared in @file{wchar.h}.
-@end deftypefun
-
-The restriction mentioned above for the @code{mbsrtowcs} function applies
-here also. There is no possibility of directly controlling the number of
-input characters. One has to place the NUL wide character at the correct
-place or control the consumed input indirectly via the available output
-array size (the @var{len} parameter).
-
-@comment wchar.h
-@comment GNU
-@deftypefun size_t mbsnrtowcs (wchar_t *restrict @var{dst}, const char **restrict @var{src}, size_t @var{nmc}, size_t @var{len}, mbstate_t *restrict @var{ps})
-The @code{mbsnrtowcs} function is very similar to the @code{mbsrtowcs}
-function. All the parameters are the same except for @var{nmc} which is
-new. The return value is the same as for @code{mbsrtowcs}.
-
-This new parameter specifies how many bytes at most can be used from the
-multibyte character string. In other words, the multibyte character
-string @code{*@var{src}} need not be NUL-terminated. But if a NUL byte is
-found within the @var{nmc} first bytes of the string, the conversion
-stops here.
-
-This function is a GNU extension. It is meant to work around the
-problems mentioned above. Now it is possible to convert a buffer with
-multibyte character text piece for piece without having to care about
-inserting NUL bytes and the effect of NUL bytes on the conversion state.
-@end deftypefun
-
-A function to convert a multibyte string into a wide character string
-and display it could be written like this (this is not a really useful
-example):
-
-@smallexample
-void
-showmbs (const char *src, FILE *fp)
-@{
- mbstate_t state;
- int cnt = 0;
- memset (&state, '\0', sizeof (state));
- while (1)
- @{
- wchar_t linebuf[100];
- const char *endp = strchr (src, '\n');
- size_t n;
-
- /* @r{Exit if there is no more line.} */
- if (endp == NULL)
- break;
-
- n = mbsnrtowcs (linebuf, &src, endp - src, 99, &state);
- linebuf[n] = L'\0';
- fprintf (fp, "line %d: \"%S\"\n", linebuf);
- @}
-@}
-@end smallexample
-
-There is no problem with the state after a call to @code{mbsnrtowcs}.
-Since we don't insert characters in the strings which were not in there
-right from the beginning and we use @var{state} only for the conversion
-of the given buffer, there is no problem with altering the state.
-
-@comment wchar.h
-@comment GNU
-@deftypefun size_t wcsnrtombs (char *restrict @var{dst}, const wchar_t **restrict @var{src}, size_t @var{nwc}, size_t @var{len}, mbstate_t *restrict @var{ps})
-The @code{wcsnrtombs} function implements the conversion from wide
-character strings to multibyte character strings. It is similar to
-@code{wcsrtombs} but, just like @code{mbsnrtowcs}, it takes an extra
-parameter, which specifies the length of the input string.
-
-No more than @var{nwc} wide characters from the input string
-@code{*@var{src}} are converted. If the input string contains a NUL
-wide character in the first @var{nwc} characters, the conversion stops at
-this place.
-
-The @code{wcsnrtombs} function is a GNU extension and just like
-@code{mbsnrtowcs} helps in situations where no NUL-terminated input
-strings are available.
-@end deftypefun
-
-
-@node Multibyte Conversion Example
-@subsection A Complete Multibyte Conversion Example
-
-The example programs given in the last sections are only brief and do
-not contain all the error checking etc. Presented here is a complete
-and documented example. It features the @code{mbrtowc} function but it
-should be easy to derive versions using the other functions.
-
-@smallexample
-int
-file_mbsrtowcs (int input, int output)
-@{
- /* @r{Note the use of @code{MB_LEN_MAX}.}
- @r{@code{MB_CUR_MAX} cannot portably be used here.} */
- char buffer[BUFSIZ + MB_LEN_MAX];
- mbstate_t state;
- int filled = 0;
- int eof = 0;
-
- /* @r{Initialize the state.} */
- memset (&state, '\0', sizeof (state));
-
- while (!eof)
- @{
- ssize_t nread;
- ssize_t nwrite;
- char *inp = buffer;
- wchar_t outbuf[BUFSIZ];
- wchar_t *outp = outbuf;
-
- /* @r{Fill up the buffer from the input file.} */
- nread = read (input, buffer + filled, BUFSIZ);
- if (nread < 0)
- @{
- perror ("read");
- return 0;
- @}
- /* @r{If we reach end of file, make a note to read no more.} */
- if (nread == 0)
- eof = 1;
-
- /* @r{@code{filled} is now the number of bytes in @code{buffer}.} */
- filled += nread;
-
- /* @r{Convert those bytes to wide characters--as many as we can.} */
- while (1)
- @{
- size_t thislen = mbrtowc (outp, inp, filled, &state);
- /* @r{Stop converting at invalid character;}
- @r{this can mean we have read just the first part}
- @r{of a valid character.} */
- if (thislen == (size_t) -1)
- break;
- /* @r{We want to handle embedded NUL bytes}
- @r{but the return value is 0. Correct this.} */
- if (thislen == 0)
- thislen = 1;
- /* @r{Advance past this character.} */
- inp += thislen;
- filled -= thislen;
- ++outp;
- @}
-
- /* @r{Write the wide characters we just made.} */
- nwrite = write (output, outbuf,
- (outp - outbuf) * sizeof (wchar_t));
- if (nwrite < 0)
- @{
- perror ("write");
- return 0;
- @}
-
- /* @r{See if we have a @emph{real} invalid character.} */
- if ((eof && filled > 0) || filled >= MB_CUR_MAX)
- @{
- error (0, 0, "invalid multibyte character");
- return 0;
- @}
-
- /* @r{If any characters must be carried forward,}
- @r{put them at the beginning of @code{buffer}.} */
- if (filled > 0)
- memmove (inp, buffer, filled);
- @}
-
- return 1;
-@}
-@end smallexample
-
-
-@node Non-reentrant Conversion
-@section Non-reentrant Conversion Function
-
-The functions described in the previous chapter are defined in
-@w{Amendment 1} to @w{ISO C90}, but the original @w{ISO C90} standard
-also contained functions for character set conversion. The reason that
-these original functions are not described first is that they are almost
-entirely useless.
-
-The problem is that all the conversion functions described in the
-original @w{ISO C90} use a local state. Using a local state implies that
-multiple conversions at the same time (not only when using threads)
-cannot be done, and that you cannot first convert single characters and
-then strings since you cannot tell the conversion functions which state
-to use.
-
-These original functions are therefore usable only in a very limited set
-of situations. One must complete converting the entire string before
-starting a new one, and each string/text must be converted with the same
-function (there is no problem with the library itself; it is guaranteed
-that no library function changes the state of any of these functions).
-@strong{For the above reasons it is highly requested that the functions
-described in the previous section be used in place of non-reentrant
-conversion functions.}
-
-@menu
-* Non-reentrant Character Conversion:: Non-reentrant Conversion of Single
- Characters.
-* Non-reentrant String Conversion:: Non-reentrant Conversion of Strings.
-* Shift State:: States in Non-reentrant Functions.
-@end menu
-
-@node Non-reentrant Character Conversion
-@subsection Non-reentrant Conversion of Single Characters
-
-@comment stdlib.h
-@comment ISO
-@deftypefun int mbtowc (wchar_t *restrict @var{result}, const char *restrict @var{string}, size_t @var{size})
-The @code{mbtowc} (``multibyte to wide character'') function when called
-with non-null @var{string} converts the first multibyte character
-beginning at @var{string} to its corresponding wide character code. It
-stores the result in @code{*@var{result}}.
-
-@code{mbtowc} never examines more than @var{size} bytes. (The idea is
-to supply for @var{size} the number of bytes of data you have in hand.)
-
-@code{mbtowc} with non-null @var{string} distinguishes three
-possibilities: the first @var{size} bytes at @var{string} start with
-valid multibyte characters, they start with an invalid byte sequence or
-just part of a character, or @var{string} points to an empty string (a
-null character).
-
-For a valid multibyte character, @code{mbtowc} converts it to a wide
-character and stores that in @code{*@var{result}}, and returns the
-number of bytes in that character (always at least @math{1} and never
-more than @var{size}).
-
-For an invalid byte sequence, @code{mbtowc} returns @math{-1}. For an
-empty string, it returns @math{0}, also storing @code{'\0'} in
-@code{*@var{result}}.
-
-If the multibyte character code uses shift characters, then
-@code{mbtowc} maintains and updates a shift state as it scans. If you
-call @code{mbtowc} with a null pointer for @var{string}, that
-initializes the shift state to its standard initial value. It also
-returns nonzero if the multibyte character code in use actually has a
-shift state. @xref{Shift State}.
-@end deftypefun
-
-@comment stdlib.h
-@comment ISO
-@deftypefun int wctomb (char *@var{string}, wchar_t @var{wchar})
-The @code{wctomb} (``wide character to multibyte'') function converts
-the wide character code @var{wchar} to its corresponding multibyte
-character sequence, and stores the result in bytes starting at
-@var{string}. At most @code{MB_CUR_MAX} characters are stored.
-
-@code{wctomb} with non-null @var{string} distinguishes three
-possibilities for @var{wchar}: a valid wide character code (one that can
-be translated to a multibyte character), an invalid code, and @code{L'\0'}.
-
-Given a valid code, @code{wctomb} converts it to a multibyte character,
-storing the bytes starting at @var{string}. Then it returns the number
-of bytes in that character (always at least @math{1} and never more
-than @code{MB_CUR_MAX}).
-
-If @var{wchar} is an invalid wide character code, @code{wctomb} returns
-@math{-1}. If @var{wchar} is @code{L'\0'}, it returns @code{0}, also
-storing @code{'\0'} in @code{*@var{string}}.
-
-If the multibyte character code uses shift characters, then
-@code{wctomb} maintains and updates a shift state as it scans. If you
-call @code{wctomb} with a null pointer for @var{string}, that
-initializes the shift state to its standard initial value. It also
-returns nonzero if the multibyte character code in use actually has a
-shift state. @xref{Shift State}.
-
-Calling this function with a @var{wchar} argument of zero when
-@var{string} is not null has the side-effect of reinitializing the
-stored shift state @emph{as well as} storing the multibyte character
-@code{'\0'} and returning @math{0}.
-@end deftypefun
-
-Similar to @code{mbrlen} there is also a non-reentrant function which
-computes the length of a multibyte character. It can be defined in
-terms of @code{mbtowc}.
-
-@comment stdlib.h
-@comment ISO
-@deftypefun int mblen (const char *@var{string}, size_t @var{size})
-The @code{mblen} function with a non-null @var{string} argument returns
-the number of bytes that make up the multibyte character beginning at
-@var{string}, never examining more than @var{size} bytes. (The idea is
-to supply for @var{size} the number of bytes of data you have in hand.)
-
-The return value of @code{mblen} distinguishes three possibilities: the
-first @var{size} bytes at @var{string} start with valid multibyte
-characters, they start with an invalid byte sequence or just part of a
-character, or @var{string} points to an empty string (a null character).
-
-For a valid multibyte character, @code{mblen} returns the number of
-bytes in that character (always at least @code{1} and never more than
-@var{size}). For an invalid byte sequence, @code{mblen} returns
-@math{-1}. For an empty string, it returns @math{0}.
-
-If the multibyte character code uses shift characters, then @code{mblen}
-maintains and updates a shift state as it scans. If you call
-@code{mblen} with a null pointer for @var{string}, that initializes the
-shift state to its standard initial value. It also returns a nonzero
-value if the multibyte character code in use actually has a shift state.
-@xref{Shift State}.
-
-@pindex stdlib.h
-The function @code{mblen} is declared in @file{stdlib.h}.
-@end deftypefun
-
-
-@node Non-reentrant String Conversion
-@subsection Non-reentrant Conversion of Strings
-
-For convenience the @w{ISO C90} standard also defines functions to
-convert entire strings instead of single characters. These functions
-suffer from the same problems as their reentrant counterparts from
-@w{Amendment 1} to @w{ISO C90}; see @ref{Converting Strings}.
-
-@comment stdlib.h
-@comment ISO
-@deftypefun size_t mbstowcs (wchar_t *@var{wstring}, const char *@var{string}, size_t @var{size})
-The @code{mbstowcs} (``multibyte string to wide character string'')
-function converts the null-terminated string of multibyte characters
-@var{string} to an array of wide character codes, storing not more than
-@var{size} wide characters into the array beginning at @var{wstring}.
-The terminating null character counts towards the size, so if @var{size}
-is less than the actual number of wide characters resulting from
-@var{string}, no terminating null character is stored.
-
-The conversion of characters from @var{string} begins in the initial
-shift state.
-
-If an invalid multibyte character sequence is found, the @code{mbstowcs}
-function returns a value of @math{-1}. Otherwise, it returns the number
-of wide characters stored in the array @var{wstring}. This number does
-not include the terminating null character, which is present if the
-number is less than @var{size}.
-
-Here is an example showing how to convert a string of multibyte
-characters, allocating enough space for the result.
-
-@smallexample
-wchar_t *
-mbstowcs_alloc (const char *string)
-@{
- size_t size = strlen (string) + 1;
- wchar_t *buf = xmalloc (size * sizeof (wchar_t));
-
- size = mbstowcs (buf, string, size);
- if (size == (size_t) -1)
- return NULL;
- buf = xrealloc (buf, (size + 1) * sizeof (wchar_t));
- return buf;
-@}
-@end smallexample
-
-@end deftypefun
-
-@comment stdlib.h
-@comment ISO
-@deftypefun size_t wcstombs (char *@var{string}, const wchar_t *@var{wstring}, size_t @var{size})
-The @code{wcstombs} (``wide character string to multibyte string'')
-function converts the null-terminated wide character array @var{wstring}
-into a string containing multibyte characters, storing not more than
-@var{size} bytes starting at @var{string}, followed by a terminating
-null character if there is room. The conversion of characters begins in
-the initial shift state.
-
-The terminating null character counts towards the size, so if @var{size}
-is less than or equal to the number of bytes needed in @var{wstring}, no
-terminating null character is stored.
-
-If a code that does not correspond to a valid multibyte character is
-found, the @code{wcstombs} function returns a value of @math{-1}.
-Otherwise, the return value is the number of bytes stored in the array
-@var{string}. This number does not include the terminating null character,
-which is present if the number is less than @var{size}.
-@end deftypefun
-
-@node Shift State
-@subsection States in Non-reentrant Functions
-
-In some multibyte character codes, the @emph{meaning} of any particular
-byte sequence is not fixed; it depends on what other sequences have come
-earlier in the same string. Typically there are just a few sequences that
-can change the meaning of other sequences; these few are called
-@dfn{shift sequences} and we say that they set the @dfn{shift state} for
-other sequences that follow.
-
-To illustrate shift state and shift sequences, suppose we decide that
-the sequence @code{0200} (just one byte) enters Japanese mode, in which
-pairs of bytes in the range from @code{0240} to @code{0377} are single
-characters, while @code{0201} enters Latin-1 mode, in which single bytes
-in the range from @code{0240} to @code{0377} are characters, and
-interpreted according to the ISO Latin-1 character set. This is a
-multibyte code which has two alternative shift states (``Japanese mode''
-and ``Latin-1 mode''), and two shift sequences that specify particular
-shift states.
-
-When the multibyte character code in use has shift states, then
-@code{mblen}, @code{mbtowc}, and @code{wctomb} must maintain and update
-the current shift state as they scan the string. To make this work
-properly, you must follow these rules:
-
-@itemize @bullet
-@item
-Before starting to scan a string, call the function with a null pointer
-for the multibyte character address---for example, @code{mblen (NULL,
-0)}. This initializes the shift state to its standard initial value.
-
-@item
-Scan the string one character at a time, in order. Do not ``back up''
-and rescan characters already scanned, and do not intersperse the
-processing of different strings.
-@end itemize
-
-Here is an example of using @code{mblen} following these rules:
-
-@smallexample
-void
-scan_string (char *s)
-@{
- int length = strlen (s);
-
- /* @r{Initialize shift state.} */
- mblen (NULL, 0);
-
- while (1)
- @{
- int thischar = mblen (s, length);
- /* @r{Deal with end of string and invalid characters.} */
- if (thischar == 0)
- break;
- if (thischar == -1)
- @{
- error ("invalid multibyte character");
- break;
- @}
- /* @r{Advance past this character.} */
- s += thischar;
- length -= thischar;
- @}
-@}
-@end smallexample
-
-The functions @code{mblen}, @code{mbtowc} and @code{wctomb} are not
-reentrant when using a multibyte code that uses a shift state. However,
-no other library functions call these functions, so you don't have to
-worry that the shift state will be changed mysteriously.
-
-
-@node Generic Charset Conversion
-@section Generic Charset Conversion
-
-The conversion functions mentioned so far in this chapter all had in
-common that they operate on character sets that are not directly
-specified by the functions. The multibyte encoding used is specified by
-the currently selected locale for the @code{LC_CTYPE} category. The
-wide character set is fixed by the implementation (in the case of GNU C
-library it is always UCS-4 encoded @w{ISO 10646}.
-
-This has of course several problems when it comes to general character
-conversion:
-
-@itemize @bullet
-@item
-For every conversion where neither the source nor the destination
-character set is the character set of the locale for the @code{LC_CTYPE}
-category, one has to change the @code{LC_CTYPE} locale using
-@code{setlocale}.
-
-Changing the @code{LC_TYPE} locale introduces major problems for the rest
-of the programs since several more functions (e.g., the character
-classification functions, @pxref{Classification of Characters}) use the
-@code{LC_CTYPE} category.
-
-@item
-Parallel conversions to and from different character sets are not
-possible since the @code{LC_CTYPE} selection is global and shared by all
-threads.
-
-@item
-If neither the source nor the destination character set is the character
-set used for @code{wchar_t} representation, there is at least a two-step
-process necessary to convert a text using the functions above. One would
-have to select the source character set as the multibyte encoding,
-convert the text into a @code{wchar_t} text, select the destination
-character set as the multibyte encoding, and convert the wide character
-text to the multibyte (@math{=} destination) character set.
-
-Even if this is possible (which is not guaranteed) it is a very tiring
-work. Plus it suffers from the other two raised points even more due to
-the steady changing of the locale.
-@end itemize
-
-The XPG2 standard defines a completely new set of functions which has
-none of these limitations. They are not at all coupled to the selected
-locales, and they have no constraints on the character sets selected for
-source and destination. Only the set of available conversions limits
-them. The standard does not specify that any conversion at all must be
-available. Such availability is a measure of the quality of the
-implementation.
-
-In the following text first the interface to @code{iconv} and then the
-conversion function, will be described. Comparisons with other
-implementations will show what obstacles stand in the way of portable
-applications. Finally, the implementation is described in so far as might
-interest the advanced user who wants to extend conversion capabilities.
-
-@menu
-* Generic Conversion Interface:: Generic Character Set Conversion Interface.
-* iconv Examples:: A complete @code{iconv} example.
-* Other iconv Implementations:: Some Details about other @code{iconv}
- Implementations.
-* glibc iconv Implementation:: The @code{iconv} Implementation in the GNU C
- library.
-@end menu
-
-@node Generic Conversion Interface
-@subsection Generic Character Set Conversion Interface
-
-This set of functions follows the traditional cycle of using a resource:
-open--use--close. The interface consists of three functions, each of
-which implements one step.
-
-Before the interfaces are described it is necessary to introduce a
-data type. Just like other open--use--close interfaces the functions
-introduced here work using handles and the @file{iconv.h} header
-defines a special type for the handles used.
-
-@comment iconv.h
-@comment XPG2
-@deftp {Data Type} iconv_t
-This data type is an abstract type defined in @file{iconv.h}. The user
-must not assume anything about the definition of this type; it must be
-completely opaque.
-
-Objects of this type can get assigned handles for the conversions using
-the @code{iconv} functions. The objects themselves need not be freed, but
-the conversions for which the handles stand for have to.
-@end deftp
-
-@noindent
-The first step is the function to create a handle.
-
-@comment iconv.h
-@comment XPG2
-@deftypefun iconv_t iconv_open (const char *@var{tocode}, const char *@var{fromcode})
-The @code{iconv_open} function has to be used before starting a
-conversion. The two parameters this function takes determine the
-source and destination character set for the conversion, and if the
-implementation has the possibility to perform such a conversion, the
-function returns a handle.
-
-If the wanted conversion is not available, the @code{iconv_open} function
-returns @code{(iconv_t) -1}. In this case the global variable
-@code{errno} can have the following values:
-
-@table @code
-@item EMFILE
-The process already has @code{OPEN_MAX} file descriptors open.
-@item ENFILE
-The system limit of open file is reached.
-@item ENOMEM
-Not enough memory to carry out the operation.
-@item EINVAL
-The conversion from @var{fromcode} to @var{tocode} is not supported.
-@end table
-
-It is not possible to use the same descriptor in different threads to
-perform independent conversions. The data structures associated
-with the descriptor include information about the conversion state.
-This must not be messed up by using it in different conversions.
-
-An @code{iconv} descriptor is like a file descriptor as for every use a
-new descriptor must be created. The descriptor does not stand for all
-of the conversions from @var{fromset} to @var{toset}.
-
-The GNU C library implementation of @code{iconv_open} has one
-significant extension to other implementations. To ease the extension
-of the set of available conversions, the implementation allows storing
-the necessary files with data and code in an arbitrary number of
-directories. How this extension must be written will be explained below
-(@pxref{glibc iconv Implementation}). Here it is only important to say
-that all directories mentioned in the @code{GCONV_PATH} environment
-variable are considered only if they contain a file @file{gconv-modules}.
-These directories need not necessarily be created by the system
-administrator. In fact, this extension is introduced to help users
-writing and using their own, new conversions. Of course, this does not
-work for security reasons in SUID binaries; in this case only the system
-directory is considered and this normally is
-@file{@var{prefix}/lib/gconv}. The @code{GCONV_PATH} environment variable
-is examined exactly once at the first call of the @code{iconv_open}
-function. Later modifications of the variable have no effect.
-
-@pindex iconv.h
-The @code{iconv_open} function was introduced early in the X/Open
-Portability Guide, @w{version 2}. It is supported by all commercial
-Unices as it is required for the Unix branding. However, the quality and
-completeness of the implementation varies widely. The @code{iconv_open}
-function is declared in @file{iconv.h}.
-@end deftypefun
-
-The @code{iconv} implementation can associate large data structure with
-the handle returned by @code{iconv_open}. Therefore, it is crucial to
-free all the resources once all conversions are carried out and the
-conversion is not needed anymore.
-
-@comment iconv.h
-@comment XPG2
-@deftypefun int iconv_close (iconv_t @var{cd})
-The @code{iconv_close} function frees all resources associated with the
-handle @var{cd}, which must have been returned by a successful call to
-the @code{iconv_open} function.
-
-If the function call was successful the return value is @math{0}.
-Otherwise it is @math{-1} and @code{errno} is set appropriately.
-Defined error are:
-
-@table @code
-@item EBADF
-The conversion descriptor is invalid.
-@end table
-
-@pindex iconv.h
-The @code{iconv_close} function was introduced together with the rest
-of the @code{iconv} functions in XPG2 and is declared in @file{iconv.h}.
-@end deftypefun
-
-The standard defines only one actual conversion function. This has,
-therefore, the most general interface: it allows conversion from one
-buffer to another. Conversion from a file to a buffer, vice versa, or
-even file to file can be implemented on top of it.
-
-@comment iconv.h
-@comment XPG2
-@deftypefun size_t iconv (iconv_t @var{cd}, char **@var{inbuf}, size_t *@var{inbytesleft}, char **@var{outbuf}, size_t *@var{outbytesleft})
-@cindex stateful
-The @code{iconv} function converts the text in the input buffer
-according to the rules associated with the descriptor @var{cd} and
-stores the result in the output buffer. It is possible to call the
-function for the same text several times in a row since for stateful
-character sets the necessary state information is kept in the data
-structures associated with the descriptor.
-
-The input buffer is specified by @code{*@var{inbuf}} and it contains
-@code{*@var{inbytesleft}} bytes. The extra indirection is necessary for
-communicating the used input back to the caller (see below). It is
-important to note that the buffer pointer is of type @code{char} and the
-length is measured in bytes even if the input text is encoded in wide
-characters.
-
-The output buffer is specified in a similar way. @code{*@var{outbuf}}
-points to the beginning of the buffer with at least
-@code{*@var{outbytesleft}} bytes room for the result. The buffer
-pointer again is of type @code{char} and the length is measured in
-bytes. If @var{outbuf} or @code{*@var{outbuf}} is a null pointer, the
-conversion is performed but no output is available.
-
-If @var{inbuf} is a null pointer, the @code{iconv} function performs the
-necessary action to put the state of the conversion into the initial
-state. This is obviously a no-op for non-stateful encodings, but if the
-encoding has a state, such a function call might put some byte sequences
-in the output buffer, which perform the necessary state changes. The
-next call with @var{inbuf} not being a null pointer then simply goes on
-from the initial state. It is important that the programmer never makes
-any assumption as to whether the conversion has to deal with states. Even
-if the input and output character sets are not stateful, the
-implementation might still have to keep states. This is due to the
-implementation chosen for the GNU C library as it is described below.
-Therefore an @code{iconv} call to reset the state should always be
-performed if some protocol requires this for the output text.
-
-The conversion stops for one of three reasons. The first is that all
-characters from the input buffer are converted. This actually can mean
-two things: either all bytes from the input buffer are consumed or
-there are some bytes at the end of the buffer that possibly can form a
-complete character but the input is incomplete. The second reason for a
-stop is that the output buffer is full. And the third reason is that
-the input contains invalid characters.
-
-In all of these cases the buffer pointers after the last successful
-conversion, for input and output buffer, are stored in @var{inbuf} and
-@var{outbuf}, and the available room in each buffer is stored in
-@var{inbytesleft} and @var{outbytesleft}.
-
-Since the character sets selected in the @code{iconv_open} call can be
-almost arbitrary, there can be situations where the input buffer contains
-valid characters, which have no identical representation in the output
-character set. The behavior in this situation is undefined. The
-@emph{current} behavior of the GNU C library in this situation is to
-return with an error immediately. This certainly is not the most
-desirable solution; therefore, future versions will provide better ones,
-but they are not yet finished.
-
-If all input from the input buffer is successfully converted and stored
-in the output buffer, the function returns the number of non-reversible
-conversions performed. In all other cases the return value is
-@code{(size_t) -1} and @code{errno} is set appropriately. In such cases
-the value pointed to by @var{inbytesleft} is nonzero.
-
-@table @code
-@item EILSEQ
-The conversion stopped because of an invalid byte sequence in the input.
-After the call, @code{*@var{inbuf}} points at the first byte of the
-invalid byte sequence.
-
-@item E2BIG
-The conversion stopped because it ran out of space in the output buffer.
-
-@item EINVAL
-The conversion stopped because of an incomplete byte sequence at the end
-of the input buffer.
-
-@item EBADF
-The @var{cd} argument is invalid.
-@end table
-
-@pindex iconv.h
-The @code{iconv} function was introduced in the XPG2 standard and is
-declared in the @file{iconv.h} header.
-@end deftypefun
-
-The definition of the @code{iconv} function is quite good overall. It
-provides quite flexible functionality. The only problems lie in the
-boundary cases, which are incomplete byte sequences at the end of the
-input buffer and invalid input. A third problem, which is not really
-a design problem, is the way conversions are selected. The standard
-does not say anything about the legitimate names, a minimal set of
-available conversions. We will see how this negatively impacts other
-implementations, as demonstrated below.
-
-@node iconv Examples
-@subsection A complete @code{iconv} example
-
-The example below features a solution for a common problem. Given that
-one knows the internal encoding used by the system for @code{wchar_t}
-strings, one often is in the position to read text from a file and store
-it in wide character buffers. One can do this using @code{mbsrtowcs},
-but then we run into the problems discussed above.
-
-@smallexample
-int
-file2wcs (int fd, const char *charset, wchar_t *outbuf, size_t avail)
-@{
- char inbuf[BUFSIZ];
- size_t insize = 0;
- char *wrptr = (char *) outbuf;
- int result = 0;
- iconv_t cd;
-
- cd = iconv_open ("WCHAR_T", charset);
- if (cd == (iconv_t) -1)
- @{
- /* @r{Something went wrong.} */
- if (errno == EINVAL)
- error (0, 0, "conversion from '%s' to wchar_t not available",
- charset);
- else
- perror ("iconv_open");
-
- /* @r{Terminate the output string.} */
- *outbuf = L'\0';
-
- return -1;
- @}
-
- while (avail > 0)
- @{
- size_t nread;
- size_t nconv;
- char *inptr = inbuf;
-
- /* @r{Read more input.} */
- nread = read (fd, inbuf + insize, sizeof (inbuf) - insize);
- if (nread == 0)
- @{
- /* @r{When we come here the file is completely read.}
- @r{This still could mean there are some unused}
- @r{characters in the @code{inbuf}. Put them back.} */
- if (lseek (fd, -insize, SEEK_CUR) == -1)
- result = -1;
-
- /* @r{Now write out the byte sequence to get into the}
- @r{initial state if this is necessary.} */
- iconv (cd, NULL, NULL, &wrptr, &avail);
-
- break;
- @}
- insize += nread;
-
- /* @r{Do the conversion.} */
- nconv = iconv (cd, &inptr, &insize, &wrptr, &avail);
- if (nconv == (size_t) -1)
- @{
- /* @r{Not everything went right. It might only be}
- @r{an unfinished byte sequence at the end of the}
- @r{buffer. Or it is a real problem.} */
- if (errno == EINVAL)
- /* @r{This is harmless. Simply move the unused}
- @r{bytes to the beginning of the buffer so that}
- @r{they can be used in the next round.} */
- memmove (inbuf, inptr, insize);
- else
- @{
- /* @r{It is a real problem. Maybe we ran out of}
- @r{space in the output buffer or we have invalid}
- @r{input. In any case back the file pointer to}
- @r{the position of the last processed byte.} */
- lseek (fd, -insize, SEEK_CUR);
- result = -1;
- break;
- @}
- @}
- @}
-
- /* @r{Terminate the output string.} */
- if (avail >= sizeof (wchar_t))
- *((wchar_t *) wrptr) = L'\0';
-
- if (iconv_close (cd) != 0)
- perror ("iconv_close");
-
- return (wchar_t *) wrptr - outbuf;
-@}
-@end smallexample
-
-@cindex stateful
-This example shows the most important aspects of using the @code{iconv}
-functions. It shows how successive calls to @code{iconv} can be used to
-convert large amounts of text. The user does not have to care about
-stateful encodings as the functions take care of everything.
-
-An interesting point is the case where @code{iconv} returns an error and
-@code{errno} is set to @code{EINVAL}. This is not really an error in the
-transformation. It can happen whenever the input character set contains
-byte sequences of more than one byte for some character and texts are not
-processed in one piece. In this case there is a chance that a multibyte
-sequence is cut. The caller can then simply read the remainder of the
-takes and feed the offending bytes together with new character from the
-input to @code{iconv} and continue the work. The internal state kept in
-the descriptor is @emph{not} unspecified after such an event as is the
-case with the conversion functions from the @w{ISO C} standard.
-
-The example also shows the problem of using wide character strings with
-@code{iconv}. As explained in the description of the @code{iconv}
-function above, the function always takes a pointer to a @code{char}
-array and the available space is measured in bytes. In the example, the
-output buffer is a wide character buffer; therefore, we use a local
-variable @var{wrptr} of type @code{char *}, which is used in the
-@code{iconv} calls.
-
-This looks rather innocent but can lead to problems on platforms that
-have tight restriction on alignment. Therefore the caller of @code{iconv}
-has to make sure that the pointers passed are suitable for access of
-characters from the appropriate character set. Since, in the
-above case, the input parameter to the function is a @code{wchar_t}
-pointer, this is the case (unless the user violates alignment when
-computing the parameter). But in other situations, especially when
-writing generic functions where one does not know what type of character
-set one uses and, therefore, treats text as a sequence of bytes, it might
-become tricky.
-
-@node Other iconv Implementations
-@subsection Some Details about other @code{iconv} Implementations
-
-This is not really the place to discuss the @code{iconv} implementation
-of other systems but it is necessary to know a bit about them to write
-portable programs. The above mentioned problems with the specification
-of the @code{iconv} functions can lead to portability issues.
-
-The first thing to notice is that, due to the large number of character
-sets in use, it is certainly not practical to encode the conversions
-directly in the C library. Therefore, the conversion information must
-come from files outside the C library. This is usually done in one or
-both of the following ways:
-
-@itemize @bullet
-@item
-The C library contains a set of generic conversion functions which can
-read the needed conversion tables and other information from data files.
-These files get loaded when necessary.
-
-This solution is problematic as it requires a great deal of effort to
-apply to all character sets (potentially an infinite set). The
-differences in the structure of the different character sets is so large
-that many different variants of the table-processing functions must be
-developed. In addition, the generic nature of these functions make them
-slower than specifically implemented functions.
-
-@item
-The C library only contains a framework which can dynamically load
-object files and execute the conversion functions contained therein.
-
-This solution provides much more flexibility. The C library itself
-contains only very little code and therefore reduces the general memory
-footprint. Also, with a documented interface between the C library and
-the loadable modules it is possible for third parties to extend the set
-of available conversion modules. A drawback of this solution is that
-dynamic loading must be available.
-@end itemize
-
-Some implementations in commercial Unices implement a mixture of these
-possibilities; the majority implement only the second solution. Using
-loadable modules moves the code out of the library itself and keeps
-the door open for extensions and improvements, but this design is also
-limiting on some platforms since not many platforms support dynamic
-loading in statically linked programs. On platforms without this
-capability it is therefore not possible to use this interface in
-statically linked programs. The GNU C library has, on ELF platforms, no
-problems with dynamic loading in these situations; therefore, this
-point is moot. The danger is that one gets acquainted with this situation
-and forgets about the restrictions on other systems.
-
-A second thing to know about other @code{iconv} implementations is that
-the number of available conversions is often very limited. Some
-implementations provide, in the standard release (not special
-international or developer releases), at most 100 to 200 conversion
-possibilities. This does not mean 200 different character sets are
-supported; for example, conversions from one character set to a set of 10
-others might count as 10 conversions. Together with the other direction
-this makes 20 conversion possibilities used up by one character set. One
-can imagine the thin coverage these platform provide. Some Unix vendors
-even provide only a handful of conversions which renders them useless for
-almost all uses.
-
-This directly leads to a third and probably the most problematic point.
-The way the @code{iconv} conversion functions are implemented on all
-known Unix systems and the availability of the conversion functions from
-character set @math{@cal{A}} to @math{@cal{B}} and the conversion from
-@math{@cal{B}} to @math{@cal{C}} does @emph{not} imply that the
-conversion from @math{@cal{A}} to @math{@cal{C}} is available.
-
-This might not seem unreasonable and problematic at first, but it is a
-quite big problem as one will notice shortly after hitting it. To show
-the problem we assume to write a program which has to convert from
-@math{@cal{A}} to @math{@cal{C}}. A call like
-
-@smallexample
-cd = iconv_open ("@math{@cal{C}}", "@math{@cal{A}}");
-@end smallexample
-
-@noindent
-fails according to the assumption above. But what does the program
-do now? The conversion is necessary; therefore, simply giving up is not
-an option.
-
-This is a nuisance. The @code{iconv} function should take care of this.
-But how should the program proceed from here on? If it tries to convert
-to character set @math{@cal{B}}, first the two @code{iconv_open}
-calls
-
-@smallexample
-cd1 = iconv_open ("@math{@cal{B}}", "@math{@cal{A}}");
-@end smallexample
-
-@noindent
-and
-
-@smallexample
-cd2 = iconv_open ("@math{@cal{C}}", "@math{@cal{B}}");
-@end smallexample
-
-@noindent
-will succeed, but how to find @math{@cal{B}}?
-
-Unfortunately, the answer is: there is no general solution. On some
-systems guessing might help. On those systems most character sets can
-convert to and from UTF-8 encoded @w{ISO 10646} or Unicode text. Beside
-this only some very system-specific methods can help. Since the
-conversion functions come from loadable modules and these modules must
-be stored somewhere in the filesystem, one @emph{could} try to find them
-and determine from the available file which conversions are available
-and whether there is an indirect route from @math{@cal{A}} to
-@math{@cal{C}}.
-
-This example shows one of the design errors of @code{iconv} mentioned
-above. It should at least be possible to determine the list of available
-conversion programmatically so that if @code{iconv_open} says there is no
-such conversion, one could make sure this also is true for indirect
-routes.
-
-@node glibc iconv Implementation
-@subsection The @code{iconv} Implementation in the GNU C library
-
-After reading about the problems of @code{iconv} implementations in the
-last section it is certainly good to note that the implementation in
-the GNU C library has none of the problems mentioned above. What
-follows is a step-by-step analysis of the points raised above. The
-evaluation is based on the current state of the development (as of
-January 1999). The development of the @code{iconv} functions is not
-complete, but basic functionality has solidified.
-
-The GNU C library's @code{iconv} implementation uses shared loadable
-modules to implement the conversions. A very small number of
-conversions are built into the library itself but these are only rather
-trivial conversions.
-
-All the benefits of loadable modules are available in the GNU C library
-implementation. This is especially appealing since the interface is
-well documented (see below), and it, therefore, is easy to write new
-conversion modules. The drawback of using loadable objects is not a
-problem in the GNU C library, at least on ELF systems. Since the
-library is able to load shared objects even in statically linked
-binaries, static linking need not be forbidden in case one wants to use
-@code{iconv}.
-
-The second mentioned problem is the number of supported conversions.
-Currently, the GNU C library supports more than 150 character sets. The
-way the implementation is designed the number of supported conversions
-is greater than 22350 (@math{150} times @math{149}). If any conversion
-from or to a character set is missing, it can be added easily.
-
-Particularly impressive as it may be, this high number is due to the
-fact that the GNU C library implementation of @code{iconv} does not have
-the third problem mentioned above (i.e., whenever there is a conversion
-from a character set @math{@cal{A}} to @math{@cal{B}} and from
-@math{@cal{B}} to @math{@cal{C}} it is always possible to convert from
-@math{@cal{A}} to @math{@cal{C}} directly). If the @code{iconv_open}
-returns an error and sets @code{errno} to @code{EINVAL}, there is no
-known way, directly or indirectly, to perform the wanted conversion.
-
-@cindex triangulation
-Triangulation is achieved by providing for each character set a
-conversion from and to UCS-4 encoded @w{ISO 10646}. Using @w{ISO 10646}
-as an intermediate representation it is possible to @dfn{triangulate}
-(i.e., convert with an intermediate representation).
-
-There is no inherent requirement to provide a conversion to @w{ISO
-10646} for a new character set, and it is also possible to provide other
-conversions where neither source nor destination character set is @w{ISO
-10646}. The existing set of conversions is simply meant to cover all
-conversions that might be of interest.
-
-@cindex ISO-2022-JP
-@cindex EUC-JP
-All currently available conversions use the triangulation method above,
-making conversion run unnecessarily slow. If, for example, somebody
-often needs the conversion from ISO-2022-JP to EUC-JP, a quicker solution
-would involve direct conversion between the two character sets, skipping
-the input to @w{ISO 10646} first. The two character sets of interest
-are much more similar to each other than to @w{ISO 10646}.
-
-In such a situation one easily can write a new conversion and provide it
-as a better alternative. The GNU C library @code{iconv} implementation
-would automatically use the module implementing the conversion if it is
-specified to be more efficient.
-
-@subsubsection Format of @file{gconv-modules} files
-
-All information about the available conversions comes from a file named
-@file{gconv-modules} which can be found in any of the directories along
-the @code{GCONV_PATH}. The @file{gconv-modules} files are line-oriented
-text files, where each of the lines has one of the following formats:
-
-@itemize @bullet
-@item
-If the first non-whitespace character is a @kbd{#} the line contains only
-comments and is ignored.
-
-@item
-Lines starting with @code{alias} define an alias name for a character
-set. Two more words are expected on the line. The first word
-defines the alias name, and the second defines the original name of the
-character set. The effect is that it is possible to use the alias name
-in the @var{fromset} or @var{toset} parameters of @code{iconv_open} and
-achieve the same result as when using the real character set name.
-
-This is quite important as a character set has often many different
-names. There is normally an official name but this need not correspond to
-the most popular name. Beside this many character sets have special
-names that are somehow constructed. For example, all character sets
-specified by the ISO have an alias of the form @code{ISO-IR-@var{nnn}}
-where @var{nnn} is the registration number. This allows programs which
-know about the registration number to construct character set names and
-use them in @code{iconv_open} calls. More on the available names and
-aliases follows below.
-
-@item
-Lines starting with @code{module} introduce an available conversion
-module. These lines must contain three or four more words.
-
-The first word specifies the source character set, the second word the
-destination character set of conversion implemented in this module, and
-the third word is the name of the loadable module. The filename is
-constructed by appending the usual shared object suffix (normally
-@file{.so}) and this file is then supposed to be found in the same
-directory the @file{gconv-modules} file is in. The last word on the line,
-which is optional, is a numeric value representing the cost of the
-conversion. If this word is missing, a cost of @math{1} is assumed. The
-numeric value itself does not matter that much; what counts are the
-relative values of the sums of costs for all possible conversion paths.
-Below is a more precise description of the use of the cost value.
-@end itemize
-
-Returning to the example above where one has written a module to directly
-convert from ISO-2022-JP to EUC-JP and back. All that has to be done is
-to put the new module, let its name be ISO2022JP-EUCJP.so, in a directory
-and add a file @file{gconv-modules} with the following content in the
-same directory:
-
-@smallexample
-module ISO-2022-JP// EUC-JP// ISO2022JP-EUCJP 1
-module EUC-JP// ISO-2022-JP// ISO2022JP-EUCJP 1
-@end smallexample
-
-To see why this is sufficient, it is necessary to understand how the
-conversion used by @code{iconv} (and described in the descriptor) is
-selected. The approach to this problem is quite simple.
-
-At the first call of the @code{iconv_open} function the program reads
-all available @file{gconv-modules} files and builds up two tables: one
-containing all the known aliases and another that contains the
-information about the conversions and which shared object implements
-them.
-
-@subsubsection Finding the conversion path in @code{iconv}
-
-The set of available conversions form a directed graph with weighted
-edges. The weights on the edges are the costs specified in the
-@file{gconv-modules} files. The @code{iconv_open} function uses an
-algorithm suitable for search for the best path in such a graph and so
-constructs a list of conversions which must be performed in succession
-to get the transformation from the source to the destination character
-set.
-
-Explaining why the above @file{gconv-modules} files allows the
-@code{iconv} implementation to resolve the specific ISO-2022-JP to
-EUC-JP conversion module instead of the conversion coming with the
-library itself is straightforward. Since the latter conversion takes two
-steps (from ISO-2022-JP to @w{ISO 10646} and then from @w{ISO 10646} to
-EUC-JP), the cost is @math{1+1 = 2}. The above @file{gconv-modules}
-file, however, specifies that the new conversion modules can perform this
-conversion with only the cost of @math{1}.
-
-A mysterious item about the @file{gconv-modules} file above (and also
-the file coming with the GNU C library) are the names of the character
-sets specified in the @code{module} lines. Why do almost all the names
-end in @code{//}? And this is not all: the names can actually be
-regular expressions. At this point in time this mystery should not be
-revealed, unless you have the relevant spell-casting materials: ashes
-from an original @w{DOS 6.2} boot disk burnt in effigy, a crucifix
-blessed by St.@: Emacs, assorted herbal roots from Central America, sand
-from Cebu, etc. Sorry! @strong{The part of the implementation where
-this is used is not yet finished. For now please simply follow the
-existing examples. It'll become clearer once it is. --drepper}
-
-A last remark about the @file{gconv-modules} is about the names not
-ending with @code{//}. Aa character set named @code{INTERNAL} is often
-mentioned. From the discussion above and the chosen name it should have
-become clear that this is the name for the representation used in the
-intermediate step of the triangulation. We have said that this is UCS-4
-but actually that is not quite right. The UCS-4 specification also
-includes the specification of the byte ordering used. Since a UCS-4 value
-consists of four bytes, a stored value is effected by byte ordering. The
-internal representation is @emph{not} the same as UCS-4 in case the byte
-ordering of the processor (or at least the running process) is not the
-same as the one required for UCS-4. This is done for performance reasons
-as one does not want to perform unnecessary byte-swapping operations if
-one is not interested in actually seeing the result in UCS-4. To avoid
-trouble with endianess, the internal representation consistently is named
-@code{INTERNAL} even on big-endian systems where the representations are
-identical.
-
-@subsubsection @code{iconv} module data structures
-
-So far this section has described how modules are located and considered
-to be used. What remains to be described is the interface of the modules
-so that one can write new ones. This section describes the interface as
-it is in use in January 1999. The interface will change a bit in the
-future but, with luck, only in an upwardly compatible way.
-
-The definitions necessary to write new modules are publicly available
-in the non-standard header @file{gconv.h}. The following text,
-therefore, describes the definitions from this header file. First,
-however, it is necessary to get an overview.
-
-From the perspective of the user of @code{iconv} the interface is quite
-simple: the @code{iconv_open} function returns a handle that can be used
-in calls to @code{iconv}, and finally the handle is freed with a call to
-@code{iconv_close}. The problem is that the handle has to be able to
-represent the possibly long sequences of conversion steps and also the
-state of each conversion since the handle is all that is passed to the
-@code{iconv} function. Therefore, the data structures are really the
-elements necessary to understanding the implementation.
-
-We need two different kinds of data structures. The first describes the
-conversion and the second describes the state etc. There are really two
-type definitions like this in @file{gconv.h}.
-@pindex gconv.h
-
-@comment gconv.h
-@comment GNU
-@deftp {Data type} {struct __gconv_step}
-This data structure describes one conversion a module can perform. For
-each function in a loaded module with conversion functions there is
-exactly one object of this type. This object is shared by all users of
-the conversion (i.e., this object does not contain any information
-corresponding to an actual conversion; it only describes the conversion
-itself).
-
-@table @code
-@item struct __gconv_loaded_object *__shlib_handle
-@itemx const char *__modname
-@itemx int __counter
-All these elements of the structure are used internally in the C library
-to coordinate loading and unloading the shared. One must not expect any
-of the other elements to be available or initialized.
-
-@item const char *__from_name
-@itemx const char *__to_name
-@code{__from_name} and @code{__to_name} contain the names of the source and
-destination character sets. They can be used to identify the actual
-conversion to be carried out since one module might implement conversions
-for more than one character set and/or direction.
-
-@item gconv_fct __fct
-@itemx gconv_init_fct __init_fct
-@itemx gconv_end_fct __end_fct
-These elements contain pointers to the functions in the loadable module.
-The interface will be explained below.
-
-@item int __min_needed_from
-@itemx int __max_needed_from
-@itemx int __min_needed_to
-@itemx int __max_needed_to;
-These values have to be supplied in the init function of the module. The
-@code{__min_needed_from} value specifies how many bytes a character of
-the source character set at least needs. The @code{__max_needed_from}
-specifies the maximum value that also includes possible shift sequences.
-
-The @code{__min_needed_to} and @code{__max_needed_to} values serve the
-same purpose as @code{__min_needed_from} and @code{__max_needed_from} but
-this time for the destination character set.
-
-It is crucial that these values be accurate since otherwise the
-conversion functions will have problems or not work at all.
-
-@item int __stateful
-This element must also be initialized by the init function.
-@code{int __stateful} is nonzero if the source character set is stateful.
-Otherwise it is zero.
-
-@item void *__data
-This element can be used freely by the conversion functions in the
-module. @code{void *__data} can be used to communicate extra information
-from one call to another. @code{void *__data} need not be initialized if
-not needed at all. If @code{void *__data} element is assigned a pointer
-to dynamically allocated memory (presumably in the init function) it has
-to be made sure that the end function deallocates the memory. Otherwise
-the application will leak memory.
-
-It is important to be aware that this data structure is shared by all
-users of this specification conversion and therefore the @code{__data}
-element must not contain data specific to one specific use of the
-conversion function.
-@end table
-@end deftp
-
-@comment gconv.h
-@comment GNU
-@deftp {Data type} {struct __gconv_step_data}
-This is the data structure that contains the information specific to
-each use of the conversion functions.
-
-
-@table @code
-@item char *__outbuf
-@itemx char *__outbufend
-These elements specify the output buffer for the conversion step. The
-@code{__outbuf} element points to the beginning of the buffer, and
-@code{__outbufend} points to the byte following the last byte in the
-buffer. The conversion function must not assume anything about the size
-of the buffer but it can be safely assumed the there is room for at
-least one complete character in the output buffer.
-
-Once the conversion is finished, if the conversion is the last step, the
-@code{__outbuf} element must be modified to point after the last byte
-written into the buffer to signal how much output is available. If this
-conversion step is not the last one, the element must not be modified.
-The @code{__outbufend} element must not be modified.
-
-@item int __is_last
-This element is nonzero if this conversion step is the last one. This
-information is necessary for the recursion. See the description of the
-conversion function internals below. This element must never be
-modified.
-
-@item int __invocation_counter
-The conversion function can use this element to see how many calls of
-the conversion function already happened. Some character sets require a
-certain prolog when generating output, and by comparing this value with
-zero, one can find out whether it is the first call and whether,
-therefore, the prolog should be emitted. This element must never be
-modified.
-
-@item int __internal_use
-This element is another one rarely used but needed in certain
-situations. It is assigned a nonzero value in case the conversion
-functions are used to implement @code{mbsrtowcs} et.al.@: (i.e., the
-function is not used directly through the @code{iconv} interface).
-
-This sometimes makes a difference as it is expected that the
-@code{iconv} functions are used to translate entire texts while the
-@code{mbsrtowcs} functions are normally used only to convert single
-strings and might be used multiple times to convert entire texts.
-
-But in this situation we would have problem complying with some rules of
-the character set specification. Some character sets require a prolog
-which must appear exactly once for an entire text. If a number of
-@code{mbsrtowcs} calls are used to convert the text, only the first call
-must add the prolog. However, because there is no communication between the
-different calls of @code{mbsrtowcs}, the conversion functions have no
-possibility to find this out. The situation is different for sequences
-of @code{iconv} calls since the handle allows access to the needed
-information.
-
-The @code{int __internal_use} element is mostly used together with
-@code{__invocation_counter} as follows:
-
-@smallexample
-if (!data->__internal_use
- && data->__invocation_counter == 0)
- /* @r{Emit prolog.} */
- ...
-@end smallexample
-
-This element must never be modified.
-
-@item mbstate_t *__statep
-The @code{__statep} element points to an object of type @code{mbstate_t}
-(@pxref{Keeping the state}). The conversion of a stateful character
-set must use the object pointed to by @code{__statep} to store
-information about the conversion state. The @code{__statep} element
-itself must never be modified.
-
-@item mbstate_t __state
-This element must @emph{never} be used directly. It is only part of
-this structure to have the needed space allocated.
-@end table
-@end deftp
-
-@subsubsection @code{iconv} module interfaces
-
-With the knowledge about the data structures we now can describe the
-conversion function itself. To understand the interface a bit of
-knowledge is necessary about the functionality in the C library that
-loads the objects with the conversions.
-
-It is often the case that one conversion is used more than once (i.e.,
-there are several @code{iconv_open} calls for the same set of character
-sets during one program run). The @code{mbsrtowcs} et.al.@: functions in
-the GNU C library also use the @code{iconv} functionality, which
-increases the number of uses of the same functions even more.
-
-Because of this multiple use of conversions, the modules do not get
-loaded exclusively for one conversion. Instead a module once loaded can
-be used by an arbitrary number of @code{iconv} or @code{mbsrtowcs} calls
-at the same time. The splitting of the information between conversion-
-function-specific information and conversion data makes this possible.
-The last section showed the two data structures used to do this.
-
-This is of course also reflected in the interface and semantics of the
-functions that the modules must provide. There are three functions that
-must have the following names:
-
-@table @code
-@item gconv_init
-The @code{gconv_init} function initializes the conversion function
-specific data structure. This very same object is shared by all
-conversions that use this conversion and, therefore, no state information
-about the conversion itself must be stored in here. If a module
-implements more than one conversion, the @code{gconv_init} function will
-be called multiple times.
-
-@item gconv_end
-The @code{gconv_end} function is responsible for freeing all resources
-allocated by the @code{gconv_init} function. If there is nothing to do,
-this function can be missing. Special care must be taken if the module
-implements more than one conversion and the @code{gconv_init} function
-does not allocate the same resources for all conversions.
-
-@item gconv
-This is the actual conversion function. It is called to convert one
-block of text. It gets passed the conversion step information
-initialized by @code{gconv_init} and the conversion data, specific to
-this use of the conversion functions.
-@end table
-
-There are three data types defined for the three module interface
-functions and these define the interface.
-
-@comment gconv.h
-@comment GNU
-@deftypevr {Data type} int {(*__gconv_init_fct)} (struct __gconv_step *)
-This specifies the interface of the initialization function of the
-module. It is called exactly once for each conversion the module
-implements.
-
-As explained in the description of the @code{struct __gconv_step} data
-structure above the initialization function has to initialize parts of
-it.
-
-@table @code
-@item __min_needed_from
-@itemx __max_needed_from
-@itemx __min_needed_to
-@itemx __max_needed_to
-These elements must be initialized to the exact numbers of the minimum
-and maximum number of bytes used by one character in the source and
-destination character sets, respectively. If the characters all have the
-same size, the minimum and maximum values are the same.
-
-@item __stateful
-This element must be initialized to an nonzero value if the source
-character set is stateful. Otherwise it must be zero.
-@end table
-
-If the initialization function needs to communicate some information
-to the conversion function, this communication can happen using the
-@code{__data} element of the @code{__gconv_step} structure. But since
-this data is shared by all the conversions, it must not be modified by
-the conversion function. The example below shows how this can be used.
-
-@smallexample
-#define MIN_NEEDED_FROM 1
-#define MAX_NEEDED_FROM 4
-#define MIN_NEEDED_TO 4
-#define MAX_NEEDED_TO 4
-
-int
-gconv_init (struct __gconv_step *step)
-@{
- /* @r{Determine which direction.} */
- struct iso2022jp_data *new_data;
- enum direction dir = illegal_dir;
- enum variant var = illegal_var;
- int result;
-
- if (__strcasecmp (step->__from_name, "ISO-2022-JP//") == 0)
- @{
- dir = from_iso2022jp;
- var = iso2022jp;
- @}
- else if (__strcasecmp (step->__to_name, "ISO-2022-JP//") == 0)
- @{
- dir = to_iso2022jp;
- var = iso2022jp;
- @}
- else if (__strcasecmp (step->__from_name, "ISO-2022-JP-2//") == 0)
- @{
- dir = from_iso2022jp;
- var = iso2022jp2;
- @}
- else if (__strcasecmp (step->__to_name, "ISO-2022-JP-2//") == 0)
- @{
- dir = to_iso2022jp;
- var = iso2022jp2;
- @}
-
- result = __GCONV_NOCONV;
- if (dir != illegal_dir)
- @{
- new_data = (struct iso2022jp_data *)
- malloc (sizeof (struct iso2022jp_data));
-
- result = __GCONV_NOMEM;
- if (new_data != NULL)
- @{
- new_data->dir = dir;
- new_data->var = var;
- step->__data = new_data;
-
- if (dir == from_iso2022jp)
- @{
- step->__min_needed_from = MIN_NEEDED_FROM;
- step->__max_needed_from = MAX_NEEDED_FROM;
- step->__min_needed_to = MIN_NEEDED_TO;
- step->__max_needed_to = MAX_NEEDED_TO;
- @}
- else
- @{
- step->__min_needed_from = MIN_NEEDED_TO;
- step->__max_needed_from = MAX_NEEDED_TO;
- step->__min_needed_to = MIN_NEEDED_FROM;
- step->__max_needed_to = MAX_NEEDED_FROM + 2;
- @}
-
- /* @r{Yes, this is a stateful encoding.} */
- step->__stateful = 1;
-
- result = __GCONV_OK;
- @}
- @}
-
- return result;
-@}
-@end smallexample
-
-The function first checks which conversion is wanted. The module from
-which this function is taken implements four different conversions;
-which one is selected can be determined by comparing the names. The
-comparison should always be done without paying attention to the case.
-
-Next, a data structure, which contains the necessary information about
-which conversion is selected, is allocated. The data structure
-@code{struct iso2022jp_data} is locally defined since, outside the
-module, this data is not used at all. Please note that if all four
-conversions this modules supports are requested there are four data
-blocks.
-
-One interesting thing is the initialization of the @code{__min_} and
-@code{__max_} elements of the step data object. A single ISO-2022-JP
-character can consist of one to four bytes. Therefore the
-@code{MIN_NEEDED_FROM} and @code{MAX_NEEDED_FROM} macros are defined
-this way. The output is always the @code{INTERNAL} character set (aka
-UCS-4) and therefore each character consists of exactly four bytes. For
-the conversion from @code{INTERNAL} to ISO-2022-JP we have to take into
-account that escape sequences might be necessary to switch the character
-sets. Therefore the @code{__max_needed_to} element for this direction
-gets assigned @code{MAX_NEEDED_FROM + 2}. This takes into account the
-two bytes needed for the escape sequences to single the switching. The
-asymmetry in the maximum values for the two directions can be explained
-easily: when reading ISO-2022-JP text, escape sequences can be handled
-alone (i.e., it is not necessary to process a real character since the
-effect of the escape sequence can be recorded in the state information).
-The situation is different for the other direction. Since it is in
-general not known which character comes next, one cannot emit escape
-sequences to change the state in advance. This means the escape
-sequences that have to be emitted together with the next character.
-Therefore one needs more room than only for the character itself.
-
-The possible return values of the initialization function are:
-
-@table @code
-@item __GCONV_OK
-The initialization succeeded
-@item __GCONV_NOCONV
-The requested conversion is not supported in the module. This can
-happen if the @file{gconv-modules} file has errors.
-@item __GCONV_NOMEM
-Memory required to store additional information could not be allocated.
-@end table
-@end deftypevr
-
-The function called before the module is unloaded is significantly
-easier. It often has nothing at all to do; in which case it can be left
-out completely.
-
-@comment gconv.h
-@comment GNU
-@deftypevr {Data type} void {(*__gconv_end_fct)} (struct gconv_step *)
-The task of this function is to free all resources allocated in the
-initialization function. Therefore only the @code{__data} element of
-the object pointed to by the argument is of interest. Continuing the
-example from the initialization function, the finalization function
-looks like this:
-
-@smallexample
-void
-gconv_end (struct __gconv_step *data)
-@{
- free (data->__data);
-@}
-@end smallexample
-@end deftypevr
-
-The most important function is the conversion function itself, which can
-get quite complicated for complex character sets. But since this is not
-of interest here, we will only describe a possible skeleton for the
-conversion function.
-
-@comment gconv.h
-@comment GNU
-@deftypevr {Data type} int {(*__gconv_fct)} (struct __gconv_step *, struct __gconv_step_data *, const char **, const char *, size_t *, int)
-The conversion function can be called for two basic reason: to convert
-text or to reset the state. From the description of the @code{iconv}
-function it can be seen why the flushing mode is necessary. What mode
-is selected is determined by the sixth argument, an integer. This
-argument being nonzero means that flushing is selected.
-
-Common to both modes is where the output buffer can be found. The
-information about this buffer is stored in the conversion step data. A
-pointer to this information is passed as the second argument to this
-function. The description of the @code{struct __gconv_step_data}
-structure has more information on the conversion step data.
-
-@cindex stateful
-What has to be done for flushing depends on the source character set.
-If the source character set is not stateful, nothing has to be done.
-Otherwise the function has to emit a byte sequence to bring the state
-object into the initial state. Once this all happened the other
-conversion modules in the chain of conversions have to get the same
-chance. Whether another step follows can be determined from the
-@code{__is_last} element of the step data structure to which the first
-parameter points.
-
-The more interesting mode is when actual text has to be converted. The
-first step in this case is to convert as much text as possible from the
-input buffer and store the result in the output buffer. The start of the
-input buffer is determined by the third argument which is a pointer to a
-pointer variable referencing the beginning of the buffer. The fourth
-argument is a pointer to the byte right after the last byte in the buffer.
-
-The conversion has to be performed according to the current state if the
-character set is stateful. The state is stored in an object pointed to
-by the @code{__statep} element of the step data (second argument). Once
-either the input buffer is empty or the output buffer is full the
-conversion stops. At this point, the pointer variable referenced by the
-third parameter must point to the byte following the last processed
-byte (i.e., if all of the input is consumed, this pointer and the fourth
-parameter have the same value).
-
-What now happens depends on whether this step is the last one. If it is
-the last step, the only thing that has to be done is to update the
-@code{__outbuf} element of the step data structure to point after the
-last written byte. This update gives the caller the information on how
-much text is available in the output buffer. In addition, the variable
-pointed to by the fifth parameter, which is of type @code{size_t}, must
-be incremented by the number of characters (@emph{not bytes}) that were
-converted in a non-reversible way. Then, the function can return.
-
-In case the step is not the last one, the later conversion functions have
-to get a chance to do their work. Therefore, the appropriate conversion
-function has to be called. The information about the functions is
-stored in the conversion data structures, passed as the first parameter.
-This information and the step data are stored in arrays, so the next
-element in both cases can be found by simple pointer arithmetic:
-
-@smallexample
-int
-gconv (struct __gconv_step *step, struct __gconv_step_data *data,
- const char **inbuf, const char *inbufend, size_t *written,
- int do_flush)
-@{
- struct __gconv_step *next_step = step + 1;
- struct __gconv_step_data *next_data = data + 1;
- ...
-@end smallexample
-
-The @code{next_step} pointer references the next step information and
-@code{next_data} the next data record. The call of the next function
-therefore will look similar to this:
-
-@smallexample
- next_step->__fct (next_step, next_data, &outerr, outbuf,
- written, 0)
-@end smallexample
-
-But this is not yet all. Once the function call returns the conversion
-function might have some more to do. If the return value of the function
-is @code{__GCONV_EMPTY_INPUT}, more room is available in the output
-buffer. Unless the input buffer is empty the conversion, functions start
-all over again and process the rest of the input buffer. If the return
-value is not @code{__GCONV_EMPTY_INPUT}, something went wrong and we have
-to recover from this.
-
-A requirement for the conversion function is that the input buffer
-pointer (the third argument) always point to the last character that
-was put in converted form into the output buffer. This is trivially
-true after the conversion performed in the current step, but if the
-conversion functions deeper downstream stop prematurely, not all
-characters from the output buffer are consumed and, therefore, the input
-buffer pointers must be backed off to the right position.
-
-Correcting the input buffers is easy to do if the input and output
-character sets have a fixed width for all characters. In this situation
-we can compute how many characters are left in the output buffer and,
-therefore, can correct the input buffer pointer appropriately with a
-similar computation. Things are getting tricky if either character set
-has characters represented with variable length byte sequences, and it
-gets even more complicated if the conversion has to take care of the
-state. In these cases the conversion has to be performed once again, from
-the known state before the initial conversion (i.e., if necessary the
-state of the conversion has to be reset and the conversion loop has to be
-executed again). The difference now is that it is known how much input
-must be created, and the conversion can stop before converting the first
-unused character. Once this is done the input buffer pointers must be
-updated again and the function can return.
-
-One final thing should be mentioned. If it is necessary for the
-conversion to know whether it is the first invocation (in case a prolog
-has to be emitted), the conversion function should increment the
-@code{__invocation_counter} element of the step data structure just
-before returning to the caller. See the description of the @code{struct
-__gconv_step_data} structure above for more information on how this can
-be used.
-
-The return value must be one of the following values:
-
-@table @code
-@item __GCONV_EMPTY_INPUT
-All input was consumed and there is room left in the output buffer.
-@item __GCONV_FULL_OUTPUT
-No more room in the output buffer. In case this is not the last step
-this value is propagated down from the call of the next conversion
-function in the chain.
-@item __GCONV_INCOMPLETE_INPUT
-The input buffer is not entirely empty since it contains an incomplete
-character sequence.
-@end table
-
-The following example provides a framework for a conversion function.
-In case a new conversion has to be written the holes in this
-implementation have to be filled and that is it.
-
-@smallexample
-int
-gconv (struct __gconv_step *step, struct __gconv_step_data *data,
- const char **inbuf, const char *inbufend, size_t *written,
- int do_flush)
-@{
- struct __gconv_step *next_step = step + 1;
- struct __gconv_step_data *next_data = data + 1;
- gconv_fct fct = next_step->__fct;
- int status;
-
- /* @r{If the function is called with no input this means we have}
- @r{to reset to the initial state. The possibly partly}
- @r{converted input is dropped.} */
- if (do_flush)
- @{
- status = __GCONV_OK;
-
- /* @r{Possible emit a byte sequence which put the state object}
- @r{into the initial state.} */
-
- /* @r{Call the steps down the chain if there are any but only}
- @r{if we successfully emitted the escape sequence.} */
- if (status == __GCONV_OK && ! data->__is_last)
- status = fct (next_step, next_data, NULL, NULL,
- written, 1);
- @}
- else
- @{
- /* @r{We preserve the initial values of the pointer variables.} */
- const char *inptr = *inbuf;
- char *outbuf = data->__outbuf;
- char *outend = data->__outbufend;
- char *outptr;
-
- do
- @{
- /* @r{Remember the start value for this round.} */
- inptr = *inbuf;
- /* @r{The outbuf buffer is empty.} */
- outptr = outbuf;
-
- /* @r{For stateful encodings the state must be safe here.} */
-
- /* @r{Run the conversion loop. @code{status} is set}
- @r{appropriately afterwards.} */
-
- /* @r{If this is the last step, leave the loop. There is}
- @r{nothing we can do.} */
- if (data->__is_last)
- @{
- /* @r{Store information about how many bytes are}
- @r{available.} */
- data->__outbuf = outbuf;
-
- /* @r{If any non-reversible conversions were performed,}
- @r{add the number to @code{*written}.} */
-
- break;
- @}
-
- /* @r{Write out all output which was produced.} */
- if (outbuf > outptr)
- @{
- const char *outerr = data->__outbuf;
- int result;
-
- result = fct (next_step, next_data, &outerr,
- outbuf, written, 0);
-
- if (result != __GCONV_EMPTY_INPUT)
- @{
- if (outerr != outbuf)
- @{
- /* @r{Reset the input buffer pointer. We}
- @r{document here the complex case.} */
- size_t nstatus;
-
- /* @r{Reload the pointers.} */
- *inbuf = inptr;
- outbuf = outptr;
-
- /* @r{Possibly reset the state.} */
-
- /* @r{Redo the conversion, but this time}
- @r{the end of the output buffer is at}
- @r{@code{outerr}.} */
- @}
-
- /* @r{Change the status.} */
- status = result;
- @}
- else
- /* @r{All the output is consumed, we can make}
- @r{ another run if everything was ok.} */
- if (status == __GCONV_FULL_OUTPUT)
- status = __GCONV_OK;
- @}
- @}
- while (status == __GCONV_OK);
-
- /* @r{We finished one use of this step.} */
- ++data->__invocation_counter;
- @}
-
- return status;
-@}
-@end smallexample
-@end deftypevr
-
-This information should be sufficient to write new modules. Anybody
-doing so should also take a look at the available source code in the GNU
-C library sources. It contains many examples of working and optimized
-modules.
-
+@node Character Set Handling, Locales, String and Array Utilities, Top +@c %MENU% Support for extended character sets +@chapter Character Set Handling + +@ifnottex +@macro cal{text} +\text\ +@end macro +@end ifnottex + +Character sets used in the early days of computing had only six, seven, +or eight bits for each character: there was never a case where more than +eight bits (one byte) were used to represent a single character. The +limitations of this approach became more apparent as more people +grappled with non-Roman character sets, where not all the characters +that make up a language's character set can be represented by @math{2^8} +choices. This chapter shows the functionality that was added to the C +library to support multiple character sets. + +@menu +* Extended Char Intro:: Introduction to Extended Characters. +* Charset Function Overview:: Overview about Character Handling + Functions. +* Restartable multibyte conversion:: Restartable multibyte conversion + Functions. +* Non-reentrant Conversion:: Non-reentrant Conversion Function. +* Generic Charset Conversion:: Generic Charset Conversion. +@end menu + + +@node Extended Char Intro +@section Introduction to Extended Characters + +A variety of solutions is available to overcome the differences between +character sets with a 1:1 relation between bytes and characters and +character sets with ratios of 2:1 or 4:1. The remainder of this +section gives a few examples to help understand the design decisions +made while developing the functionality of the @w{C library}. + +@cindex internal representation +A distinction we have to make right away is between internal and +external representation. @dfn{Internal representation} means the +representation used by a program while keeping the text in memory. +External representations are used when text is stored or transmitted +through some communication channel. Examples of external +representations include files waiting in a directory to be +read and parsed. + +Traditionally there has been no difference between the two representations. +It was equally comfortable and useful to use the same single-byte +representation internally and externally. This comfort level decreases +with more and larger character sets. + +One of the problems to overcome with the internal representation is +handling text that is externally encoded using different character +sets. Assume a program that reads two texts and compares them using +some metric. The comparison can be usefully done only if the texts are +internally kept in a common format. + +@cindex wide character +For such a common format (@math{=} character set) eight bits are certainly +no longer enough. So the smallest entity will have to grow: @dfn{wide +characters} will now be used. Instead of one byte per character, two or +four will be used instead. (Three are not good to address in memory and +more than four bytes seem not to be necessary). + +@cindex Unicode +@cindex ISO 10646 +As shown in some other part of this manual, +@c !!! Ahem, wide char string functions are not yet covered -- drepper +a completely new family has been created of functions that can handle wide +character texts in memory. The most commonly used character sets for such +internal wide character representations are Unicode and @w{ISO 10646} +(also known as UCS for Universal Character Set). Unicode was originally +planned as a 16-bit character set; whereas, @w{ISO 10646} was designed to +be a 31-bit large code space. The two standards are practically identical. +They have the same character repertoire and code table, but Unicode specifies +added semantics. At the moment, only characters in the first @code{0x10000} +code positions (the so-called Basic Multilingual Plane, BMP) have been +assigned, but the assignment of more specialized characters outside this +16-bit space is already in progress. A number of encodings have been +defined for Unicode and @w{ISO 10646} characters: +@cindex UCS-2 +@cindex UCS-4 +@cindex UTF-8 +@cindex UTF-16 +UCS-2 is a 16-bit word that can only represent characters +from the BMP, UCS-4 is a 32-bit word than can represent any Unicode +and @w{ISO 10646} character, UTF-8 is an ASCII compatible encoding where +ASCII characters are represented by ASCII bytes and non-ASCII characters +by sequences of 2-6 non-ASCII bytes, and finally UTF-16 is an extension +of UCS-2 in which pairs of certain UCS-2 words can be used to encode +non-BMP characters up to @code{0x10ffff}. + +To represent wide characters the @code{char} type is not suitable. For +this reason the @w{ISO C} standard introduces a new type that is +designed to keep one character of a wide character string. To maintain +the similarity there is also a type corresponding to @code{int} for +those functions that take a single wide character. + +@comment stddef.h +@comment ISO +@deftp {Data type} wchar_t +This data type is used as the base type for wide character strings. +In other words, arrays of objects of this type are the equivalent of +@code{char[]} for multibyte character strings. The type is defined in +@file{stddef.h}. + +The @w{ISO C90} standard, where @code{wchar_t} was introduced, does not +say anything specific about the representation. It only requires that +this type is capable of storing all elements of the basic character set. +Therefore it would be legitimate to define @code{wchar_t} as @code{char}, +which might make sense for embedded systems. + +But for GNU systems @code{wchar_t} is always 32 bits wide and, therefore, +capable of representing all UCS-4 values and, therefore, covering all of +@w{ISO 10646}. Some Unix systems define @code{wchar_t} as a 16-bit type +and thereby follow Unicode very strictly. This definition is perfectly +fine with the standard, but it also means that to represent all +characters from Unicode and @w{ISO 10646} one has to use UTF-16 surrogate +characters, which is in fact a multi-wide-character encoding. But +resorting to multi-wide-character encoding contradicts the purpose of the +@code{wchar_t} type. +@end deftp + +@comment wchar.h +@comment ISO +@deftp {Data type} wint_t +@code{wint_t} is a data type used for parameters and variables that +contain a single wide character. As the name suggests this type is the +equivalent of @code{int} when using the normal @code{char} strings. The +types @code{wchar_t} and @code{wint_t} often have the same +representation if their size is 32 bits wide but if @code{wchar_t} is +defined as @code{char} the type @code{wint_t} must be defined as +@code{int} due to the parameter promotion. + +@pindex wchar.h +This type is defined in @file{wchar.h} and was introduced in +@w{Amendment 1} to @w{ISO C90}. +@end deftp + +As there are for the @code{char} data type macros are available for +specifying the minimum and maximum value representable in an object of +type @code{wchar_t}. + +@comment wchar.h +@comment ISO +@deftypevr Macro wint_t WCHAR_MIN +The macro @code{WCHAR_MIN} evaluates to the minimum value representable +by an object of type @code{wint_t}. + +This macro was introduced in @w{Amendment 1} to @w{ISO C90}. +@end deftypevr + +@comment wchar.h +@comment ISO +@deftypevr Macro wint_t WCHAR_MAX +The macro @code{WCHAR_MAX} evaluates to the maximum value representable +by an object of type @code{wint_t}. + +This macro was introduced in @w{Amendment 1} to @w{ISO C90}. +@end deftypevr + +Another special wide character value is the equivalent to @code{EOF}. + +@comment wchar.h +@comment ISO +@deftypevr Macro wint_t WEOF +The macro @code{WEOF} evaluates to a constant expression of type +@code{wint_t} whose value is different from any member of the extended +character set. + +@code{WEOF} need not be the same value as @code{EOF} and unlike +@code{EOF} it also need @emph{not} be negative. In other words, sloppy +code like + +@smallexample +@{ + int c; + ... + while ((c = getc (fp)) < 0) + ... +@} +@end smallexample + +@noindent +has to be rewritten to use @code{WEOF} explicitly when wide characters +are used: + +@smallexample +@{ + wint_t c; + ... + while ((c = wgetc (fp)) != WEOF) + ... +@} +@end smallexample + +@pindex wchar.h +This macro was introduced in @w{Amendment 1} to @w{ISO C90} and is +defined in @file{wchar.h}. +@end deftypevr + + +These internal representations present problems when it comes to storing +and transmittal. Because each single wide character consists of more +than one byte, they are effected by byte-ordering. Thus, machines with +different endianesses would see different values when accessing the same +data. This byte ordering concern also applies for communication protocols +that are all byte-based and, thereforet require that the sender has to +decide about splitting the wide character in bytes. A last (but not least +important) point is that wide characters often require more storage space +than a customized byte-oriented character set. + +@cindex multibyte character +@cindex EBCDIC + For all the above reasons, an external encoding that is different +from the internal encoding is often used if the latter is UCS-2 or UCS-4. +The external encoding is byte-based and can be chosen appropriately for +the environment and for the texts to be handled. A variety of different +character sets can be used for this external encoding (information that +will not be exhaustively presented here--instead, a description of the +major groups will suffice). All of the ASCII-based character sets +[_bkoz_: do you mean Roman character sets? If not, what do you mean +here?] fulfill one requirement: they are "filesystem safe." This means +that the character @code{'/'} is used in the encoding @emph{only} to +represent itself. Things are a bit different for character sets like +EBCDIC (Extended Binary Coded Decimal Interchange Code, a character set +family used by IBM), but if the operation system does not understand +EBCDIC directly the parameters-to-system calls have to be converted first +anyhow. + +@itemize @bullet +@item +The simplest character sets are single-byte character sets. There can +be only up to 256 characters (for @w{8 bit} character sets), which is +not sufficient to cover all languages but might be sufficient to handle +a specific text. Handling of a @w{8 bit} character sets is simple. This +is not true for other kinds presented later, and therefore, the +application one uses might require the use of @w{8 bit} character sets. + +@cindex ISO 2022 +@item +The @w{ISO 2022} standard defines a mechanism for extended character +sets where one character @emph{can} be represented by more than one +byte. This is achieved by associating a state with the text. +Characters that can be used to change the state can be embedded in the +text. Each byte in the text might have a different interpretation in each +state. The state might even influence whether a given byte stands for a +character on its own or whether it has to be combined with some more +bytes. + +@cindex EUC +@cindex Shift_JIS +@cindex SJIS +In most uses of @w{ISO 2022} the defined character sets do not allow +state changes that cover more than the next character. This has the +big advantage that whenever one can identify the beginning of the byte +sequence of a character one can interpret a text correctly. Examples of +character sets using this policy are the various EUC character sets +(used by Sun's operations systems, EUC-JP, EUC-KR, EUC-TW, and EUC-CN) +or Shift_JIS (SJIS, a Japanese encoding). + +But there are also character sets using a state that is valid for more +than one character and has to be changed by another byte sequence. +Examples for this are ISO-2022-JP, ISO-2022-KR, and ISO-2022-CN. + +@item +@cindex ISO 6937 +Early attempts to fix 8 bit character sets for other languages using the +Roman alphabet lead to character sets like @w{ISO 6937}. Here bytes +representing characters like the acute accent do not produce output +themselves: one has to combine them with other characters to get the +desired result. For example, the byte sequence @code{0xc2 0x61} +(non-spacing acute accent, followed by lower-case `a') to get the ``small +a with acute'' character. To get the acute accent character on its own, +one has to write @code{0xc2 0x20} (the non-spacing acute followed by a +space). + +Character sets like @w[ISO 6937] are used in some embedded systems such +as teletex. + +@item +@cindex UTF-8 +Instead of converting the Unicode or @w{ISO 10646} text used internally, +it is often also sufficient to simply use an encoding different than +UCS-2/UCS-4. The Unicode and @w{ISO 10646} standards even specify such an +encoding: UTF-8. This encoding is able to represent all of @w{ISO +10646} 31 bits in a byte string of length one to six. + +@cindex UTF-7 +There were a few other attempts to encode @w{ISO 10646} such as UTF-7, +but UTF-8 is today the only encoding that should be used. In fact, with +any luck UTF-8 will soon be the only external encoding that has to be +supported. It proves to be universally usable and its only disadvantage +is that it favors Roman languages by making the byte string +representation of other scripts (Cyrillic, Greek, Asian scripts) longer +than necessary if using a specific character set for these scripts. +Methods like the Unicode compression scheme can alleviate these +problems. +@end itemize + +The question remaining is: how to select the character set or encoding +to use. The answer: you cannot decide about it yourself, it is decided +by the developers of the system or the majority of the users. Since the +goal is interoperability one has to use whatever the other people one +works with use. If there are no constraints, the selection is based on +the requirements the expected circle of users will have. In other words, +if a project is expected to be used in only, say, Russia it is fine to use +KOI8-R or a similar character set. But if at the same time people from, +say, Greece are participating one should use a character set that allows +all people to collaborate. + +The most widely useful solution seems to be: go with the most general +character set, namely @w{ISO 10646}. Use UTF-8 as the external encoding +and problems about users not being able to use their own language +adequately are a thing of the past. + +One final comment about the choice of the wide character representation +is necessary at this point. We have said above that the natural choice +is using Unicode or @w{ISO 10646}. This is not required, but at least +encouraged, by the @w{ISO C} standard. The standard defines at least a +macro @code{__STDC_ISO_10646__} that is only defined on systems where +the @code{wchar_t} type encodes @w{ISO 10646} characters. If this +symbol is not defined one should avoid making assumptions about the wide +character representation. If the programmer uses only the functions +provided by the C library to handle wide character strings there should +be no compatibility problems with other systems. + +@node Charset Function Overview +@section Overview about Character Handling Functions + +A Unix @w{C library} contains three different sets of functions in two +families to handle character set conversion. One of the function families +(the most commonly used) is specified in the @w{ISO C90} standard and, +therefore, is portable even beyond the Unix world. Unfortunately this +family is the least useful one. These functions should be avoided +whenever possible, especially when developing libraries (as opposed to +applications). + +The second family of functions got introduced in the early Unix standards +(XPG2) and is still part of the latest and greatest Unix standard: +@w{Unix 98}. It is also the most powerful and useful set of functions. +But we will start with the functions defined in @w{Amendment 1} to +@w{ISO C90}. + +@node Restartable multibyte conversion +@section Restartable Multibyte Conversion Functions + +The @w{ISO C} standard defines functions to convert strings from a +multibyte representation to wide character strings. There are a number +of peculiarities: + +@itemize @bullet +@item +The character set assumed for the multibyte encoding is not specified +as an argument to the functions. Instead the character set specified by +the @code{LC_CTYPE} category of the current locale is used; see +@ref{Locale Categories}. + +@item +The functions handling more than one character at a time require NUL +terminated strings as the argument (i.e., converting blocks of text +does not work unless one can add a NUL byte at an appropriate place). +The GNU C library contains some extensions to the standard that allow +specifying a size, but basically they also expect terminated strings. +@end itemize + +Despite these limitations the @w{ISO C} functions can be used in many +contexts. In graphical user interfaces, for instance, it is not +uncommon to have functions that require text to be displayed in a wide +character string if the text is not simple ASCII. The text itself might +come from a file with translations and the user should decide about the +current locale, which determines the translation and therefore also the +external encoding used. In such a situation (and many others) the +functions described here are perfect. If more freedom while performing +the conversion is necessary take a look at the @code{iconv} functions +(@pxref{Generic Charset Conversion}). + +@menu +* Selecting the Conversion:: Selecting the conversion and its properties. +* Keeping the state:: Representing the state of the conversion. +* Converting a Character:: Converting Single Characters. +* Converting Strings:: Converting Multibyte and Wide Character + Strings. +* Multibyte Conversion Example:: A Complete Multibyte Conversion Example. +@end menu + +@node Selecting the Conversion +@subsection Selecting the conversion and its properties + +We already said above that the currently selected locale for the +@code{LC_CTYPE} category decides about the conversion that is performed +by the functions we are about to describe. Each locale uses its own +character set (given as an argument to @code{localedef}) and this is the +one assumed as the external multibyte encoding. The wide character +character set always is UCS-4, at least on GNU systems. + +A characteristic of each multibyte character set is the maximum number +of bytes that can be necessary to represent one character. This +information is quite important when writing code that uses the +conversion functions (as shown in the examples below). +The @w{ISO C} standard defines two macros that provide this information. + + +@comment limits.h +@comment ISO +@deftypevr Macro int MB_LEN_MAX +@code{MB_LEN_MAX} specifies the maximum number of bytes in the multibyte +sequence for a single character in any of the supported locales. It is +a compile-time constant and is defined in @file{limits.h}. +@pindex limits.h +@end deftypevr + +@comment stdlib.h +@comment ISO +@deftypevr Macro int MB_CUR_MAX +@code{MB_CUR_MAX} expands into a positive integer expression that is the +maximum number of bytes in a multibyte character in the current locale. +The value is never greater than @code{MB_LEN_MAX}. Unlike +@code{MB_LEN_MAX} this macro need not be a compile-time constant, and in +the GNU C library it is not. + +@pindex stdlib.h +@code{MB_CUR_MAX} is defined in @file{stdlib.h}. +@end deftypevr + +Two different macros are necessary since strictly @w{ISO C90} compilers +do not allow variable length array definitions, but still it is desirable +to avoid dynamic allocation. This incomplete piece of code shows the +problem: + +@smallexample +@{ + char buf[MB_LEN_MAX]; + ssize_t len = 0; + + while (! feof (fp)) + @{ + fread (&buf[len], 1, MB_CUR_MAX - len, fp); + /* @r{... process} buf */ + len -= used; + @} +@} +@end smallexample + +The code in the inner loop is expected to have always enough bytes in +the array @var{buf} to convert one multibyte character. The array +@var{buf} has to be sized statically since many compilers do not allow a +variable size. The @code{fread} call makes sure that @code{MB_CUR_MAX} +bytes are always available in @var{buf}. Note that it isn't +a problem if @code{MB_CUR_MAX} is not a compile-time constant. + + +@node Keeping the state +@subsection Representing the state of the conversion + +@cindex stateful +In the introduction of this chapter it was said that certain character +sets use a @dfn{stateful} encoding. That is, the encoded values depend +in some way on the previous bytes in the text. + +Since the conversion functions allow converting a text in more than one +step we must have a way to pass this information from one call of the +functions to another. + +@comment wchar.h +@comment ISO +@deftp {Data type} mbstate_t +@cindex shift state +A variable of type @code{mbstate_t} can contain all the information +about the @dfn{shift state} needed from one call to a conversion +function to another. + +@pindex wchar.h +@code{mbstate_t} is defined in @file{wchar.h}. It was introduced in +@w{Amendment 1} to @w{ISO C90}. +@end deftp + +To use objects of type @code{mbstate_t} the programmer has to define such +objects (normally as local variables on the stack) and pass a pointer to +the object to the conversion functions. This way the conversion function +can update the object if the current multibyte character set is stateful. + +There is no specific function or initializer to put the state object in +any specific state. The rules are that the object should always +represent the initial state before the first use, and this is achieved by +clearing the whole variable with code such as follows: + +@smallexample +@{ + mbstate_t state; + memset (&state, '\0', sizeof (state)); + /* @r{from now on @var{state} can be used.} */ + ... +@} +@end smallexample + +When using the conversion functions to generate output it is often +necessary to test whether the current state corresponds to the initial +state. This is necessary, for example, to decide whether to emit +escape sequences to set the state to the initial state at certain +sequence points. Communication protocols often require this. + +@comment wchar.h +@comment ISO +@deftypefun int mbsinit (const mbstate_t *@var{ps}) +The @code {mbsinit} function determines whether the state object pointed +to by @var{ps} is in the initial state. If @var{ps} is a null pointer or +the object is in the initial state the return value is nonzero. Otherwise +it is zero. + +@pindex wchar.h +@code {mbsinit} was introduced in @w{Amendment 1} to @w{ISO C90} and is +declared in @file{wchar.h}. +@end deftypefun + +Code using @code {mbsinit} often looks similar to this: + +@c Fix the example to explicitly say how to generate the escape sequence +@c to restore the initial state. +@smallexample +@{ + mbstate_t state; + memset (&state, '\0', sizeof (state)); + /* @r{Use @var{state}.} */ + ... + if (! mbsinit (&state)) + @{ + /* @r{Emit code to return to initial state.} */ + const wchar_t empty[] = L""; + const wchar_t *srcp = empty; + wcsrtombs (outbuf, &srcp, outbuflen, &state); + @} + ... +@} +@end smallexample + +The code to emit the escape sequence to get back to the initial state is +interesting. The @code{wcsrtombs} function can be used to determine the +necessary output code (@pxref{Converting Strings}). Please note that on +GNU systems it is not necessary to perform this extra action for the +conversion from multibyte text to wide character text since the wide +character encoding is not stateful. But there is nothing mentioned in +any standard that prohibits making @code{wchar_t} using a stateful +encoding. + +@node Converting a Character +@subsection Converting Single Characters + +The most fundamental of the conversion functions are those dealing with +single characters. Please note that this does not always mean single +bytes. But since there is very often a subset of the multibyte +character set that consists of single byte sequences, there are +functions to help with converting bytes. Frequently, ASCII is a subpart +of the multibyte character set. In such a scenario, each ASCII character +stands for itself, and all other characters have at least a first byte +that is beyond the range @math{0} to @math{127}. + +@comment wchar.h +@comment ISO +@deftypefun wint_t btowc (int @var{c}) +The @code{btowc} function (``byte to wide character'') converts a valid +single byte character @var{c} in the initial shift state into the wide +character equivalent using the conversion rules from the currently +selected locale of the @code{LC_CTYPE} category. + +If @code{(unsigned char) @var{c}} is no valid single byte multibyte +character or if @var{c} is @code{EOF}, the function returns @code{WEOF}. + +Please note the restriction of @var{c} being tested for validity only in +the initial shift state. No @code{mbstate_t} object is used from +which the state information is taken, and the function also does not use +any static state. + +@pindex wchar.h +The @code{btowc} function was introduced in @w{Amendment 1} to @w{ISO C90} +and is declared in @file{wchar.h}. +@end deftypefun + +Despite the limitation that the single byte value always is interpreted +in the initial state this function is actually useful most of the time. +Most characters are either entirely single-byte character sets or they +are extension to ASCII. But then it is possible to write code like this +(not that this specific example is very useful): + +@smallexample +wchar_t * +itow (unsigned long int val) +@{ + static wchar_t buf[30]; + wchar_t *wcp = &buf[29]; + *wcp = L'\0'; + while (val != 0) + @{ + *--wcp = btowc ('0' + val % 10); + val /= 10; + @} + if (wcp == &buf[29]) + *--wcp = L'0'; + return wcp; +@} +@end smallexample + +Why is it necessary to use such a complicated implementation and not +simply cast @code{'0' + val % 10} to a wide character? The answer is +that there is no guarantee that one can perform this kind of arithmetic +on the character of the character set used for @code{wchar_t} +representation. In other situations the bytes are not constant at +compile time and so the compiler cannot do the work. In situations like +this it is necessary @code{btowc}. + +@noindent +There also is a function for the conversion in the other direction. + +@comment wchar.h +@comment ISO +@deftypefun int wctob (wint_t @var{c}) +The @code{wctob} function (``wide character to byte'') takes as the +parameter a valid wide character. If the multibyte representation for +this character in the initial state is exactly one byte long, the return +value of this function is this character. Otherwise the return value is +@code{EOF}. + +@pindex wchar.h +@code{wctob} was introduced in @w{Amendment 1} to @w{ISO C90} and +is declared in @file{wchar.h}. +@end deftypefun + +There are more general functions to convert single character from +multibyte representation to wide characters and vice versa. These +functions pose no limit on the length of the multibyte representation +and they also do not require it to be in the initial state. + +@comment wchar.h +@comment ISO +@deftypefun size_t mbrtowc (wchar_t *restrict @var{pwc}, const char *restrict @var{s}, size_t @var{n}, mbstate_t *restrict @var{ps}) +@cindex stateful +The @code{mbrtowc} function (``multibyte restartable to wide +character'') converts the next multibyte character in the string pointed +to by @var{s} into a wide character and stores it in the wide character +string pointed to by @var{pwc}. The conversion is performed according +to the locale currently selected for the @code{LC_CTYPE} category. If +the conversion for the character set used in the locale requires a state, +the multibyte string is interpreted in the state represented by the +object pointed to by @var{ps}. If @var{ps} is a null pointer, a static, +internal state variable used only by the @code{mbrtowc} function is +used. + +If the next multibyte character corresponds to the NUL wide character, +the return value of the function is @math{0} and the state object is +afterwards in the initial state. If the next @var{n} or fewer bytes +form a correct multibyte character, the return value is the number of +bytes starting from @var{s} that form the multibyte character. The +conversion state is updated according to the bytes consumed in the +conversion. In both cases the wide character (either the @code{L'\0'} +or the one found in the conversion) is stored in the string pointed to +by @var{pwc} if @var{pwc} is not null. + +If the first @var{n} bytes of the multibyte string possibly form a valid +multibyte character but there are more than @var{n} bytes needed to +complete it, the return value of the function is @code{(size_t) -2} and +no value is stored. Please note that this can happen even if @var{n} +has a value greater than or equal to @code{MB_CUR_MAX} since the input +might contain redundant shift sequences. + +If the first @code{n} bytes of the multibyte string cannot possibly form +a valid multibyte character, no value is stored, the global variable +@code{errno} is set to the value @code{EILSEQ}, and the function returns +@code{(size_t) -1}. The conversion state is afterwards undefined. + +@pindex wchar.h +@code{mbrtowc} was introduced in @w{Amendment 1} to @w{ISO C90} and +is declared in @file{wchar.h}. +@end deftypefun + +Use of @code{mbrtowc} is straightforward. A function that copies a +multibyte string into a wide character string while at the same time +converting all lowercase characters into uppercase could look like this +(this is not the final version, just an example; it has no error +checking, and sometimes leaks memory): + +@smallexample +wchar_t * +mbstouwcs (const char *s) +@{ + size_t len = strlen (s); + wchar_t *result = malloc ((len + 1) * sizeof (wchar_t)); + wchar_t *wcp = result; + wchar_t tmp[1]; + mbstate_t state; + size_t nbytes; + + memset (&state, '\0', sizeof (state)); + while ((nbytes = mbrtowc (tmp, s, len, &state)) > 0) + @{ + if (nbytes >= (size_t) -2) + /* Invalid input string. */ + return NULL; + *result++ = towupper (tmp[0]); + len -= nbytes; + s += nbytes; + @} + return result; +@} +@end smallexample + +The use of @code{mbrtowc} should be clear. A single wide character is +stored in @code{@var{tmp}[0]}, and the number of consumed bytes is stored +in the variable @var{nbytes}. If the conversion is successful, the +uppercase variant of the wide character is stored in the @var{result} +array and the pointer to the input string and the number of available +bytes is adjusted. + +The only non-obvious thing about @code{mbrtowc} might be the way memory +is allocated for the result. The above code uses the fact that there +can never be more wide characters in the converted results than there are +bytes in the multibyte input string. This method yields a pessimistic +guess about the size of the result, and if many wide character strings +have to be constructed this way or if the strings are long, the extra +memory required to be allocated because the input string contains +multibyte characters might be significant. The allocated memory block can +be resized to the correct size before returning it, but a better solution +might be to allocate just the right amount of space for the result right +away. Unfortunately there is no function to compute the length of the wide +character string directly from the multibyte string. There is, however, a +function that does part of the work. + +@comment wchar.h +@comment ISO +@deftypefun size_t mbrlen (const char *restrict @var{s}, size_t @var{n}, mbstate_t *@var{ps}) +The @code{mbrlen} function (``multibyte restartable length'') computes +the number of at most @var{n} bytes starting at @var{s}, which form the +next valid and complete multibyte character. + +If the next multibyte character corresponds to the NUL wide character, +the return value is @math{0}. If the next @var{n} bytes form a valid +multibyte character, the number of bytes belonging to this multibyte +character byte sequence is returned. + +If the the first @var{n} bytes possibly form a valid multibyte +character but the character is incomplete, the return value is +@code{(size_t) -2}. Otherwise the multibyte character sequence is invalid +and the return value is @code{(size_t) -1}. + +The multibyte sequence is interpreted in the state represented by the +object pointed to by @var{ps}. If @var{ps} is a null pointer, a state +object local to @code{mbrlen} is used. + +@pindex wchar.h +@code{mbrlen} was introduced in @w{Amendment 1} to @w{ISO C90} and +is declared in @file{wchar.h}. +@end deftypefun + +The attentive reader now will note that @code{mbrlen} can be implemented +as + +@smallexample +mbrtowc (NULL, s, n, ps != NULL ? ps : &internal) +@end smallexample + +This is true and in fact is mentioned in the official specification. +How can this function be used to determine the length of the wide +character string created from a multibyte character string? It is not +directly usable, but we can define a function @code{mbslen} using it: + +@smallexample +size_t +mbslen (const char *s) +@{ + mbstate_t state; + size_t result = 0; + size_t nbytes; + memset (&state, '\0', sizeof (state)); + while ((nbytes = mbrlen (s, MB_LEN_MAX, &state)) > 0) + @{ + if (nbytes >= (size_t) -2) + /* @r{Something is wrong.} */ + return (size_t) -1; + s += nbytes; + ++result; + @} + return result; +@} +@end smallexample + +This function simply calls @code{mbrlen} for each multibyte character +in the string and counts the number of function calls. Please note that +we here use @code{MB_LEN_MAX} as the size argument in the @code{mbrlen} +call. This is acceptable since a) this value is larger then the length of +the longest multibyte character sequence and b) we know that the string +@var{s} ends with a NUL byte, which cannot be part of any other multibyte +character sequence but the one representing the NUL wide character. +Therefore, the @code{mbrlen} function will never read invalid memory. + +Now that this function is available (just to make this clear, this +function is @emph{not} part of the GNU C library) we can compute the +number of wide character required to store the converted multibyte +character string @var{s} using + +@smallexample +wcs_bytes = (mbslen (s) + 1) * sizeof (wchar_t); +@end smallexample + +Please note that the @code{mbslen} function is quite inefficient. The +implementation of @code{mbstouwcs} with @code{mbslen} would have to +perform the conversion of the multibyte character input string twice, and +this conversion might be quite expensive. So it is necessary to think +about the consequences of using the easier but imprecise method before +doing the work twice. + +@comment wchar.h +@comment ISO +@deftypefun size_t wcrtomb (char *restrict @var{s}, wchar_t @var{wc}, mbstate_t *restrict @var{ps}) +The @code{wcrtomb} function (``wide character restartable to +multibyte'') converts a single wide character into a multibyte string +corresponding to that wide character. + +If @var{s} is a null pointer, the function resets the state stored in +the objects pointed to by @var{ps} (or the internal @code{mbstate_t} +object) to the initial state. This can also be achieved by a call like +this: + +@smallexample +wcrtombs (temp_buf, L'\0', ps) +@end smallexample + +@noindent +since, if @var{s} is a null pointer, @code{wcrtomb} performs as if it +writes into an internal buffer, which is guaranteed to be large enough. + +If @var{wc} is the NUL wide character, @code{wcrtomb} emits, if +necessary, a shift sequence to get the state @var{ps} into the initial +state followed by a single NUL byte, which is stored in the string +@var{s}. + +Otherwise a byte sequence (possibly including shift sequences) is written +into the string @var{s}. This only happens if @var{wc} is a valid wide +character (i.e., it has a multibyte representation in the character set +selected by locale of the @code{LC_CTYPE} category). If @var{wc} is no +valid wide character, nothing is stored in the strings @var{s}, +@code{errno} is set to @code{EILSEQ}, the conversion state in @var{ps} +is undefined and the return value is @code{(size_t) -1}. + +If no error occurred the function returns the number of bytes stored in +the string @var{s}. This includes all bytes representing shift +sequences. + +One word about the interface of the function: there is no parameter +specifying the length of the array @var{s}. Instead the function +assumes that there are at least @code{MB_CUR_MAX} bytes available since +this is the maximum length of any byte sequence representing a single +character. So the caller has to make sure that there is enough space +available, otherwise buffer overruns can occur. + +@pindex wchar.h +@code{wcrtomb} was introduced in @w{Amendment 1} to @w{ISO C90} and is +declared in @file{wchar.h}. +@end deftypefun + +Using @code{wcrtomb} is as easy as using @code{mbrtowc}. The following +example appends a wide character string to a multibyte character string. +Again, the code is not really useful (or correct), it is simply here to +demonstrate the use and some problems. + +@smallexample +char * +mbscatwcs (char *s, size_t len, const wchar_t *ws) +@{ + mbstate_t state; + /* @r{Find the end of the existing string.} */ + char *wp = strchr (s, '\0'); + len -= wp - s; + memset (&state, '\0', sizeof (state)); + do + @{ + size_t nbytes; + if (len < MB_CUR_LEN) + @{ + /* @r{We cannot guarantee that the next} + @r{character fits into the buffer, so} + @r{return an error.} */ + errno = E2BIG; + return NULL; + @} + nbytes = wcrtomb (wp, *ws, &state); + if (nbytes == (size_t) -1) + /* @r{Error in the conversion.} */ + return NULL; + len -= nbytes; + wp += nbytes; + @} + while (*ws++ != L'\0'); + return s; +@} +@end smallexample + +First the function has to find the end of the string currently in the +array @var{s}. The @code{strchr} call does this very efficiently since a +requirement for multibyte character representations is that the NUL byte +is never used except to represent itself (and in this context, the end +of the string). + +After initializing the state object the loop is entered where the first +task is to make sure there is enough room in the array @var{s}. We +abort if there are not at least @code{MB_CUR_LEN} bytes available. This +is not always optimal but we have no other choice. We might have less +than @code{MB_CUR_LEN} bytes available but the next multibyte character +might also be only one byte long. At the time the @code{wcrtomb} call +returns it is too late to decide whether the buffer was large enough. If +this solution is unsuitable, there is a very slow but more accurate +solution. + +@smallexample + ... + if (len < MB_CUR_LEN) + @{ + mbstate_t temp_state; + memcpy (&temp_state, &state, sizeof (state)); + if (wcrtomb (NULL, *ws, &temp_state) > len) + @{ + /* @r{We cannot guarantee that the next} + @r{character fits into the buffer, so} + @r{return an error.} */ + errno = E2BIG; + return NULL; + @} + @} + ... +@end smallexample + +Here we perform the conversion that might overflow the buffer so that +we are afterwards in the position to make an exact decision about the +buffer size. Please note the @code{NULL} argument for the destination +buffer in the new @code{wcrtomb} call; since we are not interested in the +converted text at this point, this is a nice way to express this. The +most unusual thing about this piece of code certainly is the duplication +of the conversion state object, but if a change of the state is necessary +to emit the next multibyte character, we want to have the same shift state +change performed in the real conversion. Therefore, we have to preserve +the initial shift state information. + +There are certainly many more and even better solutions to this problem. +This example is only provided for educational purposes. + +@node Converting Strings +@subsection Converting Multibyte and Wide Character Strings + +The functions described in the previous section only convert a single +character at a time. Most operations to be performed in real-world +programs include strings and therefore the @w{ISO C} standard also +defines conversions on entire strings. However, the defined set of +functions is quite limited; therefore, the GNU C library contains a few +extensions that can help in some important situations. + +@comment wchar.h +@comment ISO +@deftypefun size_t mbsrtowcs (wchar_t *restrict @var{dst}, const char **restrict @var{src}, size_t @var{len}, mbstate_t *restrict @var{ps}) +The @code{mbsrtowcs} function (``multibyte string restartable to wide +character string'') converts an NUL-terminated multibyte character +string at @code{*@var{src}} into an equivalent wide character string, +including the NUL wide character at the end. The conversion is started +using the state information from the object pointed to by @var{ps} or +from an internal object of @code{mbsrtowcs} if @var{ps} is a null +pointer. Before returning, the state object is updated to match the state +after the last converted character. The state is the initial state if the +terminating NUL byte is reached and converted. + +If @var{dst} is not a null pointer, the result is stored in the array +pointed to by @var{dst}; otherwise, the conversion result is not +available since it is stored in an internal buffer. + +If @var{len} wide characters are stored in the array @var{dst} before +reaching the end of the input string, the conversion stops and @var{len} +is returned. If @var{dst} is a null pointer, @var{len} is never checked. + +Another reason for a premature return from the function call is if the +input string contains an invalid multibyte sequence. In this case the +global variable @code{errno} is set to @code{EILSEQ} and the function +returns @code{(size_t) -1}. + +@c XXX The ISO C9x draft seems to have a problem here. It says that PS +@c is not updated if DST is NULL. This is not said straightforward and +@c none of the other functions is described like this. It would make sense +@c to define the function this way but I don't think it is meant like this. + +In all other cases the function returns the number of wide characters +converted during this call. If @var{dst} is not null, @code{mbsrtowcs} +stores in the pointer pointed to by @var{src} either a null pointer (if +the NUL byte in the input string was reached) or the address of the byte +following the last converted multibyte character. + +@pindex wchar.h +@code{mbsrtowcs} was introduced in @w{Amendment 1} to @w{ISO C90} and is +declared in @file{wchar.h}. +@end deftypefun + +The definition of the @code{mbsrtowcs} function has one important +limitation. The requirement that @var{dst} has to be a NUL-terminated +string provides problems if one wants to convert buffers with text. A +buffer is normally no collection of NUL-terminated strings but instead a +continuous collection of lines, separated by newline characters. Now +assume that a function to convert one line from a buffer is needed. Since +the line is not NUL-terminated, the source pointer cannot directly point +into the unmodified text buffer. This means, either one inserts the NUL +byte at the appropriate place for the time of the @code{mbsrtowcs} +function call (which is not doable for a read-only buffer or in a +multi-threaded application) or one copies the line in an extra buffer +where it can be terminated by a NUL byte. Note that it is not in general +possible to limit the number of characters to convert by setting the +parameter @var{len} to any specific value. Since it is not known how +many bytes each multibyte character sequence is in length, one can only +guess. + +@cindex stateful +There is still a problem with the method of NUL-terminating a line right +after the newline character, which could lead to very strange results. +As said in the description of the @code{mbsrtowcs} function above the +conversion state is guaranteed to be in the initial shift state after +processing the NUL byte at the end of the input string. But this NUL +byte is not really part of the text (i.e., the conversion state after +the newline in the original text could be something different than the +initial shift state and therefore the first character of the next line +is encoded using this state). But the state in question is never +accessible to the user since the conversion stops after the NUL byte +(which resets the state). Most stateful character sets in use today +require that the shift state after a newline be the initial state--but +this is not a strict guarantee. Therefore, simply NUL-terminating a +piece of a running text is not always an adequate solution and, +therefore, should never be used in generally used code. + +The generic conversion interface (@pxref{Generic Charset Conversion}) +does not have this limitation (it simply works on buffers, not +strings), and the GNU C library contains a set of functions that take +additional parameters specifying the maximal number of bytes that are +consumed from the input string. This way the problem of +@code{mbsrtowcs}'s example above could be solved by determining the line +length and passing this length to the function. + +@comment wchar.h +@comment ISO +@deftypefun size_t wcsrtombs (char *restrict @var{dst}, const wchar_t **restrict @var{src}, size_t @var{len}, mbstate_t *restrict @var{ps}) +The @code{wcsrtombs} function (``wide character string restartable to +multibyte string'') converts the NUL-terminated wide character string at +@code{*@var{src}} into an equivalent multibyte character string and +stores the result in the array pointed to by @var{dst}. The NUL wide +character is also converted. The conversion starts in the state +described in the object pointed to by @var{ps} or by a state object +locally to @code{wcsrtombs} in case @var{ps} is a null pointer. If +@var{dst} is a null pointer, the conversion is performed as usual but the +result is not available. If all characters of the input string were +successfully converted and if @var{dst} is not a null pointer, the +pointer pointed to by @var{src} gets assigned a null pointer. + +If one of the wide characters in the input string has no valid multibyte +character equivalent, the conversion stops early, sets the global +variable @code{errno} to @code{EILSEQ}, and returns @code{(size_t) -1}. + +Another reason for a premature stop is if @var{dst} is not a null +pointer and the next converted character would require more than +@var{len} bytes in total to the array @var{dst}. In this case (and if +@var{dest} is not a null pointer) the pointer pointed to by @var{src} is +assigned a value pointing to the wide character right after the last one +successfully converted. + +Except in the case of an encoding error the return value of the +@code{wcsrtombs} function is the number of bytes in all the multibyte +character sequences stored in @var{dst}. Before returning the state in +the object pointed to by @var{ps} (or the internal object in case +@var{ps} is a null pointer) is updated to reflect the state after the +last conversion. The state is the initial shift state in case the +terminating NUL wide character was converted. + +@pindex wchar.h +The @code{wcsrtombs} function was introduced in @w{Amendment 1} to +@w{ISO C90} and is declared in @file{wchar.h}. +@end deftypefun + +The restriction mentioned above for the @code{mbsrtowcs} function applies +here also. There is no possibility of directly controlling the number of +input characters. One has to place the NUL wide character at the correct +place or control the consumed input indirectly via the available output +array size (the @var{len} parameter). + +@comment wchar.h +@comment GNU +@deftypefun size_t mbsnrtowcs (wchar_t *restrict @var{dst}, const char **restrict @var{src}, size_t @var{nmc}, size_t @var{len}, mbstate_t *restrict @var{ps}) +The @code{mbsnrtowcs} function is very similar to the @code{mbsrtowcs} +function. All the parameters are the same except for @var{nmc}, which is +new. The return value is the same as for @code{mbsrtowcs}. + +This new parameter specifies how many bytes at most can be used from the +multibyte character string. In other words, the multibyte character +string @code{*@var{src}} need not be NUL-terminated. But if a NUL byte +is found within the @var{nmc} first bytes of the string, the conversion +stops here. + +This function is a GNU extension. It is meant to work around the +problems mentioned above. Now it is possible to convert a buffer with +multibyte character text piece for piece without having to care about +inserting NUL bytes and the effect of NUL bytes on the conversion state. +@end deftypefun + +A function to convert a multibyte string into a wide character string +and display it could be written like this (this is not a really useful +example): + +@smallexample +void +showmbs (const char *src, FILE *fp) +@{ + mbstate_t state; + int cnt = 0; + memset (&state, '\0', sizeof (state)); + while (1) + @{ + wchar_t linebuf[100]; + const char *endp = strchr (src, '\n'); + size_t n; + + /* @r{Exit if there is no more line.} */ + if (endp == NULL) + break; + + n = mbsnrtowcs (linebuf, &src, endp - src, 99, &state); + linebuf[n] = L'\0'; + fprintf (fp, "line %d: \"%S\"\n", linebuf); + @} +@} +@end smallexample + +There is no problem with the state after a call to @code{mbsnrtowcs}. +Since we don't insert characters in the strings that were not in there +right from the beginning and we use @var{state} only for the conversion +of the given buffer, there is no problem with altering the state. + +@comment wchar.h +@comment GNU +@deftypefun size_t wcsnrtombs (char *restrict @var{dst}, const wchar_t **restrict @var{src}, size_t @var{nwc}, size_t @var{len}, mbstate_t *restrict @var{ps}) +The @code{wcsnrtombs} function implements the conversion from wide +character strings to multibyte character strings. It is similar to +@code{wcsrtombs} but, just like @code{mbsnrtowcs}, it takes an extra +parameter, which specifies the length of the input string. + +No more than @var{nwc} wide characters from the input string +@code{*@var{src}} are converted. If the input string contains a NUL +wide character in the first @var{nwc} characters, the conversion stops at +this place. + +The @code{wcsnrtombs} function is a GNU extension and just like +@code{mbsnrtowcs} helps in situations where no NUL-terminated input +strings are available. +@end deftypefun + + +@node Multibyte Conversion Example +@subsection A Complete Multibyte Conversion Example + +The example programs given in the last sections are only brief and do +not contain all the error checking, etc. Presented here is a complete +and documented example. It features the @code{mbrtowc} function but it +should be easy to derive versions using the other functions. + +@smallexample +int +file_mbsrtowcs (int input, int output) +@{ + /* @r{Note the use of @code{MB_LEN_MAX}.} + @r{@code{MB_CUR_MAX} cannot portably be used here.} */ + char buffer[BUFSIZ + MB_LEN_MAX]; + mbstate_t state; + int filled = 0; + int eof = 0; + + /* @r{Initialize the state.} */ + memset (&state, '\0', sizeof (state)); + + while (!eof) + @{ + ssize_t nread; + ssize_t nwrite; + char *inp = buffer; + wchar_t outbuf[BUFSIZ]; + wchar_t *outp = outbuf; + + /* @r{Fill up the buffer from the input file.} */ + nread = read (input, buffer + filled, BUFSIZ); + if (nread < 0) + @{ + perror ("read"); + return 0; + @} + /* @r{If we reach end of file, make a note to read no more.} */ + if (nread == 0) + eof = 1; + + /* @r{@code{filled} is now the number of bytes in @code{buffer}.} */ + filled += nread; + + /* @r{Convert those bytes to wide characters--as many as we can.} */ + while (1) + @{ + size_t thislen = mbrtowc (outp, inp, filled, &state); + /* @r{Stop converting at invalid character;} + @r{this can mean we have read just the first part} + @r{of a valid character.} */ + if (thislen == (size_t) -1) + break; + /* @r{We want to handle embedded NUL bytes} + @r{but the return value is 0. Correct this.} */ + if (thislen == 0) + thislen = 1; + /* @r{Advance past this character.} */ + inp += thislen; + filled -= thislen; + ++outp; + @} + + /* @r{Write the wide characters we just made.} */ + nwrite = write (output, outbuf, + (outp - outbuf) * sizeof (wchar_t)); + if (nwrite < 0) + @{ + perror ("write"); + return 0; + @} + + /* @r{See if we have a @emph{real} invalid character.} */ + if ((eof && filled > 0) || filled >= MB_CUR_MAX) + @{ + error (0, 0, "invalid multibyte character"); + return 0; + @} + + /* @r{If any characters must be carried forward,} + @r{put them at the beginning of @code{buffer}.} */ + if (filled > 0) + memmove (inp, buffer, filled); + @} + + return 1; +@} +@end smallexample + + +@node Non-reentrant Conversion +@section Non-reentrant Conversion Function + +The functions described in the previous chapter are defined in +@w{Amendment 1} to @w{ISO C90}, but the original @w{ISO C90} standard +also contained functions for character set conversion. The reason that +these original functions are not described first is that they are almost +entirely useless. + +The problem is that all the conversion functions described in the +original @w{ISO C90} use a local state. Using a local state implies that +multiple conversions at the same time (not only when using threads) +cannot be done, and that you cannot first convert single characters and +then strings since you cannot tell the conversion functions which state +to use. + +These original functions are therefore usable only in a very limited set +of situations. One must complete converting the entire string before +starting a new one, and each string/text must be converted with the same +function (there is no problem with the library itself; it is guaranteed +that no library function changes the state of any of these functions). +@strong{For the above reasons it is highly requested that the functions +described in the previous section be used in place of non-reentrant +conversion functions.} + +@menu +* Non-reentrant Character Conversion:: Non-reentrant Conversion of Single + Characters. +* Non-reentrant String Conversion:: Non-reentrant Conversion of Strings. +* Shift State:: States in Non-reentrant Functions. +@end menu + +@node Non-reentrant Character Conversion +@subsection Non-reentrant Conversion of Single Characters + +@comment stdlib.h +@comment ISO +@deftypefun int mbtowc (wchar_t *restrict @var{result}, const char *restrict @var{string}, size_t @var{size}) +The @code{mbtowc} (``multibyte to wide character'') function when called +with non-null @var{string} converts the first multibyte character +beginning at @var{string} to its corresponding wide character code. It +stores the result in @code{*@var{result}}. + +@code{mbtowc} never examines more than @var{size} bytes. (The idea is +to supply for @var{size} the number of bytes of data you have in hand.) + +@code{mbtowc} with non-null @var{string} distinguishes three +possibilities: the first @var{size} bytes at @var{string} start with +valid multibyte characters, they start with an invalid byte sequence or +just part of a character, or @var{string} points to an empty string (a +null character). + +For a valid multibyte character, @code{mbtowc} converts it to a wide +character and stores that in @code{*@var{result}}, and returns the +number of bytes in that character (always at least @math{1} and never +more than @var{size}). + +For an invalid byte sequence, @code{mbtowc} returns @math{-1}. For an +empty string, it returns @math{0}, also storing @code{'\0'} in +@code{*@var{result}}. + +If the multibyte character code uses shift characters, then +@code{mbtowc} maintains and updates a shift state as it scans. If you +call @code{mbtowc} with a null pointer for @var{string}, that +initializes the shift state to its standard initial value. It also +returns nonzero if the multibyte character code in use actually has a +shift state. @xref{Shift State}. +@end deftypefun + +@comment stdlib.h +@comment ISO +@deftypefun int wctomb (char *@var{string}, wchar_t @var{wchar}) +The @code{wctomb} (``wide character to multibyte'') function converts +the wide character code @var{wchar} to its corresponding multibyte +character sequence, and stores the result in bytes starting at +@var{string}. At most @code{MB_CUR_MAX} characters are stored. + +@code{wctomb} with non-null @var{string} distinguishes three +possibilities for @var{wchar}: a valid wide character code (one that can +be translated to a multibyte character), an invalid code, and +@code{L'\0'}. + +Given a valid code, @code{wctomb} converts it to a multibyte character, +storing the bytes starting at @var{string}. Then it returns the number +of bytes in that character (always at least @math{1} and never more +than @code{MB_CUR_MAX}). + +If @var{wchar} is an invalid wide character code, @code{wctomb} returns +@math{-1}. If @var{wchar} is @code{L'\0'}, it returns @code{0}, also +storing @code{'\0'} in @code{*@var{string}}. + +If the multibyte character code uses shift characters, then +@code{wctomb} maintains and updates a shift state as it scans. If you +call @code{wctomb} with a null pointer for @var{string}, that +initializes the shift state to its standard initial value. It also +returns nonzero if the multibyte character code in use actually has a +shift state. @xref{Shift State}. + +Calling this function with a @var{wchar} argument of zero when +@var{string} is not null has the side-effect of reinitializing the +stored shift state @emph{as well as} storing the multibyte character +@code{'\0'} and returning @math{0}. +@end deftypefun + +Similar to @code{mbrlen} there is also a non-reentrant function that +computes the length of a multibyte character. It can be defined in +terms of @code{mbtowc}. + +@comment stdlib.h +@comment ISO +@deftypefun int mblen (const char *@var{string}, size_t @var{size}) +The @code{mblen} function with a non-null @var{string} argument returns +the number of bytes that make up the multibyte character beginning at +@var{string}, never examining more than @var{size} bytes. (The idea is +to supply for @var{size} the number of bytes of data you have in hand.) + +The return value of @code{mblen} distinguishes three possibilities: the +first @var{size} bytes at @var{string} start with valid multibyte +characters, they start with an invalid byte sequence or just part of a +character, or @var{string} points to an empty string (a null character). + +For a valid multibyte character, @code{mblen} returns the number of +bytes in that character (always at least @code{1} and never more than +@var{size}). For an invalid byte sequence, @code{mblen} returns +@math{-1}. For an empty string, it returns @math{0}. + +If the multibyte character code uses shift characters, then @code{mblen} +maintains and updates a shift state as it scans. If you call +@code{mblen} with a null pointer for @var{string}, that initializes the +shift state to its standard initial value. It also returns a nonzero +value if the multibyte character code in use actually has a shift state. +@xref{Shift State}. + +@pindex stdlib.h +The function @code{mblen} is declared in @file{stdlib.h}. +@end deftypefun + + +@node Non-reentrant String Conversion +@subsection Non-reentrant Conversion of Strings + +For convenience the @w{ISO C90} standard also defines functions to +convert entire strings instead of single characters. These functions +suffer from the same problems as their reentrant counterparts from +@w{Amendment 1} to @w{ISO C90}; see @ref{Converting Strings}. + +@comment stdlib.h +@comment ISO +@deftypefun size_t mbstowcs (wchar_t *@var{wstring}, const char *@var{string}, size_t @var{size}) +The @code{mbstowcs} (``multibyte string to wide character string'') +function converts the null-terminated string of multibyte characters +@var{string} to an array of wide character codes, storing not more than +@var{size} wide characters into the array beginning at @var{wstring}. +The terminating null character counts towards the size, so if @var{size} +is less than the actual number of wide characters resulting from +@var{string}, no terminating null character is stored. + +The conversion of characters from @var{string} begins in the initial +shift state. + +If an invalid multibyte character sequence is found, the @code{mbstowcs} +function returns a value of @math{-1}. Otherwise, it returns the number +of wide characters stored in the array @var{wstring}. This number does +not include the terminating null character, which is present if the +number is less than @var{size}. + +Here is an example showing how to convert a string of multibyte +characters, allocating enough space for the result. + +@smallexample +wchar_t * +mbstowcs_alloc (const char *string) +@{ + size_t size = strlen (string) + 1; + wchar_t *buf = xmalloc (size * sizeof (wchar_t)); + + size = mbstowcs (buf, string, size); + if (size == (size_t) -1) + return NULL; + buf = xrealloc (buf, (size + 1) * sizeof (wchar_t)); + return buf; +@} +@end smallexample + +@end deftypefun + +@comment stdlib.h +@comment ISO +@deftypefun size_t wcstombs (char *@var{string}, const wchar_t *@var{wstring}, size_t @var{size}) +The @code{wcstombs} (``wide character string to multibyte string'') +function converts the null-terminated wide character array @var{wstring} +into a string containing multibyte characters, storing not more than +@var{size} bytes starting at @var{string}, followed by a terminating +null character if there is room. The conversion of characters begins in +the initial shift state. + +The terminating null character counts towards the size, so if @var{size} +is less than or equal to the number of bytes needed in @var{wstring}, no +terminating null character is stored. + +If a code that does not correspond to a valid multibyte character is +found, the @code{wcstombs} function returns a value of @math{-1}. +Otherwise, the return value is the number of bytes stored in the array +@var{string}. This number does not include the terminating null character, +which is present if the number is less than @var{size}. +@end deftypefun + +@node Shift State +@subsection States in Non-reentrant Functions + +In some multibyte character codes, the @emph{meaning} of any particular +byte sequence is not fixed; it depends on what other sequences have come +earlier in the same string. Typically there are just a few sequences that +can change the meaning of other sequences; these few are called +@dfn{shift sequences} and we say that they set the @dfn{shift state} for +other sequences that follow. + +To illustrate shift state and shift sequences, suppose we decide that +the sequence @code{0200} (just one byte) enters Japanese mode, in which +pairs of bytes in the range from @code{0240} to @code{0377} are single +characters, while @code{0201} enters Latin-1 mode, in which single bytes +in the range from @code{0240} to @code{0377} are characters, and +interpreted according to the ISO Latin-1 character set. This is a +multibyte code that has two alternative shift states (``Japanese mode'' +and ``Latin-1 mode''), and two shift sequences that specify particular +shift states. + +When the multibyte character code in use has shift states, then +@code{mblen}, @code{mbtowc}, and @code{wctomb} must maintain and update +the current shift state as they scan the string. To make this work +properly, you must follow these rules: + +@itemize @bullet +@item +Before starting to scan a string, call the function with a null pointer +for the multibyte character address---for example, @code{mblen (NULL, +0)}. This initializes the shift state to its standard initial value. + +@item +Scan the string one character at a time, in order. Do not ``back up'' +and rescan characters already scanned, and do not intersperse the +processing of different strings. +@end itemize + +Here is an example of using @code{mblen} following these rules: + +@smallexample +void +scan_string (char *s) +@{ + int length = strlen (s); + + /* @r{Initialize shift state.} */ + mblen (NULL, 0); + + while (1) + @{ + int thischar = mblen (s, length); + /* @r{Deal with end of string and invalid characters.} */ + if (thischar == 0) + break; + if (thischar == -1) + @{ + error ("invalid multibyte character"); + break; + @} + /* @r{Advance past this character.} */ + s += thischar; + length -= thischar; + @} +@} +@end smallexample + +The functions @code{mblen}, @code{mbtowc} and @code{wctomb} are not +reentrant when using a multibyte code that uses a shift state. However, +no other library functions call these functions, so you don't have to +worry that the shift state will be changed mysteriously. + + +@node Generic Charset Conversion +@section Generic Charset Conversion + +The conversion functions mentioned so far in this chapter all had in +common that they operate on character sets that are not directly +specified by the functions. The multibyte encoding used is specified by +the currently selected locale for the @code{LC_CTYPE} category. The +wide character set is fixed by the implementation (in the case of GNU C +library it is always UCS-4 encoded @w{ISO 10646}. + +This has of course several problems when it comes to general character +conversion: + +@itemize @bullet +@item +For every conversion where neither the source nor the destination +character set is the character set of the locale for the @code{LC_CTYPE} +category, one has to change the @code{LC_CTYPE} locale using +@code{setlocale}. + +Changing the @code{LC_TYPE} locale introduces major problems for the rest +of the programs since several more functions (e.g., the character +classification functions, @pxref{Classification of Characters}) use the +@code{LC_CTYPE} category. + +@item +Parallel conversions to and from different character sets are not +possible since the @code{LC_CTYPE} selection is global and shared by all +threads. + +@item +If neither the source nor the destination character set is the character +set used for @code{wchar_t} representation, there is at least a two-step +process necessary to convert a text using the functions above. One would +have to select the source character set as the multibyte encoding, +convert the text into a @code{wchar_t} text, select the destination +character set as the multibyte encoding, and convert the wide character +text to the multibyte (@math{=} destination) character set. + +Even if this is possible (which is not guaranteed) it is a very tiring +work. Plus it suffers from the other two raised points even more due to +the steady changing of the locale. +@end itemize + +The XPG2 standard defines a completely new set of functions, which has +none of these limitations. They are not at all coupled to the selected +locales, and they have no constraints on the character sets selected for +source and destination. Only the set of available conversions limits +them. The standard does not specify that any conversion at all must be +available. Such availability is a measure of the quality of the +implementation. + +In the following text first the interface to @code{iconv} and then the +conversion function, will be described. Comparisons with other +implementations will show what obstacles stand in the way of portable +applications. Finally, the implementation is described in so far as might +interest the advanced user who wants to extend conversion capabilities. + +@menu +* Generic Conversion Interface:: Generic Character Set Conversion Interface. +* iconv Examples:: A complete @code{iconv} example. +* Other iconv Implementations:: Some Details about other @code{iconv} + Implementations. +* glibc iconv Implementation:: The @code{iconv} Implementation in the GNU C + library. +@end menu + +@node Generic Conversion Interface +@subsection Generic Character Set Conversion Interface + +This set of functions follows the traditional cycle of using a resource: +open--use--close. The interface consists of three functions, each of +which implements one step. + +Before the interfaces are described it is necessary to introduce a +data type. Just like other open--use--close interfaces the functions +introduced here work using handles and the @file{iconv.h} header +defines a special type for the handles used. + +@comment iconv.h +@comment XPG2 +@deftp {Data Type} iconv_t +This data type is an abstract type defined in @file{iconv.h}. The user +must not assume anything about the definition of this type; it must be +completely opaque. + +Objects of this type can get assigned handles for the conversions using +the @code{iconv} functions. The objects themselves need not be freed, but +the conversions for which the handles stand for have to. +@end deftp + +@noindent +The first step is the function to create a handle. + +@comment iconv.h +@comment XPG2 +@deftypefun iconv_t iconv_open (const char *@var{tocode}, const char *@var{fromcode}) +The @code{iconv_open} function has to be used before starting a +conversion. The two parameters this function takes determine the +source and destination character set for the conversion, and if the +implementation has the possibility to perform such a conversion, the +function returns a handle. + +If the wanted conversion is not available, the @code{iconv_open} function +returns @code{(iconv_t) -1}. In this case the global variable +@code{errno} can have the following values: + +@table @code +@item EMFILE +The process already has @code{OPEN_MAX} file descriptors open. +@item ENFILE +The system limit of open file is reached. +@item ENOMEM +Not enough memory to carry out the operation. +@item EINVAL +The conversion from @var{fromcode} to @var{tocode} is not supported. +@end table + +It is not possible to use the same descriptor in different threads to +perform independent conversions. The data structures associated +with the descriptor include information about the conversion state. +This must not be messed up by using it in different conversions. + +An @code{iconv} descriptor is like a file descriptor as for every use a +new descriptor must be created. The descriptor does not stand for all +of the conversions from @var{fromset} to @var{toset}. + +The GNU C library implementation of @code{iconv_open} has one +significant extension to other implementations. To ease the extension +of the set of available conversions, the implementation allows storing +the necessary files with data and code in an arbitrary number of +directories. How this extension must be written will be explained below +(@pxref{glibc iconv Implementation}). Here it is only important to say +that all directories mentioned in the @code{GCONV_PATH} environment +variable are considered only if they contain a file @file{gconv-modules}. +These directories need not necessarily be created by the system +administrator. In fact, this extension is introduced to help users +writing and using their own, new conversions. Of course, this does not +work for security reasons in SUID binaries; in this case only the system +directory is considered and this normally is +@file{@var{prefix}/lib/gconv}. The @code{GCONV_PATH} environment +variable is examined exactly once at the first call of the +@code{iconv_open} function. Later modifications of the variable have no +effect. + +@pindex iconv.h +The @code{iconv_open} function was introduced early in the X/Open +Portability Guide, @w{version 2}. It is supported by all commercial +Unices as it is required for the Unix branding. However, the quality and +completeness of the implementation varies widely. The @code{iconv_open} +function is declared in @file{iconv.h}. +@end deftypefun + +The @code{iconv} implementation can associate large data structure with +the handle returned by @code{iconv_open}. Therefore, it is crucial to +free all the resources once all conversions are carried out and the +conversion is not needed anymore. + +@comment iconv.h +@comment XPG2 +@deftypefun int iconv_close (iconv_t @var{cd}) +The @code{iconv_close} function frees all resources associated with the +handle @var{cd}, which must have been returned by a successful call to +the @code{iconv_open} function. + +If the function call was successful the return value is @math{0}. +Otherwise it is @math{-1} and @code{errno} is set appropriately. +Defined error are: + +@table @code +@item EBADF +The conversion descriptor is invalid. +@end table + +@pindex iconv.h +The @code{iconv_close} function was introduced together with the rest +of the @code{iconv} functions in XPG2 and is declared in @file{iconv.h}. +@end deftypefun + +The standard defines only one actual conversion function. This has, +therefore, the most general interface: it allows conversion from one +buffer to another. Conversion from a file to a buffer, vice versa, or +even file to file can be implemented on top of it. + +@comment iconv.h +@comment XPG2 +@deftypefun size_t iconv (iconv_t @var{cd}, char **@var{inbuf}, size_t *@var{inbytesleft}, char **@var{outbuf}, size_t *@var{outbytesleft}) +@cindex stateful +The @code{iconv} function converts the text in the input buffer +according to the rules associated with the descriptor @var{cd} and +stores the result in the output buffer. It is possible to call the +function for the same text several times in a row since for stateful +character sets the necessary state information is kept in the data +structures associated with the descriptor. + +The input buffer is specified by @code{*@var{inbuf}} and it contains +@code{*@var{inbytesleft}} bytes. The extra indirection is necessary for +communicating the used input back to the caller (see below). It is +important to note that the buffer pointer is of type @code{char} and the +length is measured in bytes even if the input text is encoded in wide +characters. + +The output buffer is specified in a similar way. @code{*@var{outbuf}} +points to the beginning of the buffer with at least +@code{*@var{outbytesleft}} bytes room for the result. The buffer +pointer again is of type @code{char} and the length is measured in +bytes. If @var{outbuf} or @code{*@var{outbuf}} is a null pointer, the +conversion is performed but no output is available. + +If @var{inbuf} is a null pointer, the @code{iconv} function performs the +necessary action to put the state of the conversion into the initial +state. This is obviously a no-op for non-stateful encodings, but if the +encoding has a state, such a function call might put some byte sequences +in the output buffer, which perform the necessary state changes. The +next call with @var{inbuf} not being a null pointer then simply goes on +from the initial state. It is important that the programmer never makes +any assumption as to whether the conversion has to deal with states. +Even if the input and output character sets are not stateful, the +implementation might still have to keep states. This is due to the +implementation chosen for the GNU C library as it is described below. +Therefore an @code{iconv} call to reset the state should always be +performed if some protocol requires this for the output text. + +The conversion stops for one of three reasons. The first is that all +characters from the input buffer are converted. This actually can mean +two things: either all bytes from the input buffer are consumed or +there are some bytes at the end of the buffer that possibly can form a +complete character but the input is incomplete. The second reason for a +stop is that the output buffer is full. And the third reason is that +the input contains invalid characters. + +In all of these cases the buffer pointers after the last successful +conversion, for input and output buffer, are stored in @var{inbuf} and +@var{outbuf}, and the available room in each buffer is stored in +@var{inbytesleft} and @var{outbytesleft}. + +Since the character sets selected in the @code{iconv_open} call can be +almost arbitrary, there can be situations where the input buffer contains +valid characters, which have no identical representation in the output +character set. The behavior in this situation is undefined. The +@emph{current} behavior of the GNU C library in this situation is to +return with an error immediately. This certainly is not the most +desirable solution; therefore, future versions will provide better ones, +but they are not yet finished. + +If all input from the input buffer is successfully converted and stored +in the output buffer, the function returns the number of non-reversible +conversions performed. In all other cases the return value is +@code{(size_t) -1} and @code{errno} is set appropriately. In such cases +the value pointed to by @var{inbytesleft} is nonzero. + +@table @code +@item EILSEQ +The conversion stopped because of an invalid byte sequence in the input. +After the call, @code{*@var{inbuf}} points at the first byte of the +invalid byte sequence. + +@item E2BIG +The conversion stopped because it ran out of space in the output buffer. + +@item EINVAL +The conversion stopped because of an incomplete byte sequence at the end +of the input buffer. + +@item EBADF +The @var{cd} argument is invalid. +@end table + +@pindex iconv.h +The @code{iconv} function was introduced in the XPG2 standard and is +declared in the @file{iconv.h} header. +@end deftypefun + +The definition of the @code{iconv} function is quite good overall. It +provides quite flexible functionality. The only problems lie in the +boundary cases, which are incomplete byte sequences at the end of the +input buffer and invalid input. A third problem, which is not really +a design problem, is the way conversions are selected. The standard +does not say anything about the legitimate names, a minimal set of +available conversions. We will see how this negatively impacts other +implementations, as demonstrated below. + +@node iconv Examples +@subsection A complete @code{iconv} example + +The example below features a solution for a common problem. Given that +one knows the internal encoding used by the system for @code{wchar_t} +strings, one often is in the position to read text from a file and store +it in wide character buffers. One can do this using @code{mbsrtowcs}, +but then we run into the problems discussed above. + +@smallexample +int +file2wcs (int fd, const char *charset, wchar_t *outbuf, size_t avail) +@{ + char inbuf[BUFSIZ]; + size_t insize = 0; + char *wrptr = (char *) outbuf; + int result = 0; + iconv_t cd; + + cd = iconv_open ("WCHAR_T", charset); + if (cd == (iconv_t) -1) + @{ + /* @r{Something went wrong.} */ + if (errno == EINVAL) + error (0, 0, "conversion from '%s' to wchar_t not available", + charset); + else + perror ("iconv_open"); + + /* @r{Terminate the output string.} */ + *outbuf = L'\0'; + + return -1; + @} + + while (avail > 0) + @{ + size_t nread; + size_t nconv; + char *inptr = inbuf; + + /* @r{Read more input.} */ + nread = read (fd, inbuf + insize, sizeof (inbuf) - insize); + if (nread == 0) + @{ + /* @r{When we come here the file is completely read.} + @r{This still could mean there are some unused} + @r{characters in the @code{inbuf}. Put them back.} */ + if (lseek (fd, -insize, SEEK_CUR) == -1) + result = -1; + + /* @r{Now write out the byte sequence to get into the} + @r{initial state if this is necessary.} */ + iconv (cd, NULL, NULL, &wrptr, &avail); + + break; + @} + insize += nread; + + /* @r{Do the conversion.} */ + nconv = iconv (cd, &inptr, &insize, &wrptr, &avail); + if (nconv == (size_t) -1) + @{ + /* @r{Not everything went right. It might only be} + @r{an unfinished byte sequence at the end of the} + @r{buffer. Or it is a real problem.} */ + if (errno == EINVAL) + /* @r{This is harmless. Simply move the unused} + @r{bytes to the beginning of the buffer so that} + @r{they can be used in the next round.} */ + memmove (inbuf, inptr, insize); + else + @{ + /* @r{It is a real problem. Maybe we ran out of} + @r{space in the output buffer or we have invalid} + @r{input. In any case back the file pointer to} + @r{the position of the last processed byte.} */ + lseek (fd, -insize, SEEK_CUR); + result = -1; + break; + @} + @} + @} + + /* @r{Terminate the output string.} */ + if (avail >= sizeof (wchar_t)) + *((wchar_t *) wrptr) = L'\0'; + + if (iconv_close (cd) != 0) + perror ("iconv_close"); + + return (wchar_t *) wrptr - outbuf; +@} +@end smallexample + +@cindex stateful +This example shows the most important aspects of using the @code{iconv} +functions. It shows how successive calls to @code{iconv} can be used to +convert large amounts of text. The user does not have to care about +stateful encodings as the functions take care of everything. + +An interesting point is the case where @code{iconv} returns an error and +@code{errno} is set to @code{EINVAL}. This is not really an error in the +transformation. It can happen whenever the input character set contains +byte sequences of more than one byte for some character and texts are not +processed in one piece. In this case there is a chance that a multibyte +sequence is cut. The caller can then simply read the remainder of the +takes and feed the offending bytes together with new character from the +input to @code{iconv} and continue the work. The internal state kept in +the descriptor is @emph{not} unspecified after such an event as is the +case with the conversion functions from the @w{ISO C} standard. + +The example also shows the problem of using wide character strings with +@code{iconv}. As explained in the description of the @code{iconv} +function above, the function always takes a pointer to a @code{char} +array and the available space is measured in bytes. In the example, the +output buffer is a wide character buffer; therefore, we use a local +variable @var{wrptr} of type @code{char *}, which is used in the +@code{iconv} calls. + +This looks rather innocent but can lead to problems on platforms that +have tight restriction on alignment. Therefore the caller of @code{iconv} +has to make sure that the pointers passed are suitable for access of +characters from the appropriate character set. Since, in the +above case, the input parameter to the function is a @code{wchar_t} +pointer, this is the case (unless the user violates alignment when +computing the parameter). But in other situations, especially when +writing generic functions where one does not know what type of character +set one uses and, therefore, treats text as a sequence of bytes, it might +become tricky. + +@node Other iconv Implementations +@subsection Some Details about other @code{iconv} Implementations + +This is not really the place to discuss the @code{iconv} implementation +of other systems but it is necessary to know a bit about them to write +portable programs. The above mentioned problems with the specification +of the @code{iconv} functions can lead to portability issues. + +The first thing to notice is that, due to the large number of character +sets in use, it is certainly not practical to encode the conversions +directly in the C library. Therefore, the conversion information must +come from files outside the C library. This is usually done in one or +both of the following ways: + +@itemize @bullet +@item +The C library contains a set of generic conversion functions that can +read the needed conversion tables and other information from data files. +These files get loaded when necessary. + +This solution is problematic as it requires a great deal of effort to +apply to all character sets (potentially an infinite set). The +differences in the structure of the different character sets is so large +that many different variants of the table-processing functions must be +developed. In addition, the generic nature of these functions make them +slower than specifically implemented functions. + +@item +The C library only contains a framework that can dynamically load +object files and execute the conversion functions contained therein. + +This solution provides much more flexibility. The C library itself +contains only very little code and therefore reduces the general memory +footprint. Also, with a documented interface between the C library and +the loadable modules it is possible for third parties to extend the set +of available conversion modules. A drawback of this solution is that +dynamic loading must be available. +@end itemize + +Some implementations in commercial Unices implement a mixture of these +possibilities; the majority implement only the second solution. Using +loadable modules moves the code out of the library itself and keeps +the door open for extensions and improvements, but this design is also +limiting on some platforms since not many platforms support dynamic +loading in statically linked programs. On platforms without this +capability it is therefore not possible to use this interface in +statically linked programs. The GNU C library has, on ELF platforms, no +problems with dynamic loading in these situations; therefore, this +point is moot. The danger is that one gets acquainted with this +situation and forgets about the restrictions on other systems. + +A second thing to know about other @code{iconv} implementations is that +the number of available conversions is often very limited. Some +implementations provide, in the standard release (not special +international or developer releases), at most 100 to 200 conversion +possibilities. This does not mean 200 different character sets are +supported; for example, conversions from one character set to a set of 10 +others might count as 10 conversions. Together with the other direction +this makes 20 conversion possibilities used up by one character set. One +can imagine the thin coverage these platform provide. Some Unix vendors +even provide only a handful of conversions, which renders them useless for +almost all uses. + +This directly leads to a third and probably the most problematic point. +The way the @code{iconv} conversion functions are implemented on all +known Unix systems and the availability of the conversion functions from +character set @math{@cal{A}} to @math{@cal{B}} and the conversion from +@math{@cal{B}} to @math{@cal{C}} does @emph{not} imply that the +conversion from @math{@cal{A}} to @math{@cal{C}} is available. + +This might not seem unreasonable and problematic at first, but it is a +quite big problem as one will notice shortly after hitting it. To show +the problem we assume to write a program that has to convert from +@math{@cal{A}} to @math{@cal{C}}. A call like + +@smallexample +cd = iconv_open ("@math{@cal{C}}", "@math{@cal{A}}"); +@end smallexample + +@noindent +fails according to the assumption above. But what does the program +do now? The conversion is necessary; therefore, simply giving up is not +an option. + +This is a nuisance. The @code{iconv} function should take care of this. +But how should the program proceed from here on? If it tries to convert +to character set @math{@cal{B}}, first the two @code{iconv_open} +calls + +@smallexample +cd1 = iconv_open ("@math{@cal{B}}", "@math{@cal{A}}"); +@end smallexample + +@noindent +and + +@smallexample +cd2 = iconv_open ("@math{@cal{C}}", "@math{@cal{B}}"); +@end smallexample + +@noindent +will succeed, but how to find @math{@cal{B}}? + +Unfortunately, the answer is: there is no general solution. On some +systems guessing might help. On those systems most character sets can +convert to and from UTF-8 encoded @w{ISO 10646} or Unicode text. Beside +this only some very system-specific methods can help. Since the +conversion functions come from loadable modules and these modules must +be stored somewhere in the filesystem, one @emph{could} try to find them +and determine from the available file which conversions are available +and whether there is an indirect route from @math{@cal{A}} to +@math{@cal{C}}. + +This example shows one of the design errors of @code{iconv} mentioned +above. It should at least be possible to determine the list of available +conversion programmatically so that if @code{iconv_open} says there is no +such conversion, one could make sure this also is true for indirect +routes. + +@node glibc iconv Implementation +@subsection The @code{iconv} Implementation in the GNU C library + +After reading about the problems of @code{iconv} implementations in the +last section it is certainly good to note that the implementation in +the GNU C library has none of the problems mentioned above. What +follows is a step-by-step analysis of the points raised above. The +evaluation is based on the current state of the development (as of +January 1999). The development of the @code{iconv} functions is not +complete, but basic functionality has solidified. + +The GNU C library's @code{iconv} implementation uses shared loadable +modules to implement the conversions. A very small number of +conversions are built into the library itself but these are only rather +trivial conversions. + +All the benefits of loadable modules are available in the GNU C library +implementation. This is especially appealing since the interface is +well documented (see below), and it, therefore, is easy to write new +conversion modules. The drawback of using loadable objects is not a +problem in the GNU C library, at least on ELF systems. Since the +library is able to load shared objects even in statically linked +binaries, static linking need not be forbidden in case one wants to use +@code{iconv}. + +The second mentioned problem is the number of supported conversions. +Currently, the GNU C library supports more than 150 character sets. The +way the implementation is designed the number of supported conversions +is greater than 22350 (@math{150} times @math{149}). If any conversion +from or to a character set is missing, it can be added easily. + +Particularly impressive as it may be, this high number is due to the +fact that the GNU C library implementation of @code{iconv} does not have +the third problem mentioned above (i.e., whenever there is a conversion +from a character set @math{@cal{A}} to @math{@cal{B}} and from +@math{@cal{B}} to @math{@cal{C}} it is always possible to convert from +@math{@cal{A}} to @math{@cal{C}} directly). If the @code{iconv_open} +returns an error and sets @code{errno} to @code{EINVAL}, there is no +known way, directly or indirectly, to perform the wanted conversion. + +@cindex triangulation +Triangulation is achieved by providing for each character set a +conversion from and to UCS-4 encoded @w{ISO 10646}. Using @w{ISO 10646} +as an intermediate representation it is possible to @dfn{triangulate} +(i.e., convert with an intermediate representation). + +There is no inherent requirement to provide a conversion to @w{ISO +10646} for a new character set, and it is also possible to provide other +conversions where neither source nor destination character set is @w{ISO +10646}. The existing set of conversions is simply meant to cover all +conversions that might be of interest. + +@cindex ISO-2022-JP +@cindex EUC-JP +All currently available conversions use the triangulation method above, +making conversion run unnecessarily slow. If, for example, somebody +often needs the conversion from ISO-2022-JP to EUC-JP, a quicker solution +would involve direct conversion between the two character sets, skipping +the input to @w{ISO 10646} first. The two character sets of interest +are much more similar to each other than to @w{ISO 10646}. + +In such a situation one easily can write a new conversion and provide it +as a better alternative. The GNU C library @code{iconv} implementation +would automatically use the module implementing the conversion if it is +specified to be more efficient. + +@subsubsection Format of @file{gconv-modules} files + +All information about the available conversions comes from a file named +@file{gconv-modules}, which can be found in any of the directories along +the @code{GCONV_PATH}. The @file{gconv-modules} files are line-oriented +text files, where each of the lines has one of the following formats: + +@itemize @bullet +@item +If the first non-whitespace character is a @kbd{#} the line contains only +comments and is ignored. + +@item +Lines starting with @code{alias} define an alias name for a character +set. Two more words are expected on the line. The first word +defines the alias name, and the second defines the original name of the +character set. The effect is that it is possible to use the alias name +in the @var{fromset} or @var{toset} parameters of @code{iconv_open} and +achieve the same result as when using the real character set name. + +This is quite important as a character set has often many different +names. There is normally an official name but this need not correspond to +the most popular name. Beside this many character sets have special +names that are somehow constructed. For example, all character sets +specified by the ISO have an alias of the form @code{ISO-IR-@var{nnn}} +where @var{nnn} is the registration number. This allows programs that +know about the registration number to construct character set names and +use them in @code{iconv_open} calls. More on the available names and +aliases follows below. + +@item +Lines starting with @code{module} introduce an available conversion +module. These lines must contain three or four more words. + +The first word specifies the source character set, the second word the +destination character set of conversion implemented in this module, and +the third word is the name of the loadable module. The filename is +constructed by appending the usual shared object suffix (normally +@file{.so}) and this file is then supposed to be found in the same +directory the @file{gconv-modules} file is in. The last word on the line, +which is optional, is a numeric value representing the cost of the +conversion. If this word is missing, a cost of @math{1} is assumed. The +numeric value itself does not matter that much; what counts are the +relative values of the sums of costs for all possible conversion paths. +Below is a more precise description of the use of the cost value. +@end itemize + +Returning to the example above where one has written a module to directly +convert from ISO-2022-JP to EUC-JP and back. All that has to be done is +to put the new module, let its name be ISO2022JP-EUCJP.so, in a directory +and add a file @file{gconv-modules} with the following content in the +same directory: + +@smallexample +module ISO-2022-JP// EUC-JP// ISO2022JP-EUCJP 1 +module EUC-JP// ISO-2022-JP// ISO2022JP-EUCJP 1 +@end smallexample + +To see why this is sufficient, it is necessary to understand how the +conversion used by @code{iconv} (and described in the descriptor) is +selected. The approach to this problem is quite simple. + +At the first call of the @code{iconv_open} function the program reads +all available @file{gconv-modules} files and builds up two tables: one +containing all the known aliases and another that contains the +information about the conversions and which shared object implements +them. + +@subsubsection Finding the conversion path in @code{iconv} + +The set of available conversions form a directed graph with weighted +edges. The weights on the edges are the costs specified in the +@file{gconv-modules} files. The @code{iconv_open} function uses an +algorithm suitable for search for the best path in such a graph and so +constructs a list of conversions that must be performed in succession +to get the transformation from the source to the destination character +set. + +Explaining why the above @file{gconv-modules} files allows the +@code{iconv} implementation to resolve the specific ISO-2022-JP to +EUC-JP conversion module instead of the conversion coming with the +library itself is straightforward. Since the latter conversion takes two +steps (from ISO-2022-JP to @w{ISO 10646} and then from @w{ISO 10646} to +EUC-JP), the cost is @math{1+1 = 2}. The above @file{gconv-modules} +file, however, specifies that the new conversion modules can perform this +conversion with only the cost of @math{1}. + +A mysterious item about the @file{gconv-modules} file above (and also +the file coming with the GNU C library) are the names of the character +sets specified in the @code{module} lines. Why do almost all the names +end in @code{//}? And this is not all: the names can actually be +regular expressions. At this point in time this mystery should not be +revealed, unless you have the relevant spell-casting materials: ashes +from an original @w{DOS 6.2} boot disk burnt in effigy, a crucifix +blessed by St.@: Emacs, assorted herbal roots from Central America, sand +from Cebu, etc. Sorry! @strong{The part of the implementation where +this is used is not yet finished. For now please simply follow the +existing examples. It'll become clearer once it is. --drepper} + +A last remark about the @file{gconv-modules} is about the names not +ending with @code{//}. A character set named @code{INTERNAL} is often +mentioned. From the discussion above and the chosen name it should have +become clear that this is the name for the representation used in the +intermediate step of the triangulation. We have said that this is UCS-4 +but actually that is not quite right. The UCS-4 specification also +includes the specification of the byte ordering used. Since a UCS-4 value +consists of four bytes, a stored value is effected by byte ordering. The +internal representation is @emph{not} the same as UCS-4 in case the byte +ordering of the processor (or at least the running process) is not the +same as the one required for UCS-4. This is done for performance reasons +as one does not want to perform unnecessary byte-swapping operations if +one is not interested in actually seeing the result in UCS-4. To avoid +trouble with endianess, the internal representation consistently is named +@code{INTERNAL} even on big-endian systems where the representations are +identical. + +@subsubsection @code{iconv} module data structures + +So far this section has described how modules are located and considered +to be used. What remains to be described is the interface of the modules +so that one can write new ones. This section describes the interface as +it is in use in January 1999. The interface will change a bit in the +future but, with luck, only in an upwardly compatible way. + +The definitions necessary to write new modules are publicly available +in the non-standard header @file{gconv.h}. The following text, +therefore, describes the definitions from this header file. First, +however, it is necessary to get an overview. + +From the perspective of the user of @code{iconv} the interface is quite +simple: the @code{iconv_open} function returns a handle that can be used +in calls to @code{iconv}, and finally the handle is freed with a call to +@code{iconv_close}. The problem is that the handle has to be able to +represent the possibly long sequences of conversion steps and also the +state of each conversion since the handle is all that is passed to the +@code{iconv} function. Therefore, the data structures are really the +elements necessary to understanding the implementation. + +We need two different kinds of data structures. The first describes the +conversion and the second describes the state etc. There are really two +type definitions like this in @file{gconv.h}. +@pindex gconv.h + +@comment gconv.h +@comment GNU +@deftp {Data type} {struct __gconv_step} +This data structure describes one conversion a module can perform. For +each function in a loaded module with conversion functions there is +exactly one object of this type. This object is shared by all users of +the conversion (i.e., this object does not contain any information +corresponding to an actual conversion; it only describes the conversion +itself). + +@table @code +@item struct __gconv_loaded_object *__shlib_handle +@itemx const char *__modname +@itemx int __counter +All these elements of the structure are used internally in the C library +to coordinate loading and unloading the shared. One must not expect any +of the other elements to be available or initialized. + +@item const char *__from_name +@itemx const char *__to_name +@code{__from_name} and @code{__to_name} contain the names of the source and +destination character sets. They can be used to identify the actual +conversion to be carried out since one module might implement conversions +for more than one character set and/or direction. + +@item gconv_fct __fct +@itemx gconv_init_fct __init_fct +@itemx gconv_end_fct __end_fct +These elements contain pointers to the functions in the loadable module. +The interface will be explained below. + +@item int __min_needed_from +@itemx int __max_needed_from +@itemx int __min_needed_to +@itemx int __max_needed_to; +These values have to be supplied in the init function of the module. The +@code{__min_needed_from} value specifies how many bytes a character of +the source character set at least needs. The @code{__max_needed_from} +specifies the maximum value that also includes possible shift sequences. + +The @code{__min_needed_to} and @code{__max_needed_to} values serve the +same purpose as @code{__min_needed_from} and @code{__max_needed_from} but +this time for the destination character set. + +It is crucial that these values be accurate since otherwise the +conversion functions will have problems or not work at all. + +@item int __stateful +This element must also be initialized by the init function. +@code{int __stateful} is nonzero if the source character set is stateful. +Otherwise it is zero. + +@item void *__data +This element can be used freely by the conversion functions in the +module. @code{void *__data} can be used to communicate extra information +from one call to another. @code{void *__data} need not be initialized if +not needed at all. If @code{void *__data} element is assigned a pointer +to dynamically allocated memory (presumably in the init function) it has +to be made sure that the end function deallocates the memory. Otherwise +the application will leak memory. + +It is important to be aware that this data structure is shared by all +users of this specification conversion and therefore the @code{__data} +element must not contain data specific to one specific use of the +conversion function. +@end table +@end deftp + +@comment gconv.h +@comment GNU +@deftp {Data type} {struct __gconv_step_data} +This is the data structure that contains the information specific to +each use of the conversion functions. + + +@table @code +@item char *__outbuf +@itemx char *__outbufend +These elements specify the output buffer for the conversion step. The +@code{__outbuf} element points to the beginning of the buffer, and +@code{__outbufend} points to the byte following the last byte in the +buffer. The conversion function must not assume anything about the size +of the buffer but it can be safely assumed the there is room for at +least one complete character in the output buffer. + +Once the conversion is finished, if the conversion is the last step, the +@code{__outbuf} element must be modified to point after the last byte +written into the buffer to signal how much output is available. If this +conversion step is not the last one, the element must not be modified. +The @code{__outbufend} element must not be modified. + +@item int __is_last +This element is nonzero if this conversion step is the last one. This +information is necessary for the recursion. See the description of the +conversion function internals below. This element must never be +modified. + +@item int __invocation_counter +The conversion function can use this element to see how many calls of +the conversion function already happened. Some character sets require a +certain prolog when generating output, and by comparing this value with +zero, one can find out whether it is the first call and whether, +therefore, the prolog should be emitted. This element must never be +modified. + +@item int __internal_use +This element is another one rarely used but needed in certain +situations. It is assigned a nonzero value in case the conversion +functions are used to implement @code{mbsrtowcs} et.al.@: (i.e., the +function is not used directly through the @code{iconv} interface). + +This sometimes makes a difference as it is expected that the +@code{iconv} functions are used to translate entire texts while the +@code{mbsrtowcs} functions are normally used only to convert single +strings and might be used multiple times to convert entire texts. + +But in this situation we would have problem complying with some rules of +the character set specification. Some character sets require a prolog, +which must appear exactly once for an entire text. If a number of +@code{mbsrtowcs} calls are used to convert the text, only the first call +must add the prolog. However, because there is no communication between the +different calls of @code{mbsrtowcs}, the conversion functions have no +possibility to find this out. The situation is different for sequences +of @code{iconv} calls since the handle allows access to the needed +information. + +The @code{int __internal_use} element is mostly used together with +@code{__invocation_counter} as follows: + +@smallexample +if (!data->__internal_use + && data->__invocation_counter == 0) + /* @r{Emit prolog.} */ + ... +@end smallexample + +This element must never be modified. + +@item mbstate_t *__statep +The @code{__statep} element points to an object of type @code{mbstate_t} +(@pxref{Keeping the state}). The conversion of a stateful character +set must use the object pointed to by @code{__statep} to store +information about the conversion state. The @code{__statep} element +itself must never be modified. + +@item mbstate_t __state +This element must @emph{never} be used directly. It is only part of +this structure to have the needed space allocated. +@end table +@end deftp + +@subsubsection @code{iconv} module interfaces + +With the knowledge about the data structures we now can describe the +conversion function itself. To understand the interface a bit of +knowledge is necessary about the functionality in the C library that +loads the objects with the conversions. + +It is often the case that one conversion is used more than once (i.e., +there are several @code{iconv_open} calls for the same set of character +sets during one program run). The @code{mbsrtowcs} et.al.@: functions in +the GNU C library also use the @code{iconv} functionality, which +increases the number of uses of the same functions even more. + +Because of this multiple use of conversions, the modules do not get +loaded exclusively for one conversion. Instead a module once loaded can +be used by an arbitrary number of @code{iconv} or @code{mbsrtowcs} calls +at the same time. The splitting of the information between conversion- +function-specific information and conversion data makes this possible. +The last section showed the two data structures used to do this. + +This is of course also reflected in the interface and semantics of the +functions that the modules must provide. There are three functions that +must have the following names: + +@table @code +@item gconv_init +The @code{gconv_init} function initializes the conversion function +specific data structure. This very same object is shared by all +conversions that use this conversion and, therefore, no state information +about the conversion itself must be stored in here. If a module +implements more than one conversion, the @code{gconv_init} function will +be called multiple times. + +@item gconv_end +The @code{gconv_end} function is responsible for freeing all resources +allocated by the @code{gconv_init} function. If there is nothing to do, +this function can be missing. Special care must be taken if the module +implements more than one conversion and the @code{gconv_init} function +does not allocate the same resources for all conversions. + +@item gconv +This is the actual conversion function. It is called to convert one +block of text. It gets passed the conversion step information +initialized by @code{gconv_init} and the conversion data, specific to +this use of the conversion functions. +@end table + +There are three data types defined for the three module interface +functions and these define the interface. + +@comment gconv.h +@comment GNU +@deftypevr {Data type} int {(*__gconv_init_fct)} (struct __gconv_step *) +This specifies the interface of the initialization function of the +module. It is called exactly once for each conversion the module +implements. + +As explained in the description of the @code{struct __gconv_step} data +structure above the initialization function has to initialize parts of +it. + +@table @code +@item __min_needed_from +@itemx __max_needed_from +@itemx __min_needed_to +@itemx __max_needed_to +These elements must be initialized to the exact numbers of the minimum +and maximum number of bytes used by one character in the source and +destination character sets, respectively. If the characters all have the +same size, the minimum and maximum values are the same. + +@item __stateful +This element must be initialized to an nonzero value if the source +character set is stateful. Otherwise it must be zero. +@end table + +If the initialization function needs to communicate some information +to the conversion function, this communication can happen using the +@code{__data} element of the @code{__gconv_step} structure. But since +this data is shared by all the conversions, it must not be modified by +the conversion function. The example below shows how this can be used. + +@smallexample +#define MIN_NEEDED_FROM 1 +#define MAX_NEEDED_FROM 4 +#define MIN_NEEDED_TO 4 +#define MAX_NEEDED_TO 4 + +int +gconv_init (struct __gconv_step *step) +@{ + /* @r{Determine which direction.} */ + struct iso2022jp_data *new_data; + enum direction dir = illegal_dir; + enum variant var = illegal_var; + int result; + + if (__strcasecmp (step->__from_name, "ISO-2022-JP//") == 0) + @{ + dir = from_iso2022jp; + var = iso2022jp; + @} + else if (__strcasecmp (step->__to_name, "ISO-2022-JP//") == 0) + @{ + dir = to_iso2022jp; + var = iso2022jp; + @} + else if (__strcasecmp (step->__from_name, "ISO-2022-JP-2//") == 0) + @{ + dir = from_iso2022jp; + var = iso2022jp2; + @} + else if (__strcasecmp (step->__to_name, "ISO-2022-JP-2//") == 0) + @{ + dir = to_iso2022jp; + var = iso2022jp2; + @} + + result = __GCONV_NOCONV; + if (dir != illegal_dir) + @{ + new_data = (struct iso2022jp_data *) + malloc (sizeof (struct iso2022jp_data)); + + result = __GCONV_NOMEM; + if (new_data != NULL) + @{ + new_data->dir = dir; + new_data->var = var; + step->__data = new_data; + + if (dir == from_iso2022jp) + @{ + step->__min_needed_from = MIN_NEEDED_FROM; + step->__max_needed_from = MAX_NEEDED_FROM; + step->__min_needed_to = MIN_NEEDED_TO; + step->__max_needed_to = MAX_NEEDED_TO; + @} + else + @{ + step->__min_needed_from = MIN_NEEDED_TO; + step->__max_needed_from = MAX_NEEDED_TO; + step->__min_needed_to = MIN_NEEDED_FROM; + step->__max_needed_to = MAX_NEEDED_FROM + 2; + @} + + /* @r{Yes, this is a stateful encoding.} */ + step->__stateful = 1; + + result = __GCONV_OK; + @} + @} + + return result; +@} +@end smallexample + +The function first checks which conversion is wanted. The module from +which this function is taken implements four different conversions; +which one is selected can be determined by comparing the names. The +comparison should always be done without paying attention to the case. + +Next, a data structure, which contains the necessary information about +which conversion is selected, is allocated. The data structure +@code{struct iso2022jp_data} is locally defined since, outside the +module, this data is not used at all. Please note that if all four +conversions this modules supports are requested there are four data +blocks. + +One interesting thing is the initialization of the @code{__min_} and +@code{__max_} elements of the step data object. A single ISO-2022-JP +character can consist of one to four bytes. Therefore the +@code{MIN_NEEDED_FROM} and @code{MAX_NEEDED_FROM} macros are defined +this way. The output is always the @code{INTERNAL} character set (aka +UCS-4) and therefore each character consists of exactly four bytes. For +the conversion from @code{INTERNAL} to ISO-2022-JP we have to take into +account that escape sequences might be necessary to switch the character +sets. Therefore the @code{__max_needed_to} element for this direction +gets assigned @code{MAX_NEEDED_FROM + 2}. This takes into account the +two bytes needed for the escape sequences to single the switching. The +asymmetry in the maximum values for the two directions can be explained +easily: when reading ISO-2022-JP text, escape sequences can be handled +alone (i.e., it is not necessary to process a real character since the +effect of the escape sequence can be recorded in the state information). +The situation is different for the other direction. Since it is in +general not known which character comes next, one cannot emit escape +sequences to change the state in advance. This means the escape +sequences that have to be emitted together with the next character. +Therefore one needs more room than only for the character itself. + +The possible return values of the initialization function are: + +@table @code +@item __GCONV_OK +The initialization succeeded +@item __GCONV_NOCONV +The requested conversion is not supported in the module. This can +happen if the @file{gconv-modules} file has errors. +@item __GCONV_NOMEM +Memory required to store additional information could not be allocated. +@end table +@end deftypevr + +The function called before the module is unloaded is significantly +easier. It often has nothing at all to do; in which case it can be left +out completely. + +@comment gconv.h +@comment GNU +@deftypevr {Data type} void {(*__gconv_end_fct)} (struct gconv_step *) +The task of this function is to free all resources allocated in the +initialization function. Therefore only the @code{__data} element of +the object pointed to by the argument is of interest. Continuing the +example from the initialization function, the finalization function +looks like this: + +@smallexample +void +gconv_end (struct __gconv_step *data) +@{ + free (data->__data); +@} +@end smallexample +@end deftypevr + +The most important function is the conversion function itself, which can +get quite complicated for complex character sets. But since this is not +of interest here, we will only describe a possible skeleton for the +conversion function. + +@comment gconv.h +@comment GNU +@deftypevr {Data type} int {(*__gconv_fct)} (struct __gconv_step *, struct __gconv_step_data *, const char **, const char *, size_t *, int) +The conversion function can be called for two basic reason: to convert +text or to reset the state. From the description of the @code{iconv} +function it can be seen why the flushing mode is necessary. What mode +is selected is determined by the sixth argument, an integer. This +argument being nonzero means that flushing is selected. + +Common to both modes is where the output buffer can be found. The +information about this buffer is stored in the conversion step data. A +pointer to this information is passed as the second argument to this +function. The description of the @code{struct __gconv_step_data} +structure has more information on the conversion step data. + +@cindex stateful +What has to be done for flushing depends on the source character set. +If the source character set is not stateful, nothing has to be done. +Otherwise the function has to emit a byte sequence to bring the state +object into the initial state. Once this all happened the other +conversion modules in the chain of conversions have to get the same +chance. Whether another step follows can be determined from the +@code{__is_last} element of the step data structure to which the first +parameter points. + +The more interesting mode is when actual text has to be converted. The +first step in this case is to convert as much text as possible from the +input buffer and store the result in the output buffer. The start of the +input buffer is determined by the third argument, which is a pointer to a +pointer variable referencing the beginning of the buffer. The fourth +argument is a pointer to the byte right after the last byte in the buffer. + +The conversion has to be performed according to the current state if the +character set is stateful. The state is stored in an object pointed to +by the @code{__statep} element of the step data (second argument). Once +either the input buffer is empty or the output buffer is full the +conversion stops. At this point, the pointer variable referenced by the +third parameter must point to the byte following the last processed +byte (i.e., if all of the input is consumed, this pointer and the fourth +parameter have the same value). + +What now happens depends on whether this step is the last one. If it is +the last step, the only thing that has to be done is to update the +@code{__outbuf} element of the step data structure to point after the +last written byte. This update gives the caller the information on how +much text is available in the output buffer. In addition, the variable +pointed to by the fifth parameter, which is of type @code{size_t}, must +be incremented by the number of characters (@emph{not bytes}) that were +converted in a non-reversible way. Then, the function can return. + +In case the step is not the last one, the later conversion functions have +to get a chance to do their work. Therefore, the appropriate conversion +function has to be called. The information about the functions is +stored in the conversion data structures, passed as the first parameter. +This information and the step data are stored in arrays, so the next +element in both cases can be found by simple pointer arithmetic: + +@smallexample +int +gconv (struct __gconv_step *step, struct __gconv_step_data *data, + const char **inbuf, const char *inbufend, size_t *written, + int do_flush) +@{ + struct __gconv_step *next_step = step + 1; + struct __gconv_step_data *next_data = data + 1; + ... +@end smallexample + +The @code{next_step} pointer references the next step information and +@code{next_data} the next data record. The call of the next function +therefore will look similar to this: + +@smallexample + next_step->__fct (next_step, next_data, &outerr, outbuf, + written, 0) +@end smallexample + +But this is not yet all. Once the function call returns the conversion +function might have some more to do. If the return value of the function +is @code{__GCONV_EMPTY_INPUT}, more room is available in the output +buffer. Unless the input buffer is empty the conversion, functions start +all over again and process the rest of the input buffer. If the return +value is not @code{__GCONV_EMPTY_INPUT}, something went wrong and we have +to recover from this. + +A requirement for the conversion function is that the input buffer +pointer (the third argument) always point to the last character that +was put in converted form into the output buffer. This is trivially +true after the conversion performed in the current step, but if the +conversion functions deeper downstream stop prematurely, not all +characters from the output buffer are consumed and, therefore, the input +buffer pointers must be backed off to the right position. + +Correcting the input buffers is easy to do if the input and output +character sets have a fixed width for all characters. In this situation +we can compute how many characters are left in the output buffer and, +therefore, can correct the input buffer pointer appropriately with a +similar computation. Things are getting tricky if either character set +has characters represented with variable length byte sequences, and it +gets even more complicated if the conversion has to take care of the +state. In these cases the conversion has to be performed once again, from +the known state before the initial conversion (i.e., if necessary the +state of the conversion has to be reset and the conversion loop has to be +executed again). The difference now is that it is known how much input +must be created, and the conversion can stop before converting the first +unused character. Once this is done the input buffer pointers must be +updated again and the function can return. + +One final thing should be mentioned. If it is necessary for the +conversion to know whether it is the first invocation (in case a prolog +has to be emitted), the conversion function should increment the +@code{__invocation_counter} element of the step data structure just +before returning to the caller. See the description of the @code{struct +__gconv_step_data} structure above for more information on how this can +be used. + +The return value must be one of the following values: + +@table @code +@item __GCONV_EMPTY_INPUT +All input was consumed and there is room left in the output buffer. +@item __GCONV_FULL_OUTPUT +No more room in the output buffer. In case this is not the last step +this value is propagated down from the call of the next conversion +function in the chain. +@item __GCONV_INCOMPLETE_INPUT +The input buffer is not entirely empty since it contains an incomplete +character sequence. +@end table + +The following example provides a framework for a conversion function. +In case a new conversion has to be written the holes in this +implementation have to be filled and that is it. + +@smallexample +int +gconv (struct __gconv_step *step, struct __gconv_step_data *data, + const char **inbuf, const char *inbufend, size_t *written, + int do_flush) +@{ + struct __gconv_step *next_step = step + 1; + struct __gconv_step_data *next_data = data + 1; + gconv_fct fct = next_step->__fct; + int status; + + /* @r{If the function is called with no input this means we have} + @r{to reset to the initial state. The possibly partly} + @r{converted input is dropped.} */ + if (do_flush) + @{ + status = __GCONV_OK; + + /* @r{Possible emit a byte sequence which put the state object} + @r{into the initial state.} */ + + /* @r{Call the steps down the chain if there are any but only} + @r{if we successfully emitted the escape sequence.} */ + if (status == __GCONV_OK && ! data->__is_last) + status = fct (next_step, next_data, NULL, NULL, + written, 1); + @} + else + @{ + /* @r{We preserve the initial values of the pointer variables.} */ + const char *inptr = *inbuf; + char *outbuf = data->__outbuf; + char *outend = data->__outbufend; + char *outptr; + + do + @{ + /* @r{Remember the start value for this round.} */ + inptr = *inbuf; + /* @r{The outbuf buffer is empty.} */ + outptr = outbuf; + + /* @r{For stateful encodings the state must be safe here.} */ + + /* @r{Run the conversion loop. @code{status} is set} + @r{appropriately afterwards.} */ + + /* @r{If this is the last step, leave the loop. There is} + @r{nothing we can do.} */ + if (data->__is_last) + @{ + /* @r{Store information about how many bytes are} + @r{available.} */ + data->__outbuf = outbuf; + + /* @r{If any non-reversible conversions were performed,} + @r{add the number to @code{*written}.} */ + + break; + @} + + /* @r{Write out all output that was produced.} */ + if (outbuf > outptr) + @{ + const char *outerr = data->__outbuf; + int result; + + result = fct (next_step, next_data, &outerr, + outbuf, written, 0); + + if (result != __GCONV_EMPTY_INPUT) + @{ + if (outerr != outbuf) + @{ + /* @r{Reset the input buffer pointer. We} + @r{document here the complex case.} */ + size_t nstatus; + + /* @r{Reload the pointers.} */ + *inbuf = inptr; + outbuf = outptr; + + /* @r{Possibly reset the state.} */ + + /* @r{Redo the conversion, but this time} + @r{the end of the output buffer is at} + @r{@code{outerr}.} */ + @} + + /* @r{Change the status.} */ + status = result; + @} + else + /* @r{All the output is consumed, we can make} + @r{ another run if everything was ok.} */ + if (status == __GCONV_FULL_OUTPUT) + status = __GCONV_OK; + @} + @} + while (status == __GCONV_OK); + + /* @r{We finished one use of this step.} */ + ++data->__invocation_counter; + @} + + return status; +@} +@end smallexample +@end deftypevr + +This information should be sufficient to write new modules. Anybody +doing so should also take a look at the available source code in the GNU +C library sources. It contains many examples of working and optimized +modules. + @c File charset.texi edited October 2001 by Dennis Grace, IBM Corporation
\ No newline at end of file |