File: libc.info, Node: Top, Next: Introduction, Prev: (dir), Up: (dir)
Main Menu
*********
This is Edition 0.10, last updated 2001-07-06, of `The GNU C Library
Reference Manual', for Version 2.3.x of the GNU C Library.
* Menu:
* Introduction:: Purpose of the GNU C Library.
* Error Reporting:: How library functions report errors.
* Memory:: Allocating virtual memory and controlling
paging.
* Character Handling:: Character testing and conversion functions.
* String and Array Utilities:: Utilities for copying and comparing strings
and arrays.
* Character Set Handling:: Support for extended character sets.
* Locales:: The country and language can affect the
behavior of library functions.
* Message Translation:: How to make the program speak the user's
language.
* Searching and Sorting:: General searching and sorting functions.
* Pattern Matching:: Matching shell ``globs'' and regular
expressions.
* I/O Overview:: Introduction to the I/O facilities.
* I/O on Streams:: High-level, portable I/O facilities.
* Low-Level I/O:: Low-level, less portable I/O.
* File System Interface:: Functions for manipulating files.
* Pipes and FIFOs:: A simple interprocess communication
mechanism.
* Sockets:: A more complicated IPC mechanism, with
networking support.
* Low-Level Terminal Interface:: How to change the characteristics of a
terminal device.
* Syslog:: System logging and messaging.
* Mathematics:: Math functions, useful constants, random
numbers.
* Arithmetic:: Low level arithmetic functions.
* Date and Time:: Functions for getting the date and time and
formatting them nicely.
* Resource Usage And Limitation:: Functions for examining resource usage and
getting and setting limits.
* Non-Local Exits:: Jumping out of nested function calls.
* Signal Handling:: How to send, block, and handle signals.
* Program Basics:: Writing the beginning and end of your
program.
* Processes:: How to create processes and run other
programs.
* Job Control:: All about process groups and sessions.
* Name Service Switch:: Accessing system databases.
* Users and Groups:: How users are identified and classified.
* System Management:: Controlling the system and getting
information about it.
* System Configuration:: Parameters describing operating system
limits.
* Cryptographic Functions:: DES encryption and password handling.
* Debugging Support:: Functions to help debugging applications.
Add-ons
* POSIX Threads:: The standard threads library.
Appendices
* Language Features:: C language features provided by the library.
* Library Summary:: A summary showing the syntax, header file,
and derivation of each library feature.
* Installation:: How to install the GNU C library.
* Maintenance:: How to enhance and port the GNU C Library.
* Contributors:: Who wrote what parts of the GNU C library.
* Free Manuals:: Free Software Needs Free Documentation.
* Copying:: The GNU Lesser General Public License says
how you can copy and share the GNU C Library.
* Documentation License:: This manual is under the GNU Free
Documentation License.
Indices
* Concept Index:: Index of concepts and names.
* Type Index:: Index of types and type qualifiers.
* Function Index:: Index of functions and function-like macros.
* Variable Index:: Index of variables and variable-like macros.
* File Index:: Index of programs and files.
--- The Detailed Node Listing ---
Introduction
* Getting Started:: What this manual is for and how to use it.
* Standards and Portability:: Standards and sources upon which the GNU
C library is based.
* Using the Library:: Some practical uses for the library.
* Roadmap to the Manual:: Overview of the remaining chapters in
this manual.
Standards and Portability
* ISO C:: The international standard for the C
programming language.
* POSIX:: The ISO/IEC 9945 (aka IEEE 1003) standards
for operating systems.
* Berkeley Unix:: BSD and SunOS.
* SVID:: The System V Interface Description.
* XPG:: The X/Open Portability Guide.
Using the Library
* Header Files:: How to include the header files in your
programs.
* Macro Definitions:: Some functions in the library may really
be implemented as macros.
* Reserved Names:: The C standard reserves some names for
the library, and some for users.
* Feature Test Macros:: How to control what names are defined.
Error Reporting
* Checking for Errors:: How errors are reported by library functions.
* Error Codes:: Error code macros; all of these expand
into integer constant values.
* Error Messages:: Mapping error codes onto error messages.
Memory
* Memory Concepts:: An introduction to concepts and terminology.
* Memory Allocation:: Allocating storage for your program data
* Locking Pages:: Preventing page faults
* Resizing the Data Segment:: `brk', `sbrk'
Memory Allocation
* Memory Allocation and C:: How to get different kinds of allocation in C.
* Unconstrained Allocation:: The `malloc' facility allows fully general
dynamic allocation.
* Allocation Debugging:: Finding memory leaks and not freed memory.
* Obstacks:: Obstacks are less general than malloc
but more efficient and convenient.
* Variable Size Automatic:: Allocation of variable-sized blocks
of automatic storage that are freed when the
calling function returns.
Unconstrained Allocation
* Basic Allocation:: Simple use of `malloc'.
* Malloc Examples:: Examples of `malloc'. `xmalloc'.
* Freeing after Malloc:: Use `free' to free a block you
got with `malloc'.
* Changing Block Size:: Use `realloc' to make a block
bigger or smaller.
* Allocating Cleared Space:: Use `calloc' to allocate a
block and clear it.
* Efficiency and Malloc:: Efficiency considerations in use of
these functions.
* Aligned Memory Blocks:: Allocating specially aligned memory.
* Malloc Tunable Parameters:: Use `mallopt' to adjust allocation
parameters.
* Heap Consistency Checking:: Automatic checking for errors.
* Hooks for Malloc:: You can use these hooks for debugging
programs that use `malloc'.
* Statistics of Malloc:: Getting information about how much
memory your program is using.
* Summary of Malloc:: Summary of `malloc' and related functions.
Allocation Debugging
* Tracing malloc:: How to install the tracing functionality.
* Using the Memory Debugger:: Example programs excerpts.
* Tips for the Memory Debugger:: Some more or less clever ideas.
* Interpreting the traces:: What do all these lines mean?
Obstacks
* Creating Obstacks:: How to declare an obstack in your program.
* Preparing for Obstacks:: Preparations needed before you can
use obstacks.
* Allocation in an Obstack:: Allocating objects in an obstack.
* Freeing Obstack Objects:: Freeing objects in an obstack.
* Obstack Functions:: The obstack functions are both
functions and macros.
* Growing Objects:: Making an object bigger by stages.
* Extra Fast Growing:: Extra-high-efficiency (though more
complicated) growing objects.
* Status of an Obstack:: Inquiries about the status of an obstack.
* Obstacks Data Alignment:: Controlling alignment of objects in obstacks.
* Obstack Chunks:: How obstacks obtain and release chunks;
efficiency considerations.
* Summary of Obstacks::
Variable Size Automatic
* Alloca Example:: Example of using `alloca'.
* Advantages of Alloca:: Reasons to use `alloca'.
* Disadvantages of Alloca:: Reasons to avoid `alloca'.
* GNU C Variable-Size Arrays:: Only in GNU C, here is an alternative
method of allocating dynamically and
freeing automatically.
Locking Pages
* Why Lock Pages:: Reasons to read this section.
* Locked Memory Details:: Everything you need to know locked
memory
* Page Lock Functions:: Here's how to do it.
Character Handling
* Classification of Characters:: Testing whether characters are
letters, digits, punctuation, etc.
* Case Conversion:: Case mapping, and the like.
* Classification of Wide Characters:: Character class determination for
wide characters.
* Using Wide Char Classes:: Notes on using the wide character
classes.
* Wide Character Case Conversion:: Mapping of wide characters.
String and Array Utilities
* Representation of Strings:: Introduction to basic concepts.
* String/Array Conventions:: Whether to use a string function or an
arbitrary array function.
* String Length:: Determining the length of a string.
* Copying and Concatenation:: Functions to copy the contents of strings
and arrays.
* String/Array Comparison:: Functions for byte-wise and character-wise
comparison.
* Collation Functions:: Functions for collating strings.
* Search Functions:: Searching for a specific element or substring.
* Finding Tokens in a String:: Splitting a string into tokens by looking
for delimiters.
* strfry:: Function for flash-cooking a string.
* Trivial Encryption:: Obscuring data.
* Encode Binary Data:: Encoding and Decoding of Binary Data.
* Argz and Envz Vectors:: Null-separated string vectors.
Argz and Envz Vectors
* Argz Functions:: Operations on argz vectors.
* Envz Functions:: Additional operations on environment vectors.
Character Set Handling
* Extended Char Intro:: Introduction to Extended Characters.
* Charset Function Overview:: Overview about Character Handling
Functions.
* Restartable multibyte conversion:: Restartable multibyte conversion
Functions.
* Non-reentrant Conversion:: Non-reentrant Conversion Function.
* Generic Charset Conversion:: Generic Charset Conversion.
Restartable multibyte conversion
* Selecting the Conversion:: Selecting the conversion and its properties.
* Keeping the state:: Representing the state of the conversion.
* Converting a Character:: Converting Single Characters.
* Converting Strings:: Converting Multibyte and Wide Character
Strings.
* Multibyte Conversion Example:: A Complete Multibyte Conversion Example.
Non-reentrant Conversion
* Non-reentrant Character Conversion:: Non-reentrant Conversion of Single
Characters.
* Non-reentrant String Conversion:: Non-reentrant Conversion of Strings.
* Shift State:: States in Non-reentrant Functions.
Generic Charset Conversion
* Generic Conversion Interface:: Generic Character Set Conversion Interface.
* iconv Examples:: A complete `iconv' example.
* Other iconv Implementations:: Some Details about other `iconv'
Implementations.
* glibc iconv Implementation:: The `iconv' Implementation in the GNU C
library.
Locales
* Effects of Locale:: Actions affected by the choice of
locale.
* Choosing Locale:: How the user specifies a locale.
* Locale Categories:: Different purposes for which you can
select a locale.
* Setting the Locale:: How a program specifies the locale
with library functions.
* Standard Locales:: Locale names available on all systems.
* Locale Information:: How to access the information for the locale.
* Formatting Numbers:: A dedicated function to format numbers.
* Yes-or-No Questions:: Check a Response against the locale.
Locale Information
* The Lame Way to Locale Data:: ISO C's `localeconv'.
* The Elegant and Fast Way:: X/Open's `nl_langinfo'.
The Lame Way to Locale Data
* General Numeric:: Parameters for formatting numbers and
currency amounts.
* Currency Symbol:: How to print the symbol that identifies an
amount of money (e.g. `$').
* Sign of Money Amount:: How to print the (positive or negative) sign
for a monetary amount, if one exists.
Message Translation
* Message catalogs a la X/Open:: The `catgets' family of functions.
* The Uniforum approach:: The `gettext' family of functions.
Message catalogs a la X/Open
* The catgets Functions:: The `catgets' function family.
* The message catalog files:: Format of the message catalog files.
* The gencat program:: How to generate message catalogs files which
can be used by the functions.
* Common Usage:: How to use the `catgets' interface.
The Uniforum approach
* Message catalogs with gettext:: The `gettext' family of functions.
* Helper programs for gettext:: Programs to handle message catalogs
for `gettext'.
Message catalogs with gettext
* Translation with gettext:: What has to be done to translate a message.
* Locating gettext catalog:: How to determine which catalog to be used.
* Advanced gettext functions:: Additional functions for more complicated
situations.
* Charset conversion in gettext:: How to specify the output character set
`gettext' uses.
* GUI program problems:: How to use `gettext' in GUI programs.
* Using gettextized software:: The possibilities of the user to influence
the way `gettext' works.
Searching and Sorting
* Comparison Functions:: Defining how to compare two objects.
Since the sort and search facilities
are general, you have to specify the
ordering.
* Array Search Function:: The `bsearch' function.
* Array Sort Function:: The `qsort' function.
* Search/Sort Example:: An example program.
* Hash Search Function:: The `hsearch' function.
* Tree Search Function:: The `tsearch' function.
Pattern Matching
* Wildcard Matching:: Matching a wildcard pattern against a single string.
* Globbing:: Finding the files that match a wildcard pattern.
* Regular Expressions:: Matching regular expressions against strings.
* Word Expansion:: Expanding shell variables, nested commands,
arithmetic, and wildcards.
This is what the shell does with shell commands.
Globbing
* Calling Glob:: Basic use of `glob'.
* Flags for Globbing:: Flags that enable various options in `glob'.
* More Flags for Globbing:: GNU specific extensions to `glob'.
Regular Expressions
* POSIX Regexp Compilation:: Using `regcomp' to prepare to match.
* Flags for POSIX Regexps:: Syntax variations for `regcomp'.
* Matching POSIX Regexps:: Using `regexec' to match the compiled
pattern that you get from `regcomp'.
* Regexp Subexpressions:: Finding which parts of the string were matched.
* Subexpression Complications:: Find points of which parts were matched.
* Regexp Cleanup:: Freeing storage; reporting errors.
Word Expansion
* Expansion Stages:: What word expansion does to a string.
* Calling Wordexp:: How to call `wordexp'.
* Flags for Wordexp:: Options you can enable in `wordexp'.
* Wordexp Example:: A sample program that does word expansion.
* Tilde Expansion:: Details of how tilde expansion works.
* Variable Substitution:: Different types of variable substitution.
I/O Overview
* I/O Concepts:: Some basic information and terminology.
* File Names:: How to refer to a file.
I/O Concepts
* Streams and File Descriptors:: The GNU Library provides two ways
to access the contents of files.
* File Position:: The number of bytes from the
beginning of the file.
File Names
* Directories:: Directories contain entries for files.
* File Name Resolution:: A file name specifies how to look up a file.
* File Name Errors:: Error conditions relating to file names.
* File Name Portability:: File name portability and syntax issues.
I/O on Streams
* Streams:: About the data type representing a stream.
* Standard Streams:: Streams to the standard input and output
devices are created for you.
* Opening Streams:: How to create a stream to talk to a file.
* Closing Streams:: Close a stream when you are finished with it.
* Streams and Threads:: Issues with streams in threaded programs.
* Streams and I18N:: Streams in internationalized applications.
* Simple Output:: Unformatted output by characters and lines.
* Character Input:: Unformatted input by characters and words.
* Line Input:: Reading a line or a record from a stream.
* Unreading:: Peeking ahead/pushing back input just read.
* Block Input/Output:: Input and output operations on blocks of data.
* Formatted Output:: `printf' and related functions.
* Customizing Printf:: You can define new conversion specifiers for
`printf' and friends.
* Formatted Input:: `scanf' and related functions.
* EOF and Errors:: How you can tell if an I/O error happens.
* Error Recovery:: What you can do about errors.
* Binary Streams:: Some systems distinguish between text files
and binary files.
* File Positioning:: About random-access streams.
* Portable Positioning:: Random access on peculiar ISO C systems.
* Stream Buffering:: How to control buffering of streams.
* Other Kinds of Streams:: Streams that do not necessarily correspond
to an open file.
* Formatted Messages:: Print strictly formatted messages.
Unreading
* Unreading Idea:: An explanation of unreading with pictures.
* How Unread:: How to call `ungetc' to do unreading.
Formatted Output
* Formatted Output Basics:: Some examples to get you started.
* Output Conversion Syntax:: General syntax of conversion
specifications.
* Table of Output Conversions:: Summary of output conversions and
what they do.
* Integer Conversions:: Details about formatting of integers.
* Floating-Point Conversions:: Details about formatting of
floating-point numbers.
* Other Output Conversions:: Details about formatting of strings,
characters, pointers, and the like.
* Formatted Output Functions:: Descriptions of the actual functions.
* Dynamic Output:: Functions that allocate memory for the output.
* Variable Arguments Output:: `vprintf' and friends.
* Parsing a Template String:: What kinds of args does a given template
call for?
* Example of Parsing:: Sample program using `parse_printf_format'.
Customizing Printf
* Registering New Conversions:: Using `register_printf_function'
to register a new output conversion.
* Conversion Specifier Options:: The handler must be able to get
the options specified in the
template when it is called.
* Defining the Output Handler:: Defining the handler and arginfo
functions that are passed as arguments
to `register_printf_function'.
* Printf Extension Example:: How to define a `printf'
handler function.
* Predefined Printf Handlers:: Predefined `printf' handlers.
Formatted Input
* Formatted Input Basics:: Some basics to get you started.
* Input Conversion Syntax:: Syntax of conversion specifications.
* Table of Input Conversions:: Summary of input conversions and what they do.
* Numeric Input Conversions:: Details of conversions for reading numbers.
* String Input Conversions:: Details of conversions for reading strings.
* Dynamic String Input:: String conversions that `malloc' the buffer.
* Other Input Conversions:: Details of miscellaneous other conversions.
* Formatted Input Functions:: Descriptions of the actual functions.
* Variable Arguments Input:: `vscanf' and friends.
Stream Buffering
* Buffering Concepts:: Terminology is defined here.
* Flushing Buffers:: How to ensure that output buffers are flushed.
* Controlling Buffering:: How to specify what kind of buffering to use.
Other Kinds of Streams
* String Streams:: Streams that get data from or put data in
a string or memory buffer.
* Obstack Streams:: Streams that store data in an obstack.
* Custom Streams:: Defining your own streams with an arbitrary
input data source and/or output data sink.
Custom Streams
* Streams and Cookies:: The "cookie" records where to fetch or
store data that is read or written.
* Hook Functions:: How you should define the four "hook
functions" that a custom stream needs.
Formatted Messages
* Printing Formatted Messages:: The `fmtmsg' function.
* Adding Severity Classes:: Add more severity classes.
* Example:: How to use `fmtmsg' and `addseverity'.
Low-Level I/O
* Opening and Closing Files:: How to open and close file
descriptors.
* I/O Primitives:: Reading and writing data.
* File Position Primitive:: Setting a descriptor's file
position.
* Descriptors and Streams:: Converting descriptor to stream
or vice-versa.
* Stream/Descriptor Precautions:: Precautions needed if you use both
descriptors and streams.
* Scatter-Gather:: Fast I/O to discontinuous buffers.
* Memory-mapped I/O:: Using files like memory.
* Waiting for I/O:: How to check for input or output
on multiple file descriptors.
* Synchronizing I/O:: Making sure all I/O actions completed.
* Asynchronous I/O:: Perform I/O in parallel.
* Control Operations:: Various other operations on file
descriptors.
* Duplicating Descriptors:: Fcntl commands for duplicating
file descriptors.
* Descriptor Flags:: Fcntl commands for manipulating
flags associated with file
descriptors.
* File Status Flags:: Fcntl commands for manipulating
flags associated with open files.
* File Locks:: Fcntl commands for implementing
file locking.
* Interrupt Input:: Getting an asynchronous signal when
input arrives.
* IOCTLs:: Generic I/O Control operations.
Stream/Descriptor Precautions
* Linked Channels:: Dealing with channels sharing a file position.
* Independent Channels:: Dealing with separately opened, unlinked channels.
* Cleaning Streams:: Cleaning a stream makes it safe to use
another channel.
Asynchronous I/O
* Asynchronous Reads/Writes:: Asynchronous Read and Write Operations.
* Status of AIO Operations:: Getting the Status of AIO Operations.
* Synchronizing AIO Operations:: Getting into a consistent state.
* Cancel AIO Operations:: Cancellation of AIO Operations.
* Configuration of AIO:: How to optimize the AIO implementation.
File Status Flags
* Access Modes:: Whether the descriptor can read or write.
* Open-time Flags:: Details of `open'.
* Operating Modes:: Special modes to control I/O operations.
* Getting File Status Flags:: Fetching and changing these flags.
File System Interface
* Working Directory:: This is used to resolve relative
file names.
* Accessing Directories:: Finding out what files a directory
contains.
* Working with Directory Trees:: Apply actions to all files or a selectable
subset of a directory hierarchy.
* Hard Links:: Adding alternate names to a file.
* Symbolic Links:: A file that ``points to'' a file name.
* Deleting Files:: How to delete a file, and what that means.
* Renaming Files:: Changing a file's name.
* Creating Directories:: A system call just for creating a directory.
* File Attributes:: Attributes of individual files.
* Making Special Files:: How to create special files.
* Temporary Files:: Naming and creating temporary files.
Accessing Directories
* Directory Entries:: Format of one directory entry.
* Opening a Directory:: How to open a directory stream.
* Reading/Closing Directory:: How to read directory entries from the stream.
* Simple Directory Lister:: A very simple directory listing program.
* Random Access Directory:: Rereading part of the directory
already read with the same stream.
* Scanning Directory Content:: Get entries for user selected subset of
contents in given directory.
* Simple Directory Lister Mark II:: Revised version of the program.
File Attributes
* Attribute Meanings:: The names of the file attributes,
and what their values mean.
* Reading Attributes:: How to read the attributes of a file.
* Testing File Type:: Distinguishing ordinary files,
directories, links...
* File Owner:: How ownership for new files is determined,
and how to change it.
* Permission Bits:: How information about a file's access
mode is stored.
* Access Permission:: How the system decides who can access a file.
* Setting Permissions:: How permissions for new files are assigned,
and how to change them.
* Testing File Access:: How to find out if your process can
access a file.
* File Times:: About the time attributes of a file.
* File Size:: Manually changing the size of a file.
Pipes and FIFOs
* Creating a Pipe:: Making a pipe with the `pipe' function.
* Pipe to a Subprocess:: Using a pipe to communicate with a
child process.
* FIFO Special Files:: Making a FIFO special file.
* Pipe Atomicity:: When pipe (or FIFO) I/O is atomic.
Sockets
* Socket Concepts:: Basic concepts you need to know about.
* Communication Styles::Stream communication, datagrams and other styles.
* Socket Addresses:: How socket names (``addresses'') work.
* Interface Naming:: Identifying specific network interfaces.
* Local Namespace:: Details about the local namespace.
* Internet Namespace:: Details about the Internet namespace.
* Misc Namespaces:: Other namespaces not documented fully here.
* Open/Close Sockets:: Creating sockets and destroying them.
* Connections:: Operations on sockets with connection state.
* Datagrams:: Operations on datagram sockets.
* Inetd:: Inetd is a daemon that starts servers on request.
The most convenient way to write a server
is to make it work with Inetd.
* Socket Options:: Miscellaneous low-level socket options.
* Networks Database:: Accessing the database of network names.
Socket Addresses
* Address Formats:: About `struct sockaddr'.
* Setting Address:: Binding an address to a socket.
* Reading Address:: Reading the address of a socket.
Local Namespace
* Concepts: Local Namespace Concepts. What you need to understand.
* Details: Local Namespace Details. Address format, symbolic names, etc.
* Example: Local Socket Example. Example of creating a socket.
Internet Namespace
* Internet Address Formats:: How socket addresses are specified in the
Internet namespace.
* Host Addresses:: All about host addresses of Internet host.
* Protocols Database:: Referring to protocols by name.
* Ports:: Internet port numbers.
* Services Database:: Ports may have symbolic names.
* Byte Order:: Different hosts may use different byte
ordering conventions; you need to
canonicalize host address and port number.
* Inet Example:: Putting it all together.
Host Addresses
* Abstract Host Addresses:: What a host number consists of.
* Data type: Host Address Data Type. Data type for a host number.
* Functions: Host Address Functions. Functions to operate on them.
* Names: Host Names. Translating host names to host numbers.
Open/Close Sockets
* Creating a Socket:: How to open a socket.
* Closing a Socket:: How to close a socket.
* Socket Pairs:: These are created like pipes.
Connections
* Connecting:: What the client program must do.
* Listening:: How a server program waits for requests.
* Accepting Connections:: What the server does when it gets a request.
* Who is Connected:: Getting the address of the
other side of a connection.
* Transferring Data:: How to send and receive data.
* Byte Stream Example:: An example program: a client for communicating
over a byte stream socket in the Internet namespace.
* Server Example:: A corresponding server program.
* Out-of-Band Data:: This is an advanced feature.
Transferring Data
* Sending Data:: Sending data with `send'.
* Receiving Data:: Reading data with `recv'.
* Socket Data Options:: Using `send' and `recv'.
Datagrams
* Sending Datagrams:: Sending packets on a datagram socket.
* Receiving Datagrams:: Receiving packets on a datagram socket.
* Datagram Example:: An example program: packets sent over a
datagram socket in the local namespace.
* Example Receiver:: Another program, that receives those packets.
Inetd
* Inetd Servers::
* Configuring Inetd::
Socket Options
* Socket Option Functions:: The basic functions for setting and getting
socket options.
* Socket-Level Options:: Details of the options at the socket level.
Low-Level Terminal Interface
* Is It a Terminal:: How to determine if a file is a terminal
device, and what its name is.
* I/O Queues:: About flow control and typeahead.
* Canonical or Not:: Two basic styles of input processing.
* Terminal Modes:: How to examine and modify flags controlling
details of terminal I/O: echoing,
signals, editing. Posix.
* BSD Terminal Modes:: BSD compatible terminal mode setting
* Line Control:: Sending break sequences, clearing
terminal buffers ...
* Noncanon Example:: How to read single characters without echo.
* Pseudo-Terminals:: How to open a pseudo-terminal.
Terminal Modes
* Mode Data Types:: The data type `struct termios' and
related types.
* Mode Functions:: Functions to read and set the terminal
attributes.
* Setting Modes:: The right way to set terminal attributes
reliably.
* Input Modes:: Flags controlling low-level input handling.
* Output Modes:: Flags controlling low-level output handling.
* Control Modes:: Flags controlling serial port behavior.
* Local Modes:: Flags controlling high-level input handling.
* Line Speed:: How to read and set the terminal line speed.
* Special Characters:: Characters that have special effects,
and how to change them.
* Noncanonical Input:: Controlling how long to wait for input.
Special Characters
* Editing Characters:: Special characters that terminate lines and
delete text, and other editing functions.
* Signal Characters:: Special characters that send or raise signals
to or for certain classes of processes.
* Start/Stop Characters:: Special characters that suspend or resume
suspended output.
* Other Special:: Other special characters for BSD systems:
they can discard output, and print status.
Pseudo-Terminals
* Allocation:: Allocating a pseudo terminal.
* Pseudo-Terminal Pairs:: How to open both sides of a
pseudo-terminal in a single operation.
Syslog
* Overview of Syslog:: Overview of a system's Syslog facility
* Submitting Syslog Messages:: Functions to submit messages to Syslog
Submitting Syslog Messages
* openlog:: Open connection to Syslog
* syslog; vsyslog:: Submit message to Syslog
* closelog:: Close connection to Syslog
* setlogmask:: Cause certain messages to be ignored
* Syslog Example:: Example of all of the above
Mathematics
* Mathematical Constants:: Precise numeric values for often-used
constants.
* Trig Functions:: Sine, cosine, tangent, and friends.
* Inverse Trig Functions:: Arcsine, arccosine, etc.
* Exponents and Logarithms:: Also pow and sqrt.
* Hyperbolic Functions:: sinh, cosh, tanh, etc.
* Special Functions:: Bessel, gamma, erf.
* Errors in Math Functions:: Known Maximum Errors in Math Functions.
* Pseudo-Random Numbers:: Functions for generating pseudo-random
numbers.
* FP Function Optimizations:: Fast code or small code.
Pseudo-Random Numbers
* ISO Random:: `rand' and friends.
* BSD Random:: `random' and friends.
* SVID Random:: `drand48' and friends.
Arithmetic
* Integers:: Basic integer types and concepts
* Integer Division:: Integer division with guaranteed rounding.
* Floating Point Numbers:: Basic concepts. IEEE 754.
* Floating Point Classes:: The five kinds of floating-point number.
* Floating Point Errors:: When something goes wrong in a calculation.
* Rounding:: Controlling how results are rounded.
* Control Functions:: Saving and restoring the FPU's state.
* Arithmetic Functions:: Fundamental operations provided by the library.
* Complex Numbers:: The types. Writing complex constants.
* Operations on Complex:: Projection, conjugation, decomposition.
* Parsing of Numbers:: Converting strings to numbers.
* System V Number Conversion:: An archaic way to convert numbers to strings.
Floating Point Errors
* FP Exceptions:: IEEE 754 math exceptions and how to detect them.
* Infinity and NaN:: Special values returned by calculations.
* Status bit operations:: Checking for exceptions after the fact.
* Math Error Reporting:: How the math functions report errors.
Arithmetic Functions
* Absolute Value:: Absolute values of integers and floats.
* Normalization Functions:: Extracting exponents and putting them back.
* Rounding Functions:: Rounding floats to integers.
* Remainder Functions:: Remainders on division, precisely defined.
* FP Bit Twiddling:: Sign bit adjustment. Adding epsilon.
* FP Comparison Functions:: Comparisons without risk of exceptions.
* Misc FP Arithmetic:: Max, min, positive difference, multiply-add.
Parsing of Numbers
* Parsing of Integers:: Functions for conversion of integer values.
* Parsing of Floats:: Functions for conversion of floating-point
values.
Date and Time
* Time Basics:: Concepts and definitions.
* Elapsed Time:: Data types to represent elapsed times
* Processor And CPU Time:: Time a program has spent executing.
* Calendar Time:: Manipulation of ``real'' dates and times.
* Setting an Alarm:: Sending a signal after a specified time.
* Sleeping:: Waiting for a period of time.
Processor And CPU Time
* CPU Time:: The `clock' function.
* Processor Time:: The `times' function.
Calendar Time
* Simple Calendar Time:: Facilities for manipulating calendar time.
* High-Resolution Calendar:: A time representation with greater precision.
* Broken-down Time:: Facilities for manipulating local time.
* High Accuracy Clock:: Maintaining a high accuracy system clock.
* Formatting Calendar Time:: Converting times to strings.
* Parsing Date and Time:: Convert textual time and date information back
into broken-down time values.
* TZ Variable:: How users specify the time zone.
* Time Zone Functions:: Functions to examine or specify the time zone.
* Time Functions Example:: An example program showing use of some of
the time functions.
Parsing Date and Time
* Low-Level Time String Parsing:: Interpret string according to given format.
* General Time String Parsing:: User-friendly function to parse data and
time strings.
Resource Usage And Limitation
* Resource Usage:: Measuring various resources used.
* Limits on Resources:: Specifying limits on resource usage.
* Priority:: Reading or setting process run priority.
* Memory Resources:: Querying memory available resources.
* Processor Resources:: Learn about the processors available.
Priority
* Absolute Priority:: The first tier of priority. Posix
* Realtime Scheduling:: Scheduling among the process nobility
* Basic Scheduling Functions:: Get/set scheduling policy, priority
* Traditional Scheduling:: Scheduling among the vulgar masses
* CPU Affinity:: Limiting execution to certain CPUs
Traditional Scheduling
* Traditional Scheduling Intro::
* Traditional Scheduling Functions::
Memory Resources
* Memory Subsystem:: Overview about traditional Unix memory handling.
* Query Memory Parameters:: How to get information about the memory
subsystem?
Non-Local Exits
* Intro: Non-Local Intro. When and how to use these facilities.
* Details: Non-Local Details. Functions for non-local exits.
* Non-Local Exits and Signals:: Portability issues.
* System V contexts:: Complete context control a la System V.
Signal Handling
* Concepts of Signals:: Introduction to the signal facilities.
* Standard Signals:: Particular kinds of signals with
standard names and meanings.
* Signal Actions:: Specifying what happens when a
particular signal is delivered.
* Defining Handlers:: How to write a signal handler function.
* Interrupted Primitives:: Signal handlers affect use of `open',
`read', `write' and other functions.
* Generating Signals:: How to send a signal to a process.
* Blocking Signals:: Making the system hold signals temporarily.
* Waiting for a Signal:: Suspending your program until a signal
arrives.
* Signal Stack:: Using a Separate Signal Stack.
* BSD Signal Handling:: Additional functions for backward
compatibility with BSD.
Concepts of Signals
* Kinds of Signals:: Some examples of what can cause a signal.
* Signal Generation:: Concepts of why and how signals occur.
* Delivery of Signal:: Concepts of what a signal does to the
process.
Standard Signals
* Program Error Signals:: Used to report serious program errors.
* Termination Signals:: Used to interrupt and/or terminate the
program.
* Alarm Signals:: Used to indicate expiration of timers.
* Asynchronous I/O Signals:: Used to indicate input is available.
* Job Control Signals:: Signals used to support job control.
* Operation Error Signals:: Used to report operational system errors.
* Miscellaneous Signals:: Miscellaneous Signals.
* Signal Messages:: Printing a message describing a signal.
Signal Actions
* Basic Signal Handling:: The simple `signal' function.
* Advanced Signal Handling:: The more powerful `sigaction' function.
* Signal and Sigaction:: How those two functions interact.
* Sigaction Function Example:: An example of using the sigaction function.
* Flags for Sigaction:: Specifying options for signal handling.
* Initial Signal Actions:: How programs inherit signal actions.
Defining Handlers
* Handler Returns:: Handlers that return normally, and what
this means.
* Termination in Handler:: How handler functions terminate a program.
* Longjmp in Handler:: Nonlocal transfer of control out of a
signal handler.
* Signals in Handler:: What happens when signals arrive while
the handler is already occupied.
* Merged Signals:: When a second signal arrives before the
first is handled.
* Nonreentrancy:: Do not call any functions unless you know they
are reentrant with respect to signals.
* Atomic Data Access:: A single handler can run in the middle of
reading or writing a single object.
Atomic Data Access
* Non-atomic Example:: A program illustrating interrupted access.
* Types: Atomic Types. Data types that guarantee no interruption.
* Usage: Atomic Usage. Proving that interruption is harmless.
Generating Signals
* Signaling Yourself:: A process can send a signal to itself.
* Signaling Another Process:: Send a signal to another process.
* Permission for kill:: Permission for using `kill'.
* Kill Example:: Using `kill' for Communication.
Blocking Signals
* Why Block:: The purpose of blocking signals.
* Signal Sets:: How to specify which signals to
block.
* Process Signal Mask:: Blocking delivery of signals to your
process during normal execution.
* Testing for Delivery:: Blocking to Test for Delivery of
a Signal.
* Blocking for Handler:: Blocking additional signals while a
handler is being run.
* Checking for Pending Signals:: Checking for Pending Signals
* Remembering a Signal:: How you can get almost the same
effect as blocking a signal, by
handling it and setting a flag
to be tested later.
Waiting for a Signal
* Using Pause:: The simple way, using `pause'.
* Pause Problems:: Why the simple way is often not very good.
* Sigsuspend:: Reliably waiting for a specific signal.
BSD Signal Handling
* BSD Handler:: BSD Function to Establish a Handler.
* Blocking in BSD:: BSD Functions for Blocking Signals.
Program Basics
* Program Arguments:: Parsing your program's command-line arguments.
* Environment Variables:: Less direct parameters affecting your program
* System Calls:: Requesting service from the system
* Program Termination:: Telling the system you're done; return status
Program Arguments
* Argument Syntax:: By convention, options start with a hyphen.
* Parsing Program Arguments:: Ways to parse program options and arguments.
Parsing Program Arguments
* Getopt:: Parsing program options using `getopt'.
* Argp:: Parsing program options using `argp_parse'.
* Suboptions:: Some programs need more detailed options.
* Suboptions Example:: This shows how it could be done for `mount'.
Environment Variables
* Environment Access:: How to get and set the values of
environment variables.
* Standard Environment:: These environment variables have
standard interpretations.
Program Termination
* Normal Termination:: If a program calls `exit', a
process terminates normally.
* Exit Status:: The `exit status' provides information
about why the process terminated.
* Cleanups on Exit:: A process can run its own cleanup
functions upon normal termination.
* Aborting a Program:: The `abort' function causes
abnormal program termination.
* Termination Internals:: What happens when a process terminates.
Processes
* Running a Command:: The easy way to run another program.
* Process Creation Concepts:: An overview of the hard way to do it.
* Process Identification:: How to get the process ID of a process.
* Creating a Process:: How to fork a child process.
* Executing a File:: How to make a process execute another program.
* Process Completion:: How to tell when a child process has completed.
* Process Completion Status:: How to interpret the status value
returned from a child process.
* BSD Wait Functions:: More functions, for backward compatibility.
* Process Creation Example:: A complete example program.
Job Control
* Concepts of Job Control:: Jobs can be controlled by a shell.
* Job Control is Optional:: Not all POSIX systems support job control.
* Controlling Terminal:: How a process gets its controlling terminal.
* Access to the Terminal:: How processes share the controlling terminal.
* Orphaned Process Groups:: Jobs left after the user logs out.
* Implementing a Shell:: What a shell must do to implement job control.
* Functions for Job Control:: Functions to control process groups.
Implementing a Shell
* Data Structures:: Introduction to the sample shell.
* Initializing the Shell:: What the shell must do to take
responsibility for job control.
* Launching Jobs:: Creating jobs to execute commands.
* Foreground and Background:: Putting a job in foreground of background.
* Stopped and Terminated Jobs:: Reporting job status.
* Continuing Stopped Jobs:: How to continue a stopped job in
the foreground or background.
* Missing Pieces:: Other parts of the shell.
Functions for Job Control
* Identifying the Terminal:: Determining the controlling terminal's name.
* Process Group Functions:: Functions for manipulating process groups.
* Terminal Access Functions:: Functions for controlling terminal access.
Name Service Switch
* NSS Basics:: What is this NSS good for.
* NSS Configuration File:: Configuring NSS.
* NSS Module Internals:: How does it work internally.
* Extending NSS:: What to do to add services or databases.
NSS Configuration File
* Services in the NSS configuration:: Service names in the NSS configuration.
* Actions in the NSS configuration:: React appropriately to the lookup result.
* Notes on NSS Configuration File:: Things to take care about while
configuring NSS.
NSS Module Internals
* NSS Module Names:: Construction of the interface function of
the NSS modules.
* NSS Modules Interface:: Programming interface in the NSS module
functions.
Extending NSS
* Adding another Service to NSS:: What is to do to add a new service.
* NSS Module Function Internals:: Guidelines for writing new NSS
service functions.
Users and Groups
* User and Group IDs:: Each user has a unique numeric ID;
likewise for groups.
* Process Persona:: The user IDs and group IDs of a process.
* Why Change Persona:: Why a program might need to change
its user and/or group IDs.
* How Change Persona:: Changing the user and group IDs.
* Reading Persona:: How to examine the user and group IDs.
* Setting User ID:: Functions for setting the user ID.
* Setting Groups:: Functions for setting the group IDs.
* Enable/Disable Setuid:: Turning setuid access on and off.
* Setuid Program Example:: The pertinent parts of one sample program.
* Tips for Setuid:: How to avoid granting unlimited access.
* Who Logged In:: Getting the name of the user who logged in,
or of the real user ID of the current process.
* User Accounting Database:: Keeping information about users and various
actions in databases.
* User Database:: Functions and data structures for
accessing the user database.
* Group Database:: Functions and data structures for
accessing the group database.
* Database Example:: Example program showing the use of database
inquiry functions.
* Netgroup Database:: Functions for accessing the netgroup database.
User Accounting Database
* Manipulating the Database:: Scanning and modifying the user
accounting database.
* XPG Functions:: A standardized way for doing the same thing.
* Logging In and Out:: Functions from BSD that modify the user
accounting database.
User Database
* User Data Structure:: What each user record contains.
* Lookup User:: How to look for a particular user.
* Scanning All Users:: Scanning the list of all users, one by one.
* Writing a User Entry:: How a program can rewrite a user's record.
Group Database
* Group Data Structure:: What each group record contains.
* Lookup Group:: How to look for a particular group.
* Scanning All Groups:: Scanning the list of all groups.
Netgroup Database
* Netgroup Data:: Data in the Netgroup database and where
it comes from.
* Lookup Netgroup:: How to look for a particular netgroup.
* Netgroup Membership:: How to test for netgroup membership.
System Management
* Host Identification:: Determining the name of the machine.
* Platform Type:: Determining operating system and basic
machine type
* Filesystem Handling:: Controlling/querying mounts
* System Parameters:: Getting and setting various system parameters
Filesystem Handling
* Mount Information:: What is or could be mounted?
* Mount-Unmount-Remount:: Controlling what is mounted and how
Mount Information
* fstab:: The `fstab' file
* mtab:: The `mtab' file
* Other Mount Information:: Other (non-libc) sources of mount information
System Configuration
* General Limits:: Constants and functions that describe
various process-related limits that have
one uniform value for any given machine.
* System Options:: Optional POSIX features.
* Version Supported:: Version numbers of POSIX.1 and POSIX.2.
* Sysconf:: Getting specific configuration values
of general limits and system options.
* Minimums:: Minimum values for general limits.
* Limits for Files:: Size limitations that pertain to individual files.
These can vary between file systems
or even from file to file.
* Options for Files:: Optional features that some files may support.
* File Minimums:: Minimum values for file limits.
* Pathconf:: Getting the limit values for a particular file.
* Utility Limits:: Capacity limits of some POSIX.2 utility programs.
* Utility Minimums:: Minimum allowable values of those limits.
* String Parameters:: Getting the default search path.
Sysconf
* Sysconf Definition:: Detailed specifications of `sysconf'.
* Constants for Sysconf:: The list of parameters `sysconf' can read.
* Examples of Sysconf:: How to use `sysconf' and the parameter
macros properly together.
Cryptographic Functions
* Legal Problems:: This software can get you locked up, or worse.
* getpass:: Prompting the user for a password.
* crypt:: A one-way function for passwords.
* DES Encryption:: Routines for DES encryption.
Debugging Support
* Backtraces:: Obtaining and printing a back trace of the
current stack.
POSIX Threads
* Basic Thread Operations:: Creating, terminating, and waiting for threads.
* Thread Attributes:: Tuning thread scheduling.
* Cancellation:: Stopping a thread before it's done.
* Cleanup Handlers:: Deallocating resources when a thread is
canceled.
* Mutexes:: One way to synchronize threads.
* Condition Variables:: Another way.
* POSIX Semaphores:: And a third way.
* Thread-Specific Data:: Variables with different values in
different threads.
* Threads and Signal Handling:: Why you should avoid mixing the two, and
how to do it if you must.
* Threads and Fork:: Interactions between threads and the
`fork' function.
* Streams and Fork:: Interactions between stdio streams and
`fork'.
* Miscellaneous Thread Functions:: A grab bag of utility routines.
Language Features
* Consistency Checking:: Using `assert' to abort if
something ``impossible'' happens.
* Variadic Functions:: Defining functions with varying numbers
of args.
* Null Pointer Constant:: The macro `NULL'.
* Important Data Types:: Data types for object sizes.
* Data Type Measurements:: Parameters of data type representations.
Variadic Functions
* Why Variadic:: Reasons for making functions take
variable arguments.
* How Variadic:: How to define and call variadic functions.
* Variadic Example:: A complete example.
How Variadic
* Variadic Prototypes:: How to make a prototype for a function
with variable arguments.
* Receiving Arguments:: Steps you must follow to access the
optional argument values.
* How Many Arguments:: How to decide whether there are more arguments.
* Calling Variadics:: Things you need to know about calling
variable arguments functions.
* Argument Macros:: Detailed specification of the macros
for accessing variable arguments.
* Old Varargs:: The pre-ISO way of defining variadic functions.
Data Type Measurements
* Width of Type:: How many bits does an integer type hold?
* Range of Type:: What are the largest and smallest values
that an integer type can hold?
* Floating Type Macros:: Parameters that measure the floating point types.
* Structure Measurement:: Getting measurements on structure types.
Floating Type Macros
* Floating Point Concepts:: Definitions of terminology.
* Floating Point Parameters:: Details of specific macros.
* IEEE Floating Point:: The measurements for one common
representation.
Installation
* Configuring and compiling:: How to compile and test GNU libc.
* Running make install:: How to install it once you've got it
compiled.
* Tools for Compilation:: You'll need these first.
* Supported Configurations:: What it runs on, what it doesn't.
* Linux:: Specific advice for GNU/Linux systems.
* Reporting Bugs:: So they'll get fixed.
Maintenance
* Source Layout:: How to add new functions or header files
to the GNU C library.
* Porting:: How to port the GNU C library to
a new machine or operating system.
Porting
* Hierarchy Conventions:: The layout of the `sysdeps' hierarchy.
* Porting to Unix:: Porting the library to an average
Unix-like system.
File: libc.info, Node: Introduction, Next: Error Reporting, Prev: Top, Up: Top
1 Introduction
**************
The C language provides no built-in facilities for performing such
common operations as input/output, memory management, string
manipulation, and the like. Instead, these facilities are defined in a
standard "library", which you compile and link with your programs.
The GNU C library, described in this document, defines all of the
library functions that are specified by the ISO C standard, as well as
additional features specific to POSIX and other derivatives of the Unix
operating system, and extensions specific to the GNU system.
The purpose of this manual is to tell you how to use the facilities
of the GNU library. We have mentioned which features belong to which
standards to help you identify things that are potentially non-portable
to other systems. But the emphasis in this manual is not on strict
portability.
* Menu:
* Getting Started:: What this manual is for and how to use it.
* Standards and Portability:: Standards and sources upon which the GNU
C library is based.
* Using the Library:: Some practical uses for the library.
* Roadmap to the Manual:: Overview of the remaining chapters in
this manual.
File: libc.info, Node: Getting Started, Next: Standards and Portability, Up: Introduction
1.1 Getting Started
===================
This manual is written with the assumption that you are at least
somewhat familiar with the C programming language and basic programming
concepts. Specifically, familiarity with ISO standard C (*note ISO
C::), rather than "traditional" pre-ISO C dialects, is assumed.
The GNU C library includes several "header files", each of which
provides definitions and declarations for a group of related facilities;
this information is used by the C compiler when processing your program.
For example, the header file `stdio.h' declares facilities for
performing input and output, and the header file `string.h' declares
string processing utilities. The organization of this manual generally
follows the same division as the header files.
If you are reading this manual for the first time, you should read
all of the introductory material and skim the remaining chapters.
There are a _lot_ of functions in the GNU C library and it's not
realistic to expect that you will be able to remember exactly _how_ to
use each and every one of them. It's more important to become
generally familiar with the kinds of facilities that the library
provides, so that when you are writing your programs you can recognize
_when_ to make use of library functions, and _where_ in this manual you
can find more specific information about them.
File: libc.info, Node: Standards and Portability, Next: Using the Library, Prev: Getting Started, Up: Introduction
1.2 Standards and Portability
=============================
This section discusses the various standards and other sources that the
GNU C library is based upon. These sources include the ISO C and POSIX
standards, and the System V and Berkeley Unix implementations.
The primary focus of this manual is to tell you how to make effective
use of the GNU library facilities. But if you are concerned about
making your programs compatible with these standards, or portable to
operating systems other than GNU, this can affect how you use the
library. This section gives you an overview of these standards, so that
you will know what they are when they are mentioned in other parts of
the manual.
*Note Library Summary::, for an alphabetical list of the functions
and other symbols provided by the library. This list also states which
standards each function or symbol comes from.
* Menu:
* ISO C:: The international standard for the C
programming language.
* POSIX:: The ISO/IEC 9945 (aka IEEE 1003) standards
for operating systems.
* Berkeley Unix:: BSD and SunOS.
* SVID:: The System V Interface Description.
* XPG:: The X/Open Portability Guide.
File: libc.info, Node: ISO C, Next: POSIX, Up: Standards and Portability
1.2.1 ISO C
-----------
The GNU C library is compatible with the C standard adopted by the
American National Standards Institute (ANSI): `American National
Standard X3.159-1989--"ANSI C"' and later by the International
Standardization Organization (ISO): `ISO/IEC 9899:1990, "Programming
languages--C"'. We here refer to the standard as ISO C since this is
the more general standard in respect of ratification. The header files
and library facilities that make up the GNU library are a superset of
those specified by the ISO C standard.
If you are concerned about strict adherence to the ISO C standard,
you should use the `-ansi' option when you compile your programs with
the GNU C compiler. This tells the compiler to define _only_ ISO
standard features from the library header files, unless you explicitly
ask for additional features. *Note Feature Test Macros::, for
information on how to do this.
Being able to restrict the library to include only ISO C features is
important because ISO C puts limitations on what names can be defined
by the library implementation, and the GNU extensions don't fit these
limitations. *Note Reserved Names::, for more information about these
restrictions.
This manual does not attempt to give you complete details on the
differences between ISO C and older dialects. It gives advice on how
to write programs to work portably under multiple C dialects, but does
not aim for completeness.
File: libc.info, Node: POSIX, Next: Berkeley Unix, Prev: ISO C, Up: Standards and Portability
1.2.2 POSIX (The Portable Operating System Interface)
-----------------------------------------------------
The GNU library is also compatible with the ISO "POSIX" family of
standards, known more formally as the "Portable Operating System
Interface for Computer Environments" (ISO/IEC 9945). They were also
published as ANSI/IEEE Std 1003. POSIX is derived mostly from various
versions of the Unix operating system.
The library facilities specified by the POSIX standards are a
superset of those required by ISO C; POSIX specifies additional
features for ISO C functions, as well as specifying new additional
functions. In general, the additional requirements and functionality
defined by the POSIX standards are aimed at providing lower-level
support for a particular kind of operating system environment, rather
than general programming language support which can run in many diverse
operating system environments.
The GNU C library implements all of the functions specified in
`ISO/IEC 9945-1:1996, the POSIX System Application Program Interface',
commonly referred to as POSIX.1. The primary extensions to the ISO C
facilities specified by this standard include file system interface
primitives (*note File System Interface::), device-specific terminal
control functions (*note Low-Level Terminal Interface::), and process
control functions (*note Processes::).
Some facilities from `ISO/IEC 9945-2:1993, the POSIX Shell and
Utilities standard' (POSIX.2) are also implemented in the GNU library.
These include utilities for dealing with regular expressions and other
pattern matching facilities (*note Pattern Matching::).
File: libc.info, Node: Berkeley Unix, Next: SVID, Prev: POSIX, Up: Standards and Portability
1.2.3 Berkeley Unix
-------------------
The GNU C library defines facilities from some versions of Unix which
are not formally standardized, specifically from the 4.2 BSD, 4.3 BSD,
and 4.4 BSD Unix systems (also known as "Berkeley Unix") and from
"SunOS" (a popular 4.2 BSD derivative that includes some Unix System V
functionality). These systems support most of the ISO C and POSIX
facilities, and 4.4 BSD and newer releases of SunOS in fact support
them all.
The BSD facilities include symbolic links (*note Symbolic Links::),
the `select' function (*note Waiting for I/O::), the BSD signal
functions (*note BSD Signal Handling::), and sockets (*note Sockets::).
File: libc.info, Node: SVID, Next: XPG, Prev: Berkeley Unix, Up: Standards and Portability
1.2.4 SVID (The System V Interface Description)
-----------------------------------------------
The "System V Interface Description" (SVID) is a document describing
the AT&T Unix System V operating system. It is to some extent a
superset of the POSIX standard (*note POSIX::).
The GNU C library defines most of the facilities required by the SVID
that are not also required by the ISO C or POSIX standards, for
compatibility with System V Unix and other Unix systems (such as
SunOS) which include these facilities. However, many of the more
obscure and less generally useful facilities required by the SVID are
not included. (In fact, Unix System V itself does not provide them
all.)
The supported facilities from System V include the methods for
inter-process communication and shared memory, the `hsearch' and
`drand48' families of functions, `fmtmsg' and several of the
mathematical functions.
File: libc.info, Node: XPG, Prev: SVID, Up: Standards and Portability
1.2.5 XPG (The X/Open Portability Guide)
----------------------------------------
The X/Open Portability Guide, published by the X/Open Company, Ltd., is
a more general standard than POSIX. X/Open owns the Unix copyright and
the XPG specifies the requirements for systems which are intended to be
a Unix system.
The GNU C library complies to the X/Open Portability Guide, Issue
4.2, with all extensions common to XSI (X/Open System Interface)
compliant systems and also all X/Open UNIX extensions.
The additions on top of POSIX are mainly derived from functionality
available in System V and BSD systems. Some of the really bad mistakes
in System V systems were corrected, though. Since fulfilling the XPG
standard with the Unix extensions is a precondition for getting the
Unix brand chances are good that the functionality is available on
commercial systems.
File: libc.info, Node: Using the Library, Next: Roadmap to the Manual, Prev: Standards and Portability, Up: Introduction
1.3 Using the Library
=====================
This section describes some of the practical issues involved in using
the GNU C library.
* Menu:
* Header Files:: How to include the header files in your
programs.
* Macro Definitions:: Some functions in the library may really
be implemented as macros.
* Reserved Names:: The C standard reserves some names for
the library, and some for users.
* Feature Test Macros:: How to control what names are defined.
File: libc.info, Node: Header Files, Next: Macro Definitions, Up: Using the Library
1.3.1 Header Files
------------------
Libraries for use by C programs really consist of two parts: "header
files" that define types and macros and declare variables and
functions; and the actual library or "archive" that contains the
definitions of the variables and functions.
(Recall that in C, a "declaration" merely provides information that
a function or variable exists and gives its type. For a function
declaration, information about the types of its arguments might be
provided as well. The purpose of declarations is to allow the compiler
to correctly process references to the declared variables and functions.
A "definition", on the other hand, actually allocates storage for a
variable or says what a function does.)
In order to use the facilities in the GNU C library, you should be
sure that your program source files include the appropriate header
files. This is so that the compiler has declarations of these
facilities available and can correctly process references to them.
Once your program has been compiled, the linker resolves these
references to the actual definitions provided in the archive file.
Header files are included into a program source file by the
`#include' preprocessor directive. The C language supports two forms
of this directive; the first,
#include "HEADER"
is typically used to include a header file HEADER that you write
yourself; this would contain definitions and declarations describing the
interfaces between the different parts of your particular application.
By contrast,
#include <file.h>
is typically used to include a header file `file.h' that contains
definitions and declarations for a standard library. This file would
normally be installed in a standard place by your system administrator.
You should use this second form for the C library header files.
Typically, `#include' directives are placed at the top of the C
source file, before any other code. If you begin your source files with
some comments explaining what the code in the file does (a good idea),
put the `#include' directives immediately afterwards, following the
feature test macro definition (*note Feature Test Macros::).
For more information about the use of header files and `#include'
directives, *note Header Files: (cpp.info)Header Files.
The GNU C library provides several header files, each of which
contains the type and macro definitions and variable and function
declarations for a group of related facilities. This means that your
programs may need to include several header files, depending on exactly
which facilities you are using.
Some library header files include other library header files
automatically. However, as a matter of programming style, you should
not rely on this; it is better to explicitly include all the header
files required for the library facilities you are using. The GNU C
library header files have been written in such a way that it doesn't
matter if a header file is accidentally included more than once;
including a header file a second time has no effect. Likewise, if your
program needs to include multiple header files, the order in which they
are included doesn't matter.
*Compatibility Note:* Inclusion of standard header files in any
order and any number of times works in any ISO C implementation.
However, this has traditionally not been the case in many older C
implementations.
Strictly speaking, you don't _have to_ include a header file to use
a function it declares; you could declare the function explicitly
yourself, according to the specifications in this manual. But it is
usually better to include the header file because it may define types
and macros that are not otherwise available and because it may define
more efficient macro replacements for some functions. It is also a sure
way to have the correct declaration.
File: libc.info, Node: Macro Definitions, Next: Reserved Names, Prev: Header Files, Up: Using the Library
1.3.2 Macro Definitions of Functions
------------------------------------
If we describe something as a function in this manual, it may have a
macro definition as well. This normally has no effect on how your
program runs--the macro definition does the same thing as the function
would. In particular, macro equivalents for library functions evaluate
arguments exactly once, in the same way that a function call would. The
main reason for these macro definitions is that sometimes they can
produce an inline expansion that is considerably faster than an actual
function call.
Taking the address of a library function works even if it is also
defined as a macro. This is because, in this context, the name of the
function isn't followed by the left parenthesis that is syntactically
necessary to recognize a macro call.
You might occasionally want to avoid using the macro definition of a
function--perhaps to make your program easier to debug. There are two
ways you can do this:
* You can avoid a macro definition in a specific use by enclosing
the name of the function in parentheses. This works because the
name of the function doesn't appear in a syntactic context where
it is recognizable as a macro call.
* You can suppress any macro definition for a whole source file by
using the `#undef' preprocessor directive, unless otherwise stated
explicitly in the description of that facility.
For example, suppose the header file `stdlib.h' declares a function
named `abs' with
extern int abs (int);
and also provides a macro definition for `abs'. Then, in:
#include <stdlib.h>
int f (int *i) { return abs (++*i); }
the reference to `abs' might refer to either a macro or a function. On
the other hand, in each of the following examples the reference is to a
function and not a macro.
#include <stdlib.h>
int g (int *i) { return (abs) (++*i); }
#undef abs
int h (int *i) { return abs (++*i); }
Since macro definitions that double for a function behave in exactly
the same way as the actual function version, there is usually no need
for any of these methods. In fact, removing macro definitions usually
just makes your program slower.
File: libc.info, Node: Reserved Names, Next: Feature Test Macros, Prev: Macro Definitions, Up: Using the Library
1.3.3 Reserved Names
--------------------
The names of all library types, macros, variables and functions that
come from the ISO C standard are reserved unconditionally; your program
*may not* redefine these names. All other library names are reserved
if your program explicitly includes the header file that defines or
declares them. There are several reasons for these restrictions:
* Other people reading your code could get very confused if you were
using a function named `exit' to do something completely different
from what the standard `exit' function does, for example.
Preventing this situation helps to make your programs easier to
understand and contributes to modularity and maintainability.
* It avoids the possibility of a user accidentally redefining a
library function that is called by other library functions. If
redefinition were allowed, those other functions would not work
properly.
* It allows the compiler to do whatever special optimizations it
pleases on calls to these functions, without the possibility that
they may have been redefined by the user. Some library
facilities, such as those for dealing with variadic arguments
(*note Variadic Functions::) and non-local exits (*note Non-Local
Exits::), actually require a considerable amount of cooperation on
the part of the C compiler, and with respect to the
implementation, it might be easier for the compiler to treat these
as built-in parts of the language.
In addition to the names documented in this manual, reserved names
include all external identifiers (global functions and variables) that
begin with an underscore (`_') and all identifiers regardless of use
that begin with either two underscores or an underscore followed by a
capital letter are reserved names. This is so that the library and
header files can define functions, variables, and macros for internal
purposes without risk of conflict with names in user programs.
Some additional classes of identifier names are reserved for future
extensions to the C language or the POSIX.1 environment. While using
these names for your own purposes right now might not cause a problem,
they do raise the possibility of conflict with future versions of the C
or POSIX standards, so you should avoid these names.
* Names beginning with a capital `E' followed a digit or uppercase
letter may be used for additional error code names. *Note Error
Reporting::.
* Names that begin with either `is' or `to' followed by a lowercase
letter may be used for additional character testing and conversion
functions. *Note Character Handling::.
* Names that begin with `LC_' followed by an uppercase letter may be
used for additional macros specifying locale attributes. *Note
Locales::.
* Names of all existing mathematics functions (*note Mathematics::)
suffixed with `f' or `l' are reserved for corresponding functions
that operate on `float' and `long double' arguments, respectively.
* Names that begin with `SIG' followed by an uppercase letter are
reserved for additional signal names. *Note Standard Signals::.
* Names that begin with `SIG_' followed by an uppercase letter are
reserved for additional signal actions. *Note Basic Signal
Handling::.
* Names beginning with `str', `mem', or `wcs' followed by a
lowercase letter are reserved for additional string and array
functions. *Note String and Array Utilities::.
* Names that end with `_t' are reserved for additional type names.
In addition, some individual header files reserve names beyond those
that they actually define. You only need to worry about these
restrictions if your program includes that particular header file.
* The header file `dirent.h' reserves names prefixed with `d_'.
* The header file `fcntl.h' reserves names prefixed with `l_', `F_',
`O_', and `S_'.
* The header file `grp.h' reserves names prefixed with `gr_'.
* The header file `limits.h' reserves names suffixed with `_MAX'.
* The header file `pwd.h' reserves names prefixed with `pw_'.
* The header file `signal.h' reserves names prefixed with `sa_' and
`SA_'.
* The header file `sys/stat.h' reserves names prefixed with `st_'
and `S_'.
* The header file `sys/times.h' reserves names prefixed with `tms_'.
* The header file `termios.h' reserves names prefixed with `c_',
`V', `I', `O', and `TC'; and names prefixed with `B' followed by a
digit.
File: libc.info, Node: Feature Test Macros, Prev: Reserved Names, Up: Using the Library
1.3.4 Feature Test Macros
-------------------------
The exact set of features available when you compile a source file is
controlled by which "feature test macros" you define.
If you compile your programs using `gcc -ansi', you get only the
ISO C library features, unless you explicitly request additional
features by defining one or more of the feature macros. *Note GNU CC
Command Options: (gcc.info)Invoking GCC, for more information about GCC
options.
You should define these macros by using `#define' preprocessor
directives at the top of your source code files. These directives
_must_ come before any `#include' of a system header file. It is best
to make them the very first thing in the file, preceded only by
comments. You could also use the `-D' option to GCC, but it's better
if you make the source files indicate their own meaning in a
self-contained way.
This system exists to allow the library to conform to multiple
standards. Although the different standards are often described as
supersets of each other, they are usually incompatible because larger
standards require functions with names that smaller ones reserve to the
user program. This is not mere pedantry -- it has been a problem in
practice. For instance, some non-GNU programs define functions named
`getline' that have nothing to do with this library's `getline'. They
would not be compilable if all features were enabled indiscriminately.
This should not be used to verify that a program conforms to a
limited standard. It is insufficient for this purpose, as it will not
protect you from including header files outside the standard, or
relying on semantics undefined within the standard.
-- Macro: _POSIX_SOURCE
If you define this macro, then the functionality from the POSIX.1
standard (IEEE Standard 1003.1) is available, as well as all of the
ISO C facilities.
The state of `_POSIX_SOURCE' is irrelevant if you define the macro
`_POSIX_C_SOURCE' to a positive integer.
-- Macro: _POSIX_C_SOURCE
Define this macro to a positive integer to control which POSIX
functionality is made available. The greater the value of this
macro, the more functionality is made available.
If you define this macro to a value greater than or equal to `1',
then the functionality from the 1990 edition of the POSIX.1
standard (IEEE Standard 1003.1-1990) is made available.
If you define this macro to a value greater than or equal to `2',
then the functionality from the 1992 edition of the POSIX.2
standard (IEEE Standard 1003.2-1992) is made available.
If you define this macro to a value greater than or equal to
`199309L', then the functionality from the 1993 edition of the
POSIX.1b standard (IEEE Standard 1003.1b-1993) is made available.
Greater values for `_POSIX_C_SOURCE' will enable future extensions.
The POSIX standards process will define these values as necessary,
and the GNU C Library should support them some time after they
become standardized. The 1996 edition of POSIX.1 (ISO/IEC 9945-1:
1996) states that if you define `_POSIX_C_SOURCE' to a value
greater than or equal to `199506L', then the functionality from
the 1996 edition is made available.
-- Macro: _BSD_SOURCE
If you define this macro, functionality derived from 4.3 BSD Unix
is included as well as the ISO C, POSIX.1, and POSIX.2 material.
Some of the features derived from 4.3 BSD Unix conflict with the
corresponding features specified by the POSIX.1 standard. If this
macro is defined, the 4.3 BSD definitions take precedence over the
POSIX definitions.
Due to the nature of some of the conflicts between 4.3 BSD and
POSIX.1, you need to use a special "BSD compatibility library"
when linking programs compiled for BSD compatibility. This is
because some functions must be defined in two different ways, one
of them in the normal C library, and one of them in the
compatibility library. If your program defines `_BSD_SOURCE', you
must give the option `-lbsd-compat' to the compiler or linker when
linking the program, to tell it to find functions in this special
compatibility library before looking for them in the normal C
library.
-- Macro: _SVID_SOURCE
If you define this macro, functionality derived from SVID is
included as well as the ISO C, POSIX.1, POSIX.2, and X/Open
material.
-- Macro: _XOPEN_SOURCE
-- Macro: _XOPEN_SOURCE_EXTENDED
If you define this macro, functionality described in the X/Open
Portability Guide is included. This is a superset of the POSIX.1
and POSIX.2 functionality and in fact `_POSIX_SOURCE' and
`_POSIX_C_SOURCE' are automatically defined.
As the unification of all Unices, functionality only available in
BSD and SVID is also included.
If the macro `_XOPEN_SOURCE_EXTENDED' is also defined, even more
functionality is available. The extra functions will make all
functions available which are necessary for the X/Open Unix brand.
If the macro `_XOPEN_SOURCE' has the value 500 this includes all
functionality described so far plus some new definitions from the
Single Unix Specification, version 2.
-- Macro: _LARGEFILE_SOURCE
If this macro is defined some extra functions are available which
rectify a few shortcomings in all previous standards.
Specifically, the functions `fseeko' and `ftello' are available.
Without these functions the difference between the ISO C interface
(`fseek', `ftell') and the low-level POSIX interface (`lseek')
would lead to problems.
This macro was introduced as part of the Large File Support
extension (LFS).
-- Macro: _LARGEFILE64_SOURCE
If you define this macro an additional set of functions is made
available which enables 32 bit systems to use files of sizes beyond
the usual limit of 2GB. This interface is not available if the
system does not support files that large. On systems where the
natural file size limit is greater than 2GB (i.e., on 64 bit
systems) the new functions are identical to the replaced functions.
The new functionality is made available by a new set of types and
functions which replace the existing ones. The names of these new
objects contain `64' to indicate the intention, e.g., `off_t' vs.
`off64_t' and `fseeko' vs. `fseeko64'.
This macro was introduced as part of the Large File Support
extension (LFS). It is a transition interface for the period when
64 bit offsets are not generally used (see `_FILE_OFFSET_BITS').
-- Macro: _FILE_OFFSET_BITS
This macro determines which file system interface shall be used,
one replacing the other. Whereas `_LARGEFILE64_SOURCE' makes the
64 bit interface available as an additional interface,
`_FILE_OFFSET_BITS' allows the 64 bit interface to replace the old
interface.
If `_FILE_OFFSET_BITS' is undefined, or if it is defined to the
value `32', nothing changes. The 32 bit interface is used and
types like `off_t' have a size of 32 bits on 32 bit systems.
If the macro is defined to the value `64', the large file interface
replaces the old interface. I.e., the functions are not made
available under different names (as they are with
`_LARGEFILE64_SOURCE'). Instead the old function names now
reference the new functions, e.g., a call to `fseeko' now indeed
calls `fseeko64'.
This macro should only be selected if the system provides
mechanisms for handling large files. On 64 bit systems this macro
has no effect since the `*64' functions are identical to the
normal functions.
This macro was introduced as part of the Large File Support
extension (LFS).
-- Macro: _ISOC99_SOURCE
Until the revised ISO C standard is widely adopted the new features
are not automatically enabled. The GNU libc nevertheless has a
complete implementation of the new standard and to enable the new
features the macro `_ISOC99_SOURCE' should be defined.
-- Macro: _GNU_SOURCE
If you define this macro, everything is included: ISO C89,
ISO C99, POSIX.1, POSIX.2, BSD, SVID, X/Open, LFS, and GNU
extensions. In the cases where POSIX.1 conflicts with BSD, the
POSIX definitions take precedence.
If you want to get the full effect of `_GNU_SOURCE' but make the
BSD definitions take precedence over the POSIX definitions, use
this sequence of definitions:
#define _GNU_SOURCE
#define _BSD_SOURCE
#define _SVID_SOURCE
Note that if you do this, you must link your program with the BSD
compatibility library by passing the `-lbsd-compat' option to the
compiler or linker. *Note:* If you forget to do this, you may get
very strange errors at run time.
-- Macro: _REENTRANT
-- Macro: _THREAD_SAFE
If you define one of these macros, reentrant versions of several
functions get declared. Some of the functions are specified in
POSIX.1c but many others are only available on a few other systems
or are unique to GNU libc. The problem is the delay in the
standardization of the thread safe C library interface.
Unlike on some other systems, no special version of the C library
must be used for linking. There is only one version but while
compiling this it must have been specified to compile as thread
safe.
We recommend you use `_GNU_SOURCE' in new programs. If you don't
specify the `-ansi' option to GCC and don't define any of these macros
explicitly, the effect is the same as defining `_POSIX_C_SOURCE' to 2
and `_POSIX_SOURCE', `_SVID_SOURCE', and `_BSD_SOURCE' to 1.
When you define a feature test macro to request a larger class of
features, it is harmless to define in addition a feature test macro for
a subset of those features. For example, if you define
`_POSIX_C_SOURCE', then defining `_POSIX_SOURCE' as well has no effect.
Likewise, if you define `_GNU_SOURCE', then defining either
`_POSIX_SOURCE' or `_POSIX_C_SOURCE' or `_SVID_SOURCE' as well has no
effect.
Note, however, that the features of `_BSD_SOURCE' are not a subset of
any of the other feature test macros supported. This is because it
defines BSD features that take precedence over the POSIX features that
are requested by the other macros. For this reason, defining
`_BSD_SOURCE' in addition to the other feature test macros does have an
effect: it causes the BSD features to take priority over the conflicting
POSIX features.
File: libc.info, Node: Roadmap to the Manual, Prev: Using the Library, Up: Introduction
1.4 Roadmap to the Manual
=========================
Here is an overview of the contents of the remaining chapters of this
manual.
* *Note Error Reporting::, describes how errors detected by the
library are reported.
* *Note Language Features::, contains information about library
support for standard parts of the C language, including things
like the `sizeof' operator and the symbolic constant `NULL', how
to write functions accepting variable numbers of arguments, and
constants describing the ranges and other properties of the
numerical types. There is also a simple debugging mechanism which
allows you to put assertions in your code, and have diagnostic
messages printed if the tests fail.
* *Note Memory::, describes the GNU library's facilities for
managing and using virtual and real memory, including dynamic
allocation of virtual memory. If you do not know in advance how
much memory your program needs, you can allocate it dynamically
instead, and manipulate it via pointers.
* *Note Character Handling::, contains information about character
classification functions (such as `isspace') and functions for
performing case conversion.
* *Note String and Array Utilities::, has descriptions of functions
for manipulating strings (null-terminated character arrays) and
general byte arrays, including operations such as copying and
comparison.
* *Note I/O Overview::, gives an overall look at the input and output
facilities in the library, and contains information about basic
concepts such as file names.
* *Note I/O on Streams::, describes I/O operations involving streams
(or `FILE *' objects). These are the normal C library functions
from `stdio.h'.
* *Note Low-Level I/O::, contains information about I/O operations
on file descriptors. File descriptors are a lower-level mechanism
specific to the Unix family of operating systems.
* *Note File System Interface::, has descriptions of operations on
entire files, such as functions for deleting and renaming them and
for creating new directories. This chapter also contains
information about how you can access the attributes of a file,
such as its owner and file protection modes.
* *Note Pipes and FIFOs::, contains information about simple
interprocess communication mechanisms. Pipes allow communication
between two related processes (such as between a parent and
child), while FIFOs allow communication between processes sharing
a common file system on the same machine.
* *Note Sockets::, describes a more complicated interprocess
communication mechanism that allows processes running on different
machines to communicate over a network. This chapter also
contains information about Internet host addressing and how to use
the system network databases.
* *Note Low-Level Terminal Interface::, describes how you can change
the attributes of a terminal device. If you want to disable echo
of characters typed by the user, for example, read this chapter.
* *Note Mathematics::, contains information about the math library
functions. These include things like random-number generators and
remainder functions on integers as well as the usual trigonometric
and exponential functions on floating-point numbers.
* *Note Low-Level Arithmetic Functions: Arithmetic, describes
functions for simple arithmetic, analysis of floating-point
values, and reading numbers from strings.
* *Note Searching and Sorting::, contains information about functions
for searching and sorting arrays. You can use these functions on
any kind of array by providing an appropriate comparison function.
* *Note Pattern Matching::, presents functions for matching regular
expressions and shell file name patterns, and for expanding words
as the shell does.
* *Note Date and Time::, describes functions for measuring both
calendar time and CPU time, as well as functions for setting
alarms and timers.
* *Note Character Set Handling::, contains information about
manipulating characters and strings using character sets larger
than will fit in the usual `char' data type.
* *Note Locales::, describes how selecting a particular country or
language affects the behavior of the library. For example, the
locale affects collation sequences for strings and how monetary
values are formatted.
* *Note Non-Local Exits::, contains descriptions of the `setjmp' and
`longjmp' functions. These functions provide a facility for
`goto'-like jumps which can jump from one function to another.
* *Note Signal Handling::, tells you all about signals--what they
are, how to establish a handler that is called when a particular
kind of signal is delivered, and how to prevent signals from
arriving during critical sections of your program.
* *Note Program Basics::, tells how your programs can access their
command-line arguments and environment variables.
* *Note Processes::, contains information about how to start new
processes and run programs.
* *Note Job Control::, describes functions for manipulating process
groups and the controlling terminal. This material is probably
only of interest if you are writing a shell or other program which
handles job control specially.
* *Note Name Service Switch::, describes the services which are
available for looking up names in the system databases, how to
determine which service is used for which database, and how these
services are implemented so that contributors can design their own
services.
* *Note User Database::, and *Note Group Database::, tell you how to
access the system user and group databases.
* *Note System Management::, describes functions for controlling and
getting information about the hardware and software configuration
your program is executing under.
* *Note System Configuration::, tells you how you can get
information about various operating system limits. Most of these
parameters are provided for compatibility with POSIX.
* *Note Library Summary::, gives a summary of all the functions,
variables, and macros in the library, with complete data types and
function prototypes, and says what standard or system each is
derived from.
* *Note Maintenance::, explains how to build and install the GNU C
library on your system, how to report any bugs you might find, and
how to add new functions or port the library to a new system.
If you already know the name of the facility you are interested in,
you can look it up in *Note Library Summary::. This gives you a
summary of its syntax and a pointer to where you can find a more
detailed description. This appendix is particularly useful if you just
want to verify the order and type of arguments to a function, for
example. It also tells you what standard or system each function,
variable, or macro is derived from.
File: libc.info, Node: Error Reporting, Next: Memory, Prev: Introduction, Up: Top
2 Error Reporting
*****************
Many functions in the GNU C library detect and report error conditions,
and sometimes your programs need to check for these error conditions.
For example, when you open an input file, you should verify that the
file was actually opened correctly, and print an error message or take
other appropriate action if the call to the library function failed.
This chapter describes how the error reporting facility works. Your
program should include the header file `errno.h' to use this facility.
* Menu:
* Checking for Errors:: How errors are reported by library functions.
* Error Codes:: Error code macros; all of these expand
into integer constant values.
* Error Messages:: Mapping error codes onto error messages.
File: libc.info, Node: Checking for Errors, Next: Error Codes, Up: Error Reporting
2.1 Checking for Errors
=======================
Most library functions return a special value to indicate that they have
failed. The special value is typically `-1', a null pointer, or a
constant such as `EOF' that is defined for that purpose. But this
return value tells you only that an error has occurred. To find out
what kind of error it was, you need to look at the error code stored in
the variable `errno'. This variable is declared in the header file
`errno.h'.
-- Variable: volatile int errno
The variable `errno' contains the system error number. You can
change the value of `errno'.
Since `errno' is declared `volatile', it might be changed
asynchronously by a signal handler; see *Note Defining Handlers::.
However, a properly written signal handler saves and restores the
value of `errno', so you generally do not need to worry about this
possibility except when writing signal handlers.
The initial value of `errno' at program startup is zero. Many
library functions are guaranteed to set it to certain nonzero
values when they encounter certain kinds of errors. These error
conditions are listed for each function. These functions do not
change `errno' when they succeed; thus, the value of `errno' after
a successful call is not necessarily zero, and you should not use
`errno' to determine _whether_ a call failed. The proper way to
do that is documented for each function. _If_ the call failed,
you can examine `errno'.
Many library functions can set `errno' to a nonzero value as a
result of calling other library functions which might fail. You
should assume that any library function might alter `errno' when
the function returns an error.
*Portability Note:* ISO C specifies `errno' as a "modifiable
lvalue" rather than as a variable, permitting it to be implemented
as a macro. For example, its expansion might involve a function
call, like `*_errno ()'. In fact, that is what it is on the GNU
system itself. The GNU library, on non-GNU systems, does whatever
is right for the particular system.
There are a few library functions, like `sqrt' and `atan', that
return a perfectly legitimate value in case of an error, but also
set `errno'. For these functions, if you want to check to see
whether an error occurred, the recommended method is to set `errno'
to zero before calling the function, and then check its value
afterward.
All the error codes have symbolic names; they are macros defined in
`errno.h'. The names start with `E' and an upper-case letter or digit;
you should consider names of this form to be reserved names. *Note
Reserved Names::.
The error code values are all positive integers and are all distinct,
with one exception: `EWOULDBLOCK' and `EAGAIN' are the same. Since the
values are distinct, you can use them as labels in a `switch'
statement; just don't use both `EWOULDBLOCK' and `EAGAIN'. Your
program should not make any other assumptions about the specific values
of these symbolic constants.
The value of `errno' doesn't necessarily have to correspond to any
of these macros, since some library functions might return other error
codes of their own for other situations. The only values that are
guaranteed to be meaningful for a particular library function are the
ones that this manual lists for that function.
On non-GNU systems, almost any system call can return `EFAULT' if it
is given an invalid pointer as an argument. Since this could only
happen as a result of a bug in your program, and since it will not
happen on the GNU system, we have saved space by not mentioning
`EFAULT' in the descriptions of individual functions.
In some Unix systems, many system calls can also return `EFAULT' if
given as an argument a pointer into the stack, and the kernel for some
obscure reason fails in its attempt to extend the stack. If this ever
happens, you should probably try using statically or dynamically
allocated memory instead of stack memory on that system.
File: libc.info, Node: Error Codes, Next: Error Messages, Prev: Checking for Errors, Up: Error Reporting
2.2 Error Codes
===============
The error code macros are defined in the header file `errno.h'. All of
them expand into integer constant values. Some of these error codes
can't occur on the GNU system, but they can occur using the GNU library
on other systems.
-- Macro: int EPERM
Operation not permitted; only the owner of the file (or other
resource) or processes with special privileges can perform the
operation.
-- Macro: int ENOENT
No such file or directory. This is a "file doesn't exist" error
for ordinary files that are referenced in contexts where they are
expected to already exist.
-- Macro: int ESRCH
No process matches the specified process ID.
-- Macro: int EINTR
Interrupted function call; an asynchronous signal occurred and
prevented completion of the call. When this happens, you should
try the call again.
You can choose to have functions resume after a signal that is
handled, rather than failing with `EINTR'; see *Note Interrupted
Primitives::.
-- Macro: int EIO
Input/output error; usually used for physical read or write errors.
-- Macro: int ENXIO
No such device or address. The system tried to use the device
represented by a file you specified, and it couldn't find the
device. This can mean that the device file was installed
incorrectly, or that the physical device is missing or not
correctly attached to the computer.
-- Macro: int E2BIG
Argument list too long; used when the arguments passed to a new
program being executed with one of the `exec' functions (*note
Executing a File::) occupy too much memory space. This condition
never arises in the GNU system.
-- Macro: int ENOEXEC
Invalid executable file format. This condition is detected by the
`exec' functions; see *Note Executing a File::.
-- Macro: int EBADF
Bad file descriptor; for example, I/O on a descriptor that has been
closed or reading from a descriptor open only for writing (or vice
versa).
-- Macro: int ECHILD
There are no child processes. This error happens on operations
that are supposed to manipulate child processes, when there aren't
any processes to manipulate.
-- Macro: int EDEADLK
Deadlock avoided; allocating a system resource would have resulted
in a deadlock situation. The system does not guarantee that it
will notice all such situations. This error means you got lucky
and the system noticed; it might just hang. *Note File Locks::,
for an example.
-- Macro: int ENOMEM
No memory available. The system cannot allocate more virtual
memory because its capacity is full.
-- Macro: int EACCES
Permission denied; the file permissions do not allow the attempted
operation.
-- Macro: int EFAULT
Bad address; an invalid pointer was detected. In the GNU system,
this error never happens; you get a signal instead.
-- Macro: int ENOTBLK
A file that isn't a block special file was given in a situation
that requires one. For example, trying to mount an ordinary file
as a file system in Unix gives this error.
-- Macro: int EBUSY
Resource busy; a system resource that can't be shared is already
in use. For example, if you try to delete a file that is the root
of a currently mounted filesystem, you get this error.
-- Macro: int EEXIST
File exists; an existing file was specified in a context where it
only makes sense to specify a new file.
-- Macro: int EXDEV
An attempt to make an improper link across file systems was
detected. This happens not only when you use `link' (*note Hard
Links::) but also when you rename a file with `rename' (*note
Renaming Files::).
-- Macro: int ENODEV
The wrong type of device was given to a function that expects a
particular sort of device.
-- Macro: int ENOTDIR
A file that isn't a directory was specified when a directory is
required.
-- Macro: int EISDIR
File is a directory; you cannot open a directory for writing, or
create or remove hard links to it.
-- Macro: int EINVAL
Invalid argument. This is used to indicate various kinds of
problems with passing the wrong argument to a library function.
-- Macro: int EMFILE
The current process has too many files open and can't open any
more. Duplicate descriptors do count toward this limit.
In BSD and GNU, the number of open files is controlled by a
resource limit that can usually be increased. If you get this
error, you might want to increase the `RLIMIT_NOFILE' limit or
make it unlimited; *note Limits on Resources::.
-- Macro: int ENFILE
There are too many distinct file openings in the entire system.
Note that any number of linked channels count as just one file
opening; see *Note Linked Channels::. This error never occurs in
the GNU system.
-- Macro: int ENOTTY
Inappropriate I/O control operation, such as trying to set terminal
modes on an ordinary file.
-- Macro: int ETXTBSY
An attempt to execute a file that is currently open for writing, or
write to a file that is currently being executed. Often using a
debugger to run a program is considered having it open for writing
and will cause this error. (The name stands for "text file
busy".) This is not an error in the GNU system; the text is
copied as necessary.
-- Macro: int EFBIG
File too big; the size of a file would be larger than allowed by
the system.
-- Macro: int ENOSPC
No space left on device; write operation on a file failed because
the disk is full.
-- Macro: int ESPIPE
Invalid seek operation (such as on a pipe).
-- Macro: int EROFS
An attempt was made to modify something on a read-only file system.
-- Macro: int EMLINK
Too many links; the link count of a single file would become too
large. `rename' can cause this error if the file being renamed
already has as many links as it can take (*note Renaming Files::).
-- Macro: int EPIPE
Broken pipe; there is no process reading from the other end of a
pipe. Every library function that returns this error code also
generates a `SIGPIPE' signal; this signal terminates the program
if not handled or blocked. Thus, your program will never actually
see `EPIPE' unless it has handled or blocked `SIGPIPE'.
-- Macro: int EDOM
Domain error; used by mathematical functions when an argument
value does not fall into the domain over which the function is
defined.
-- Macro: int ERANGE
Range error; used by mathematical functions when the result value
is not representable because of overflow or underflow.
-- Macro: int EAGAIN
Resource temporarily unavailable; the call might work if you try
again later. The macro `EWOULDBLOCK' is another name for `EAGAIN';
they are always the same in the GNU C library.
This error can happen in a few different situations:
* An operation that would block was attempted on an object that
has non-blocking mode selected. Trying the same operation
again will block until some external condition makes it
possible to read, write, or connect (whatever the operation).
You can use `select' to find out when the operation will be
possible; *note Waiting for I/O::.
*Portability Note:* In many older Unix systems, this condition
was indicated by `EWOULDBLOCK', which was a distinct error
code different from `EAGAIN'. To make your program portable,
you should check for both codes and treat them the same.
* A temporary resource shortage made an operation impossible.
`fork' can return this error. It indicates that the shortage
is expected to pass, so your program can try the call again
later and it may succeed. It is probably a good idea to
delay for a few seconds before trying it again, to allow time
for other processes to release scarce resources. Such
shortages are usually fairly serious and affect the whole
system, so usually an interactive program should report the
error to the user and return to its command loop.
-- Macro: int EWOULDBLOCK
In the GNU C library, this is another name for `EAGAIN' (above).
The values are always the same, on every operating system.
C libraries in many older Unix systems have `EWOULDBLOCK' as a
separate error code.
-- Macro: int EINPROGRESS
An operation that cannot complete immediately was initiated on an
object that has non-blocking mode selected. Some functions that
must always block (such as `connect'; *note Connecting::) never
return `EAGAIN'. Instead, they return `EINPROGRESS' to indicate
that the operation has begun and will take some time. Attempts to
manipulate the object before the call completes return `EALREADY'.
You can use the `select' function to find out when the pending
operation has completed; *note Waiting for I/O::.
-- Macro: int EALREADY
An operation is already in progress on an object that has
non-blocking mode selected.
-- Macro: int ENOTSOCK
A file that isn't a socket was specified when a socket is required.
-- Macro: int EMSGSIZE
The size of a message sent on a socket was larger than the
supported maximum size.
-- Macro: int EPROTOTYPE
The socket type does not support the requested communications
protocol.
-- Macro: int ENOPROTOOPT
You specified a socket option that doesn't make sense for the
particular protocol being used by the socket. *Note Socket
Options::.
-- Macro: int EPROTONOSUPPORT
The socket domain does not support the requested communications
protocol (perhaps because the requested protocol is completely
invalid). *Note Creating a Socket::.
-- Macro: int ESOCKTNOSUPPORT
The socket type is not supported.
-- Macro: int EOPNOTSUPP
The operation you requested is not supported. Some socket
functions don't make sense for all types of sockets, and others
may not be implemented for all communications protocols. In the
GNU system, this error can happen for many calls when the object
does not support the particular operation; it is a generic
indication that the server knows nothing to do for that call.
-- Macro: int EPFNOSUPPORT
The socket communications protocol family you requested is not
supported.
-- Macro: int EAFNOSUPPORT
The address family specified for a socket is not supported; it is
inconsistent with the protocol being used on the socket. *Note
Sockets::.
-- Macro: int EADDRINUSE
The requested socket address is already in use. *Note Socket
Addresses::.
-- Macro: int EADDRNOTAVAIL
The requested socket address is not available; for example, you
tried to give a socket a name that doesn't match the local host
name. *Note Socket Addresses::.
-- Macro: int ENETDOWN
A socket operation failed because the network was down.
-- Macro: int ENETUNREACH
A socket operation failed because the subnet containing the remote
host was unreachable.
-- Macro: int ENETRESET
A network connection was reset because the remote host crashed.
-- Macro: int ECONNABORTED
A network connection was aborted locally.
-- Macro: int ECONNRESET
A network connection was closed for reasons outside the control of
the local host, such as by the remote machine rebooting or an
unrecoverable protocol violation.
-- Macro: int ENOBUFS
The kernel's buffers for I/O operations are all in use. In GNU,
this error is always synonymous with `ENOMEM'; you may get one or
the other from network operations.
-- Macro: int EISCONN
You tried to connect a socket that is already connected. *Note
Connecting::.
-- Macro: int ENOTCONN
The socket is not connected to anything. You get this error when
you try to transmit data over a socket, without first specifying a
destination for the data. For a connectionless socket (for
datagram protocols, such as UDP), you get `EDESTADDRREQ' instead.
-- Macro: int EDESTADDRREQ
No default destination address was set for the socket. You get
this error when you try to transmit data over a connectionless
socket, without first specifying a destination for the data with
`connect'.
-- Macro: int ESHUTDOWN
The socket has already been shut down.
-- Macro: int ETOOMANYREFS
???
-- Macro: int ETIMEDOUT
A socket operation with a specified timeout received no response
during the timeout period.
-- Macro: int ECONNREFUSED
A remote host refused to allow the network connection (typically
because it is not running the requested service).
-- Macro: int ELOOP
Too many levels of symbolic links were encountered in looking up a
file name. This often indicates a cycle of symbolic links.
-- Macro: int ENAMETOOLONG
Filename too long (longer than `PATH_MAX'; *note Limits for
Files::) or host name too long (in `gethostname' or `sethostname';
*note Host Identification::).
-- Macro: int EHOSTDOWN
The remote host for a requested network connection is down.
-- Macro: int EHOSTUNREACH
The remote host for a requested network connection is not
reachable.
-- Macro: int ENOTEMPTY
Directory not empty, where an empty directory was expected.
Typically, this error occurs when you are trying to delete a
directory.
-- Macro: int EPROCLIM
This means that the per-user limit on new process would be
exceeded by an attempted `fork'. *Note Limits on Resources::, for
details on the `RLIMIT_NPROC' limit.
-- Macro: int EUSERS
The file quota system is confused because there are too many users.
-- Macro: int EDQUOT
The user's disk quota was exceeded.
-- Macro: int ESTALE
Stale NFS file handle. This indicates an internal confusion in
the NFS system which is due to file system rearrangements on the
server host. Repairing this condition usually requires unmounting
and remounting the NFS file system on the local host.
-- Macro: int EREMOTE
An attempt was made to NFS-mount a remote file system with a file
name that already specifies an NFS-mounted file. (This is an
error on some operating systems, but we expect it to work properly
on the GNU system, making this error code impossible.)
-- Macro: int EBADRPC
???
-- Macro: int ERPCMISMATCH
???
-- Macro: int EPROGUNAVAIL
???
-- Macro: int EPROGMISMATCH
???
-- Macro: int EPROCUNAVAIL
???
-- Macro: int ENOLCK
No locks available. This is used by the file locking facilities;
see *Note File Locks::. This error is never generated by the GNU
system, but it can result from an operation to an NFS server
running another operating system.
-- Macro: int EFTYPE
Inappropriate file type or format. The file was the wrong type
for the operation, or a data file had the wrong format.
On some systems `chmod' returns this error if you try to set the
sticky bit on a non-directory file; *note Setting Permissions::.
-- Macro: int EAUTH
???
-- Macro: int ENEEDAUTH
???
-- Macro: int ENOSYS
Function not implemented. This indicates that the function called
is not implemented at all, either in the C library itself or in the
operating system. When you get this error, you can be sure that
this particular function will always fail with `ENOSYS' unless you
install a new version of the C library or the operating system.
-- Macro: int ENOTSUP
Not supported. A function returns this error when certain
parameter values are valid, but the functionality they request is
not available. This can mean that the function does not implement
a particular command or option value or flag bit at all. For
functions that operate on some object given in a parameter, such
as a file descriptor or a port, it might instead mean that only
_that specific object_ (file descriptor, port, etc.) is unable to
support the other parameters given; different file descriptors
might support different ranges of parameter values.
If the entire function is not available at all in the
implementation, it returns `ENOSYS' instead.
-- Macro: int EILSEQ
While decoding a multibyte character the function came along an
invalid or an incomplete sequence of bytes or the given wide
character is invalid.
-- Macro: int EBACKGROUND
In the GNU system, servers supporting the `term' protocol return
this error for certain operations when the caller is not in the
foreground process group of the terminal. Users do not usually
see this error because functions such as `read' and `write'
translate it into a `SIGTTIN' or `SIGTTOU' signal. *Note Job
Control::, for information on process groups and these signals.
-- Macro: int EDIED
In the GNU system, opening a file returns this error when the file
is translated by a program and the translator program dies while
starting up, before it has connected to the file.
-- Macro: int ED
The experienced user will know what is wrong.
-- Macro: int EGREGIOUS
You did *what*?
-- Macro: int EIEIO
Go home and have a glass of warm, dairy-fresh milk.
-- Macro: int EGRATUITOUS
This error code has no purpose.
-- Macro: int EBADMSG
-- Macro: int EIDRM
-- Macro: int EMULTIHOP
-- Macro: int ENODATA
-- Macro: int ENOLINK
-- Macro: int ENOMSG
-- Macro: int ENOSR
-- Macro: int ENOSTR
-- Macro: int EOVERFLOW
-- Macro: int EPROTO
-- Macro: int ETIME
-- Macro: int ECANCELED
Operation canceled; an asynchronous operation was canceled before
it completed. *Note Asynchronous I/O::. When you call
`aio_cancel', the normal result is for the operations affected to
complete with this error; *note Cancel AIO Operations::.
_The following error codes are defined by the Linux/i386 kernel.
They are not yet documented._
-- Macro: int ERESTART
-- Macro: int ECHRNG
-- Macro: int EL2NSYNC
-- Macro: int EL3HLT
-- Macro: int EL3RST
-- Macro: int ELNRNG
-- Macro: int EUNATCH
-- Macro: int ENOCSI
-- Macro: int EL2HLT
-- Macro: int EBADE
-- Macro: int EBADR
-- Macro: int EXFULL
-- Macro: int ENOANO
-- Macro: int EBADRQC
-- Macro: int EBADSLT
-- Macro: int EDEADLOCK
-- Macro: int EBFONT
-- Macro: int ENONET
-- Macro: int ENOPKG
-- Macro: int EADV
-- Macro: int ESRMNT
-- Macro: int ECOMM
-- Macro: int EDOTDOT
-- Macro: int ENOTUNIQ
-- Macro: int EBADFD
-- Macro: int EREMCHG
-- Macro: int ELIBACC
-- Macro: int ELIBBAD
-- Macro: int ELIBSCN
-- Macro: int ELIBMAX
-- Macro: int ELIBEXEC
-- Macro: int ESTRPIPE
-- Macro: int EUCLEAN
-- Macro: int ENOTNAM
-- Macro: int ENAVAIL
-- Macro: int EISNAM
-- Macro: int EREMOTEIO
-- Macro: int ENOMEDIUM
-- Macro: int EMEDIUMTYPE
File: libc.info, Node: Error Messages, Prev: Error Codes, Up: Error Reporting
2.3 Error Messages
==================
The library has functions and variables designed to make it easy for
your program to report informative error messages in the customary
format about the failure of a library call. The functions `strerror'
and `perror' give you the standard error message for a given error
code; the variable `program_invocation_short_name' gives you convenient
access to the name of the program that encountered the error.
-- Function: char * strerror (int ERRNUM)
The `strerror' function maps the error code (*note Checking for
Errors::) specified by the ERRNUM argument to a descriptive error
message string. The return value is a pointer to this string.
The value ERRNUM normally comes from the variable `errno'.
You should not modify the string returned by `strerror'. Also, if
you make subsequent calls to `strerror', the string might be
overwritten. (But it's guaranteed that no library function ever
calls `strerror' behind your back.)
The function `strerror' is declared in `string.h'.
-- Function: char * strerror_r (int ERRNUM, char *BUF, size_t N)
The `strerror_r' function works like `strerror' but instead of
returning the error message in a statically allocated buffer
shared by all threads in the process, it returns a private copy
for the thread. This might be either some permanent global data or
a message string in the user supplied buffer starting at BUF with
the length of N bytes.
At most N characters are written (including the NUL byte) so it is
up to the user to select the buffer large enough.
This function should always be used in multi-threaded programs
since there is no way to guarantee the string returned by
`strerror' really belongs to the last call of the current thread.
This function `strerror_r' is a GNU extension and it is declared in
`string.h'.
-- Function: void perror (const char *MESSAGE)
This function prints an error message to the stream `stderr'; see
*Note Standard Streams::. The orientation of `stderr' is not
changed.
If you call `perror' with a MESSAGE that is either a null pointer
or an empty string, `perror' just prints the error message
corresponding to `errno', adding a trailing newline.
If you supply a non-null MESSAGE argument, then `perror' prefixes
its output with this string. It adds a colon and a space
character to separate the MESSAGE from the error string
corresponding to `errno'.
The function `perror' is declared in `stdio.h'.
`strerror' and `perror' produce the exact same message for any given
error code; the precise text varies from system to system. On the GNU
system, the messages are fairly short; there are no multi-line messages
or embedded newlines. Each error message begins with a capital letter
and does not include any terminating punctuation.
*Compatibility Note:* The `strerror' function was introduced in
ISO C89. Many older C systems do not support this function yet.
Many programs that don't read input from the terminal are designed to
exit if any system call fails. By convention, the error message from
such a program should start with the program's name, sans directories.
You can find that name in the variable `program_invocation_short_name';
the full file name is stored the variable `program_invocation_name'.
-- Variable: char * program_invocation_name
This variable's value is the name that was used to invoke the
program running in the current process. It is the same as
`argv[0]'. Note that this is not necessarily a useful file name;
often it contains no directory names. *Note Program Arguments::.
-- Variable: char * program_invocation_short_name
This variable's value is the name that was used to invoke the
program running in the current process, with directory names
removed. (That is to say, it is the same as
`program_invocation_name' minus everything up to the last slash,
if any.)
The library initialization code sets up both of these variables
before calling `main'.
*Portability Note:* These two variables are GNU extensions. If you
want your program to work with non-GNU libraries, you must save the
value of `argv[0]' in `main', and then strip off the directory names
yourself. We added these extensions to make it possible to write
self-contained error-reporting subroutines that require no explicit
cooperation from `main'.
Here is an example showing how to handle failure to open a file
correctly. The function `open_sesame' tries to open the named file for
reading and returns a stream if successful. The `fopen' library
function returns a null pointer if it couldn't open the file for some
reason. In that situation, `open_sesame' constructs an appropriate
error message using the `strerror' function, and terminates the
program. If we were going to make some other library calls before
passing the error code to `strerror', we'd have to save it in a local
variable instead, because those other library functions might overwrite
`errno' in the meantime.
#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
FILE *
open_sesame (char *name)
{
FILE *stream;
errno = 0;
stream = fopen (name, "r");
if (stream == NULL)
{
fprintf (stderr, "%s: Couldn't open file %s; %s\n",
program_invocation_short_name, name, strerror (errno));
exit (EXIT_FAILURE);
}
else
return stream;
}
Using `perror' has the advantage that the function is portable and
available on all systems implementing ISO C. But often the text
`perror' generates is not what is wanted and there is no way to extend
or change what `perror' does. The GNU coding standard, for instance,
requires error messages to be preceded by the program name and programs
which read some input files should should provide information about the
input file name and the line number in case an error is encountered
while reading the file. For these occasions there are two functions
available which are widely used throughout the GNU project. These
functions are declared in `error.h'.
-- Function: void error (int STATUS, int ERRNUM, const char *FORMAT,
...)
The `error' function can be used to report general problems during
program execution. The FORMAT argument is a format string just
like those given to the `printf' family of functions. The
arguments required for the format can follow the FORMAT parameter.
Just like `perror', `error' also can report an error code in
textual form. But unlike `perror' the error value is explicitly
passed to the function in the ERRNUM parameter. This elimintates
the problem mentioned above that the error reporting function must
be called immediately after the function causing the error since
otherwise `errno' might have a different value.
The `error' prints first the program name. If the application
defined a global variable `error_print_progname' and points it to a
function this function will be called to print the program name.
Otherwise the string from the global variable `program_name' is
used. The program name is followed by a colon and a space which
in turn is followed by the output produced by the format string.
If the ERRNUM parameter is non-zero the format string output is
followed by a colon and a space, followed by the error message for
the error code ERRNUM. In any case is the output terminated with
a newline.
The output is directed to the `stderr' stream. If the `stderr'
wasn't oriented before the call it will be narrow-oriented
afterwards.
The function will return unless the STATUS parameter has a
non-zero value. In this case the function will call `exit' with
the STATUS value for its parameter and therefore never return. If
`error' returns the global variable `error_message_count' is
incremented by one to keep track of the number of errors reported.
-- Function: void error_at_line (int STATUS, int ERRNUM, const char
*FNAME, unsigned int LINENO, const char *FORMAT, ...)
The `error_at_line' function is very similar to the `error'
function. The only difference are the additional parameters FNAME
and LINENO. The handling of the other parameters is identical to
that of `error' except that between the program name and the string
generated by the format string additional text is inserted.
Directly following the program name a colon, followed by the file
name pointer to by FNAME, another colon, and a value of LINENO is
printed.
This additional output of course is meant to be used to locate an
error in an input file (like a programming language source code
file etc).
If the global variable `error_one_per_line' is set to a non-zero
value `error_at_line' will avoid printing consecutive messages for
the same file anem line. Repetition which are not directly
following each other are not caught.
Just like `error' this function only returned if STATUS is zero.
Otherwise `exit' is called with the non-zero value. If `error'
returns the global variable `error_message_count' is incremented
by one to keep track of the number of errors reported.
As mentioned above the `error' and `error_at_line' functions can be
customized by defining a variable named `error_print_progname'.
-- Variable: void (* error_print_progname ) (void)
If the `error_print_progname' variable is defined to a non-zero
value the function pointed to is called by `error' or
`error_at_line'. It is expected to print the program name or do
something similarly useful.
The function is expected to be print to the `stderr' stream and
must be able to handle whatever orientation the stream has.
The variable is global and shared by all threads.
-- Variable: unsigned int error_message_count
The `error_message_count' variable is incremented whenever one of
the functions `error' or `error_at_line' returns. The variable is
global and shared by all threads.
-- Variable: int error_one_per_line
The `error_one_per_line' variable influences only `error_at_line'.
Normally the `error_at_line' function creates output for every
invocation. If `error_one_per_line' is set to a non-zero value
`error_at_line' keeps track of the last file name and line number
for which an error was reported and avoid directly following
messages for the same file and line. This variable is global and
shared by all threads.
A program which read some input file and reports errors in it could look
like this:
{
char *line = NULL;
size_t len = 0;
unsigned int lineno = 0;
error_message_count = 0;
while (! feof_unlocked (fp))
{
ssize_t n = getline (&line, &len, fp);
if (n <= 0)
/* End of file or error. */
break;
++lineno;
/* Process the line. */
...
if (Detect error in line)
error_at_line (0, errval, filename, lineno,
"some error text %s", some_variable);
}
if (error_message_count != 0)
error (EXIT_FAILURE, 0, "%u errors found", error_message_count);
}
`error' and `error_at_line' are clearly the functions of choice and
enable the programmer to write applications which follow the GNU coding
standard. The GNU libc additionally contains functions which are used
in BSD for the same purpose. These functions are declared in `err.h'.
It is generally advised to not use these functions. They are included
only for compatibility.
-- Function: void warn (const char *FORMAT, ...)
The `warn' function is roughly equivalent to a call like
error (0, errno, format, the parameters)
except that the global variables `error' respects and modifies are
not used.
-- Function: void vwarn (const char *FORMAT, va_list)
The `vwarn' function is just like `warn' except that the
parameters for the handling of the format string FORMAT are passed
in as an value of type `va_list'.
-- Function: void warnx (const char *FORMAT, ...)
The `warnx' function is roughly equivalent to a call like
error (0, 0, format, the parameters)
except that the global variables `error' respects and modifies are
not used. The difference to `warn' is that no error number string
is printed.
-- Function: void vwarnx (const char *FORMAT, va_list)
The `vwarnx' function is just like `warnx' except that the
parameters for the handling of the format string FORMAT are passed
in as an value of type `va_list'.
-- Function: void err (int STATUS, const char *FORMAT, ...)
The `err' function is roughly equivalent to a call like
error (status, errno, format, the parameters)
except that the global variables `error' respects and modifies are
not used and that the program is exited even if STATUS is zero.
-- Function: void verr (int STATUS, const char *FORMAT, va_list)
The `verr' function is just like `err' except that the parameters
for the handling of the format string FORMAT are passed in as an
value of type `va_list'.
-- Function: void errx (int STATUS, const char *FORMAT, ...)
The `errx' function is roughly equivalent to a call like
error (status, 0, format, the parameters)
except that the global variables `error' respects and modifies are
not used and that the program is exited even if STATUS is zero.
The difference to `err' is that no error number string is printed.
-- Function: void verrx (int STATUS, const char *FORMAT, va_list)
The `verrx' function is just like `errx' except that the
parameters for the handling of the format string FORMAT are passed
in as an value of type `va_list'.
File: libc.info, Node: Memory, Next: Character Handling, Prev: Error Reporting, Up: Top
3 Virtual Memory Allocation And Paging
**************************************
This chapter describes how processes manage and use memory in a system
that uses the GNU C library.
The GNU C Library has several functions for dynamically allocating
virtual memory in various ways. They vary in generality and in
efficiency. The library also provides functions for controlling paging
and allocation of real memory.
* Menu:
* Memory Concepts:: An introduction to concepts and terminology.
* Memory Allocation:: Allocating storage for your program data
* Locking Pages:: Preventing page faults
* Resizing the Data Segment:: `brk', `sbrk'
Memory mapped I/O is not discussed in this chapter. *Note
Memory-mapped I/O::.
File: libc.info, Node: Memory Concepts, Next: Memory Allocation, Up: Memory
3.1 Process Memory Concepts
===========================
One of the most basic resources a process has available to it is memory.
There are a lot of different ways systems organize memory, but in a
typical one, each process has one linear virtual address space, with
addresses running from zero to some huge maximum. It need not be
contiguous; i.e. not all of these addresses actually can be used to
store data.
The virtual memory is divided into pages (4 kilobytes is typical).
Backing each page of virtual memory is a page of real memory (called a
"frame") or some secondary storage, usually disk space. The disk space
might be swap space or just some ordinary disk file. Actually, a page
of all zeroes sometimes has nothing at all backing it - there's just a
flag saying it is all zeroes.
The same frame of real memory or backing store can back multiple
virtual pages belonging to multiple processes. This is normally the
case, for example, with virtual memory occupied by GNU C library code.
The same real memory frame containing the `printf' function backs a
virtual memory page in each of the existing processes that has a
`printf' call in its program.
In order for a program to access any part of a virtual page, the page
must at that moment be backed by ("connected to") a real frame. But
because there is usually a lot more virtual memory than real memory, the
pages must move back and forth between real memory and backing store
regularly, coming into real memory when a process needs to access them
and then retreating to backing store when not needed anymore. This
movement is called "paging".
When a program attempts to access a page which is not at that moment
backed by real memory, this is known as a "page fault". When a page
fault occurs, the kernel suspends the process, places the page into a
real page frame (this is called "paging in" or "faulting in"), then
resumes the process so that from the process' point of view, the page
was in real memory all along. In fact, to the process, all pages always
seem to be in real memory. Except for one thing: the elapsed execution
time of an instruction that would normally be a few nanoseconds is
suddenly much, much, longer (because the kernel normally has to do I/O
to complete the page-in). For programs sensitive to that, the functions
described in *Note Locking Pages:: can control it.
Within each virtual address space, a process has to keep track of
what is at which addresses, and that process is called memory
allocation. Allocation usually brings to mind meting out scarce
resources, but in the case of virtual memory, that's not a major goal,
because there is generally much more of it than anyone needs. Memory
allocation within a process is mainly just a matter of making sure that
the same byte of memory isn't used to store two different things.
Processes allocate memory in two major ways: by exec and
programmatically. Actually, forking is a third way, but it's not very
interesting. *Note Creating a Process::.
Exec is the operation of creating a virtual address space for a
process, loading its basic program into it, and executing the program.
It is done by the "exec" family of functions (e.g. `execl'). The
operation takes a program file (an executable), it allocates space to
load all the data in the executable, loads it, and transfers control to
it. That data is most notably the instructions of the program (the
"text"), but also literals and constants in the program and even some
variables: C variables with the static storage class (*note Memory
Allocation and C::).
Once that program begins to execute, it uses programmatic allocation
to gain additional memory. In a C program with the GNU C library, there
are two kinds of programmatic allocation: automatic and dynamic. *Note
Memory Allocation and C::.
Memory-mapped I/O is another form of dynamic virtual memory
allocation. Mapping memory to a file means declaring that the contents
of certain range of a process' addresses shall be identical to the
contents of a specified regular file. The system makes the virtual
memory initially contain the contents of the file, and if you modify
the memory, the system writes the same modification to the file. Note
that due to the magic of virtual memory and page faults, there is no
reason for the system to do I/O to read the file, or allocate real
memory for its contents, until the program accesses the virtual memory.
*Note Memory-mapped I/O::.
Just as it programmatically allocates memory, the program can
programmatically deallocate ("free") it. You can't free the memory
that was allocated by exec. When the program exits or execs, you might
say that all its memory gets freed, but since in both cases the address
space ceases to exist, the point is really moot. *Note Program
Termination::.
A process' virtual address space is divided into segments. A
segment is a contiguous range of virtual addresses. Three important
segments are:
* The "text segment" contains a program's instructions and literals
and static constants. It is allocated by exec and stays the same
size for the life of the virtual address space.
* The "data segment" is working storage for the program. It can be
preallocated and preloaded by exec and the process can extend or
shrink it by calling functions as described in *Note Resizing the
Data Segment::. Its lower end is fixed.
* The "stack segment" contains a program stack. It grows as the
stack grows, but doesn't shrink when the stack shrinks.
File: libc.info, Node: Memory Allocation, Next: Locking Pages, Prev: Memory Concepts, Up: Memory
3.2 Allocating Storage For Program Data
=======================================
This section covers how ordinary programs manage storage for their data,
including the famous `malloc' function and some fancier facilities
special the GNU C library and GNU Compiler.
* Menu:
* Memory Allocation and C:: How to get different kinds of allocation in C.
* Unconstrained Allocation:: The `malloc' facility allows fully general
dynamic allocation.
* Allocation Debugging:: Finding memory leaks and not freed memory.
* Obstacks:: Obstacks are less general than malloc
but more efficient and convenient.
* Variable Size Automatic:: Allocation of variable-sized blocks
of automatic storage that are freed when the
calling function returns.
File: libc.info, Node: Memory Allocation and C, Next: Unconstrained Allocation, Up: Memory Allocation
3.2.1 Memory Allocation in C Programs
-------------------------------------
The C language supports two kinds of memory allocation through the
variables in C programs:
* "Static allocation" is what happens when you declare a static or
global variable. Each static or global variable defines one block
of space, of a fixed size. The space is allocated once, when your
program is started (part of the exec operation), and is never
freed.
* "Automatic allocation" happens when you declare an automatic
variable, such as a function argument or a local variable. The
space for an automatic variable is allocated when the compound
statement containing the declaration is entered, and is freed when
that compound statement is exited.
In GNU C, the size of the automatic storage can be an expression
that varies. In other C implementations, it must be a constant.
A third important kind of memory allocation, "dynamic allocation",
is not supported by C variables but is available via GNU C library
functions.
3.2.1.1 Dynamic Memory Allocation
.................................
"Dynamic memory allocation" is a technique in which programs determine
as they are running where to store some information. You need dynamic
allocation when the amount of memory you need, or how long you continue
to need it, depends on factors that are not known before the program
runs.
For example, you may need a block to store a line read from an input
file; since there is no limit to how long a line can be, you must
allocate the memory dynamically and make it dynamically larger as you
read more of the line.
Or, you may need a block for each record or each definition in the
input data; since you can't know in advance how many there will be, you
must allocate a new block for each record or definition as you read it.
When you use dynamic allocation, the allocation of a block of memory
is an action that the program requests explicitly. You call a function
or macro when you want to allocate space, and specify the size with an
argument. If you want to free the space, you do so by calling another
function or macro. You can do these things whenever you want, as often
as you want.
Dynamic allocation is not supported by C variables; there is no
storage class "dynamic", and there can never be a C variable whose
value is stored in dynamically allocated space. The only way to get
dynamically allocated memory is via a system call (which is generally
via a GNU C library function call), and the only way to refer to
dynamically allocated space is through a pointer. Because it is less
convenient, and because the actual process of dynamic allocation
requires more computation time, programmers generally use dynamic
allocation only when neither static nor automatic allocation will serve.
For example, if you want to allocate dynamically some space to hold a
`struct foobar', you cannot declare a variable of type `struct foobar'
whose contents are the dynamically allocated space. But you can
declare a variable of pointer type `struct foobar *' and assign it the
address of the space. Then you can use the operators `*' and `->' on
this pointer variable to refer to the contents of the space:
{
struct foobar *ptr
= (struct foobar *) malloc (sizeof (struct foobar));
ptr->name = x;
ptr->next = current_foobar;
current_foobar = ptr;
}
File: libc.info, Node: Unconstrained Allocation, Next: Allocation Debugging, Prev: Memory Allocation and C, Up: Memory Allocation
3.2.2 Unconstrained Allocation
------------------------------
The most general dynamic allocation facility is `malloc'. It allows
you to allocate blocks of memory of any size at any time, make them
bigger or smaller at any time, and free the blocks individually at any
time (or never).
* Menu:
* Basic Allocation:: Simple use of `malloc'.
* Malloc Examples:: Examples of `malloc'. `xmalloc'.
* Freeing after Malloc:: Use `free' to free a block you
got with `malloc'.
* Changing Block Size:: Use `realloc' to make a block
bigger or smaller.
* Allocating Cleared Space:: Use `calloc' to allocate a
block and clear it.
* Efficiency and Malloc:: Efficiency considerations in use of
these functions.
* Aligned Memory Blocks:: Allocating specially aligned memory.
* Malloc Tunable Parameters:: Use `mallopt' to adjust allocation
parameters.
* Heap Consistency Checking:: Automatic checking for errors.
* Hooks for Malloc:: You can use these hooks for debugging
programs that use `malloc'.
* Statistics of Malloc:: Getting information about how much
memory your program is using.
* Summary of Malloc:: Summary of `malloc' and related functions.
File: libc.info, Node: Basic Allocation, Next: Malloc Examples, Up: Unconstrained Allocation
3.2.2.1 Basic Memory Allocation
...............................
To allocate a block of memory, call `malloc'. The prototype for this
function is in `stdlib.h'.
-- Function: void * malloc (size_t SIZE)
This function returns a pointer to a newly allocated block SIZE
bytes long, or a null pointer if the block could not be allocated.
The contents of the block are undefined; you must initialize it
yourself (or use `calloc' instead; *note Allocating Cleared Space::).
Normally you would cast the value as a pointer to the kind of object
that you want to store in the block. Here we show an example of doing
so, and of initializing the space with zeros using the library function
`memset' (*note Copying and Concatenation::):
struct foo *ptr;
...
ptr = (struct foo *) malloc (sizeof (struct foo));
if (ptr == 0) abort ();
memset (ptr, 0, sizeof (struct foo));
You can store the result of `malloc' into any pointer variable
without a cast, because ISO C automatically converts the type `void *'
to another type of pointer when necessary. But the cast is necessary
in contexts other than assignment operators or if you might want your
code to run in traditional C.
Remember that when allocating space for a string, the argument to
`malloc' must be one plus the length of the string. This is because a
string is terminated with a null character that doesn't count in the
"length" of the string but does need space. For example:
char *ptr;
...
ptr = (char *) malloc (length + 1);
*Note Representation of Strings::, for more information about this.
File: libc.info, Node: Malloc Examples, Next: Freeing after Malloc, Prev: Basic Allocation, Up: Unconstrained Allocation
3.2.2.2 Examples of `malloc'
............................
If no more space is available, `malloc' returns a null pointer. You
should check the value of _every_ call to `malloc'. It is useful to
write a subroutine that calls `malloc' and reports an error if the
value is a null pointer, returning only if the value is nonzero. This
function is conventionally called `xmalloc'. Here it is:
void *
xmalloc (size_t size)
{
register void *value = malloc (size);
if (value == 0)
fatal ("virtual memory exhausted");
return value;
}
Here is a real example of using `malloc' (by way of `xmalloc'). The
function `savestring' will copy a sequence of characters into a newly
allocated null-terminated string:
char *
savestring (const char *ptr, size_t len)
{
register char *value = (char *) xmalloc (len + 1);
value[len] = '\0';
return (char *) memcpy (value, ptr, len);
}
The block that `malloc' gives you is guaranteed to be aligned so
that it can hold any type of data. In the GNU system, the address is
always a multiple of eight on most systems, and a multiple of 16 on
64-bit systems. Only rarely is any higher boundary (such as a page
boundary) necessary; for those cases, use `memalign', `posix_memalign'
or `valloc' (*note Aligned Memory Blocks::).
Note that the memory located after the end of the block is likely to
be in use for something else; perhaps a block already allocated by
another call to `malloc'. If you attempt to treat the block as longer
than you asked for it to be, you are liable to destroy the data that
`malloc' uses to keep track of its blocks, or you may destroy the
contents of another block. If you have already allocated a block and
discover you want it to be bigger, use `realloc' (*note Changing Block
Size::).
File: libc.info, Node: Freeing after Malloc, Next: Changing Block Size, Prev: Malloc Examples, Up: Unconstrained Allocation
3.2.2.3 Freeing Memory Allocated with `malloc'
..............................................
When you no longer need a block that you got with `malloc', use the
function `free' to make the block available to be allocated again. The
prototype for this function is in `stdlib.h'.
-- Function: void free (void *PTR)
The `free' function deallocates the block of memory pointed at by
PTR.
-- Function: void cfree (void *PTR)
This function does the same thing as `free'. It's provided for
backward compatibility with SunOS; you should use `free' instead.
Freeing a block alters the contents of the block. *Do not expect to
find any data (such as a pointer to the next block in a chain of
blocks) in the block after freeing it.* Copy whatever you need out of
the block before freeing it! Here is an example of the proper way to
free all the blocks in a chain, and the strings that they point to:
struct chain
{
struct chain *next;
char *name;
}
void
free_chain (struct chain *chain)
{
while (chain != 0)
{
struct chain *next = chain->next;
free (chain->name);
free (chain);
chain = next;
}
}
Occasionally, `free' can actually return memory to the operating
system and make the process smaller. Usually, all it can do is allow a
later call to `malloc' to reuse the space. In the meantime, the space
remains in your program as part of a free-list used internally by
`malloc'.
There is no point in freeing blocks at the end of a program, because
all of the program's space is given back to the system when the process
terminates.
File: libc.info, Node: Changing Block Size, Next: Allocating Cleared Space, Prev: Freeing after Malloc, Up: Unconstrained Allocation
3.2.2.4 Changing the Size of a Block
....................................
Often you do not know for certain how big a block you will ultimately
need at the time you must begin to use the block. For example, the
block might be a buffer that you use to hold a line being read from a
file; no matter how long you make the buffer initially, you may
encounter a line that is longer.
You can make the block longer by calling `realloc'. This function
is declared in `stdlib.h'.
-- Function: void * realloc (void *PTR, size_t NEWSIZE)
The `realloc' function changes the size of the block whose address
is PTR to be NEWSIZE.
Since the space after the end of the block may be in use, `realloc'
may find it necessary to copy the block to a new address where
more free space is available. The value of `realloc' is the new
address of the block. If the block needs to be moved, `realloc'
copies the old contents.
If you pass a null pointer for PTR, `realloc' behaves just like
`malloc (NEWSIZE)'. This can be convenient, but beware that older
implementations (before ISO C) may not support this behavior, and
will probably crash when `realloc' is passed a null pointer.
Like `malloc', `realloc' may return a null pointer if no memory
space is available to make the block bigger. When this happens, the
original block is untouched; it has not been modified or relocated.
In most cases it makes no difference what happens to the original
block when `realloc' fails, because the application program cannot
continue when it is out of memory, and the only thing to do is to give
a fatal error message. Often it is convenient to write and use a
subroutine, conventionally called `xrealloc', that takes care of the
error message as `xmalloc' does for `malloc':
void *
xrealloc (void *ptr, size_t size)
{
register void *value = realloc (ptr, size);
if (value == 0)
fatal ("Virtual memory exhausted");
return value;
}
You can also use `realloc' to make a block smaller. The reason you
would do this is to avoid tying up a lot of memory space when only a
little is needed. In several allocation implementations, making a
block smaller sometimes necessitates copying it, so it can fail if no
other space is available.
If the new size you specify is the same as the old size, `realloc'
is guaranteed to change nothing and return the same address that you
gave.
File: libc.info, Node: Allocating Cleared Space, Next: Efficiency and Malloc, Prev: Changing Block Size, Up: Unconstrained Allocation
3.2.2.5 Allocating Cleared Space
................................
The function `calloc' allocates memory and clears it to zero. It is
declared in `stdlib.h'.
-- Function: void * calloc (size_t COUNT, size_t ELTSIZE)
This function allocates a block long enough to contain a vector of
COUNT elements, each of size ELTSIZE. Its contents are cleared to
zero before `calloc' returns.
You could define `calloc' as follows:
void *
calloc (size_t count, size_t eltsize)
{
size_t size = count * eltsize;
void *value = malloc (size);
if (value != 0)
memset (value, 0, size);
return value;
}
But in general, it is not guaranteed that `calloc' calls `malloc'
internally. Therefore, if an application provides its own
`malloc'/`realloc'/`free' outside the C library, it should always
define `calloc', too.
File: libc.info, Node: Efficiency and Malloc, Next: Aligned Memory Blocks, Prev: Allocating Cleared Space, Up: Unconstrained Allocation
3.2.2.6 Efficiency Considerations for `malloc'
..............................................
As opposed to other versions, the `malloc' in the GNU C Library does
not round up block sizes to powers of two, neither for large nor for
small sizes. Neighboring chunks can be coalesced on a `free' no matter
what their size is. This makes the implementation suitable for all
kinds of allocation patterns without generally incurring high memory
waste through fragmentation.
Very large blocks (much larger than a page) are allocated with
`mmap' (anonymous or via `/dev/zero') by this implementation. This has
the great advantage that these chunks are returned to the system
immediately when they are freed. Therefore, it cannot happen that a
large chunk becomes "locked" in between smaller ones and even after
calling `free' wastes memory. The size threshold for `mmap' to be used
can be adjusted with `mallopt'. The use of `mmap' can also be disabled
completely.
File: libc.info, Node: Aligned Memory Blocks, Next: Malloc Tunable Parameters, Prev: Efficiency and Malloc, Up: Unconstrained Allocation
3.2.2.7 Allocating Aligned Memory Blocks
........................................
The address of a block returned by `malloc' or `realloc' in the GNU
system is always a multiple of eight (or sixteen on 64-bit systems).
If you need a block whose address is a multiple of a higher power of
two than that, use `memalign', `posix_memalign', or `valloc'.
`memalign' is declared in `malloc.h' and `posix_memalign' is declared
in `stdlib.h'.
With the GNU library, you can use `free' to free the blocks that
`memalign', `posix_memalign', and `valloc' return. That does not work
in BSD, however--BSD does not provide any way to free such blocks.
-- Function: void * memalign (size_t BOUNDARY, size_t SIZE)
The `memalign' function allocates a block of SIZE bytes whose
address is a multiple of BOUNDARY. The BOUNDARY must be a power
of two! The function `memalign' works by allocating a somewhat
larger block, and then returning an address within the block that
is on the specified boundary.
-- Function: int posix_memalign (void **MEMPTR, size_t ALIGNMENT,
size_t SIZE)
The `posix_memalign' function is similar to the `memalign'
function in that it returns a buffer of SIZE bytes aligned to a
multiple of ALIGNMENT. But it adds one requirement to the
parameter ALIGNMENT: the value must be a power of two multiple of
`sizeof (void *)'.
If the function succeeds in allocation memory a pointer to the
allocated memory is returned in `*MEMPTR' and the return value is
zero. Otherwise the function returns an error value indicating
the problem.
This function was introduced in POSIX 1003.1d.
-- Function: void * valloc (size_t SIZE)
Using `valloc' is like using `memalign' and passing the page size
as the value of the second argument. It is implemented like this:
void *
valloc (size_t size)
{
return memalign (getpagesize (), size);
}
*Note Query Memory Parameters:: for more information about the
memory subsystem.
File: libc.info, Node: Malloc Tunable Parameters, Next: Heap Consistency Checking, Prev: Aligned Memory Blocks, Up: Unconstrained Allocation
3.2.2.8 Malloc Tunable Parameters
.................................
You can adjust some parameters for dynamic memory allocation with the
`mallopt' function. This function is the general SVID/XPG interface,
defined in `malloc.h'.
-- Function: int mallopt (int PARAM, int VALUE)
When calling `mallopt', the PARAM argument specifies the parameter
to be set, and VALUE the new value to be set. Possible choices
for PARAM, as defined in `malloc.h', are:
`M_TRIM_THRESHOLD'
This is the minimum size (in bytes) of the top-most,
releasable chunk that will cause `sbrk' to be called with a
negative argument in order to return memory to the system.
`M_TOP_PAD'
This parameter determines the amount of extra memory to
obtain from the system when a call to `sbrk' is required. It
also specifies the number of bytes to retain when shrinking
the heap by calling `sbrk' with a negative argument. This
provides the necessary hysteresis in heap size such that
excessive amounts of system calls can be avoided.
`M_MMAP_THRESHOLD'
All chunks larger than this value are allocated outside the
normal heap, using the `mmap' system call. This way it is
guaranteed that the memory for these chunks can be returned
to the system on `free'. Note that requests smaller than
this threshold might still be allocated via `mmap'.
`M_MMAP_MAX'
The maximum number of chunks to allocate with `mmap'.
Setting this to zero disables all use of `mmap'.
File: libc.info, Node: Heap Consistency Checking, Next: Hooks for Malloc, Prev: Malloc Tunable Parameters, Up: Unconstrained Allocation
3.2.2.9 Heap Consistency Checking
.................................
You can ask `malloc' to check the consistency of dynamic memory by
using the `mcheck' function. This function is a GNU extension,
declared in `mcheck.h'.
-- Function: int mcheck (void (*ABORTFN) (enum mcheck_status STATUS))
Calling `mcheck' tells `malloc' to perform occasional consistency
checks. These will catch things such as writing past the end of a
block that was allocated with `malloc'.
The ABORTFN argument is the function to call when an inconsistency
is found. If you supply a null pointer, then `mcheck' uses a
default function which prints a message and calls `abort' (*note
Aborting a Program::). The function you supply is called with one
argument, which says what sort of inconsistency was detected; its
type is described below.
It is too late to begin allocation checking once you have allocated
anything with `malloc'. So `mcheck' does nothing in that case.
The function returns `-1' if you call it too late, and `0'
otherwise (when it is successful).
The easiest way to arrange to call `mcheck' early enough is to use
the option `-lmcheck' when you link your program; then you don't
need to modify your program source at all. Alternatively you
might use a debugger to insert a call to `mcheck' whenever the
program is started, for example these gdb commands will
automatically call `mcheck' whenever the program starts:
(gdb) break main
Breakpoint 1, main (argc=2, argv=0xbffff964) at whatever.c:10
(gdb) command 1
Type commands for when breakpoint 1 is hit, one per line.
End with a line saying just "end".
>call mcheck(0)
>continue
>end
(gdb) ...
This will however only work if no initialization function of any
object involved calls any of the `malloc' functions since `mcheck'
must be called before the first such function.
-- Function: enum mcheck_status mprobe (void *POINTER)
The `mprobe' function lets you explicitly check for inconsistencies
in a particular allocated block. You must have already called
`mcheck' at the beginning of the program, to do its occasional
checks; calling `mprobe' requests an additional consistency check
to be done at the time of the call.
The argument POINTER must be a pointer returned by `malloc' or
`realloc'. `mprobe' returns a value that says what inconsistency,
if any, was found. The values are described below.
-- Data Type: enum mcheck_status
This enumerated type describes what kind of inconsistency was
detected in an allocated block, if any. Here are the possible
values:
`MCHECK_DISABLED'
`mcheck' was not called before the first allocation. No
consistency checking can be done.
`MCHECK_OK'
No inconsistency detected.
`MCHECK_HEAD'
The data immediately before the block was modified. This
commonly happens when an array index or pointer is
decremented too far.
`MCHECK_TAIL'
The data immediately after the block was modified. This
commonly happens when an array index or pointer is
incremented too far.
`MCHECK_FREE'
The block was already freed.
Another possibility to check for and guard against bugs in the use of
`malloc', `realloc' and `free' is to set the environment variable
`MALLOC_CHECK_'. When `MALLOC_CHECK_' is set, a special (less
efficient) implementation is used which is designed to be tolerant
against simple errors, such as double calls of `free' with the same
argument, or overruns of a single byte (off-by-one bugs). Not all such
errors can be protected against, however, and memory leaks can result.
If `MALLOC_CHECK_' is set to `0', any detected heap corruption is
silently ignored; if set to `1', a diagnostic is printed on `stderr';
if set to `2', `abort' is called immediately. This can be useful
because otherwise a crash may happen much later, and the true cause for
the problem is then very hard to track down.
There is one problem with `MALLOC_CHECK_': in SUID or SGID binaries
it could possibly be exploited since diverging from the normal programs
behavior it now writes something to the standard error descriptor.
Therefore the use of `MALLOC_CHECK_' is disabled by default for SUID
and SGID binaries. It can be enabled again by the system administrator
by adding a file `/etc/suid-debug' (the content is not important it
could be empty).
So, what's the difference between using `MALLOC_CHECK_' and linking
with `-lmcheck'? `MALLOC_CHECK_' is orthogonal with respect to
`-lmcheck'. `-lmcheck' has been added for backward compatibility.
Both `MALLOC_CHECK_' and `-lmcheck' should uncover the same bugs - but
using `MALLOC_CHECK_' you don't need to recompile your application.
File: libc.info, Node: Hooks for Malloc, Next: Statistics of Malloc, Prev: Heap Consistency Checking, Up: Unconstrained Allocation
3.2.2.10 Memory Allocation Hooks
................................
The GNU C library lets you modify the behavior of `malloc', `realloc',
and `free' by specifying appropriate hook functions. You can use these
hooks to help you debug programs that use dynamic memory allocation,
for example.
The hook variables are declared in `malloc.h'.
-- Variable: __malloc_hook
The value of this variable is a pointer to the function that
`malloc' uses whenever it is called. You should define this
function to look like `malloc'; that is, like:
void *FUNCTION (size_t SIZE, const void *CALLER)
The value of CALLER is the return address found on the stack when
the `malloc' function was called. This value allows you to trace
the memory consumption of the program.
-- Variable: __realloc_hook
The value of this variable is a pointer to function that `realloc'
uses whenever it is called. You should define this function to
look like `realloc'; that is, like:
void *FUNCTION (void *PTR, size_t SIZE, const void *CALLER)
The value of CALLER is the return address found on the stack when
the `realloc' function was called. This value allows you to trace
the memory consumption of the program.
-- Variable: __free_hook
The value of this variable is a pointer to function that `free'
uses whenever it is called. You should define this function to
look like `free'; that is, like:
void FUNCTION (void *PTR, const void *CALLER)
The value of CALLER is the return address found on the stack when
the `free' function was called. This value allows you to trace the
memory consumption of the program.
-- Variable: __memalign_hook
The value of this variable is a pointer to function that `memalign'
uses whenever it is called. You should define this function to
look like `memalign'; that is, like:
void *FUNCTION (size_t ALIGNMENT, size_t SIZE, const void *CALLER)
The value of CALLER is the return address found on the stack when
the `memalign' function was called. This value allows you to
trace the memory consumption of the program.
You must make sure that the function you install as a hook for one of
these functions does not call that function recursively without
restoring the old value of the hook first! Otherwise, your program
will get stuck in an infinite recursion. Before calling the function
recursively, one should make sure to restore all the hooks to their
previous value. When coming back from the recursive call, all the
hooks should be resaved since a hook might modify itself.
-- Variable: __malloc_initialize_hook
The value of this variable is a pointer to a function that is
called once when the malloc implementation is initialized. This
is a weak variable, so it can be overridden in the application
with a definition like the following:
void (*__MALLOC_INITIALIZE_HOOK) (void) = my_init_hook;
An issue to look out for is the time at which the malloc hook
functions can be safely installed. If the hook functions call the
malloc-related functions recursively, it is necessary that malloc has
already properly initialized itself at the time when `__malloc_hook'
etc. is assigned to. On the other hand, if the hook functions provide a
complete malloc implementation of their own, it is vital that the hooks
are assigned to _before_ the very first `malloc' call has completed,
because otherwise a chunk obtained from the ordinary, un-hooked malloc
may later be handed to `__free_hook', for example.
In both cases, the problem can be solved by setting up the hooks from
within a user-defined function pointed to by
`__malloc_initialize_hook'--then the hooks will be set up safely at the
right time.
Here is an example showing how to use `__malloc_hook' and
`__free_hook' properly. It installs a function that prints out
information every time `malloc' or `free' is called. We just assume
here that `realloc' and `memalign' are not used in our program.
/* Prototypes for __malloc_hook, __free_hook */
#include <malloc.h>
/* Prototypes for our hooks. */
static void *my_init_hook (void);
static void *my_malloc_hook (size_t, const void *);
static void my_free_hook (void*, const void *);
/* Override initializing hook from the C library. */
void (*__malloc_initialize_hook) (void) = my_init_hook;
static void
my_init_hook (void)
{
old_malloc_hook = __malloc_hook;
old_free_hook = __free_hook;
__malloc_hook = my_malloc_hook;
__free_hook = my_free_hook;
}
static void *
my_malloc_hook (size_t size, const void *caller)
{
void *result;
/* Restore all old hooks */
__malloc_hook = old_malloc_hook;
__free_hook = old_free_hook;
/* Call recursively */
result = malloc (size);
/* Save underlying hooks */
old_malloc_hook = __malloc_hook;
old_free_hook = __free_hook;
/* `printf' might call `malloc', so protect it too. */
printf ("malloc (%u) returns %p\n", (unsigned int) size, result);
/* Restore our own hooks */
__malloc_hook = my_malloc_hook;
__free_hook = my_free_hook;
return result;
}
static void *
my_free_hook (void *ptr, const void *caller)
{
/* Restore all old hooks */
__malloc_hook = old_malloc_hook;
__free_hook = old_free_hook;
/* Call recursively */
free (ptr);
/* Save underlying hooks */
old_malloc_hook = __malloc_hook;
old_free_hook = __free_hook;
/* `printf' might call `free', so protect it too. */
printf ("freed pointer %p\n", ptr);
/* Restore our own hooks */
__malloc_hook = my_malloc_hook;
__free_hook = my_free_hook;
}
main ()
{
...
}
The `mcheck' function (*note Heap Consistency Checking::) works by
installing such hooks.
File: libc.info, Node: Statistics of Malloc, Next: Summary of Malloc, Prev: Hooks for Malloc, Up: Unconstrained Allocation
3.2.2.11 Statistics for Memory Allocation with `malloc'
.......................................................
You can get information about dynamic memory allocation by calling the
`mallinfo' function. This function and its associated data type are
declared in `malloc.h'; they are an extension of the standard SVID/XPG
version.
-- Data Type: struct mallinfo
This structure type is used to return information about the dynamic
memory allocator. It contains the following members:
`int arena'
This is the total size of memory allocated with `sbrk' by
`malloc', in bytes.
`int ordblks'
This is the number of chunks not in use. (The memory
allocator internally gets chunks of memory from the operating
system, and then carves them up to satisfy individual
`malloc' requests; see *Note Efficiency and Malloc::.)
`int smblks'
This field is unused.
`int hblks'
This is the total number of chunks allocated with `mmap'.
`int hblkhd'
This is the total size of memory allocated with `mmap', in
bytes.
`int usmblks'
This field is unused.
`int fsmblks'
This field is unused.
`int uordblks'
This is the total size of memory occupied by chunks handed
out by `malloc'.
`int fordblks'
This is the total size of memory occupied by free (not in
use) chunks.
`int keepcost'
This is the size of the top-most releasable chunk that
normally borders the end of the heap (i.e. the high end of
the virtual address space's data segment).
-- Function: struct mallinfo mallinfo (void)
This function returns information about the current dynamic memory
usage in a structure of type `struct mallinfo'.
File: libc.info, Node: Summary of Malloc, Prev: Statistics of Malloc, Up: Unconstrained Allocation
3.2.2.12 Summary of `malloc'-Related Functions
..............................................
Here is a summary of the functions that work with `malloc':
`void *malloc (size_t SIZE)'
Allocate a block of SIZE bytes. *Note Basic Allocation::.
`void free (void *ADDR)'
Free a block previously allocated by `malloc'. *Note Freeing
after Malloc::.
`void *realloc (void *ADDR, size_t SIZE)'
Make a block previously allocated by `malloc' larger or smaller,
possibly by copying it to a new location. *Note Changing Block
Size::.
`void *calloc (size_t COUNT, size_t ELTSIZE)'
Allocate a block of COUNT * ELTSIZE bytes using `malloc', and set
its contents to zero. *Note Allocating Cleared Space::.
`void *valloc (size_t SIZE)'
Allocate a block of SIZE bytes, starting on a page boundary.
*Note Aligned Memory Blocks::.
`void *memalign (size_t SIZE, size_t BOUNDARY)'
Allocate a block of SIZE bytes, starting on an address that is a
multiple of BOUNDARY. *Note Aligned Memory Blocks::.
`int mallopt (int PARAM, int VALUE)'
Adjust a tunable parameter. *Note Malloc Tunable Parameters::.
`int mcheck (void (*ABORTFN) (void))'
Tell `malloc' to perform occasional consistency checks on
dynamically allocated memory, and to call ABORTFN when an
inconsistency is found. *Note Heap Consistency Checking::.
`void *(*__malloc_hook) (size_t SIZE, const void *CALLER)'
A pointer to a function that `malloc' uses whenever it is called.
`void *(*__realloc_hook) (void *PTR, size_t SIZE, const void *CALLER)'
A pointer to a function that `realloc' uses whenever it is called.
`void (*__free_hook) (void *PTR, const void *CALLER)'
A pointer to a function that `free' uses whenever it is called.
`void (*__memalign_hook) (size_t SIZE, size_t ALIGNMENT, const void *CALLER)'
A pointer to a function that `memalign' uses whenever it is called.
`struct mallinfo mallinfo (void)'
Return information about the current dynamic memory usage. *Note
Statistics of Malloc::.
File: libc.info, Node: Allocation Debugging, Next: Obstacks, Prev: Unconstrained Allocation, Up: Memory Allocation
3.2.3 Allocation Debugging
--------------------------
A complicated task when programming with languages which do not use
garbage collected dynamic memory allocation is to find memory leaks.
Long running programs must assure that dynamically allocated objects are
freed at the end of their lifetime. If this does not happen the system
runs out of memory, sooner or later.
The `malloc' implementation in the GNU C library provides some
simple means to detect such leaks and obtain some information to find
the location. To do this the application must be started in a special
mode which is enabled by an environment variable. There are no speed
penalties for the program if the debugging mode is not enabled.
* Menu:
* Tracing malloc:: How to install the tracing functionality.
* Using the Memory Debugger:: Example programs excerpts.
* Tips for the Memory Debugger:: Some more or less clever ideas.
* Interpreting the traces:: What do all these lines mean?
File: libc.info, Node: Tracing malloc, Next: Using the Memory Debugger, Up: Allocation Debugging
3.2.3.1 How to install the tracing functionality
................................................
-- Function: void mtrace (void)
When the `mtrace' function is called it looks for an environment
variable named `MALLOC_TRACE'. This variable is supposed to
contain a valid file name. The user must have write access. If
the file already exists it is truncated. If the environment
variable is not set or it does not name a valid file which can be
opened for writing nothing is done. The behavior of `malloc' etc.
is not changed. For obvious reasons this also happens if the
application is installed with the SUID or SGID bit set.
If the named file is successfully opened, `mtrace' installs special
handlers for the functions `malloc', `realloc', and `free' (*note
Hooks for Malloc::). From then on, all uses of these functions
are traced and protocolled into the file. There is now of course
a speed penalty for all calls to the traced functions so tracing
should not be enabled during normal use.
This function is a GNU extension and generally not available on
other systems. The prototype can be found in `mcheck.h'.
-- Function: void muntrace (void)
The `muntrace' function can be called after `mtrace' was used to
enable tracing the `malloc' calls. If no (successful) call of
`mtrace' was made `muntrace' does nothing.
Otherwise it deinstalls the handlers for `malloc', `realloc', and
`free' and then closes the protocol file. No calls are
protocolled anymore and the program runs again at full speed.
This function is a GNU extension and generally not available on
other systems. The prototype can be found in `mcheck.h'.
File: libc.info, Node: Using the Memory Debugger, Next: Tips for the Memory Debugger, Prev: Tracing malloc, Up: Allocation Debugging
3.2.3.2 Example program excerpts
................................
Even though the tracing functionality does not influence the runtime
behavior of the program it is not a good idea to call `mtrace' in all
programs. Just imagine that you debug a program using `mtrace' and all
other programs used in the debugging session also trace their `malloc'
calls. The output file would be the same for all programs and thus is
unusable. Therefore one should call `mtrace' only if compiled for
debugging. A program could therefore start like this:
#include <mcheck.h>
int
main (int argc, char *argv[])
{
#ifdef DEBUGGING
mtrace ();
#endif
...
}
This is all what is needed if you want to trace the calls during the
whole runtime of the program. Alternatively you can stop the tracing at
any time with a call to `muntrace'. It is even possible to restart the
tracing again with a new call to `mtrace'. But this can cause
unreliable results since there may be calls of the functions which are
not called. Please note that not only the application uses the traced
functions, also libraries (including the C library itself) use these
functions.
This last point is also why it is no good idea to call `muntrace'
before the program terminated. The libraries are informed about the
termination of the program only after the program returns from `main'
or calls `exit' and so cannot free the memory they use before this time.
So the best thing one can do is to call `mtrace' as the very first
function in the program and never call `muntrace'. So the program
traces almost all uses of the `malloc' functions (except those calls
which are executed by constructors of the program or used libraries).
File: libc.info, Node: Tips for the Memory Debugger, Next: Interpreting the traces, Prev: Using the Memory Debugger, Up: Allocation Debugging
3.2.3.3 Some more or less clever ideas
......................................
You know the situation. The program is prepared for debugging and in
all debugging sessions it runs well. But once it is started without
debugging the error shows up. A typical example is a memory leak that
becomes visible only when we turn off the debugging. If you foresee
such situations you can still win. Simply use something equivalent to
the following little program:
#include <mcheck.h>
#include <signal.h>
static void
enable (int sig)
{
mtrace ();
signal (SIGUSR1, enable);
}
static void
disable (int sig)
{
muntrace ();
signal (SIGUSR2, disable);
}
int
main (int argc, char *argv[])
{
...
signal (SIGUSR1, enable);
signal (SIGUSR2, disable);
...
}
I.e., the user can start the memory debugger any time s/he wants if
the program was started with `MALLOC_TRACE' set in the environment.
The output will of course not show the allocations which happened before
the first signal but if there is a memory leak this will show up
nevertheless.
File: libc.info, Node: Interpreting the traces, Prev: Tips for the Memory Debugger, Up: Allocation Debugging
3.2.3.4 Interpreting the traces
...............................
If you take a look at the output it will look similar to this:
= Start
[0x8048209] - 0x8064cc8
[0x8048209] - 0x8064ce0
[0x8048209] - 0x8064cf8
[0x80481eb] + 0x8064c48 0x14
[0x80481eb] + 0x8064c60 0x14
[0x80481eb] + 0x8064c78 0x14
[0x80481eb] + 0x8064c90 0x14
= End
What this all means is not really important since the trace file is
not meant to be read by a human. Therefore no attention is given to
readability. Instead there is a program which comes with the GNU C
library which interprets the traces and outputs a summary in an
user-friendly way. The program is called `mtrace' (it is in fact a
Perl script) and it takes one or two arguments. In any case the name of
the file with the trace output must be specified. If an optional
argument precedes the name of the trace file this must be the name of
the program which generated the trace.
drepper$ mtrace tst-mtrace log
No memory leaks.
In this case the program `tst-mtrace' was run and it produced a
trace file `log'. The message printed by `mtrace' shows there are no
problems with the code, all allocated memory was freed afterwards.
If we call `mtrace' on the example trace given above we would get a
different outout:
drepper$ mtrace errlog
- 0x08064cc8 Free 2 was never alloc'd 0x8048209
- 0x08064ce0 Free 3 was never alloc'd 0x8048209
- 0x08064cf8 Free 4 was never alloc'd 0x8048209
Memory not freed:
-----------------
Address Size Caller
0x08064c48 0x14 at 0x80481eb
0x08064c60 0x14 at 0x80481eb
0x08064c78 0x14 at 0x80481eb
0x08064c90 0x14 at 0x80481eb
We have called `mtrace' with only one argument and so the script has
no chance to find out what is meant with the addresses given in the
trace. We can do better:
drepper$ mtrace tst errlog
- 0x08064cc8 Free 2 was never alloc'd /home/drepper/tst.c:39
- 0x08064ce0 Free 3 was never alloc'd /home/drepper/tst.c:39
- 0x08064cf8 Free 4 was never alloc'd /home/drepper/tst.c:39
Memory not freed:
-----------------
Address Size Caller
0x08064c48 0x14 at /home/drepper/tst.c:33
0x08064c60 0x14 at /home/drepper/tst.c:33
0x08064c78 0x14 at /home/drepper/tst.c:33
0x08064c90 0x14 at /home/drepper/tst.c:33
Suddenly the output makes much more sense and the user can see
immediately where the function calls causing the trouble can be found.
Interpreting this output is not complicated. There are at most two
different situations being detected. First, `free' was called for
pointers which were never returned by one of the allocation functions.
This is usually a very bad problem and what this looks like is shown in
the first three lines of the output. Situations like this are quite
rare and if they appear they show up very drastically: the program
normally crashes.
The other situation which is much harder to detect are memory leaks.
As you can see in the output the `mtrace' function collects all this
information and so can say that the program calls an allocation function
from line 33 in the source file `/home/drepper/tst-mtrace.c' four times
without freeing this memory before the program terminates. Whether
this is a real problem remains to be investigated.
File: libc.info, Node: Obstacks, Next: Variable Size Automatic, Prev: Allocation Debugging, Up: Memory Allocation
3.2.4 Obstacks
--------------
An "obstack" is a pool of memory containing a stack of objects. You
can create any number of separate obstacks, and then allocate objects in
specified obstacks. Within each obstack, the last object allocated must
always be the first one freed, but distinct obstacks are independent of
each other.
Aside from this one constraint of order of freeing, obstacks are
totally general: an obstack can contain any number of objects of any
size. They are implemented with macros, so allocation is usually very
fast as long as the objects are usually small. And the only space
overhead per object is the padding needed to start each object on a
suitable boundary.
* Menu:
* Creating Obstacks:: How to declare an obstack in your program.
* Preparing for Obstacks:: Preparations needed before you can
use obstacks.
* Allocation in an Obstack:: Allocating objects in an obstack.
* Freeing Obstack Objects:: Freeing objects in an obstack.
* Obstack Functions:: The obstack functions are both
functions and macros.
* Growing Objects:: Making an object bigger by stages.
* Extra Fast Growing:: Extra-high-efficiency (though more
complicated) growing objects.
* Status of an Obstack:: Inquiries about the status of an obstack.
* Obstacks Data Alignment:: Controlling alignment of objects in obstacks.
* Obstack Chunks:: How obstacks obtain and release chunks;
efficiency considerations.
* Summary of Obstacks::
File: libc.info, Node: Creating Obstacks, Next: Preparing for Obstacks, Up: Obstacks
3.2.4.1 Creating Obstacks
.........................
The utilities for manipulating obstacks are declared in the header file
`obstack.h'.
-- Data Type: struct obstack
An obstack is represented by a data structure of type `struct
obstack'. This structure has a small fixed size; it records the
status of the obstack and how to find the space in which objects
are allocated. It does not contain any of the objects themselves.
You should not try to access the contents of the structure
directly; use only the functions described in this chapter.
You can declare variables of type `struct obstack' and use them as
obstacks, or you can allocate obstacks dynamically like any other kind
of object. Dynamic allocation of obstacks allows your program to have a
variable number of different stacks. (You can even allocate an obstack
structure in another obstack, but this is rarely useful.)
All the functions that work with obstacks require you to specify
which obstack to use. You do this with a pointer of type `struct
obstack *'. In the following, we often say "an obstack" when strictly
speaking the object at hand is such a pointer.
The objects in the obstack are packed into large blocks called
"chunks". The `struct obstack' structure points to a chain of the
chunks currently in use.
The obstack library obtains a new chunk whenever you allocate an
object that won't fit in the previous chunk. Since the obstack library
manages chunks automatically, you don't need to pay much attention to
them, but you do need to supply a function which the obstack library
should use to get a chunk. Usually you supply a function which uses
`malloc' directly or indirectly. You must also supply a function to
free a chunk. These matters are described in the following section.
File: libc.info, Node: Preparing for Obstacks, Next: Allocation in an Obstack, Prev: Creating Obstacks, Up: Obstacks
3.2.4.2 Preparing for Using Obstacks
....................................
Each source file in which you plan to use the obstack functions must
include the header file `obstack.h', like this:
#include <obstack.h>
Also, if the source file uses the macro `obstack_init', it must
declare or define two functions or macros that will be called by the
obstack library. One, `obstack_chunk_alloc', is used to allocate the
chunks of memory into which objects are packed. The other,
`obstack_chunk_free', is used to return chunks when the objects in them
are freed. These macros should appear before any use of obstacks in
the source file.
Usually these are defined to use `malloc' via the intermediary
`xmalloc' (*note Unconstrained Allocation::). This is done with the
following pair of macro definitions:
#define obstack_chunk_alloc xmalloc
#define obstack_chunk_free free
Though the memory you get using obstacks really comes from `malloc',
using obstacks is faster because `malloc' is called less often, for
larger blocks of memory. *Note Obstack Chunks::, for full details.
At run time, before the program can use a `struct obstack' object as
an obstack, it must initialize the obstack by calling `obstack_init'.
-- Function: int obstack_init (struct obstack *OBSTACK-PTR)
Initialize obstack OBSTACK-PTR for allocation of objects. This
function calls the obstack's `obstack_chunk_alloc' function. If
allocation of memory fails, the function pointed to by
`obstack_alloc_failed_handler' is called. The `obstack_init'
function always returns 1 (Compatibility notice: Former versions of
obstack returned 0 if allocation failed).
Here are two examples of how to allocate the space for an obstack and
initialize it. First, an obstack that is a static variable:
static struct obstack myobstack;
...
obstack_init (&myobstack);
Second, an obstack that is itself dynamically allocated:
struct obstack *myobstack_ptr
= (struct obstack *) xmalloc (sizeof (struct obstack));
obstack_init (myobstack_ptr);
-- Variable: obstack_alloc_failed_handler
The value of this variable is a pointer to a function that
`obstack' uses when `obstack_chunk_alloc' fails to allocate
memory. The default action is to print a message and abort. You
should supply a function that either calls `exit' (*note Program
Termination::) or `longjmp' (*note Non-Local Exits::) and doesn't
return.
void my_obstack_alloc_failed (void)
...
obstack_alloc_failed_handler = &my_obstack_alloc_failed;
File: libc.info, Node: Allocation in an Obstack, Next: Freeing Obstack Objects, Prev: Preparing for Obstacks, Up: Obstacks
3.2.4.3 Allocation in an Obstack
................................
The most direct way to allocate an object in an obstack is with
`obstack_alloc', which is invoked almost like `malloc'.
-- Function: void * obstack_alloc (struct obstack *OBSTACK-PTR, int
SIZE)
This allocates an uninitialized block of SIZE bytes in an obstack
and returns its address. Here OBSTACK-PTR specifies which obstack
to allocate the block in; it is the address of the `struct obstack'
object which represents the obstack. Each obstack function or
macro requires you to specify an OBSTACK-PTR as the first argument.
This function calls the obstack's `obstack_chunk_alloc' function if
it needs to allocate a new chunk of memory; it calls
`obstack_alloc_failed_handler' if allocation of memory by
`obstack_chunk_alloc' failed.
For example, here is a function that allocates a copy of a string STR
in a specific obstack, which is in the variable `string_obstack':
struct obstack string_obstack;
char *
copystring (char *string)
{
size_t len = strlen (string) + 1;
char *s = (char *) obstack_alloc (&string_obstack, len);
memcpy (s, string, len);
return s;
}
To allocate a block with specified contents, use the function
`obstack_copy', declared like this:
-- Function: void * obstack_copy (struct obstack *OBSTACK-PTR, void
*ADDRESS, int SIZE)
This allocates a block and initializes it by copying SIZE bytes of
data starting at ADDRESS. It calls `obstack_alloc_failed_handler'
if allocation of memory by `obstack_chunk_alloc' failed.
-- Function: void * obstack_copy0 (struct obstack *OBSTACK-PTR, void
*ADDRESS, int SIZE)
Like `obstack_copy', but appends an extra byte containing a null
character. This extra byte is not counted in the argument SIZE.
The `obstack_copy0' function is convenient for copying a sequence of
characters into an obstack as a null-terminated string. Here is an
example of its use:
char *
obstack_savestring (char *addr, int size)
{
return obstack_copy0 (&myobstack, addr, size);
}
Contrast this with the previous example of `savestring' using `malloc'
(*note Basic Allocation::).
File: libc.info, Node: Freeing Obstack Objects, Next: Obstack Functions, Prev: Allocation in an Obstack, Up: Obstacks
3.2.4.4 Freeing Objects in an Obstack
.....................................
To free an object allocated in an obstack, use the function
`obstack_free'. Since the obstack is a stack of objects, freeing one
object automatically frees all other objects allocated more recently in
the same obstack.
-- Function: void obstack_free (struct obstack *OBSTACK-PTR, void
*OBJECT)
If OBJECT is a null pointer, everything allocated in the obstack
is freed. Otherwise, OBJECT must be the address of an object
allocated in the obstack. Then OBJECT is freed, along with
everything allocated in OBSTACK since OBJECT.
Note that if OBJECT is a null pointer, the result is an
uninitialized obstack. To free all memory in an obstack but leave it
valid for further allocation, call `obstack_free' with the address of
the first object allocated on the obstack:
obstack_free (obstack_ptr, first_object_allocated_ptr);
Recall that the objects in an obstack are grouped into chunks. When
all the objects in a chunk become free, the obstack library
automatically frees the chunk (*note Preparing for Obstacks::). Then
other obstacks, or non-obstack allocation, can reuse the space of the
chunk.
File: libc.info, Node: Obstack Functions, Next: Growing Objects, Prev: Freeing Obstack Objects, Up: Obstacks
3.2.4.5 Obstack Functions and Macros
....................................
The interfaces for using obstacks may be defined either as functions or
as macros, depending on the compiler. The obstack facility works with
all C compilers, including both ISO C and traditional C, but there are
precautions you must take if you plan to use compilers other than GNU C.
If you are using an old-fashioned non-ISO C compiler, all the obstack
"functions" are actually defined only as macros. You can call these
macros like functions, but you cannot use them in any other way (for
example, you cannot take their address).
Calling the macros requires a special precaution: namely, the first
operand (the obstack pointer) may not contain any side effects, because
it may be computed more than once. For example, if you write this:
obstack_alloc (get_obstack (), 4);
you will find that `get_obstack' may be called several times. If you
use `*obstack_list_ptr++' as the obstack pointer argument, you will get
very strange results since the incrementation may occur several times.
In ISO C, each function has both a macro definition and a function
definition. The function definition is used if you take the address of
the function without calling it. An ordinary call uses the macro
definition by default, but you can request the function definition
instead by writing the function name in parentheses, as shown here:
char *x;
void *(*funcp) ();
/* Use the macro. */
x = (char *) obstack_alloc (obptr, size);
/* Call the function. */
x = (char *) (obstack_alloc) (obptr, size);
/* Take the address of the function. */
funcp = obstack_alloc;
This is the same situation that exists in ISO C for the standard library
functions. *Note Macro Definitions::.
*Warning:* When you do use the macros, you must observe the
precaution of avoiding side effects in the first operand, even in ISO C.
If you use the GNU C compiler, this precaution is not necessary,
because various language extensions in GNU C permit defining the macros
so as to compute each argument only once.
File: libc.info, Node: Growing Objects, Next: Extra Fast Growing, Prev: Obstack Functions, Up: Obstacks
3.2.4.6 Growing Objects
.......................
Because memory in obstack chunks is used sequentially, it is possible to
build up an object step by step, adding one or more bytes at a time to
the end of the object. With this technique, you do not need to know
how much data you will put in the object until you come to the end of
it. We call this the technique of "growing objects". The special
functions for adding data to the growing object are described in this
section.
You don't need to do anything special when you start to grow an
object. Using one of the functions to add data to the object
automatically starts it. However, it is necessary to say explicitly
when the object is finished. This is done with the function
`obstack_finish'.
The actual address of the object thus built up is not known until the
object is finished. Until then, it always remains possible that you
will add so much data that the object must be copied into a new chunk.
While the obstack is in use for a growing object, you cannot use it
for ordinary allocation of another object. If you try to do so, the
space already added to the growing object will become part of the other
object.
-- Function: void obstack_blank (struct obstack *OBSTACK-PTR, int SIZE)
The most basic function for adding to a growing object is
`obstack_blank', which adds space without initializing it.
-- Function: void obstack_grow (struct obstack *OBSTACK-PTR, void
*DATA, int SIZE)
To add a block of initialized space, use `obstack_grow', which is
the growing-object analogue of `obstack_copy'. It adds SIZE bytes
of data to the growing object, copying the contents from DATA.
-- Function: void obstack_grow0 (struct obstack *OBSTACK-PTR, void
*DATA, int SIZE)
This is the growing-object analogue of `obstack_copy0'. It adds
SIZE bytes copied from DATA, followed by an additional null
character.
-- Function: void obstack_1grow (struct obstack *OBSTACK-PTR, char C)
To add one character at a time, use the function `obstack_1grow'.
It adds a single byte containing C to the growing object.
-- Function: void obstack_ptr_grow (struct obstack *OBSTACK-PTR, void
*DATA)
Adding the value of a pointer one can use the function
`obstack_ptr_grow'. It adds `sizeof (void *)' bytes containing
the value of DATA.
-- Function: void obstack_int_grow (struct obstack *OBSTACK-PTR, int
DATA)
A single value of type `int' can be added by using the
`obstack_int_grow' function. It adds `sizeof (int)' bytes to the
growing object and initializes them with the value of DATA.
-- Function: void * obstack_finish (struct obstack *OBSTACK-PTR)
When you are finished growing the object, use the function
`obstack_finish' to close it off and return its final address.
Once you have finished the object, the obstack is available for
ordinary allocation or for growing another object.
This function can return a null pointer under the same conditions
as `obstack_alloc' (*note Allocation in an Obstack::).
When you build an object by growing it, you will probably need to
know afterward how long it became. You need not keep track of this as
you grow the object, because you can find out the length from the
obstack just before finishing the object with the function
`obstack_object_size', declared as follows:
-- Function: int obstack_object_size (struct obstack *OBSTACK-PTR)
This function returns the current size of the growing object, in
bytes. Remember to call this function _before_ finishing the
object. After it is finished, `obstack_object_size' will return
zero.
If you have started growing an object and wish to cancel it, you
should finish it and then free it, like this:
obstack_free (obstack_ptr, obstack_finish (obstack_ptr));
This has no effect if no object was growing.
You can use `obstack_blank' with a negative size argument to make
the current object smaller. Just don't try to shrink it beyond zero
length--there's no telling what will happen if you do that.
File: libc.info, Node: Extra Fast Growing, Next: Status of an Obstack, Prev: Growing Objects, Up: Obstacks
3.2.4.7 Extra Fast Growing Objects
..................................
The usual functions for growing objects incur overhead for checking
whether there is room for the new growth in the current chunk. If you
are frequently constructing objects in small steps of growth, this
overhead can be significant.
You can reduce the overhead by using special "fast growth" functions
that grow the object without checking. In order to have a robust
program, you must do the checking yourself. If you do this checking in
the simplest way each time you are about to add data to the object, you
have not saved anything, because that is what the ordinary growth
functions do. But if you can arrange to check less often, or check
more efficiently, then you make the program faster.
The function `obstack_room' returns the amount of room available in
the current chunk. It is declared as follows:
-- Function: int obstack_room (struct obstack *OBSTACK-PTR)
This returns the number of bytes that can be added safely to the
current growing object (or to an object about to be started) in
obstack OBSTACK using the fast growth functions.
While you know there is room, you can use these fast growth functions
for adding data to a growing object:
-- Function: void obstack_1grow_fast (struct obstack *OBSTACK-PTR,
char C)
The function `obstack_1grow_fast' adds one byte containing the
character C to the growing object in obstack OBSTACK-PTR.
-- Function: void obstack_ptr_grow_fast (struct obstack *OBSTACK-PTR,
void *DATA)
The function `obstack_ptr_grow_fast' adds `sizeof (void *)' bytes
containing the value of DATA to the growing object in obstack
OBSTACK-PTR.
-- Function: void obstack_int_grow_fast (struct obstack *OBSTACK-PTR,
int DATA)
The function `obstack_int_grow_fast' adds `sizeof (int)' bytes
containing the value of DATA to the growing object in obstack
OBSTACK-PTR.
-- Function: void obstack_blank_fast (struct obstack *OBSTACK-PTR, int
SIZE)
The function `obstack_blank_fast' adds SIZE bytes to the growing
object in obstack OBSTACK-PTR without initializing them.
When you check for space using `obstack_room' and there is not
enough room for what you want to add, the fast growth functions are not
safe. In this case, simply use the corresponding ordinary growth
function instead. Very soon this will copy the object to a new chunk;
then there will be lots of room available again.
So, each time you use an ordinary growth function, check afterward
for sufficient space using `obstack_room'. Once the object is copied
to a new chunk, there will be plenty of space again, so the program will
start using the fast growth functions again.
Here is an example:
void
add_string (struct obstack *obstack, const char *ptr, int len)
{
while (len > 0)
{
int room = obstack_room (obstack);
if (room == 0)
{
/* Not enough room. Add one character slowly,
which may copy to a new chunk and make room. */
obstack_1grow (obstack, *ptr++);
len--;
}
else
{
if (room > len)
room = len;
/* Add fast as much as we have room for. */
len -= room;
while (room-- > 0)
obstack_1grow_fast (obstack, *ptr++);
}
}
}
File: libc.info, Node: Status of an Obstack, Next: Obstacks Data Alignment, Prev: Extra Fast Growing, Up: Obstacks
3.2.4.8 Status of an Obstack
............................
Here are functions that provide information on the current status of
allocation in an obstack. You can use them to learn about an object
while still growing it.
-- Function: void * obstack_base (struct obstack *OBSTACK-PTR)
This function returns the tentative address of the beginning of the
currently growing object in OBSTACK-PTR. If you finish the object
immediately, it will have that address. If you make it larger
first, it may outgrow the current chunk--then its address will
change!
If no object is growing, this value says where the next object you
allocate will start (once again assuming it fits in the current
chunk).
-- Function: void * obstack_next_free (struct obstack *OBSTACK-PTR)
This function returns the address of the first free byte in the
current chunk of obstack OBSTACK-PTR. This is the end of the
currently growing object. If no object is growing,
`obstack_next_free' returns the same value as `obstack_base'.
-- Function: int obstack_object_size (struct obstack *OBSTACK-PTR)
This function returns the size in bytes of the currently growing
object. This is equivalent to
obstack_next_free (OBSTACK-PTR) - obstack_base (OBSTACK-PTR)
File: libc.info, Node: Obstacks Data Alignment, Next: Obstack Chunks, Prev: Status of an Obstack, Up: Obstacks
3.2.4.9 Alignment of Data in Obstacks
.....................................
Each obstack has an "alignment boundary"; each object allocated in the
obstack automatically starts on an address that is a multiple of the
specified boundary. By default, this boundary is aligned so that the
object can hold any type of data.
To access an obstack's alignment boundary, use the macro
`obstack_alignment_mask', whose function prototype looks like this:
-- Macro: int obstack_alignment_mask (struct obstack *OBSTACK-PTR)
The value is a bit mask; a bit that is 1 indicates that the
corresponding bit in the address of an object should be 0. The
mask value should be one less than a power of 2; the effect is
that all object addresses are multiples of that power of 2. The
default value of the mask is a value that allows aligned objects
to hold any type of data: for example, if its value is 3, any type
of data can be stored at locations whose addresses are multiples
of 4. A mask value of 0 means an object can start on any multiple
of 1 (that is, no alignment is required).
The expansion of the macro `obstack_alignment_mask' is an lvalue,
so you can alter the mask by assignment. For example, this
statement:
obstack_alignment_mask (obstack_ptr) = 0;
has the effect of turning off alignment processing in the
specified obstack.
Note that a change in alignment mask does not take effect until
_after_ the next time an object is allocated or finished in the
obstack. If you are not growing an object, you can make the new
alignment mask take effect immediately by calling `obstack_finish'.
This will finish a zero-length object and then do proper alignment for
the next object.
File: libc.info, Node: Obstack Chunks, Next: Summary of Obstacks, Prev: Obstacks Data Alignment, Up: Obstacks
3.2.4.10 Obstack Chunks
.......................
Obstacks work by allocating space for themselves in large chunks, and
then parceling out space in the chunks to satisfy your requests. Chunks
are normally 4096 bytes long unless you specify a different chunk size.
The chunk size includes 8 bytes of overhead that are not actually used
for storing objects. Regardless of the specified size, longer chunks
will be allocated when necessary for long objects.
The obstack library allocates chunks by calling the function
`obstack_chunk_alloc', which you must define. When a chunk is no
longer needed because you have freed all the objects in it, the obstack
library frees the chunk by calling `obstack_chunk_free', which you must
also define.
These two must be defined (as macros) or declared (as functions) in
each source file that uses `obstack_init' (*note Creating Obstacks::).
Most often they are defined as macros like this:
#define obstack_chunk_alloc malloc
#define obstack_chunk_free free
Note that these are simple macros (no arguments). Macro definitions
with arguments will not work! It is necessary that
`obstack_chunk_alloc' or `obstack_chunk_free', alone, expand into a
function name if it is not itself a function name.
If you allocate chunks with `malloc', the chunk size should be a
power of 2. The default chunk size, 4096, was chosen because it is long
enough to satisfy many typical requests on the obstack yet short enough
not to waste too much memory in the portion of the last chunk not yet
used.
-- Macro: int obstack_chunk_size (struct obstack *OBSTACK-PTR)
This returns the chunk size of the given obstack.
Since this macro expands to an lvalue, you can specify a new chunk
size by assigning it a new value. Doing so does not affect the chunks
already allocated, but will change the size of chunks allocated for
that particular obstack in the future. It is unlikely to be useful to
make the chunk size smaller, but making it larger might improve
efficiency if you are allocating many objects whose size is comparable
to the chunk size. Here is how to do so cleanly:
if (obstack_chunk_size (obstack_ptr) < NEW-CHUNK-SIZE)
obstack_chunk_size (obstack_ptr) = NEW-CHUNK-SIZE;
File: libc.info, Node: Summary of Obstacks, Prev: Obstack Chunks, Up: Obstacks
3.2.4.11 Summary of Obstack Functions
.....................................
Here is a summary of all the functions associated with obstacks. Each
takes the address of an obstack (`struct obstack *') as its first
argument.
`void obstack_init (struct obstack *OBSTACK-PTR)'
Initialize use of an obstack. *Note Creating Obstacks::.
`void *obstack_alloc (struct obstack *OBSTACK-PTR, int SIZE)'
Allocate an object of SIZE uninitialized bytes. *Note Allocation
in an Obstack::.
`void *obstack_copy (struct obstack *OBSTACK-PTR, void *ADDRESS, int SIZE)'
Allocate an object of SIZE bytes, with contents copied from
ADDRESS. *Note Allocation in an Obstack::.
`void *obstack_copy0 (struct obstack *OBSTACK-PTR, void *ADDRESS, int SIZE)'
Allocate an object of SIZE+1 bytes, with SIZE of them copied from
ADDRESS, followed by a null character at the end. *Note
Allocation in an Obstack::.
`void obstack_free (struct obstack *OBSTACK-PTR, void *OBJECT)'
Free OBJECT (and everything allocated in the specified obstack
more recently than OBJECT). *Note Freeing Obstack Objects::.
`void obstack_blank (struct obstack *OBSTACK-PTR, int SIZE)'
Add SIZE uninitialized bytes to a growing object. *Note Growing
Objects::.
`void obstack_grow (struct obstack *OBSTACK-PTR, void *ADDRESS, int SIZE)'
Add SIZE bytes, copied from ADDRESS, to a growing object. *Note
Growing Objects::.
`void obstack_grow0 (struct obstack *OBSTACK-PTR, void *ADDRESS, int SIZE)'
Add SIZE bytes, copied from ADDRESS, to a growing object, and then
add another byte containing a null character. *Note Growing
Objects::.
`void obstack_1grow (struct obstack *OBSTACK-PTR, char DATA-CHAR)'
Add one byte containing DATA-CHAR to a growing object. *Note
Growing Objects::.
`void *obstack_finish (struct obstack *OBSTACK-PTR)'
Finalize the object that is growing and return its permanent
address. *Note Growing Objects::.
`int obstack_object_size (struct obstack *OBSTACK-PTR)'
Get the current size of the currently growing object. *Note
Growing Objects::.
`void obstack_blank_fast (struct obstack *OBSTACK-PTR, int SIZE)'
Add SIZE uninitialized bytes to a growing object without checking
that there is enough room. *Note Extra Fast Growing::.
`void obstack_1grow_fast (struct obstack *OBSTACK-PTR, char DATA-CHAR)'
Add one byte containing DATA-CHAR to a growing object without
checking that there is enough room. *Note Extra Fast Growing::.
`int obstack_room (struct obstack *OBSTACK-PTR)'
Get the amount of room now available for growing the current
object. *Note Extra Fast Growing::.
`int obstack_alignment_mask (struct obstack *OBSTACK-PTR)'
The mask used for aligning the beginning of an object. This is an
lvalue. *Note Obstacks Data Alignment::.
`int obstack_chunk_size (struct obstack *OBSTACK-PTR)'
The size for allocating chunks. This is an lvalue. *Note Obstack
Chunks::.
`void *obstack_base (struct obstack *OBSTACK-PTR)'
Tentative starting address of the currently growing object. *Note
Status of an Obstack::.
`void *obstack_next_free (struct obstack *OBSTACK-PTR)'
Address just after the end of the currently growing object. *Note
Status of an Obstack::.
File: libc.info, Node: Variable Size Automatic, Prev: Obstacks, Up: Memory Allocation
3.2.5 Automatic Storage with Variable Size
------------------------------------------
The function `alloca' supports a kind of half-dynamic allocation in
which blocks are allocated dynamically but freed automatically.
Allocating a block with `alloca' is an explicit action; you can
allocate as many blocks as you wish, and compute the size at run time.
But all the blocks are freed when you exit the function that `alloca'
was called from, just as if they were automatic variables declared in
that function. There is no way to free the space explicitly.
The prototype for `alloca' is in `stdlib.h'. This function is a BSD
extension.
-- Function: void * alloca (size_t SIZE);
The return value of `alloca' is the address of a block of SIZE
bytes of memory, allocated in the stack frame of the calling
function.
Do not use `alloca' inside the arguments of a function call--you
will get unpredictable results, because the stack space for the
`alloca' would appear on the stack in the middle of the space for the
function arguments. An example of what to avoid is `foo (x, alloca
(4), y)'.
* Menu:
* Alloca Example:: Example of using `alloca'.
* Advantages of Alloca:: Reasons to use `alloca'.
* Disadvantages of Alloca:: Reasons to avoid `alloca'.
* GNU C Variable-Size Arrays:: Only in GNU C, here is an alternative
method of allocating dynamically and
freeing automatically.
File: libc.info, Node: Alloca Example, Next: Advantages of Alloca, Up: Variable Size Automatic
3.2.5.1 `alloca' Example
........................
As an example of the use of `alloca', here is a function that opens a
file name made from concatenating two argument strings, and returns a
file descriptor or minus one signifying failure:
int
open2 (char *str1, char *str2, int flags, int mode)
{
char *name = (char *) alloca (strlen (str1) + strlen (str2) + 1);
stpcpy (stpcpy (name, str1), str2);
return open (name, flags, mode);
}
Here is how you would get the same results with `malloc' and `free':
int
open2 (char *str1, char *str2, int flags, int mode)
{
char *name = (char *) malloc (strlen (str1) + strlen (str2) + 1);
int desc;
if (name == 0)
fatal ("virtual memory exceeded");
stpcpy (stpcpy (name, str1), str2);
desc = open (name, flags, mode);
free (name);
return desc;
}
As you can see, it is simpler with `alloca'. But `alloca' has
other, more important advantages, and some disadvantages.
File: libc.info, Node: Advantages of Alloca, Next: Disadvantages of Alloca, Prev: Alloca Example, Up: Variable Size Automatic
3.2.5.2 Advantages of `alloca'
..............................
Here are the reasons why `alloca' may be preferable to `malloc':
* Using `alloca' wastes very little space and is very fast. (It is
open-coded by the GNU C compiler.)
* Since `alloca' does not have separate pools for different sizes of
block, space used for any size block can be reused for any other
size. `alloca' does not cause memory fragmentation.
* Nonlocal exits done with `longjmp' (*note Non-Local Exits::)
automatically free the space allocated with `alloca' when they exit
through the function that called `alloca'. This is the most
important reason to use `alloca'.
To illustrate this, suppose you have a function
`open_or_report_error' which returns a descriptor, like `open', if
it succeeds, but does not return to its caller if it fails. If
the file cannot be opened, it prints an error message and jumps
out to the command level of your program using `longjmp'. Let's
change `open2' (*note Alloca Example::) to use this subroutine:
int
open2 (char *str1, char *str2, int flags, int mode)
{
char *name = (char *) alloca (strlen (str1) + strlen (str2) + 1);
stpcpy (stpcpy (name, str1), str2);
return open_or_report_error (name, flags, mode);
}
Because of the way `alloca' works, the memory it allocates is
freed even when an error occurs, with no special effort required.
By contrast, the previous definition of `open2' (which uses
`malloc' and `free') would develop a memory leak if it were
changed in this way. Even if you are willing to make more changes
to fix it, there is no easy way to do so.
File: libc.info, Node: Disadvantages of Alloca, Next: GNU C Variable-Size Arrays, Prev: Advantages of Alloca, Up: Variable Size Automatic
3.2.5.3 Disadvantages of `alloca'
.................................
These are the disadvantages of `alloca' in comparison with `malloc':
* If you try to allocate more memory than the machine can provide,
you don't get a clean error message. Instead you get a fatal
signal like the one you would get from an infinite recursion;
probably a segmentation violation (*note Program Error Signals::).
* Some non-GNU systems fail to support `alloca', so it is less
portable. However, a slower emulation of `alloca' written in C is
available for use on systems with this deficiency.
File: libc.info, Node: GNU C Variable-Size Arrays, Prev: Disadvantages of Alloca, Up: Variable Size Automatic
3.2.5.4 GNU C Variable-Size Arrays
..................................
In GNU C, you can replace most uses of `alloca' with an array of
variable size. Here is how `open2' would look then:
int open2 (char *str1, char *str2, int flags, int mode)
{
char name[strlen (str1) + strlen (str2) + 1];
stpcpy (stpcpy (name, str1), str2);
return open (name, flags, mode);
}
But `alloca' is not always equivalent to a variable-sized array, for
several reasons:
* A variable size array's space is freed at the end of the scope of
the name of the array. The space allocated with `alloca' remains
until the end of the function.
* It is possible to use `alloca' within a loop, allocating an
additional block on each iteration. This is impossible with
variable-sized arrays.
*Note:* If you mix use of `alloca' and variable-sized arrays within
one function, exiting a scope in which a variable-sized array was
declared frees all blocks allocated with `alloca' during the execution
of that scope.
File: libc.info, Node: Locking Pages, Next: Resizing the Data Segment, Prev: Memory Allocation, Up: Memory
3.4 Locking Pages
=================
You can tell the system to associate a particular virtual memory page
with a real page frame and keep it that way -- i.e. cause the page to
be paged in if it isn't already and mark it so it will never be paged
out and consequently will never cause a page fault. This is called
"locking" a page.
The functions in this chapter lock and unlock the calling process'
pages.
* Menu:
* Why Lock Pages:: Reasons to read this section.
* Locked Memory Details:: Everything you need to know locked
memory
* Page Lock Functions:: Here's how to do it.
File: libc.info, Node: Why Lock Pages, Next: Locked Memory Details, Up: Locking Pages
3.4.1 Why Lock Pages
--------------------
Because page faults cause paged out pages to be paged in transparently,
a process rarely needs to be concerned about locking pages. However,
there are two reasons people sometimes are:
* Speed. A page fault is transparent only insofar as the process is
not sensitive to how long it takes to do a simple memory access.
Time-critical processes, especially realtime processes, may not be
able to wait or may not be able to tolerate variance in execution
speed.
A process that needs to lock pages for this reason probably also
needs priority among other processes for use of the CPU. *Note
Priority::.
In some cases, the programmer knows better than the system's demand
paging allocator which pages should remain in real memory to
optimize system performance. In this case, locking pages can help.
* Privacy. If you keep secrets in virtual memory and that virtual
memory gets paged out, that increases the chance that the secrets
will get out. If a password gets written out to disk swap space,
for example, it might still be there long after virtual and real
memory have been wiped clean.
Be aware that when you lock a page, that's one fewer page frame that
can be used to back other virtual memory (by the same or other
processes), which can mean more page faults, which means the system
runs more slowly. In fact, if you lock enough memory, some programs
may not be able to run at all for lack of real memory.
File: libc.info, Node: Locked Memory Details, Next: Page Lock Functions, Prev: Why Lock Pages, Up: Locking Pages
3.4.2 Locked Memory Details
---------------------------
A memory lock is associated with a virtual page, not a real frame. The
paging rule is: If a frame backs at least one locked page, don't page it
out.
Memory locks do not stack. I.e. you can't lock a particular page
twice so that it has to be unlocked twice before it is truly unlocked.
It is either locked or it isn't.
A memory lock persists until the process that owns the memory
explicitly unlocks it. (But process termination and exec cause the
virtual memory to cease to exist, which you might say means it isn't
locked any more).
Memory locks are not inherited by child processes. (But note that
on a modern Unix system, immediately after a fork, the parent's and the
child's virtual address space are backed by the same real page frames,
so the child enjoys the parent's locks). *Note Creating a Process::.
Because of its ability to impact other processes, only the superuser
can lock a page. Any process can unlock its own page.
The system sets limits on the amount of memory a process can have
locked and the amount of real memory it can have dedicated to it.
*Note Limits on Resources::.
In Linux, locked pages aren't as locked as you might think. Two
virtual pages that are not shared memory can nonetheless be backed by
the same real frame. The kernel does this in the name of efficiency
when it knows both virtual pages contain identical data, and does it
even if one or both of the virtual pages are locked.
But when a process modifies one of those pages, the kernel must get
it a separate frame and fill it with the page's data. This is known as
a "copy-on-write page fault". It takes a small amount of time and in a
pathological case, getting that frame may require I/O.
To make sure this doesn't happen to your program, don't just lock the
pages. Write to them as well, unless you know you won't write to them
ever. And to make sure you have pre-allocated frames for your stack,
enter a scope that declares a C automatic variable larger than the
maximum stack size you will need, set it to something, then return from
its scope.
File: libc.info, Node: Page Lock Functions, Prev: Locked Memory Details, Up: Locking Pages
3.4.3 Functions To Lock And Unlock Pages
----------------------------------------
The symbols in this section are declared in `sys/mman.h'. These
functions are defined by POSIX.1b, but their availability depends on
your kernel. If your kernel doesn't allow these functions, they exist
but always fail. They _are_ available with a Linux kernel.
*Portability Note:* POSIX.1b requires that when the `mlock' and
`munlock' functions are available, the file `unistd.h' define the macro
`_POSIX_MEMLOCK_RANGE' and the file `limits.h' define the macro
`PAGESIZE' to be the size of a memory page in bytes. It requires that
when the `mlockall' and `munlockall' functions are available, the
`unistd.h' file define the macro `_POSIX_MEMLOCK'. The GNU C library
conforms to this requirement.
-- Function: int mlock (const void *ADDR, size_t LEN)
`mlock' locks a range of the calling process' virtual pages.
The range of memory starts at address ADDR and is LEN bytes long.
Actually, since you must lock whole pages, it is the range of
pages that include any part of the specified range.
When the function returns successfully, each of those pages is
backed by (connected to) a real frame (is resident) and is marked
to stay that way. This means the function may cause page-ins and
have to wait for them.
When the function fails, it does not affect the lock status of any
pages.
The return value is zero if the function succeeds. Otherwise, it
is `-1' and `errno' is set accordingly. `errno' values specific
to this function are:
`ENOMEM'
* At least some of the specified address range does not
exist in the calling process' virtual address space.
* The locking would cause the process to exceed its locked
page limit.
`EPERM'
The calling process is not superuser.
`EINVAL'
LEN is not positive.
`ENOSYS'
The kernel does not provide `mlock' capability.
You can lock _all_ a process' memory with `mlockall'. You unlock
memory with `munlock' or `munlockall'.
To avoid all page faults in a C program, you have to use
`mlockall', because some of the memory a program uses is hidden
from the C code, e.g. the stack and automatic variables, and you
wouldn't know what address to tell `mlock'.
-- Function: int munlock (const void *ADDR, size_t LEN)
`munlock' unlocks a range of the calling process' virtual pages.
`munlock' is the inverse of `mlock' and functions completely
analogously to `mlock', except that there is no `EPERM' failure.
-- Function: int mlockall (int FLAGS)
`mlockall' locks all the pages in a process' virtual memory address
space, and/or any that are added to it in the future. This
includes the pages of the code, data and stack segment, as well as
shared libraries, user space kernel data, shared memory, and
memory mapped files.
FLAGS is a string of single bit flags represented by the following
macros. They tell `mlockall' which of its functions you want. All
other bits must be zero.
`MCL_CURRENT'
Lock all pages which currently exist in the calling process'
virtual address space.
`MCL_FUTURE'
Set a mode such that any pages added to the process' virtual
address space in the future will be locked from birth. This
mode does not affect future address spaces owned by the same
process so exec, which replaces a process' address space,
wipes out `MCL_FUTURE'. *Note Executing a File::.
When the function returns successfully, and you specified
`MCL_CURRENT', all of the process' pages are backed by (connected
to) real frames (they are resident) and are marked to stay that
way. This means the function may cause page-ins and have to wait
for them.
When the process is in `MCL_FUTURE' mode because it successfully
executed this function and specified `MCL_CURRENT', any system call
by the process that requires space be added to its virtual address
space fails with `errno' = `ENOMEM' if locking the additional space
would cause the process to exceed its locked page limit. In the
case that the address space addition that can't be accommodated is
stack expansion, the stack expansion fails and the kernel sends a
`SIGSEGV' signal to the process.
When the function fails, it does not affect the lock status of any
pages or the future locking mode.
The return value is zero if the function succeeds. Otherwise, it
is `-1' and `errno' is set accordingly. `errno' values specific
to this function are:
`ENOMEM'
* At least some of the specified address range does not
exist in the calling process' virtual address space.
* The locking would cause the process to exceed its locked
page limit.
`EPERM'
The calling process is not superuser.
`EINVAL'
Undefined bits in FLAGS are not zero.
`ENOSYS'
The kernel does not provide `mlockall' capability.
You can lock just specific pages with `mlock'. You unlock pages
with `munlockall' and `munlock'.
-- Function: int munlockall (void)
`munlockall' unlocks every page in the calling process' virtual
address space and turn off `MCL_FUTURE' future locking mode.
The return value is zero if the function succeeds. Otherwise, it
is `-1' and `errno' is set accordingly. The only way this
function can fail is for generic reasons that all functions and
system calls can fail, so there are no specific `errno' values.
File: libc.info, Node: Resizing the Data Segment, Prev: Locking Pages, Up: Memory
3.3 Resizing the Data Segment
=============================
The symbols in this section are declared in `unistd.h'.
You will not normally use the functions in this section, because the
functions described in *Note Memory Allocation:: are easier to use.
Those are interfaces to a GNU C Library memory allocator that uses the
functions below itself. The functions below are simple interfaces to
system calls.
-- Function: int brk (void *ADDR)
`brk' sets the high end of the calling process' data segment to
ADDR.
The address of the end of a segment is defined to be the address
of the last byte in the segment plus 1.
The function has no effect if ADDR is lower than the low end of
the data segment. (This is considered success, by the way).
The function fails if it would cause the data segment to overlap
another segment or exceed the process' data storage limit (*note
Limits on Resources::).
The function is named for a common historical case where data
storage and the stack are in the same segment. Data storage
allocation grows upward from the bottom of the segment while the
stack grows downward toward it from the top of the segment and the
curtain between them is called the "break".
The return value is zero on success. On failure, the return value
is `-1' and `errno' is set accordingly. The following `errno'
values are specific to this function:
`ENOMEM'
The request would cause the data segment to overlap another
segment or exceed the process' data storage limit.
-- Function: int sbrk (ptrdiff_t DELTA)
This function is the same as `brk' except that you specify the new
end of the data segment as an offset DELTA from the current end
and on success the return value is the address of the resulting
end of the data segment instead of zero.
This means you can use `sbrk(0)' to find out what the current end
of the data segment is.
File: libc.info, Node: Character Handling, Next: String and Array Utilities, Prev: Memory, Up: Top
4 Character Handling
********************
Programs that work with characters and strings often need to classify a
character--is it alphabetic, is it a digit, is it whitespace, and so
on--and perform case conversion operations on characters. The
functions in the header file `ctype.h' are provided for this purpose.
Since the choice of locale and character set can alter the
classifications of particular character codes, all of these functions
are affected by the current locale. (More precisely, they are affected
by the locale currently selected for character classification--the
`LC_CTYPE' category; see *Note Locale Categories::.)
The ISO C standard specifies two different sets of functions. The
one set works on `char' type characters, the other one on `wchar_t'
wide characters (*note Extended Char Intro::).
* Menu:
* Classification of Characters:: Testing whether characters are
letters, digits, punctuation, etc.
* Case Conversion:: Case mapping, and the like.
* Classification of Wide Characters:: Character class determination for
wide characters.
* Using Wide Char Classes:: Notes on using the wide character
classes.
* Wide Character Case Conversion:: Mapping of wide characters.
File: libc.info, Node: Classification of Characters, Next: Case Conversion, Up: Character Handling
4.1 Classification of Characters
================================
This section explains the library functions for classifying characters.
For example, `isalpha' is the function to test for an alphabetic
character. It takes one argument, the character to test, and returns a
nonzero integer if the character is alphabetic, and zero otherwise. You
would use it like this:
if (isalpha (c))
printf ("The character `%c' is alphabetic.\n", c);
Each of the functions in this section tests for membership in a
particular class of characters; each has a name starting with `is'.
Each of them takes one argument, which is a character to test, and
returns an `int' which is treated as a boolean value. The character
argument is passed as an `int', and it may be the constant value `EOF'
instead of a real character.
The attributes of any given character can vary between locales.
*Note Locales::, for more information on locales.
These functions are declared in the header file `ctype.h'.
-- Function: int islower (int C)
Returns true if C is a lower-case letter. The letter need not be
from the Latin alphabet, any alphabet representable is valid.
-- Function: int isupper (int C)
Returns true if C is an upper-case letter. The letter need not be
from the Latin alphabet, any alphabet representable is valid.
-- Function: int isalpha (int C)
Returns true if C is an alphabetic character (a letter). If
`islower' or `isupper' is true of a character, then `isalpha' is
also true.
In some locales, there may be additional characters for which
`isalpha' is true--letters which are neither upper case nor lower
case. But in the standard `"C"' locale, there are no such
additional characters.
-- Function: int isdigit (int C)
Returns true if C is a decimal digit (`0' through `9').
-- Function: int isalnum (int C)
Returns true if C is an alphanumeric character (a letter or
number); in other words, if either `isalpha' or `isdigit' is true
of a character, then `isalnum' is also true.
-- Function: int isxdigit (int C)
Returns true if C is a hexadecimal digit. Hexadecimal digits
include the normal decimal digits `0' through `9' and the letters
`A' through `F' and `a' through `f'.
-- Function: int ispunct (int C)
Returns true if C is a punctuation character. This means any
printing character that is not alphanumeric or a space character.
-- Function: int isspace (int C)
Returns true if C is a "whitespace" character. In the standard
`"C"' locale, `isspace' returns true for only the standard
whitespace characters:
`' ''
space
`'\f''
formfeed
`'\n''
newline
`'\r''
carriage return
`'\t''
horizontal tab
`'\v''
vertical tab
-- Function: int isblank (int C)
Returns true if C is a blank character; that is, a space or a tab.
This function was originally a GNU extension, but was added in
ISO C99.
-- Function: int isgraph (int C)
Returns true if C is a graphic character; that is, a character
that has a glyph associated with it. The whitespace characters
are not considered graphic.
-- Function: int isprint (int C)
Returns true if C is a printing character. Printing characters
include all the graphic characters, plus the space (` ') character.
-- Function: int iscntrl (int C)
Returns true if C is a control character (that is, a character that
is not a printing character).
-- Function: int isascii (int C)
Returns true if C is a 7-bit `unsigned char' value that fits into
the US/UK ASCII character set. This function is a BSD extension
and is also an SVID extension.
File: libc.info, Node: Case Conversion, Next: Classification of Wide Characters, Prev: Classification of Characters, Up: Character Handling
4.2 Case Conversion
===================
This section explains the library functions for performing conversions
such as case mappings on characters. For example, `toupper' converts
any character to upper case if possible. If the character can't be
converted, `toupper' returns it unchanged.
These functions take one argument of type `int', which is the
character to convert, and return the converted character as an `int'.
If the conversion is not applicable to the argument given, the argument
is returned unchanged.
*Compatibility Note:* In pre-ISO C dialects, instead of returning
the argument unchanged, these functions may fail when the argument is
not suitable for the conversion. Thus for portability, you may need to
write `islower(c) ? toupper(c) : c' rather than just `toupper(c)'.
These functions are declared in the header file `ctype.h'.
-- Function: int tolower (int C)
If C is an upper-case letter, `tolower' returns the corresponding
lower-case letter. If C is not an upper-case letter, C is
returned unchanged.
-- Function: int toupper (int C)
If C is a lower-case letter, `toupper' returns the corresponding
upper-case letter. Otherwise C is returned unchanged.
-- Function: int toascii (int C)
This function converts C to a 7-bit `unsigned char' value that
fits into the US/UK ASCII character set, by clearing the high-order
bits. This function is a BSD extension and is also an SVID
extension.
-- Function: int _tolower (int C)
This is identical to `tolower', and is provided for compatibility
with the SVID. *Note SVID::.
-- Function: int _toupper (int C)
This is identical to `toupper', and is provided for compatibility
with the SVID.
File: libc.info, Node: Classification of Wide Characters, Next: Using Wide Char Classes, Prev: Case Conversion, Up: Character Handling
4.3 Character class determination for wide characters
=====================================================
Amendment 1 to ISO C90 defines functions to classify wide characters.
Although the original ISO C90 standard already defined the type
`wchar_t', no functions operating on them were defined.
The general design of the classification functions for wide
characters is more general. It allows extensions to the set of
available classifications, beyond those which are always available.
The POSIX standard specifies how extensions can be made, and this is
already implemented in the GNU C library implementation of the
`localedef' program.
The character class functions are normally implemented with bitsets,
with a bitset per character. For a given character, the appropriate
bitset is read from a table and a test is performed as to whether a
certain bit is set. Which bit is tested for is determined by the class.
For the wide character classification functions this is made visible.
There is a type classification type defined, a function to retrieve this
value for a given class, and a function to test whether a given
character is in this class, using the classification value. On top of
this the normal character classification functions as used for `char'
objects can be defined.
-- Data type: wctype_t
The `wctype_t' can hold a value which represents a character class.
The only defined way to generate such a value is by using the
`wctype' function.
This type is defined in `wctype.h'.
-- Function: wctype_t wctype (const char *PROPERTY)
The `wctype' returns a value representing a class of wide
characters which is identified by the string PROPERTY. Beside
some standard properties each locale can define its own ones. In
case no property with the given name is known for the current
locale selected for the `LC_CTYPE' category, the function returns
zero.
The properties known in every locale are:
`"alnum"' `"alpha"' `"cntrl"' `"digit"'
`"graph"' `"lower"' `"print"' `"punct"'
`"space"' `"upper"' `"xdigit"'
This function is declared in `wctype.h'.
To test the membership of a character to one of the non-standard
classes the ISO C standard defines a completely new function.
-- Function: int iswctype (wint_t WC, wctype_t DESC)
This function returns a nonzero value if WC is in the character
class specified by DESC. DESC must previously be returned by a
successful call to `wctype'.
This function is declared in `wctype.h'.
To make it easier to use the commonly-used classification functions,
they are defined in the C library. There is no need to use `wctype' if
the property string is one of the known character classes. In some
situations it is desirable to construct the property strings, and then
it is important that `wctype' can also handle the standard classes.
-- Function: int iswalnum (wint_t WC)
This function returns a nonzero value if WC is an alphanumeric
character (a letter or number); in other words, if either
`iswalpha' or `iswdigit' is true of a character, then `iswalnum'
is also true.
This function can be implemented using
iswctype (wc, wctype ("alnum"))
It is declared in `wctype.h'.
-- Function: int iswalpha (wint_t WC)
Returns true if WC is an alphabetic character (a letter). If
`iswlower' or `iswupper' is true of a character, then `iswalpha'
is also true.
In some locales, there may be additional characters for which
`iswalpha' is true--letters which are neither upper case nor lower
case. But in the standard `"C"' locale, there are no such
additional characters.
This function can be implemented using
iswctype (wc, wctype ("alpha"))
It is declared in `wctype.h'.
-- Function: int iswcntrl (wint_t WC)
Returns true if WC is a control character (that is, a character
that is not a printing character).
This function can be implemented using
iswctype (wc, wctype ("cntrl"))
It is declared in `wctype.h'.
-- Function: int iswdigit (wint_t WC)
Returns true if WC is a digit (e.g., `0' through `9'). Please
note that this function does not only return a nonzero value for
_decimal_ digits, but for all kinds of digits. A consequence is
that code like the following will *not* work unconditionally for
wide characters:
n = 0;
while (iswdigit (*wc))
{
n *= 10;
n += *wc++ - L'0';
}
This function can be implemented using
iswctype (wc, wctype ("digit"))
It is declared in `wctype.h'.
-- Function: int iswgraph (wint_t WC)
Returns true if WC is a graphic character; that is, a character
that has a glyph associated with it. The whitespace characters
are not considered graphic.
This function can be implemented using
iswctype (wc, wctype ("graph"))
It is declared in `wctype.h'.
-- Function: int iswlower (wint_t WC)
Returns true if WC is a lower-case letter. The letter need not be
from the Latin alphabet, any alphabet representable is valid.
This function can be implemented using
iswctype (wc, wctype ("lower"))
It is declared in `wctype.h'.
-- Function: int iswprint (wint_t WC)
Returns true if WC is a printing character. Printing characters
include all the graphic characters, plus the space (` ') character.
This function can be implemented using
iswctype (wc, wctype ("print"))
It is declared in `wctype.h'.
-- Function: int iswpunct (wint_t WC)
Returns true if WC is a punctuation character. This means any
printing character that is not alphanumeric or a space character.
This function can be implemented using
iswctype (wc, wctype ("punct"))
It is declared in `wctype.h'.
-- Function: int iswspace (wint_t WC)
Returns true if WC is a "whitespace" character. In the standard
`"C"' locale, `iswspace' returns true for only the standard
whitespace characters:
`L' ''
space
`L'\f''
formfeed
`L'\n''
newline
`L'\r''
carriage return
`L'\t''
horizontal tab
`L'\v''
vertical tab
This function can be implemented using
iswctype (wc, wctype ("space"))
It is declared in `wctype.h'.
-- Function: int iswupper (wint_t WC)
Returns true if WC is an upper-case letter. The letter need not be
from the Latin alphabet, any alphabet representable is valid.
This function can be implemented using
iswctype (wc, wctype ("upper"))
It is declared in `wctype.h'.
-- Function: int iswxdigit (wint_t WC)
Returns true if WC is a hexadecimal digit. Hexadecimal digits
include the normal decimal digits `0' through `9' and the letters
`A' through `F' and `a' through `f'.
This function can be implemented using
iswctype (wc, wctype ("xdigit"))
It is declared in `wctype.h'.
The GNU C library also provides a function which is not defined in
the ISO C standard but which is available as a version for single byte
characters as well.
-- Function: int iswblank (wint_t WC)
Returns true if WC is a blank character; that is, a space or a tab.
This function was originally a GNU extension, but was added in
ISO C99. It is declared in `wchar.h'.
File: libc.info, Node: Using Wide Char Classes, Next: Wide Character Case Conversion, Prev: Classification of Wide Characters, Up: Character Handling
4.4 Notes on using the wide character classes
=============================================
The first note is probably not astonishing but still occasionally a
cause of problems. The `iswXXX' functions can be implemented using
macros and in fact, the GNU C library does this. They are still
available as real functions but when the `wctype.h' header is included
the macros will be used. This is the same as the `char' type versions
of these functions.
The second note covers something new. It can be best illustrated by
a (real-world) example. The first piece of code is an excerpt from the
original code. It is truncated a bit but the intention should be clear.
int
is_in_class (int c, const char *class)
{
if (strcmp (class, "alnum") == 0)
return isalnum (c);
if (strcmp (class, "alpha") == 0)
return isalpha (c);
if (strcmp (class, "cntrl") == 0)
return iscntrl (c);
...
return 0;
}
Now, with the `wctype' and `iswctype' you can avoid the `if'
cascades, but rewriting the code as follows is wrong:
int
is_in_class (int c, const char *class)
{
wctype_t desc = wctype (class);
return desc ? iswctype ((wint_t) c, desc) : 0;
}
The problem is that it is not guaranteed that the wide character
representation of a single-byte character can be found using casting.
In fact, usually this fails miserably. The correct solution to this
problem is to write the code as follows:
int
is_in_class (int c, const char *class)
{
wctype_t desc = wctype (class);
return desc ? iswctype (btowc (c), desc) : 0;
}
*Note Converting a Character::, for more information on `btowc'.
Note that this change probably does not improve the performance of the
program a lot since the `wctype' function still has to make the string
comparisons. It gets really interesting if the `is_in_class' function
is called more than once for the same class name. In this case the
variable DESC could be computed once and reused for all the calls.
Therefore the above form of the function is probably not the final one.
File: libc.info, Node: Wide Character Case Conversion, Prev: Using Wide Char Classes, Up: Character Handling
4.5 Mapping of wide characters.
===============================
The classification functions are also generalized by the ISO C
standard. Instead of just allowing the two standard mappings, a locale
can contain others. Again, the `localedef' program already supports
generating such locale data files.
-- Data Type: wctrans_t
This data type is defined as a scalar type which can hold a value
representing the locale-dependent character mapping. There is no
way to construct such a value apart from using the return value of
the `wctrans' function.
This type is defined in `wctype.h'.
-- Function: wctrans_t wctrans (const char *PROPERTY)
The `wctrans' function has to be used to find out whether a named
mapping is defined in the current locale selected for the
`LC_CTYPE' category. If the returned value is non-zero, you can
use it afterwards in calls to `towctrans'. If the return value is
zero no such mapping is known in the current locale.
Beside locale-specific mappings there are two mappings which are
guaranteed to be available in every locale:
`"tolower"' `"toupper"'
These functions are declared in `wctype.h'.
-- Function: wint_t towctrans (wint_t WC, wctrans_t DESC)
`towctrans' maps the input character WC according to the rules of
the mapping for which DESC is a descriptor, and returns the value
it finds. DESC must be obtained by a successful call to `wctrans'.
This function is declared in `wctype.h'.
For the generally available mappings, the ISO C standard defines
convenient shortcuts so that it is not necessary to call `wctrans' for
them.
-- Function: wint_t towlower (wint_t WC)
If WC is an upper-case letter, `towlower' returns the corresponding
lower-case letter. If WC is not an upper-case letter, WC is
returned unchanged.
`towlower' can be implemented using
towctrans (wc, wctrans ("tolower"))
This function is declared in `wctype.h'.
-- Function: wint_t towupper (wint_t WC)
If WC is a lower-case letter, `towupper' returns the corresponding
upper-case letter. Otherwise WC is returned unchanged.
`towupper' can be implemented using
towctrans (wc, wctrans ("toupper"))
This function is declared in `wctype.h'.
The same warnings given in the last section for the use of the wide
character classification functions apply here. It is not possible to
simply cast a `char' type value to a `wint_t' and use it as an argument
to `towctrans' calls.
File: libc.info, Node: String and Array Utilities, Next: Character Set Handling, Prev: Character Handling, Up: Top
5 String and Array Utilities
****************************
Operations on strings (or arrays of characters) are an important part of
many programs. The GNU C library provides an extensive set of string
utility functions, including functions for copying, concatenating,
comparing, and searching strings. Many of these functions can also
operate on arbitrary regions of storage; for example, the `memcpy'
function can be used to copy the contents of any kind of array.
It's fairly common for beginning C programmers to "reinvent the
wheel" by duplicating this functionality in their own code, but it pays
to become familiar with the library functions and to make use of them,
since this offers benefits in maintenance, efficiency, and portability.
For instance, you could easily compare one string to another in two
lines of C code, but if you use the built-in `strcmp' function, you're
less likely to make a mistake. And, since these library functions are
typically highly optimized, your program may run faster too.
* Menu:
* Representation of Strings:: Introduction to basic concepts.
* String/Array Conventions:: Whether to use a string function or an
arbitrary array function.
* String Length:: Determining the length of a string.
* Copying and Concatenation:: Functions to copy the contents of strings
and arrays.
* String/Array Comparison:: Functions for byte-wise and character-wise
comparison.
* Collation Functions:: Functions for collating strings.
* Search Functions:: Searching for a specific element or substring.
* Finding Tokens in a String:: Splitting a string into tokens by looking
for delimiters.
* strfry:: Function for flash-cooking a string.
* Trivial Encryption:: Obscuring data.
* Encode Binary Data:: Encoding and Decoding of Binary Data.
* Argz and Envz Vectors:: Null-separated string vectors.
File: libc.info, Node: Representation of Strings, Next: String/Array Conventions, Up: String and Array Utilities
5.1 Representation of Strings
=============================
This section is a quick summary of string concepts for beginning C
programmers. It describes how character strings are represented in C
and some common pitfalls. If you are already familiar with this
material, you can skip this section.
A "string" is an array of `char' objects. But string-valued
variables are usually declared to be pointers of type `char *'. Such
variables do not include space for the text of a string; that has to be
stored somewhere else--in an array variable, a string constant, or
dynamically allocated memory (*note Memory Allocation::). It's up to
you to store the address of the chosen memory space into the pointer
variable. Alternatively you can store a "null pointer" in the pointer
variable. The null pointer does not point anywhere, so attempting to
reference the string it points to gets an error.
"string" normally refers to multibyte character strings as opposed to
wide character strings. Wide character strings are arrays of type
`wchar_t' and as for multibyte character strings usually pointers of
type `wchar_t *' are used.
By convention, a "null character", `'\0'', marks the end of a
multibyte character string and the "null wide character", `L'\0'',
marks the end of a wide character string. For example, in testing to
see whether the `char *' variable P points to a null character marking
the end of a string, you can write `!*P' or `*P == '\0''.
A null character is quite different conceptually from a null pointer,
although both are represented by the integer `0'.
"String literals" appear in C program source as strings of
characters between double-quote characters (`"') where the initial
double-quote character is immediately preceded by a capital `L' (ell)
character (as in `L"foo"'). In ISO C, string literals can also be
formed by "string concatenation": `"a" "b"' is the same as `"ab"'. For
wide character strings one can either use `L"a" L"b"' or `L"a" "b"'.
Modification of string literals is not allowed by the GNU C compiler,
because literals are placed in read-only storage.
Character arrays that are declared `const' cannot be modified
either. It's generally good style to declare non-modifiable string
pointers to be of type `const char *', since this often allows the C
compiler to detect accidental modifications as well as providing some
amount of documentation about what your program intends to do with the
string.
The amount of memory allocated for the character array may extend
past the null character that normally marks the end of the string. In
this document, the term "allocated size" is always used to refer to the
total amount of memory allocated for the string, while the term
"length" refers to the number of characters up to (but not including)
the terminating null character.
A notorious source of program bugs is trying to put more characters
in a string than fit in its allocated size. When writing code that
extends strings or moves characters into a pre-allocated array, you
should be very careful to keep track of the length of the text and make
explicit checks for overflowing the array. Many of the library
functions _do not_ do this for you! Remember also that you need to
allocate an extra byte to hold the null character that marks the end of
the string.
Originally strings were sequences of bytes where each byte
represents a single character. This is still true today if the strings
are encoded using a single-byte character encoding. Things are
different if the strings are encoded using a multibyte encoding (for
more information on encodings see *Note Extended Char Intro::). There
is no difference in the programming interface for these two kind of
strings; the programmer has to be aware of this and interpret the byte
sequences accordingly.
But since there is no separate interface taking care of these
differences the byte-based string functions are sometimes hard to use.
Since the count parameters of these functions specify bytes a call to
`strncpy' could cut a multibyte character in the middle and put an
incomplete (and therefore unusable) byte sequence in the target buffer.
To avoid these problems later versions of the ISO C standard
introduce a second set of functions which are operating on "wide
characters" (*note Extended Char Intro::). These functions don't have
the problems the single-byte versions have since every wide character is
a legal, interpretable value. This does not mean that cutting wide
character strings at arbitrary points is without problems. It normally
is for alphabet-based languages (except for non-normalized text) but
languages based on syllables still have the problem that more than one
wide character is necessary to complete a logical unit. This is a
higher level problem which the C library functions are not designed to
solve. But it is at least good that no invalid byte sequences can be
created. Also, the higher level functions can also much easier operate
on wide character than on multibyte characters so that a general advise
is to use wide characters internally whenever text is more than simply
copied.
The remaining of this chapter will discuss the functions for handling
wide character strings in parallel with the discussion of the multibyte
character strings since there is almost always an exact equivalent
available.
File: libc.info, Node: String/Array Conventions, Next: String Length, Prev: Representation of Strings, Up: String and Array Utilities
5.2 String and Array Conventions
================================
This chapter describes both functions that work on arbitrary arrays or
blocks of memory, and functions that are specific to null-terminated
arrays of characters and wide characters.
Functions that operate on arbitrary blocks of memory have names
beginning with `mem' and `wmem' (such as `memcpy' and `wmemcpy') and
invariably take an argument which specifies the size (in bytes and wide
characters respectively) of the block of memory to operate on. The
array arguments and return values for these functions have type `void
*' or `wchar_t'. As a matter of style, the elements of the arrays used
with the `mem' functions are referred to as "bytes". You can pass any
kind of pointer to these functions, and the `sizeof' operator is useful
in computing the value for the size argument. Parameters to the `wmem'
functions must be of type `wchar_t *'. These functions are not really
usable with anything but arrays of this type.
In contrast, functions that operate specifically on strings and wide
character strings have names beginning with `str' and `wcs'
respectively (such as `strcpy' and `wcscpy') and look for a null
character to terminate the string instead of requiring an explicit size
argument to be passed. (Some of these functions accept a specified
maximum length, but they also check for premature termination with a
null character.) The array arguments and return values for these
functions have type `char *' and `wchar_t *' respectively, and the
array elements are referred to as "characters" and "wide characters".
In many cases, there are both `mem' and `str'/`wcs' versions of a
function. The one that is more appropriate to use depends on the exact
situation. When your program is manipulating arbitrary arrays or
blocks of storage, then you should always use the `mem' functions. On
the other hand, when you are manipulating null-terminated strings it is
usually more convenient to use the `str'/`wcs' functions, unless you
already know the length of the string in advance. The `wmem' functions
should be used for wide character arrays with known size.
Some of the memory and string functions take single characters as
arguments. Since a value of type `char' is automatically promoted into
an value of type `int' when used as a parameter, the functions are
declared with `int' as the type of the parameter in question. In case
of the wide character function the situation is similarly: the
parameter type for a single wide character is `wint_t' and not
`wchar_t'. This would for many implementations not be necessary since
the `wchar_t' is large enough to not be automatically promoted, but
since the ISO C standard does not require such a choice of types the
`wint_t' type is used.
File: libc.info, Node: String Length, Next: Copying and Concatenation, Prev: String/Array Conventions, Up: String and Array Utilities
5.3 String Length
=================
You can get the length of a string using the `strlen' function. This
function is declared in the header file `string.h'.
-- Function: size_t strlen (const char *S)
The `strlen' function returns the length of the null-terminated
string S in bytes. (In other words, it returns the offset of the
terminating null character within the array.)
For example,
strlen ("hello, world")
=> 12
When applied to a character array, the `strlen' function returns
the length of the string stored there, not its allocated size.
You can get the allocated size of the character array that holds a
string using the `sizeof' operator:
char string[32] = "hello, world";
sizeof (string)
=> 32
strlen (string)
=> 12
But beware, this will not work unless STRING is the character
array itself, not a pointer to it. For example:
char string[32] = "hello, world";
char *ptr = string;
sizeof (string)
=> 32
sizeof (ptr)
=> 4 /* (on a machine with 4 byte pointers) */
This is an easy mistake to make when you are working with
functions that take string arguments; those arguments are always
pointers, not arrays.
It must also be noted that for multibyte encoded strings the return
value does not have to correspond to the number of characters in
the string. To get this value the string can be converted to wide
characters and `wcslen' can be used or something like the following
code can be used:
/* The input is in `string'.
The length is expected in `n'. */
{
mbstate_t t;
char *scopy = string;
/* In initial state. */
memset (&t, '\0', sizeof (t));
/* Determine number of characters. */
n = mbsrtowcs (NULL, &scopy, strlen (scopy), &t);
}
This is cumbersome to do so if the number of characters (as
opposed to bytes) is needed often it is better to work with wide
characters.
The wide character equivalent is declared in `wchar.h'.
-- Function: size_t wcslen (const wchar_t *WS)
The `wcslen' function is the wide character equivalent to
`strlen'. The return value is the number of wide characters in the
wide character string pointed to by WS (this is also the offset of
the terminating null wide character of WS).
Since there are no multi wide character sequences making up one
character the return value is not only the offset in the array, it
is also the number of wide characters.
This function was introduced in Amendment 1 to ISO C90.
-- Function: size_t strnlen (const char *S, size_t MAXLEN)
The `strnlen' function returns the length of the string S in bytes
if this length is smaller than MAXLEN bytes. Otherwise it returns
MAXLEN. Therefore this function is equivalent to `(strlen (S) < n
? strlen (S) : MAXLEN)' but it is more efficient and works even if
the string S is not null-terminated.
char string[32] = "hello, world";
strnlen (string, 32)
=> 12
strnlen (string, 5)
=> 5
This function is a GNU extension and is declared in `string.h'.
-- Function: size_t wcsnlen (const wchar_t *WS, size_t MAXLEN)
`wcsnlen' is the wide character equivalent to `strnlen'. The
MAXLEN parameter specifies the maximum number of wide characters.
This function is a GNU extension and is declared in `wchar.h'.
File: libc.info, Node: Copying and Concatenation, Next: String/Array Comparison, Prev: String Length, Up: String and Array Utilities
5.4 Copying and Concatenation
=============================
You can use the functions described in this section to copy the contents
of strings and arrays, or to append the contents of one string to
another. The `str' and `mem' functions are declared in the header file
`string.h' while the `wstr' and `wmem' functions are declared in the
file `wchar.h'.
A helpful way to remember the ordering of the arguments to the
functions in this section is that it corresponds to an assignment
expression, with the destination array specified to the left of the
source array. All of these functions return the address of the
destination array.
Most of these functions do not work properly if the source and
destination arrays overlap. For example, if the beginning of the
destination array overlaps the end of the source array, the original
contents of that part of the source array may get overwritten before it
is copied. Even worse, in the case of the string functions, the null
character marking the end of the string may be lost, and the copy
function might get stuck in a loop trashing all the memory allocated to
your program.
All functions that have problems copying between overlapping arrays
are explicitly identified in this manual. In addition to functions in
this section, there are a few others like `sprintf' (*note Formatted
Output Functions::) and `scanf' (*note Formatted Input Functions::).
-- Function: void * memcpy (void *restrict TO, const void *restrict
FROM, size_t SIZE)
The `memcpy' function copies SIZE bytes from the object beginning
at FROM into the object beginning at TO. The behavior of this
function is undefined if the two arrays TO and FROM overlap; use
`memmove' instead if overlapping is possible.
The value returned by `memcpy' is the value of TO.
Here is an example of how you might use `memcpy' to copy the
contents of an array:
struct foo *oldarray, *newarray;
int arraysize;
...
memcpy (new, old, arraysize * sizeof (struct foo));
-- Function: wchar_t * wmemcpy (wchar_t *restrict WTO, const wchar_t
*restrict WFROM, size_t SIZE)
The `wmemcpy' function copies SIZE wide characters from the object
beginning at WFROM into the object beginning at WTO. The behavior
of this function is undefined if the two arrays WTO and WFROM
overlap; use `wmemmove' instead if overlapping is possible.
The following is a possible implementation of `wmemcpy' but there
are more optimizations possible.
wchar_t *
wmemcpy (wchar_t *restrict wto, const wchar_t *restrict wfrom,
size_t size)
{
return (wchar_t *) memcpy (wto, wfrom, size * sizeof (wchar_t));
}
The value returned by `wmemcpy' is the value of WTO.
This function was introduced in Amendment 1 to ISO C90.
-- Function: void * mempcpy (void *restrict TO, const void *restrict
FROM, size_t SIZE)
The `mempcpy' function is nearly identical to the `memcpy'
function. It copies SIZE bytes from the object beginning at
`from' into the object pointed to by TO. But instead of returning
the value of TO it returns a pointer to the byte following the
last written byte in the object beginning at TO. I.e., the value
is `((void *) ((char *) TO + SIZE))'.
This function is useful in situations where a number of objects
shall be copied to consecutive memory positions.
void *
combine (void *o1, size_t s1, void *o2, size_t s2)
{
void *result = malloc (s1 + s2);
if (result != NULL)
mempcpy (mempcpy (result, o1, s1), o2, s2);
return result;
}
This function is a GNU extension.
-- Function: wchar_t * wmempcpy (wchar_t *restrict WTO, const wchar_t
*restrict WFROM, size_t SIZE)
The `wmempcpy' function is nearly identical to the `wmemcpy'
function. It copies SIZE wide characters from the object
beginning at `wfrom' into the object pointed to by WTO. But
instead of returning the value of WTO it returns a pointer to the
wide character following the last written wide character in the
object beginning at WTO. I.e., the value is `WTO + SIZE'.
This function is useful in situations where a number of objects
shall be copied to consecutive memory positions.
The following is a possible implementation of `wmemcpy' but there
are more optimizations possible.
wchar_t *
wmempcpy (wchar_t *restrict wto, const wchar_t *restrict wfrom,
size_t size)
{
return (wchar_t *) mempcpy (wto, wfrom, size * sizeof (wchar_t));
}
This function is a GNU extension.
-- Function: void * memmove (void *TO, const void *FROM, size_t SIZE)
`memmove' copies the SIZE bytes at FROM into the SIZE bytes at TO,
even if those two blocks of space overlap. In the case of
overlap, `memmove' is careful to copy the original values of the
bytes in the block at FROM, including those bytes which also
belong to the block at TO.
The value returned by `memmove' is the value of TO.
-- Function: wchar_t * wmemmove (wchar *WTO, const wchar_t *WFROM,
size_t SIZE)
`wmemmove' copies the SIZE wide characters at WFROM into the SIZE
wide characters at WTO, even if those two blocks of space overlap.
In the case of overlap, `memmove' is careful to copy the original
values of the wide characters in the block at WFROM, including
those wide characters which also belong to the block at WTO.
The following is a possible implementation of `wmemcpy' but there
are more optimizations possible.
wchar_t *
wmempcpy (wchar_t *restrict wto, const wchar_t *restrict wfrom,
size_t size)
{
return (wchar_t *) mempcpy (wto, wfrom, size * sizeof (wchar_t));
}
The value returned by `wmemmove' is the value of WTO.
This function is a GNU extension.
-- Function: void * memccpy (void *restrict TO, const void *restrict
FROM, int C, size_t SIZE)
This function copies no more than SIZE bytes from FROM to TO,
stopping if a byte matching C is found. The return value is a
pointer into TO one byte past where C was copied, or a null
pointer if no byte matching C appeared in the first SIZE bytes of
FROM.
-- Function: void * memset (void *BLOCK, int C, size_t SIZE)
This function copies the value of C (converted to an `unsigned
char') into each of the first SIZE bytes of the object beginning
at BLOCK. It returns the value of BLOCK.
-- Function: wchar_t * wmemset (wchar_t *BLOCK, wchar_t WC, size_t
SIZE)
This function copies the value of WC into each of the first SIZE
wide characters of the object beginning at BLOCK. It returns the
value of BLOCK.
-- Function: char * strcpy (char *restrict TO, const char *restrict
FROM)
This copies characters from the string FROM (up to and including
the terminating null character) into the string TO. Like
`memcpy', this function has undefined results if the strings
overlap. The return value is the value of TO.
-- Function: wchar_t * wcscpy (wchar_t *restrict WTO, const wchar_t
*restrict WFROM)
This copies wide characters from the string WFROM (up to and
including the terminating null wide character) into the string
WTO. Like `wmemcpy', this function has undefined results if the
strings overlap. The return value is the value of WTO.
-- Function: char * strncpy (char *restrict TO, const char *restrict
FROM, size_t SIZE)
This function is similar to `strcpy' but always copies exactly
SIZE characters into TO.
If the length of FROM is more than SIZE, then `strncpy' copies
just the first SIZE characters. Note that in this case there is
no null terminator written into TO.
If the length of FROM is less than SIZE, then `strncpy' copies all
of FROM, followed by enough null characters to add up to SIZE
characters in all. This behavior is rarely useful, but it is
specified by the ISO C standard.
The behavior of `strncpy' is undefined if the strings overlap.
Using `strncpy' as opposed to `strcpy' is a way to avoid bugs
relating to writing past the end of the allocated space for TO.
However, it can also make your program much slower in one common
case: copying a string which is probably small into a potentially
large buffer. In this case, SIZE may be large, and when it is,
`strncpy' will waste a considerable amount of time copying null
characters.
-- Function: wchar_t * wcsncpy (wchar_t *restrict WTO, const wchar_t
*restrict WFROM, size_t SIZE)
This function is similar to `wcscpy' but always copies exactly
SIZE wide characters into WTO.
If the length of WFROM is more than SIZE, then `wcsncpy' copies
just the first SIZE wide characters. Note that in this case there
is no null terminator written into WTO.
If the length of WFROM is less than SIZE, then `wcsncpy' copies
all of WFROM, followed by enough null wide characters to add up to
SIZE wide characters in all. This behavior is rarely useful, but
it is specified by the ISO C standard.
The behavior of `wcsncpy' is undefined if the strings overlap.
Using `wcsncpy' as opposed to `wcscpy' is a way to avoid bugs
relating to writing past the end of the allocated space for WTO.
However, it can also make your program much slower in one common
case: copying a string which is probably small into a potentially
large buffer. In this case, SIZE may be large, and when it is,
`wcsncpy' will waste a considerable amount of time copying null
wide characters.
-- Function: char * strdup (const char *S)
This function copies the null-terminated string S into a newly
allocated string. The string is allocated using `malloc'; see
*Note Unconstrained Allocation::. If `malloc' cannot allocate
space for the new string, `strdup' returns a null pointer.
Otherwise it returns a pointer to the new string.
-- Function: wchar_t * wcsdup (const wchar_t *WS)
This function copies the null-terminated wide character string WS
into a newly allocated string. The string is allocated using
`malloc'; see *Note Unconstrained Allocation::. If `malloc'
cannot allocate space for the new string, `wcsdup' returns a null
pointer. Otherwise it returns a pointer to the new wide character
string.
This function is a GNU extension.
-- Function: char * strndup (const char *S, size_t SIZE)
This function is similar to `strdup' but always copies at most
SIZE characters into the newly allocated string.
If the length of S is more than SIZE, then `strndup' copies just
the first SIZE characters and adds a closing null terminator.
Otherwise all characters are copied and the string is terminated.
This function is different to `strncpy' in that it always
terminates the destination string.
`strndup' is a GNU extension.
-- Function: char * stpcpy (char *restrict TO, const char *restrict
FROM)
This function is like `strcpy', except that it returns a pointer to
the end of the string TO (that is, the address of the terminating
null character `to + strlen (from)') rather than the beginning.
For example, this program uses `stpcpy' to concatenate `foo' and
`bar' to produce `foobar', which it then prints.
#include <string.h>
#include <stdio.h>
int
main (void)
{
char buffer[10];
char *to = buffer;
to = stpcpy (to, "foo");
to = stpcpy (to, "bar");
puts (buffer);
return 0;
}
This function is not part of the ISO or POSIX standards, and is not
customary on Unix systems, but we did not invent it either.
Perhaps it comes from MS-DOG.
Its behavior is undefined if the strings overlap. The function is
declared in `string.h'.
-- Function: wchar_t * wcpcpy (wchar_t *restrict WTO, const wchar_t
*restrict WFROM)
This function is like `wcscpy', except that it returns a pointer to
the end of the string WTO (that is, the address of the terminating
null character `wto + strlen (wfrom)') rather than the beginning.
This function is not part of ISO or POSIX but was found useful
while developing the GNU C Library itself.
The behavior of `wcpcpy' is undefined if the strings overlap.
`wcpcpy' is a GNU extension and is declared in `wchar.h'.
-- Function: char * stpncpy (char *restrict TO, const char *restrict
FROM, size_t SIZE)
This function is similar to `stpcpy' but copies always exactly
SIZE characters into TO.
If the length of FROM is more then SIZE, then `stpncpy' copies
just the first SIZE characters and returns a pointer to the
character directly following the one which was copied last. Note
that in this case there is no null terminator written into TO.
If the length of FROM is less than SIZE, then `stpncpy' copies all
of FROM, followed by enough null characters to add up to SIZE
characters in all. This behavior is rarely useful, but it is
implemented to be useful in contexts where this behavior of the
`strncpy' is used. `stpncpy' returns a pointer to the _first_
written null character.
This function is not part of ISO or POSIX but was found useful
while developing the GNU C Library itself.
Its behavior is undefined if the strings overlap. The function is
declared in `string.h'.
-- Function: wchar_t * wcpncpy (wchar_t *restrict WTO, const wchar_t
*restrict WFROM, size_t SIZE)
This function is similar to `wcpcpy' but copies always exactly
WSIZE characters into WTO.
If the length of WFROM is more then SIZE, then `wcpncpy' copies
just the first SIZE wide characters and returns a pointer to the
wide character directly following the last non-null wide character
which was copied last. Note that in this case there is no null
terminator written into WTO.
If the length of WFROM is less than SIZE, then `wcpncpy' copies
all of WFROM, followed by enough null characters to add up to SIZE
characters in all. This behavior is rarely useful, but it is
implemented to be useful in contexts where this behavior of the
`wcsncpy' is used. `wcpncpy' returns a pointer to the _first_
written null character.
This function is not part of ISO or POSIX but was found useful
while developing the GNU C Library itself.
Its behavior is undefined if the strings overlap.
`wcpncpy' is a GNU extension and is declared in `wchar.h'.
-- Macro: char * strdupa (const char *S)
This macro is similar to `strdup' but allocates the new string
using `alloca' instead of `malloc' (*note Variable Size
Automatic::). This means of course the returned string has the
same limitations as any block of memory allocated using `alloca'.
For obvious reasons `strdupa' is implemented only as a macro; you
cannot get the address of this function. Despite this limitation
it is a useful function. The following code shows a situation
where using `malloc' would be a lot more expensive.
#include <paths.h>
#include <string.h>
#include <stdio.h>
const char path[] = _PATH_STDPATH;
int
main (void)
{
char *wr_path = strdupa (path);
char *cp = strtok (wr_path, ":");
while (cp != NULL)
{
puts (cp);
cp = strtok (NULL, ":");
}
return 0;
}
Please note that calling `strtok' using PATH directly is invalid.
It is also not allowed to call `strdupa' in the argument list of
`strtok' since `strdupa' uses `alloca' (*note Variable Size
Automatic::) can interfere with the parameter passing.
This function is only available if GNU CC is used.
-- Macro: char * strndupa (const char *S, size_t SIZE)
This function is similar to `strndup' but like `strdupa' it
allocates the new string using `alloca' *note Variable Size
Automatic::. The same advantages and limitations of `strdupa' are
valid for `strndupa', too.
This function is implemented only as a macro, just like `strdupa'.
Just as `strdupa' this macro also must not be used inside the
parameter list in a function call.
`strndupa' is only available if GNU CC is used.
-- Function: char * strcat (char *restrict TO, const char *restrict
FROM)
The `strcat' function is similar to `strcpy', except that the
characters from FROM are concatenated or appended to the end of
TO, instead of overwriting it. That is, the first character from
FROM overwrites the null character marking the end of TO.
An equivalent definition for `strcat' would be:
char *
strcat (char *restrict to, const char *restrict from)
{
strcpy (to + strlen (to), from);
return to;
}
This function has undefined results if the strings overlap.
-- Function: wchar_t * wcscat (wchar_t *restrict WTO, const wchar_t
*restrict WFROM)
The `wcscat' function is similar to `wcscpy', except that the
characters from WFROM are concatenated or appended to the end of
WTO, instead of overwriting it. That is, the first character from
WFROM overwrites the null character marking the end of WTO.
An equivalent definition for `wcscat' would be:
wchar_t *
wcscat (wchar_t *wto, const wchar_t *wfrom)
{
wcscpy (wto + wcslen (wto), wfrom);
return wto;
}
This function has undefined results if the strings overlap.
Programmers using the `strcat' or `wcscat' function (or the
following `strncat' or `wcsncar' functions for that matter) can easily
be recognized as lazy and reckless. In almost all situations the
lengths of the participating strings are known (it better should be
since how can one otherwise ensure the allocated size of the buffer is
sufficient?) Or at least, one could know them if one keeps track of the
results of the various function calls. But then it is very inefficient
to use `strcat'/`wcscat'. A lot of time is wasted finding the end of
the destination string so that the actual copying can start. This is a
common example:
/* This function concatenates arbitrarily many strings. The last
parameter must be `NULL'. */
char *
concat (const char *str, ...)
{
va_list ap, ap2;
size_t total = 1;
const char *s;
char *result;
va_start (ap, str);
/* Actually `va_copy', but this is the name more gcc versions
understand. */
__va_copy (ap2, ap);
/* Determine how much space we need. */
for (s = str; s != NULL; s = va_arg (ap, const char *))
total += strlen (s);
va_end (ap);
result = (char *) malloc (total);
if (result != NULL)
{
result[0] = '\0';
/* Copy the strings. */
for (s = str; s != NULL; s = va_arg (ap2, const char *))
strcat (result, s);
}
va_end (ap2);
return result;
}
This looks quite simple, especially the second loop where the strings
are actually copied. But these innocent lines hide a major performance
penalty. Just imagine that ten strings of 100 bytes each have to be
concatenated. For the second string we search the already stored 100
bytes for the end of the string so that we can append the next string.
For all strings in total the comparisons necessary to find the end of
the intermediate results sums up to 5500! If we combine the copying
with the search for the allocation we can write this function more
efficient:
char *
concat (const char *str, ...)
{
va_list ap;
size_t allocated = 100;
char *result = (char *) malloc (allocated);
if (result != NULL)
{
char *newp;
char *wp;
va_start (ap, str);
wp = result;
for (s = str; s != NULL; s = va_arg (ap, const char *))
{
size_t len = strlen (s);
/* Resize the allocated memory if necessary. */
if (wp + len + 1 > result + allocated)
{
allocated = (allocated + len) * 2;
newp = (char *) realloc (result, allocated);
if (newp == NULL)
{
free (result);
return NULL;
}
wp = newp + (wp - result);
result = newp;
}
wp = mempcpy (wp, s, len);
}
/* Terminate the result string. */
*wp++ = '\0';
/* Resize memory to the optimal size. */
newp = realloc (result, wp - result);
if (newp != NULL)
result = newp;
va_end (ap);
}
return result;
}
With a bit more knowledge about the input strings one could fine-tune
the memory allocation. The difference we are pointing to here is that
we don't use `strcat' anymore. We always keep track of the length of
the current intermediate result so we can safe us the search for the
end of the string and use `mempcpy'. Please note that we also don't
use `stpcpy' which might seem more natural since we handle with
strings. But this is not necessary since we already know the length of
the string and therefore can use the faster memory copying function.
The example would work for wide characters the same way.
Whenever a programmer feels the need to use `strcat' she or he
should think twice and look through the program whether the code cannot
be rewritten to take advantage of already calculated results. Again: it
is almost always unnecessary to use `strcat'.
-- Function: char * strncat (char *restrict TO, const char *restrict
FROM, size_t SIZE)
This function is like `strcat' except that not more than SIZE
characters from FROM are appended to the end of TO. A single null
character is also always appended to TO, so the total allocated
size of TO must be at least `SIZE + 1' bytes longer than its
initial length.
The `strncat' function could be implemented like this:
char *
strncat (char *to, const char *from, size_t size)
{
to[strlen (to) + size] = '\0';
strncpy (to + strlen (to), from, size);
return to;
}
The behavior of `strncat' is undefined if the strings overlap.
-- Function: wchar_t * wcsncat (wchar_t *restrict WTO, const wchar_t
*restrict WFROM, size_t SIZE)
This function is like `wcscat' except that not more than SIZE
characters from FROM are appended to the end of TO. A single null
character is also always appended to TO, so the total allocated
size of TO must be at least `SIZE + 1' bytes longer than its
initial length.
The `wcsncat' function could be implemented like this:
wchar_t *
wcsncat (wchar_t *restrict wto, const wchar_t *restrict wfrom,
size_t size)
{
wto[wcslen (to) + size] = L'\0';
wcsncpy (wto + wcslen (wto), wfrom, size);
return wto;
}
The behavior of `wcsncat' is undefined if the strings overlap.
Here is an example showing the use of `strncpy' and `strncat' (the
wide character version is equivalent). Notice how, in the call to
`strncat', the SIZE parameter is computed to avoid overflowing the
character array `buffer'.
#include <string.h>
#include <stdio.h>
#define SIZE 10
static char buffer[SIZE];
main ()
{
strncpy (buffer, "hello", SIZE);
puts (buffer);
strncat (buffer, ", world", SIZE - strlen (buffer) - 1);
puts (buffer);
}
The output produced by this program looks like:
hello
hello, wo
-- Function: void bcopy (const void *FROM, void *TO, size_t SIZE)
This is a partially obsolete alternative for `memmove', derived
from BSD. Note that it is not quite equivalent to `memmove',
because the arguments are not in the same order and there is no
return value.
-- Function: void bzero (void *BLOCK, size_t SIZE)
This is a partially obsolete alternative for `memset', derived from
BSD. Note that it is not as general as `memset', because the only
value it can store is zero.
File: libc.info, Node: String/Array Comparison, Next: Collation Functions, Prev: Copying and Concatenation, Up: String and Array Utilities
5.5 String/Array Comparison
===========================
You can use the functions in this section to perform comparisons on the
contents of strings and arrays. As well as checking for equality, these
functions can also be used as the ordering functions for sorting
operations. *Note Searching and Sorting::, for an example of this.
Unlike most comparison operations in C, the string comparison
functions return a nonzero value if the strings are _not_ equivalent
rather than if they are. The sign of the value indicates the relative
ordering of the first characters in the strings that are not
equivalent: a negative value indicates that the first string is "less"
than the second, while a positive value indicates that the first string
is "greater".
The most common use of these functions is to check only for equality.
This is canonically done with an expression like `! strcmp (s1, s2)'.
All of these functions are declared in the header file `string.h'.
-- Function: int memcmp (const void *A1, const void *A2, size_t SIZE)
The function `memcmp' compares the SIZE bytes of memory beginning
at A1 against the SIZE bytes of memory beginning at A2. The value
returned has the same sign as the difference between the first
differing pair of bytes (interpreted as `unsigned char' objects,
then promoted to `int').
If the contents of the two blocks are equal, `memcmp' returns `0'.
-- Function: int wmemcmp (const wchar_t *A1, const wchar_t *A2, size_t
SIZE)
The function `wmemcmp' compares the SIZE wide characters beginning
at A1 against the SIZE wide characters beginning at A2. The value
returned is smaller than or larger than zero depending on whether
the first differing wide character is A1 is smaller or larger than
the corresponding character in A2.
If the contents of the two blocks are equal, `wmemcmp' returns `0'.
On arbitrary arrays, the `memcmp' function is mostly useful for
testing equality. It usually isn't meaningful to do byte-wise ordering
comparisons on arrays of things other than bytes. For example, a
byte-wise comparison on the bytes that make up floating-point numbers
isn't likely to tell you anything about the relationship between the
values of the floating-point numbers.
`wmemcmp' is really only useful to compare arrays of type `wchar_t'
since the function looks at `sizeof (wchar_t)' bytes at a time and this
number of bytes is system dependent.
You should also be careful about using `memcmp' to compare objects
that can contain "holes", such as the padding inserted into structure
objects to enforce alignment requirements, extra space at the end of
unions, and extra characters at the ends of strings whose length is less
than their allocated size. The contents of these "holes" are
indeterminate and may cause strange behavior when performing byte-wise
comparisons. For more predictable results, perform an explicit
component-wise comparison.
For example, given a structure type definition like:
struct foo
{
unsigned char tag;
union
{
double f;
long i;
char *p;
} value;
};
you are better off writing a specialized comparison function to compare
`struct foo' objects instead of comparing them with `memcmp'.
-- Function: int strcmp (const char *S1, const char *S2)
The `strcmp' function compares the string S1 against S2, returning
a value that has the same sign as the difference between the first
differing pair of characters (interpreted as `unsigned char'
objects, then promoted to `int').
If the two strings are equal, `strcmp' returns `0'.
A consequence of the ordering used by `strcmp' is that if S1 is an
initial substring of S2, then S1 is considered to be "less than"
S2.
`strcmp' does not take sorting conventions of the language the
strings are written in into account. To get that one has to use
`strcoll'.
-- Function: int wcscmp (const wchar_t *WS1, const wchar_t *WS2)
The `wcscmp' function compares the wide character string WS1
against WS2. The value returned is smaller than or larger than
zero depending on whether the first differing wide character is
WS1 is smaller or larger than the corresponding character in WS2.
If the two strings are equal, `wcscmp' returns `0'.
A consequence of the ordering used by `wcscmp' is that if WS1 is
an initial substring of WS2, then WS1 is considered to be "less
than" WS2.
`wcscmp' does not take sorting conventions of the language the
strings are written in into account. To get that one has to use
`wcscoll'.
-- Function: int strcasecmp (const char *S1, const char *S2)
This function is like `strcmp', except that differences in case are
ignored. How uppercase and lowercase characters are related is
determined by the currently selected locale. In the standard `"C"'
locale the characters A" and a" do not match but in a locale which
regards these characters as parts of the alphabet they do match.
`strcasecmp' is derived from BSD.
-- Function: int wcscasecmp (const wchar_t *WS1, const wchar_T *WS2)
This function is like `wcscmp', except that differences in case are
ignored. How uppercase and lowercase characters are related is
determined by the currently selected locale. In the standard `"C"'
locale the characters A" and a" do not match but in a locale which
regards these characters as parts of the alphabet they do match.
`wcscasecmp' is a GNU extension.
-- Function: int strncmp (const char *S1, const char *S2, size_t SIZE)
This function is the similar to `strcmp', except that no more than
SIZE wide characters are compared. In other words, if the two
strings are the same in their first SIZE wide characters, the
return value is zero.
-- Function: int wcsncmp (const wchar_t *WS1, const wchar_t *WS2,
size_t SIZE)
This function is the similar to `wcscmp', except that no more than
SIZE wide characters are compared. In other words, if the two
strings are the same in their first SIZE wide characters, the
return value is zero.
-- Function: int strncasecmp (const char *S1, const char *S2, size_t N)
This function is like `strncmp', except that differences in case
are ignored. Like `strcasecmp', it is locale dependent how
uppercase and lowercase characters are related.
`strncasecmp' is a GNU extension.
-- Function: int wcsncasecmp (const wchar_t *WS1, const wchar_t *S2,
size_t N)
This function is like `wcsncmp', except that differences in case
are ignored. Like `wcscasecmp', it is locale dependent how
uppercase and lowercase characters are related.
`wcsncasecmp' is a GNU extension.
Here are some examples showing the use of `strcmp' and `strncmp'
(equivalent examples can be constructed for the wide character
functions). These examples assume the use of the ASCII character set.
(If some other character set--say, EBCDIC--is used instead, then the
glyphs are associated with different numeric codes, and the return
values and ordering may differ.)
strcmp ("hello", "hello")
=> 0 /* These two strings are the same. */
strcmp ("hello", "Hello")
=> 32 /* Comparisons are case-sensitive. */
strcmp ("hello", "world")
=> -15 /* The character `'h'' comes before `'w''. */
strcmp ("hello", "hello, world")
=> -44 /* Comparing a null character against a comma. */
strncmp ("hello", "hello, world", 5)
=> 0 /* The initial 5 characters are the same. */
strncmp ("hello, world", "hello, stupid world!!!", 5)
=> 0 /* The initial 5 characters are the same. */
-- Function: int strverscmp (const char *S1, const char *S2)
The `strverscmp' function compares the string S1 against S2,
considering them as holding indices/version numbers. Return value
follows the same conventions as found in the `strverscmp'
function. In fact, if S1 and S2 contain no digits, `strverscmp'
behaves like `strcmp'.
Basically, we compare strings normally (character by character),
until we find a digit in each string - then we enter a special
comparison mode, where each sequence of digits is taken as a
whole. If we reach the end of these two parts without noticing a
difference, we return to the standard comparison mode. There are
two types of numeric parts: "integral" and "fractional" (those
begin with a '0'). The types of the numeric parts affect the way
we sort them:
* integral/integral: we compare values as you would expect.
* fractional/integral: the fractional part is less than the
integral one. Again, no surprise.
* fractional/fractional: the things become a bit more complex.
If the common prefix contains only leading zeroes, the
longest part is less than the other one; else the comparison
behaves normally.
strverscmp ("no digit", "no digit")
=> 0 /* same behavior as strcmp. */
strverscmp ("item#99", "item#100")
=> <0 /* same prefix, but 99 < 100. */
strverscmp ("alpha1", "alpha001")
=> >0 /* fractional part inferior to integral one. */
strverscmp ("part1_f012", "part1_f01")
=> >0 /* two fractional parts. */
strverscmp ("foo.009", "foo.0")
=> <0 /* idem, but with leading zeroes only. */
This function is especially useful when dealing with filename
sorting, because filenames frequently hold indices/version numbers.
`strverscmp' is a GNU extension.
-- Function: int bcmp (const void *A1, const void *A2, size_t SIZE)
This is an obsolete alias for `memcmp', derived from BSD.
File: libc.info, Node: Collation Functions, Next: Search Functions, Prev: String/Array Comparison, Up: String and Array Utilities
5.6 Collation Functions
=======================
In some locales, the conventions for lexicographic ordering differ from
the strict numeric ordering of character codes. For example, in Spanish
most glyphs with diacritical marks such as accents are not considered
distinct letters for the purposes of collation. On the other hand, the
two-character sequence `ll' is treated as a single letter that is
collated immediately after `l'.
You can use the functions `strcoll' and `strxfrm' (declared in the
headers file `string.h') and `wcscoll' and `wcsxfrm' (declared in the
headers file `wchar') to compare strings using a collation ordering
appropriate for the current locale. The locale used by these functions
in particular can be specified by setting the locale for the
`LC_COLLATE' category; see *Note Locales::.
In the standard C locale, the collation sequence for `strcoll' is
the same as that for `strcmp'. Similarly, `wcscoll' and `wcscmp' are
the same in this situation.
Effectively, the way these functions work is by applying a mapping to
transform the characters in a string to a byte sequence that represents
the string's position in the collating sequence of the current locale.
Comparing two such byte sequences in a simple fashion is equivalent to
comparing the strings with the locale's collating sequence.
The functions `strcoll' and `wcscoll' perform this translation
implicitly, in order to do one comparison. By contrast, `strxfrm' and
`wcsxfrm' perform the mapping explicitly. If you are making multiple
comparisons using the same string or set of strings, it is likely to be
more efficient to use `strxfrm' or `wcsxfrm' to transform all the
strings just once, and subsequently compare the transformed strings
with `strcmp' or `wcscmp'.
-- Function: int strcoll (const char *S1, const char *S2)
The `strcoll' function is similar to `strcmp' but uses the
collating sequence of the current locale for collation (the
`LC_COLLATE' locale).
-- Function: int wcscoll (const wchar_t *WS1, const wchar_t *WS2)
The `wcscoll' function is similar to `wcscmp' but uses the
collating sequence of the current locale for collation (the
`LC_COLLATE' locale).
Here is an example of sorting an array of strings, using `strcoll'
to compare them. The actual sort algorithm is not written here; it
comes from `qsort' (*note Array Sort Function::). The job of the code
shown here is to say how to compare the strings while sorting them.
(Later on in this section, we will show a way to do this more
efficiently using `strxfrm'.)
/* This is the comparison function used with `qsort'. */
int
compare_elements (char **p1, char **p2)
{
return strcoll (*p1, *p2);
}
/* This is the entry point--the function to sort
strings using the locale's collating sequence. */
void
sort_strings (char **array, int nstrings)
{
/* Sort `temp_array' by comparing the strings. */
qsort (array, nstrings,
sizeof (char *), compare_elements);
}
-- Function: size_t strxfrm (char *restrict TO, const char *restrict
FROM, size_t SIZE)
The function `strxfrm' transforms the string FROM using the
collation transformation determined by the locale currently
selected for collation, and stores the transformed string in the
array TO. Up to SIZE characters (including a terminating null
character) are stored.
The behavior is undefined if the strings TO and FROM overlap; see
*Note Copying and Concatenation::.
The return value is the length of the entire transformed string.
This value is not affected by the value of SIZE, but if it is
greater or equal than SIZE, it means that the transformed string
did not entirely fit in the array TO. In this case, only as much
of the string as actually fits was stored. To get the whole
transformed string, call `strxfrm' again with a bigger output
array.
The transformed string may be longer than the original string, and
it may also be shorter.
If SIZE is zero, no characters are stored in TO. In this case,
`strxfrm' simply returns the number of characters that would be
the length of the transformed string. This is useful for
determining what size the allocated array should be. It does not
matter what TO is if SIZE is zero; TO may even be a null pointer.
-- Function: size_t wcsxfrm (wchar_t *restrict WTO, const wchar_t
*WFROM, size_t SIZE)
The function `wcsxfrm' transforms wide character string WFROM
using the collation transformation determined by the locale
currently selected for collation, and stores the transformed
string in the array WTO. Up to SIZE wide characters (including a
terminating null character) are stored.
The behavior is undefined if the strings WTO and WFROM overlap;
see *Note Copying and Concatenation::.
The return value is the length of the entire transformed wide
character string. This value is not affected by the value of
SIZE, but if it is greater or equal than SIZE, it means that the
transformed wide character string did not entirely fit in the
array WTO. In this case, only as much of the wide character
string as actually fits was stored. To get the whole transformed
wide character string, call `wcsxfrm' again with a bigger output
array.
The transformed wide character string may be longer than the
original wide character string, and it may also be shorter.
If SIZE is zero, no characters are stored in TO. In this case,
`wcsxfrm' simply returns the number of wide characters that would
be the length of the transformed wide character string. This is
useful for determining what size the allocated array should be
(remember to multiply with `sizeof (wchar_t)'). It does not
matter what WTO is if SIZE is zero; WTO may even be a null pointer.
Here is an example of how you can use `strxfrm' when you plan to do
many comparisons. It does the same thing as the previous example, but
much faster, because it has to transform each string only once, no
matter how many times it is compared with other strings. Even the time
needed to allocate and free storage is much less than the time we save,
when there are many strings.
struct sorter { char *input; char *transformed; };
/* This is the comparison function used with `qsort'
to sort an array of `struct sorter'. */
int
compare_elements (struct sorter *p1, struct sorter *p2)
{
return strcmp (p1->transformed, p2->transformed);
}
/* This is the entry point--the function to sort
strings using the locale's collating sequence. */
void
sort_strings_fast (char **array, int nstrings)
{
struct sorter temp_array[nstrings];
int i;
/* Set up `temp_array'. Each element contains
one input string and its transformed string. */
for (i = 0; i < nstrings; i++)
{
size_t length = strlen (array[i]) * 2;
char *transformed;
size_t transformed_length;
temp_array[i].input = array[i];
/* First try a buffer perhaps big enough. */
transformed = (char *) xmalloc (length);
/* Transform `array[i]'. */
transformed_length = strxfrm (transformed, array[i], length);
/* If the buffer was not large enough, resize it
and try again. */
if (transformed_length >= length)
{
/* Allocate the needed space. +1 for terminating
`NUL' character. */
transformed = (char *) xrealloc (transformed,
transformed_length + 1);
/* The return value is not interesting because we know
how long the transformed string is. */
(void) strxfrm (transformed, array[i],
transformed_length + 1);
}
temp_array[i].transformed = transformed;
}
/* Sort `temp_array' by comparing transformed strings. */
qsort (temp_array, sizeof (struct sorter),
nstrings, compare_elements);
/* Put the elements back in the permanent array
in their sorted order. */
for (i = 0; i < nstrings; i++)
array[i] = temp_array[i].input;
/* Free the strings we allocated. */
for (i = 0; i < nstrings; i++)
free (temp_array[i].transformed);
}
The interesting part of this code for the wide character version
would look like this:
void
sort_strings_fast (wchar_t **array, int nstrings)
{
...
/* Transform `array[i]'. */
transformed_length = wcsxfrm (transformed, array[i], length);
/* If the buffer was not large enough, resize it
and try again. */
if (transformed_length >= length)
{
/* Allocate the needed space. +1 for terminating
`NUL' character. */
transformed = (wchar_t *) xrealloc (transformed,
(transformed_length + 1)
* sizeof (wchar_t));
/* The return value is not interesting because we know
how long the transformed string is. */
(void) wcsxfrm (transformed, array[i],
transformed_length + 1);
}
...
Note the additional multiplication with `sizeof (wchar_t)' in the
`realloc' call.
*Compatibility Note:* The string collation functions are a new
feature of ISO C90. Older C dialects have no equivalent feature. The
wide character versions were introduced in Amendment 1 to ISO C90.
File: libc.info, Node: Search Functions, Next: Finding Tokens in a String, Prev: Collation Functions, Up: String and Array Utilities
5.7 Search Functions
====================
This section describes library functions which perform various kinds of
searching operations on strings and arrays. These functions are
declared in the header file `string.h'.
-- Function: void * memchr (const void *BLOCK, int C, size_t SIZE)
This function finds the first occurrence of the byte C (converted
to an `unsigned char') in the initial SIZE bytes of the object
beginning at BLOCK. The return value is a pointer to the located
byte, or a null pointer if no match was found.
-- Function: wchar_t * wmemchr (const wchar_t *BLOCK, wchar_t WC,
size_t SIZE)
This function finds the first occurrence of the wide character WC
in the initial SIZE wide characters of the object beginning at
BLOCK. The return value is a pointer to the located wide
character, or a null pointer if no match was found.
-- Function: void * rawmemchr (const void *BLOCK, int C)
Often the `memchr' function is used with the knowledge that the
byte C is available in the memory block specified by the
parameters. But this means that the SIZE parameter is not really
needed and that the tests performed with it at runtime (to check
whether the end of the block is reached) are not needed.
The `rawmemchr' function exists for just this situation which is
surprisingly frequent. The interface is similar to `memchr' except
that the SIZE parameter is missing. The function will look beyond
the end of the block pointed to by BLOCK in case the programmer
made an error in assuming that the byte C is present in the block.
In this case the result is unspecified. Otherwise the return
value is a pointer to the located byte.
This function is of special interest when looking for the end of a
string. Since all strings are terminated by a null byte a call
like
rawmemchr (str, '\0')
will never go beyond the end of the string.
This function is a GNU extension.
-- Function: void * memrchr (const void *BLOCK, int C, size_t SIZE)
The function `memrchr' is like `memchr', except that it searches
backwards from the end of the block defined by BLOCK and SIZE
(instead of forwards from the front).
This function is a GNU extension.
-- Function: char * strchr (const char *STRING, int C)
The `strchr' function finds the first occurrence of the character
C (converted to a `char') in the null-terminated string beginning
at STRING. The return value is a pointer to the located
character, or a null pointer if no match was found.
For example,
strchr ("hello, world", 'l')
=> "llo, world"
strchr ("hello, world", '?')
=> NULL
The terminating null character is considered to be part of the
string, so you can use this function get a pointer to the end of a
string by specifying a null character as the value of the C
argument. It would be better (but less portable) to use
`strchrnul' in this case, though.
-- Function: wchar_t * wcschr (const wchar_t *WSTRING, int WC)
The `wcschr' function finds the first occurrence of the wide
character WC in the null-terminated wide character string
beginning at WSTRING. The return value is a pointer to the
located wide character, or a null pointer if no match was found.
The terminating null character is considered to be part of the wide
character string, so you can use this function get a pointer to
the end of a wide character string by specifying a null wude
character as the value of the WC argument. It would be better
(but less portable) to use `wcschrnul' in this case, though.
-- Function: char * strchrnul (const char *STRING, int C)
`strchrnul' is the same as `strchr' except that if it does not
find the character, it returns a pointer to string's terminating
null character rather than a null pointer.
This function is a GNU extension.
-- Function: wchar_t * wcschrnul (const wchar_t *WSTRING, wchar_t WC)
`wcschrnul' is the same as `wcschr' except that if it does not
find the wide character, it returns a pointer to wide character
string's terminating null wide character rather than a null
pointer.
This function is a GNU extension.
One useful, but unusual, use of the `strchr' function is when one
wants to have a pointer pointing to the NUL byte terminating a string.
This is often written in this way:
s += strlen (s);
This is almost optimal but the addition operation duplicated a bit of
the work already done in the `strlen' function. A better solution is
this:
s = strchr (s, '\0');
There is no restriction on the second parameter of `strchr' so it
could very well also be the NUL character. Those readers thinking very
hard about this might now point out that the `strchr' function is more
expensive than the `strlen' function since we have two abort criteria.
This is right. But in the GNU C library the implementation of `strchr'
is optimized in a special way so that `strchr' actually is faster.
-- Function: char * strrchr (const char *STRING, int C)
The function `strrchr' is like `strchr', except that it searches
backwards from the end of the string STRING (instead of forwards
from the front).
For example,
strrchr ("hello, world", 'l')
=> "ld"
-- Function: wchar_t * wcsrchr (const wchar_t *WSTRING, wchar_t C)
The function `wcsrchr' is like `wcschr', except that it searches
backwards from the end of the string WSTRING (instead of forwards
from the front).
-- Function: char * strstr (const char *HAYSTACK, const char *NEEDLE)
This is like `strchr', except that it searches HAYSTACK for a
substring NEEDLE rather than just a single character. It returns
a pointer into the string HAYSTACK that is the first character of
the substring, or a null pointer if no match was found. If NEEDLE
is an empty string, the function returns HAYSTACK.
For example,
strstr ("hello, world", "l")
=> "llo, world"
strstr ("hello, world", "wo")
=> "world"
-- Function: wchar_t * wcsstr (const wchar_t *HAYSTACK, const wchar_t
*NEEDLE)
This is like `wcschr', except that it searches HAYSTACK for a
substring NEEDLE rather than just a single wide character. It
returns a pointer into the string HAYSTACK that is the first wide
character of the substring, or a null pointer if no match was
found. If NEEDLE is an empty string, the function returns
HAYSTACK.
-- Function: wchar_t * wcswcs (const wchar_t *HAYSTACK, const wchar_t
*NEEDLE)
`wcswcs' is an deprecated alias for `wcsstr'. This is the name
originally used in the X/Open Portability Guide before the
Amendment 1 to ISO C90 was published.
-- Function: char * strcasestr (const char *HAYSTACK, const char
*NEEDLE)
This is like `strstr', except that it ignores case in searching for
the substring. Like `strcasecmp', it is locale dependent how
uppercase and lowercase characters are related.
For example,
strstr ("hello, world", "L")
=> "llo, world"
strstr ("hello, World", "wo")
=> "World"
-- Function: void * memmem (const void *HAYSTACK, size_t HAYSTACK-LEN,
const void *NEEDLE, size_t NEEDLE-LEN)
This is like `strstr', but NEEDLE and HAYSTACK are byte arrays
rather than null-terminated strings. NEEDLE-LEN is the length of
NEEDLE and HAYSTACK-LEN is the length of HAYSTACK.
This function is a GNU extension.
-- Function: size_t strspn (const char *STRING, const char *SKIPSET)
The `strspn' ("string span") function returns the length of the
initial substring of STRING that consists entirely of characters
that are members of the set specified by the string SKIPSET. The
order of the characters in SKIPSET is not important.
For example,
strspn ("hello, world", "abcdefghijklmnopqrstuvwxyz")
=> 5
Note that "character" is here used in the sense of byte. In a
string using a multibyte character encoding (abstract) character
consisting of more than one byte are not treated as an entity.
Each byte is treated separately. The function is not
locale-dependent.
-- Function: size_t wcsspn (const wchar_t *WSTRING, const wchar_t
*SKIPSET)
The `wcsspn' ("wide character string span") function returns the
length of the initial substring of WSTRING that consists entirely
of wide characters that are members of the set specified by the
string SKIPSET. The order of the wide characters in SKIPSET is not
important.
-- Function: size_t strcspn (const char *STRING, const char *STOPSET)
The `strcspn' ("string complement span") function returns the
length of the initial substring of STRING that consists entirely
of characters that are _not_ members of the set specified by the
string STOPSET. (In other words, it returns the offset of the
first character in STRING that is a member of the set STOPSET.)
For example,
strcspn ("hello, world", " \t\n,.;!?")
=> 5
Note that "character" is here used in the sense of byte. In a
string using a multibyte character encoding (abstract) character
consisting of more than one byte are not treated as an entity.
Each byte is treated separately. The function is not
locale-dependent.
-- Function: size_t wcscspn (const wchar_t *WSTRING, const wchar_t
*STOPSET)
The `wcscspn' ("wide character string complement span") function
returns the length of the initial substring of WSTRING that
consists entirely of wide characters that are _not_ members of the
set specified by the string STOPSET. (In other words, it returns
the offset of the first character in STRING that is a member of
the set STOPSET.)
-- Function: char * strpbrk (const char *STRING, const char *STOPSET)
The `strpbrk' ("string pointer break") function is related to
`strcspn', except that it returns a pointer to the first character
in STRING that is a member of the set STOPSET instead of the
length of the initial substring. It returns a null pointer if no
such character from STOPSET is found.
For example,
strpbrk ("hello, world", " \t\n,.;!?")
=> ", world"
Note that "character" is here used in the sense of byte. In a
string using a multibyte character encoding (abstract) character
consisting of more than one byte are not treated as an entity.
Each byte is treated separately. The function is not
locale-dependent.
-- Function: wchar_t * wcspbrk (const wchar_t *WSTRING, const wchar_t
*STOPSET)
The `wcspbrk' ("wide character string pointer break") function is
related to `wcscspn', except that it returns a pointer to the first
wide character in WSTRING that is a member of the set STOPSET
instead of the length of the initial substring. It returns a null
pointer if no such character from STOPSET is found.
5.7.1 Compatibility String Search Functions
-------------------------------------------
-- Function: char * index (const char *STRING, int C)
`index' is another name for `strchr'; they are exactly the same.
New code should always use `strchr' since this name is defined in
ISO C while `index' is a BSD invention which never was available
on System V derived systems.
-- Function: char * rindex (const char *STRING, int C)
`rindex' is another name for `strrchr'; they are exactly the same.
New code should always use `strrchr' since this name is defined in
ISO C while `rindex' is a BSD invention which never was available
on System V derived systems.
File: libc.info, Node: Finding Tokens in a String, Next: strfry, Prev: Search Functions, Up: String and Array Utilities
5.8 Finding Tokens in a String
==============================
It's fairly common for programs to have a need to do some simple kinds
of lexical analysis and parsing, such as splitting a command string up
into tokens. You can do this with the `strtok' function, declared in
the header file `string.h'.
-- Function: char * strtok (char *restrict NEWSTRING, const char
*restrict DELIMITERS)
A string can be split into tokens by making a series of calls to
the function `strtok'.
The string to be split up is passed as the NEWSTRING argument on
the first call only. The `strtok' function uses this to set up
some internal state information. Subsequent calls to get
additional tokens from the same string are indicated by passing a
null pointer as the NEWSTRING argument. Calling `strtok' with
another non-null NEWSTRING argument reinitializes the state
information. It is guaranteed that no other library function ever
calls `strtok' behind your back (which would mess up this internal
state information).
The DELIMITERS argument is a string that specifies a set of
delimiters that may surround the token being extracted. All the
initial characters that are members of this set are discarded.
The first character that is _not_ a member of this set of
delimiters marks the beginning of the next token. The end of the
token is found by looking for the next character that is a member
of the delimiter set. This character in the original string
NEWSTRING is overwritten by a null character, and the pointer to
the beginning of the token in NEWSTRING is returned.
On the next call to `strtok', the searching begins at the next
character beyond the one that marked the end of the previous token.
Note that the set of delimiters DELIMITERS do not have to be the
same on every call in a series of calls to `strtok'.
If the end of the string NEWSTRING is reached, or if the remainder
of string consists only of delimiter characters, `strtok' returns
a null pointer.
Note that "character" is here used in the sense of byte. In a
string using a multibyte character encoding (abstract) character
consisting of more than one byte are not treated as an entity.
Each byte is treated separately. The function is not
locale-dependent.
Note that "character" is here used in the sense of byte. In a
string using a multibyte character encoding (abstract) character
consisting of more than one byte are not treated as an entity.
Each byte is treated separately. The function is not
locale-dependent.
-- Function: wchar_t * wcstok (wchar_t *NEWSTRING, const char
*DELIMITERS)
A string can be split into tokens by making a series of calls to
the function `wcstok'.
The string to be split up is passed as the NEWSTRING argument on
the first call only. The `wcstok' function uses this to set up
some internal state information. Subsequent calls to get
additional tokens from the same wide character string are
indicated by passing a null pointer as the NEWSTRING argument.
Calling `wcstok' with another non-null NEWSTRING argument
reinitializes the state information. It is guaranteed that no
other library function ever calls `wcstok' behind your back (which
would mess up this internal state information).
The DELIMITERS argument is a wide character string that specifies
a set of delimiters that may surround the token being extracted.
All the initial wide characters that are members of this set are
discarded. The first wide character that is _not_ a member of
this set of delimiters marks the beginning of the next token. The
end of the token is found by looking for the next wide character
that is a member of the delimiter set. This wide character in the
original wide character string NEWSTRING is overwritten by a null
wide character, and the pointer to the beginning of the token in
NEWSTRING is returned.
On the next call to `wcstok', the searching begins at the next
wide character beyond the one that marked the end of the previous
token. Note that the set of delimiters DELIMITERS do not have to
be the same on every call in a series of calls to `wcstok'.
If the end of the wide character string NEWSTRING is reached, or
if the remainder of string consists only of delimiter wide
characters, `wcstok' returns a null pointer.
Note that "character" is here used in the sense of byte. In a
string using a multibyte character encoding (abstract) character
consisting of more than one byte are not treated as an entity.
Each byte is treated separately. The function is not
locale-dependent.
*Warning:* Since `strtok' and `wcstok' alter the string they is
parsing, you should always copy the string to a temporary buffer before
parsing it with `strtok'/`wcstok' (*note Copying and Concatenation::).
If you allow `strtok' or `wcstok' to modify a string that came from
another part of your program, you are asking for trouble; that string
might be used for other purposes after `strtok' or `wcstok' has
modified it, and it would not have the expected value.
The string that you are operating on might even be a constant. Then
when `strtok' or `wcstok' tries to modify it, your program will get a
fatal signal for writing in read-only memory. *Note Program Error
Signals::. Even if the operation of `strtok' or `wcstok' would not
require a modification of the string (e.g., if there is exactly one
token) the string can (and in the GNU libc case will) be modified.
This is a special case of a general principle: if a part of a program
does not have as its purpose the modification of a certain data
structure, then it is error-prone to modify the data structure
temporarily.
The functions `strtok' and `wcstok' are not reentrant. *Note
Nonreentrancy::, for a discussion of where and why reentrancy is
important.
Here is a simple example showing the use of `strtok'.
#include <string.h>
#include <stddef.h>
...
const char string[] = "words separated by spaces -- and, punctuation!";
const char delimiters[] = " .,;:!-";
char *token, *cp;
...
cp = strdupa (string); /* Make writable copy. */
token = strtok (cp, delimiters); /* token => "words" */
token = strtok (NULL, delimiters); /* token => "separated" */
token = strtok (NULL, delimiters); /* token => "by" */
token = strtok (NULL, delimiters); /* token => "spaces" */
token = strtok (NULL, delimiters); /* token => "and" */
token = strtok (NULL, delimiters); /* token => "punctuation" */
token = strtok (NULL, delimiters); /* token => NULL */
The GNU C library contains two more functions for tokenizing a string
which overcome the limitation of non-reentrancy. They are only
available for multibyte character strings.
-- Function: char * strtok_r (char *NEWSTRING, const char *DELIMITERS,
char **SAVE_PTR)
Just like `strtok', this function splits the string into several
tokens which can be accessed by successive calls to `strtok_r'.
The difference is that the information about the next token is
stored in the space pointed to by the third argument, SAVE_PTR,
which is a pointer to a string pointer. Calling `strtok_r' with a
null pointer for NEWSTRING and leaving SAVE_PTR between the calls
unchanged does the job without hindering reentrancy.
This function is defined in POSIX.1 and can be found on many
systems which support multi-threading.
-- Function: char * strsep (char **STRING_PTR, const char *DELIMITER)
This function has a similar functionality as `strtok_r' with the
NEWSTRING argument replaced by the SAVE_PTR argument. The
initialization of the moving pointer has to be done by the user.
Successive calls to `strsep' move the pointer along the tokens
separated by DELIMITER, returning the address of the next token
and updating STRING_PTR to point to the beginning of the next
token.
One difference between `strsep' and `strtok_r' is that if the
input string contains more than one character from DELIMITER in a
row `strsep' returns an empty string for each pair of characters
from DELIMITER. This means that a program normally should test
for `strsep' returning an empty string before processing it.
This function was introduced in 4.3BSD and therefore is widely
available.
Here is how the above example looks like when `strsep' is used.
#include <string.h>
#include <stddef.h>
...
const char string[] = "words separated by spaces -- and, punctuation!";
const char delimiters[] = " .,;:!-";
char *running;
char *token;
...
running = strdupa (string);
token = strsep (&running, delimiters); /* token => "words" */
token = strsep (&running, delimiters); /* token => "separated" */
token = strsep (&running, delimiters); /* token => "by" */
token = strsep (&running, delimiters); /* token => "spaces" */
token = strsep (&running, delimiters); /* token => "" */
token = strsep (&running, delimiters); /* token => "" */
token = strsep (&running, delimiters); /* token => "" */
token = strsep (&running, delimiters); /* token => "and" */
token = strsep (&running, delimiters); /* token => "" */
token = strsep (&running, delimiters); /* token => "punctuation" */
token = strsep (&running, delimiters); /* token => "" */
token = strsep (&running, delimiters); /* token => NULL */
-- Function: char * basename (const char *FILENAME)
The GNU version of the `basename' function returns the last
component of the path in FILENAME. This function is the preferred
usage, since it does not modify the argument, FILENAME, and
respects trailing slashes. The prototype for `basename' can be
found in `string.h'. Note, this function is overriden by the XPG
version, if `libgen.h' is included.
Example of using GNU `basename':
#include <string.h>
int
main (int argc, char *argv[])
{
char *prog = basename (argv[0]);
if (argc < 2)
{
fprintf (stderr, "Usage %s <arg>\n", prog);
exit (1);
}
...
}
*Portability Note:* This function may produce different results on
different systems.
-- Function: char * basename (char *PATH)
This is the standard XPG defined `basename'. It is similar in
spirit to the GNU version, but may modify the PATH by removing
trailing '/' characters. If the PATH is made up entirely of '/'
characters, then "/" will be returned. Also, if PATH is `NULL' or
an empty string, then "." is returned. The prototype for the XPG
version can be found in `libgen.h'.
Example of using XPG `basename':
#include <libgen.h>
int
main (int argc, char *argv[])
{
char *prog;
char *path = strdupa (argv[0]);
prog = basename (path);
if (argc < 2)
{
fprintf (stderr, "Usage %s <arg>\n", prog);
exit (1);
}
...
}
-- Function: char * dirname (char *PATH)
The `dirname' function is the compliment to the XPG version of
`basename'. It returns the parent directory of the file specified
by PATH. If PATH is `NULL', an empty string, or contains no '/'
characters, then "." is returned. The prototype for this function
can be found in `libgen.h'.
File: libc.info, Node: strfry, Next: Trivial Encryption, Prev: Finding Tokens in a String, Up: String and Array Utilities
5.9 strfry
==========
The function below addresses the perennial programming quandary: "How do
I take good data in string form and painlessly turn it into garbage?"
This is actually a fairly simple task for C programmers who do not use
the GNU C library string functions, but for programs based on the GNU C
library, the `strfry' function is the preferred method for destroying
string data.
The prototype for this function is in `string.h'.
-- Function: char * strfry (char *STRING)
`strfry' creates a pseudorandom anagram of a string, replacing the
input with the anagram in place. For each position in the string,
`strfry' swaps it with a position in the string selected at random
(from a uniform distribution). The two positions may be the same.
The return value of `strfry' is always STRING.
*Portability Note:* This function is unique to the GNU C library.
File: libc.info, Node: Trivial Encryption, Next: Encode Binary Data, Prev: strfry, Up: String and Array Utilities
5.10 Trivial Encryption
=======================
The `memfrob' function converts an array of data to something
unrecognizable and back again. It is not encryption in its usual sense
since it is easy for someone to convert the encrypted data back to clear
text. The transformation is analogous to Usenet's "Rot13" encryption
method for obscuring offensive jokes from sensitive eyes and such.
Unlike Rot13, `memfrob' works on arbitrary binary data, not just text.
For true encryption, *Note Cryptographic Functions::.
This function is declared in `string.h'.
-- Function: void * memfrob (void *MEM, size_t LENGTH)
`memfrob' transforms (frobnicates) each byte of the data structure
at MEM, which is LENGTH bytes long, by bitwise exclusive oring it
with binary 00101010. It does the transformation in place and its
return value is always MEM.
Note that `memfrob' a second time on the same data structure
returns it to its original state.
This is a good function for hiding information from someone who
doesn't want to see it or doesn't want to see it very much. To
really prevent people from retrieving the information, use
stronger encryption such as that described in *Note Cryptographic
Functions::.
*Portability Note:* This function is unique to the GNU C library.
File: libc.info, Node: Encode Binary Data, Next: Argz and Envz Vectors, Prev: Trivial Encryption, Up: String and Array Utilities
5.11 Encode Binary Data
=======================
To store or transfer binary data in environments which only support text
one has to encode the binary data by mapping the input bytes to
characters in the range allowed for storing or transfering. SVID
systems (and nowadays XPG compliant systems) provide minimal support for
this task.
-- Function: char * l64a (long int N)
This function encodes a 32-bit input value using characters from
the basic character set. It returns a pointer to a 7 character
buffer which contains an encoded version of N. To encode a series
of bytes the user must copy the returned string to a destination
buffer. It returns the empty string if N is zero, which is
somewhat bizarre but mandated by the standard.
*Warning:* Since a static buffer is used this function should not
be used in multi-threaded programs. There is no thread-safe
alternative to this function in the C library.
*Compatibility Note:* The XPG standard states that the return
value of `l64a' is undefined if N is negative. In the GNU
implementation, `l64a' treats its argument as unsigned, so it will
return a sensible encoding for any nonzero N; however, portable
programs should not rely on this.
To encode a large buffer `l64a' must be called in a loop, once for
each 32-bit word of the buffer. For example, one could do
something like this:
char *
encode (const void *buf, size_t len)
{
/* We know in advance how long the buffer has to be. */
unsigned char *in = (unsigned char *) buf;
char *out = malloc (6 + ((len + 3) / 4) * 6 + 1);
char *cp = out, *p;
/* Encode the length. */
/* Using `htonl' is necessary so that the data can be
decoded even on machines with different byte order.
`l64a' can return a string shorter than 6 bytes, so
we pad it with encoding of 0 ('.') at the end by
hand. */
p = stpcpy (cp, l64a (htonl (len)));
cp = mempcpy (p, "......", 6 - (p - cp));
while (len > 3)
{
unsigned long int n = *in++;
n = (n << 8) | *in++;
n = (n << 8) | *in++;
n = (n << 8) | *in++;
len -= 4;
p = stpcpy (cp, l64a (htonl (n)));
cp = mempcpy (p, "......", 6 - (p - cp));
}
if (len > 0)
{
unsigned long int n = *in++;
if (--len > 0)
{
n = (n << 8) | *in++;
if (--len > 0)
n = (n << 8) | *in;
}
cp = stpcpy (cp, l64a (htonl (n)));
}
*cp = '\0';
return out;
}
It is strange that the library does not provide the complete
functionality needed but so be it.
To decode data produced with `l64a' the following function should be
used.
-- Function: long int a64l (const char *STRING)
The parameter STRING should contain a string which was produced by
a call to `l64a'. The function processes at least 6 characters of
this string, and decodes the characters it finds according to the
table below. It stops decoding when it finds a character not in
the table, rather like `atoi'; if you have a buffer which has been
broken into lines, you must be careful to skip over the
end-of-line characters.
The decoded number is returned as a `long int' value.
The `l64a' and `a64l' functions use a base 64 encoding, in which
each character of an encoded string represents six bits of an input
word. These symbols are used for the base 64 digits:
0 1 2 3 4 5 6 7
0 `.' `/' `0' `1' `2' `3' `4' `5'
8 `6' `7' `8' `9' `A' `B' `C' `D'
16 `E' `F' `G' `H' `I' `J' `K' `L'
24 `M' `N' `O' `P' `Q' `R' `S' `T'
32 `U' `V' `W' `X' `Y' `Z' `a' `b'
40 `c' `d' `e' `f' `g' `h' `i' `j'
48 `k' `l' `m' `n' `o' `p' `q' `r'
56 `s' `t' `u' `v' `w' `x' `y' `z'
This encoding scheme is not standard. There are some other encoding
methods which are much more widely used (UU encoding, MIME encoding).
Generally, it is better to use one of these encodings.
File: libc.info, Node: Argz and Envz Vectors, Prev: Encode Binary Data, Up: String and Array Utilities
5.12 Argz and Envz Vectors
==========================
"argz vectors" are vectors of strings in a contiguous block of memory,
each element separated from its neighbors by null-characters (`'\0'').
"Envz vectors" are an extension of argz vectors where each element
is a name-value pair, separated by a `'='' character (as in a Unix
environment).
* Menu:
* Argz Functions:: Operations on argz vectors.
* Envz Functions:: Additional operations on environment vectors.
File: libc.info, Node: Argz Functions, Next: Envz Functions, Up: Argz and Envz Vectors
5.12.1 Argz Functions
---------------------
Each argz vector is represented by a pointer to the first element, of
type `char *', and a size, of type `size_t', both of which can be
initialized to `0' to represent an empty argz vector. All argz
functions accept either a pointer and a size argument, or pointers to
them, if they will be modified.
The argz functions use `malloc'/`realloc' to allocate/grow argz
vectors, and so any argz vector creating using these functions may be
freed by using `free'; conversely, any argz function that may grow a
string expects that string to have been allocated using `malloc' (those
argz functions that only examine their arguments or modify them in
place will work on any sort of memory). *Note Unconstrained
Allocation::.
All argz functions that do memory allocation have a return type of
`error_t', and return `0' for success, and `ENOMEM' if an allocation
error occurs.
These functions are declared in the standard include file `argz.h'.
-- Function: error_t argz_create (char *const ARGV[], char **ARGZ,
size_t *ARGZ_LEN)
The `argz_create' function converts the Unix-style argument vector
ARGV (a vector of pointers to normal C strings, terminated by
`(char *)0'; *note Program Arguments::) into an argz vector with
the same elements, which is returned in ARGZ and ARGZ_LEN.
-- Function: error_t argz_create_sep (const char *STRING, int SEP,
char **ARGZ, size_t *ARGZ_LEN)
The `argz_create_sep' function converts the null-terminated string
STRING into an argz vector (returned in ARGZ and ARGZ_LEN) by
splitting it into elements at every occurrence of the character
SEP.
-- Function: size_t argz_count (const char *ARGZ, size_t ARG_LEN)
Returns the number of elements in the argz vector ARGZ and
ARGZ_LEN.
-- Function: void argz_extract (char *ARGZ, size_t ARGZ_LEN, char
**ARGV)
The `argz_extract' function converts the argz vector ARGZ and
ARGZ_LEN into a Unix-style argument vector stored in ARGV, by
putting pointers to every element in ARGZ into successive
positions in ARGV, followed by a terminator of `0'. ARGV must be
pre-allocated with enough space to hold all the elements in ARGZ
plus the terminating `(char *)0' (`(argz_count (ARGZ, ARGZ_LEN) +
1) * sizeof (char *)' bytes should be enough). Note that the
string pointers stored into ARGV point into ARGZ--they are not
copies--and so ARGZ must be copied if it will be changed while
ARGV is still active. This function is useful for passing the
elements in ARGZ to an exec function (*note Executing a File::).
-- Function: void argz_stringify (char *ARGZ, size_t LEN, int SEP)
The `argz_stringify' converts ARGZ into a normal string with the
elements separated by the character SEP, by replacing each `'\0''
inside ARGZ (except the last one, which terminates the string)
with SEP. This is handy for printing ARGZ in a readable manner.
-- Function: error_t argz_add (char **ARGZ, size_t *ARGZ_LEN, const
char *STR)
The `argz_add' function adds the string STR to the end of the argz
vector `*ARGZ', and updates `*ARGZ' and `*ARGZ_LEN' accordingly.
-- Function: error_t argz_add_sep (char **ARGZ, size_t *ARGZ_LEN,
const char *STR, int DELIM)
The `argz_add_sep' function is similar to `argz_add', but STR is
split into separate elements in the result at occurrences of the
character DELIM. This is useful, for instance, for adding the
components of a Unix search path to an argz vector, by using a
value of `':'' for DELIM.
-- Function: error_t argz_append (char **ARGZ, size_t *ARGZ_LEN, const
char *BUF, size_t BUF_LEN)
The `argz_append' function appends BUF_LEN bytes starting at BUF
to the argz vector `*ARGZ', reallocating `*ARGZ' to accommodate
it, and adding BUF_LEN to `*ARGZ_LEN'.
-- Function: error_t argz_delete (char **ARGZ, size_t *ARGZ_LEN, char
*ENTRY)
If ENTRY points to the beginning of one of the elements in the
argz vector `*ARGZ', the `argz_delete' function will remove this
entry and reallocate `*ARGZ', modifying `*ARGZ' and `*ARGZ_LEN'
accordingly. Note that as destructive argz functions usually
reallocate their argz argument, pointers into argz vectors such as
ENTRY will then become invalid.
-- Function: error_t argz_insert (char **ARGZ, size_t *ARGZ_LEN, char
*BEFORE, const char *ENTRY)
The `argz_insert' function inserts the string ENTRY into the argz
vector `*ARGZ' at a point just before the existing element pointed
to by BEFORE, reallocating `*ARGZ' and updating `*ARGZ' and
`*ARGZ_LEN'. If BEFORE is `0', ENTRY is added to the end instead
(as if by `argz_add'). Since the first element is in fact the
same as `*ARGZ', passing in `*ARGZ' as the value of BEFORE will
result in ENTRY being inserted at the beginning.
-- Function: char * argz_next (char *ARGZ, size_t ARGZ_LEN, const char
*ENTRY)
The `argz_next' function provides a convenient way of iterating
over the elements in the argz vector ARGZ. It returns a pointer
to the next element in ARGZ after the element ENTRY, or `0' if
there are no elements following ENTRY. If ENTRY is `0', the first
element of ARGZ is returned.
This behavior suggests two styles of iteration:
char *entry = 0;
while ((entry = argz_next (ARGZ, ARGZ_LEN, entry)))
ACTION;
(the double parentheses are necessary to make some C compilers
shut up about what they consider a questionable `while'-test) and:
char *entry;
for (entry = ARGZ;
entry;
entry = argz_next (ARGZ, ARGZ_LEN, entry))
ACTION;
Note that the latter depends on ARGZ having a value of `0' if it
is empty (rather than a pointer to an empty block of memory); this
invariant is maintained for argz vectors created by the functions
here.
-- Function: error_t argz_replace (char **ARGZ, size_t *ARGZ_LEN,
const char *STR, const char *WITH, unsigned *REPLACE_COUNT)
Replace any occurrences of the string STR in ARGZ with WITH,
reallocating ARGZ as necessary. If REPLACE_COUNT is non-zero,
`*REPLACE_COUNT' will be incremented by number of replacements
performed.
File: libc.info, Node: Envz Functions, Prev: Argz Functions, Up: Argz and Envz Vectors
5.12.2 Envz Functions
---------------------
Envz vectors are just argz vectors with additional constraints on the
form of each element; as such, argz functions can also be used on them,
where it makes sense.
Each element in an envz vector is a name-value pair, separated by a
`'='' character; if multiple `'='' characters are present in an
element, those after the first are considered part of the value, and
treated like all other non-`'\0'' characters.
If _no_ `'='' characters are present in an element, that element is
considered the name of a "null" entry, as distinct from an entry with an
empty value: `envz_get' will return `0' if given the name of null
entry, whereas an entry with an empty value would result in a value of
`""'; `envz_entry' will still find such entries, however. Null entries
can be removed with `envz_strip' function.
As with argz functions, envz functions that may allocate memory (and
thus fail) have a return type of `error_t', and return either `0' or
`ENOMEM'.
These functions are declared in the standard include file `envz.h'.
-- Function: char * envz_entry (const char *ENVZ, size_t ENVZ_LEN,
const char *NAME)
The `envz_entry' function finds the entry in ENVZ with the name
NAME, and returns a pointer to the whole entry--that is, the argz
element which begins with NAME followed by a `'='' character. If
there is no entry with that name, `0' is returned.
-- Function: char * envz_get (const char *ENVZ, size_t ENVZ_LEN, const
char *NAME)
The `envz_get' function finds the entry in ENVZ with the name NAME
(like `envz_entry'), and returns a pointer to the value portion of
that entry (following the `'=''). If there is no entry with that
name (or only a null entry), `0' is returned.
-- Function: error_t envz_add (char **ENVZ, size_t *ENVZ_LEN, const
char *NAME, const char *VALUE)
The `envz_add' function adds an entry to `*ENVZ' (updating `*ENVZ'
and `*ENVZ_LEN') with the name NAME, and value VALUE. If an entry
with the same name already exists in ENVZ, it is removed first.
If VALUE is `0', then the new entry will the special null type of
entry (mentioned above).
-- Function: error_t envz_merge (char **ENVZ, size_t *ENVZ_LEN, const
char *ENVZ2, size_t ENVZ2_LEN, int OVERRIDE)
The `envz_merge' function adds each entry in ENVZ2 to ENVZ, as if
with `envz_add', updating `*ENVZ' and `*ENVZ_LEN'. If OVERRIDE is
true, then values in ENVZ2 will supersede those with the same name
in ENVZ, otherwise not.
Null entries are treated just like other entries in this respect,
so a null entry in ENVZ can prevent an entry of the same name in
ENVZ2 from being added to ENVZ, if OVERRIDE is false.
-- Function: void envz_strip (char **ENVZ, size_t *ENVZ_LEN)
The `envz_strip' function removes any null entries from ENVZ,
updating `*ENVZ' and `*ENVZ_LEN'.
File: libc.info, Node: Character Set Handling, Next: Locales, Prev: String and Array Utilities, Up: Top
6 Character Set Handling
************************
Character sets used in the early days of computing had only six, seven,
or eight bits for each character: there was never a case where more than
eight bits (one byte) were used to represent a single character. The
limitations of this approach became more apparent as more people
grappled with non-Roman character sets, where not all the characters
that make up a language's character set can be represented by 2^8
choices. This chapter shows the functionality that was added to the C
library to support multiple character sets.
* Menu:
* Extended Char Intro:: Introduction to Extended Characters.
* Charset Function Overview:: Overview about Character Handling
Functions.
* Restartable multibyte conversion:: Restartable multibyte conversion
Functions.
* Non-reentrant Conversion:: Non-reentrant Conversion Function.
* Generic Charset Conversion:: Generic Charset Conversion.
File: libc.info, Node: Extended Char Intro, Next: Charset Function Overview, Up: Character Set Handling
6.1 Introduction to Extended Characters
=======================================
A variety of solutions is available to overcome the differences between
character sets with a 1:1 relation between bytes and characters and
character sets with ratios of 2:1 or 4:1. The remainder of this
section gives a few examples to help understand the design decisions
made while developing the functionality of the C library.
A distinction we have to make right away is between internal and
external representation. "Internal representation" means the
representation used by a program while keeping the text in memory.
External representations are used when text is stored or transmitted
through some communication channel. Examples of external
representations include files waiting in a directory to be read and
parsed.
Traditionally there has been no difference between the two
representations. It was equally comfortable and useful to use the same
single-byte representation internally and externally. This comfort
level decreases with more and larger character sets.
One of the problems to overcome with the internal representation is
handling text that is externally encoded using different character
sets. Assume a program that reads two texts and compares them using
some metric. The comparison can be usefully done only if the texts are
internally kept in a common format.
For such a common format (= character set) eight bits are certainly
no longer enough. So the smallest entity will have to grow: "wide
characters" will now be used. Instead of one byte per character, two or
four will be used instead. (Three are not good to address in memory and
more than four bytes seem not to be necessary).
As shown in some other part of this manual, a completely new family
has been created of functions that can handle wide character texts in
memory. The most commonly used character sets for such internal wide
character representations are Unicode and ISO 10646 (also known as UCS
for Universal Character Set). Unicode was originally planned as a
16-bit character set; whereas, ISO 10646 was designed to be a 31-bit
large code space. The two standards are practically identical. They
have the same character repertoire and code table, but Unicode specifies
added semantics. At the moment, only characters in the first `0x10000'
code positions (the so-called Basic Multilingual Plane, BMP) have been
assigned, but the assignment of more specialized characters outside this
16-bit space is already in progress. A number of encodings have been
defined for Unicode and ISO 10646 characters: UCS-2 is a 16-bit word
that can only represent characters from the BMP, UCS-4 is a 32-bit word
than can represent any Unicode and ISO 10646 character, UTF-8 is an
ASCII compatible encoding where ASCII characters are represented by
ASCII bytes and non-ASCII characters by sequences of 2-6 non-ASCII
bytes, and finally UTF-16 is an extension of UCS-2 in which pairs of
certain UCS-2 words can be used to encode non-BMP characters up to
`0x10ffff'.
To represent wide characters the `char' type is not suitable. For
this reason the ISO C standard introduces a new type that is designed
to keep one character of a wide character string. To maintain the
similarity there is also a type corresponding to `int' for those
functions that take a single wide character.
-- Data type: wchar_t
This data type is used as the base type for wide character strings.
In other words, arrays of objects of this type are the equivalent
of `char[]' for multibyte character strings. The type is defined
in `stddef.h'.
The ISO C90 standard, where `wchar_t' was introduced, does not say
anything specific about the representation. It only requires that
this type is capable of storing all elements of the basic
character set. Therefore it would be legitimate to define
`wchar_t' as `char', which might make sense for embedded systems.
But for GNU systems `wchar_t' is always 32 bits wide and,
therefore, capable of representing all UCS-4 values and,
therefore, covering all of ISO 10646. Some Unix systems define
`wchar_t' as a 16-bit type and thereby follow Unicode very
strictly. This definition is perfectly fine with the standard,
but it also means that to represent all characters from Unicode
and ISO 10646 one has to use UTF-16 surrogate characters, which is
in fact a multi-wide-character encoding. But resorting to
multi-wide-character encoding contradicts the purpose of the
`wchar_t' type.
-- Data type: wint_t
`wint_t' is a data type used for parameters and variables that
contain a single wide character. As the name suggests this type
is the equivalent of `int' when using the normal `char' strings.
The types `wchar_t' and `wint_t' often have the same
representation if their size is 32 bits wide but if `wchar_t' is
defined as `char' the type `wint_t' must be defined as `int' due
to the parameter promotion.
This type is defined in `wchar.h' and was introduced in
Amendment 1 to ISO C90.
As there are for the `char' data type macros are available for
specifying the minimum and maximum value representable in an object of
type `wchar_t'.
-- Macro: wint_t WCHAR_MIN
The macro `WCHAR_MIN' evaluates to the minimum value representable
by an object of type `wint_t'.
This macro was introduced in Amendment 1 to ISO C90.
-- Macro: wint_t WCHAR_MAX
The macro `WCHAR_MAX' evaluates to the maximum value representable
by an object of type `wint_t'.
This macro was introduced in Amendment 1 to ISO C90.
Another special wide character value is the equivalent to `EOF'.
-- Macro: wint_t WEOF
The macro `WEOF' evaluates to a constant expression of type
`wint_t' whose value is different from any member of the extended
character set.
`WEOF' need not be the same value as `EOF' and unlike `EOF' it
also need _not_ be negative. In other words, sloppy code like
{
int c;
...
while ((c = getc (fp)) < 0)
...
}
has to be rewritten to use `WEOF' explicitly when wide characters
are used:
{
wint_t c;
...
while ((c = wgetc (fp)) != WEOF)
...
}
This macro was introduced in Amendment 1 to ISO C90 and is defined
in `wchar.h'.
These internal representations present problems when it comes to
storing and transmittal. Because each single wide character consists
of more than one byte, they are effected by byte-ordering. Thus,
machines with different endianesses would see different values when
accessing the same data. This byte ordering concern also applies for
communication protocols that are all byte-based and, thereforet require
that the sender has to decide about splitting the wide character in
bytes. A last (but not least important) point is that wide characters
often require more storage space than a customized byte-oriented
character set.
For all the above reasons, an external encoding that is different
from the internal encoding is often used if the latter is UCS-2 or
UCS-4. The external encoding is byte-based and can be chosen
appropriately for the environment and for the texts to be handled. A
variety of different character sets can be used for this external
encoding (information that will not be exhaustively presented
here-instead, a description of the major groups will suffice). All of
the ASCII-based character sets fulfill one requirement: they are
"filesystem safe." This means that the character `'/'' is used in the
encoding _only_ to represent itself. Things are a bit different for
character sets like EBCDIC (Extended Binary Coded Decimal Interchange
Code, a character set family used by IBM), but if the operation system
does not understand EBCDIC directly the parameters-to-system calls have
to be converted first anyhow.
* The simplest character sets are single-byte character sets. There
can be only up to 256 characters (for 8 bit character sets), which
is not sufficient to cover all languages but might be sufficient
to handle a specific text. Handling of a 8 bit character sets is
simple. This is not true for other kinds presented later, and
therefore, the application one uses might require the use of 8 bit
character sets.
* The ISO 2022 standard defines a mechanism for extended character
sets where one character _can_ be represented by more than one
byte. This is achieved by associating a state with the text.
Characters that can be used to change the state can be embedded in
the text. Each byte in the text might have a different
interpretation in each state. The state might even influence
whether a given byte stands for a character on its own or whether
it has to be combined with some more bytes.
In most uses of ISO 2022 the defined character sets do not allow
state changes that cover more than the next character. This has
the big advantage that whenever one can identify the beginning of
the byte sequence of a character one can interpret a text
correctly. Examples of character sets using this policy are the
various EUC character sets (used by Sun's operations systems,
EUC-JP, EUC-KR, EUC-TW, and EUC-CN) or Shift_JIS (SJIS, a Japanese
encoding).
But there are also character sets using a state that is valid for
more than one character and has to be changed by another byte
sequence. Examples for this are ISO-2022-JP, ISO-2022-KR, and
ISO-2022-CN.
* Early attempts to fix 8 bit character sets for other languages
using the Roman alphabet lead to character sets like ISO 6937.
Here bytes representing characters like the acute accent do not
produce output themselves: one has to combine them with other
characters to get the desired result. For example, the byte
sequence `0xc2 0x61' (non-spacing acute accent, followed by
lower-case `a') to get the "small a with acute" character. To
get the acute accent character on its own, one has to write `0xc2
0x20' (the non-spacing acute followed by a space).
Character sets like ISO 6937 are used in some embedded systems such
as teletex.
* Instead of converting the Unicode or ISO 10646 text used
internally, it is often also sufficient to simply use an encoding
different than UCS-2/UCS-4. The Unicode and ISO 10646 standards
even specify such an encoding: UTF-8. This encoding is able to
represent all of ISO 10646 31 bits in a byte string of length one
to six.
There were a few other attempts to encode ISO 10646 such as UTF-7,
but UTF-8 is today the only encoding that should be used. In
fact, with any luck UTF-8 will soon be the only external encoding
that has to be supported. It proves to be universally usable and
its only disadvantage is that it favors Roman languages by making
the byte string representation of other scripts (Cyrillic, Greek,
Asian scripts) longer than necessary if using a specific character
set for these scripts. Methods like the Unicode compression
scheme can alleviate these problems.
The question remaining is: how to select the character set or
encoding to use. The answer: you cannot decide about it yourself, it
is decided by the developers of the system or the majority of the
users. Since the goal is interoperability one has to use whatever the
other people one works with use. If there are no constraints, the
selection is based on the requirements the expected circle of users
will have. In other words, if a project is expected to be used in
only, say, Russia it is fine to use KOI8-R or a similar character set.
But if at the same time people from, say, Greece are participating one
should use a character set that allows all people to collaborate.
The most widely useful solution seems to be: go with the most general
character set, namely ISO 10646. Use UTF-8 as the external encoding
and problems about users not being able to use their own language
adequately are a thing of the past.
One final comment about the choice of the wide character
representation is necessary at this point. We have said above that the
natural choice is using Unicode or ISO 10646. This is not required,
but at least encouraged, by the ISO C standard. The standard defines
at least a macro `__STDC_ISO_10646__' that is only defined on systems
where the `wchar_t' type encodes ISO 10646 characters. If this symbol
is not defined one should avoid making assumptions about the wide
character representation. If the programmer uses only the functions
provided by the C library to handle wide character strings there should
be no compatibility problems with other systems.
File: libc.info, Node: Charset Function Overview, Next: Restartable multibyte conversion, Prev: Extended Char Intro, Up: Character Set Handling
6.2 Overview about Character Handling Functions
===============================================
A Unix C library contains three different sets of functions in two
families to handle character set conversion. One of the function
families (the most commonly used) is specified in the ISO C90 standard
and, therefore, is portable even beyond the Unix world. Unfortunately
this family is the least useful one. These functions should be avoided
whenever possible, especially when developing libraries (as opposed to
applications).
The second family of functions got introduced in the early Unix
standards (XPG2) and is still part of the latest and greatest Unix
standard: Unix 98. It is also the most powerful and useful set of
functions. But we will start with the functions defined in Amendment 1
to ISO C90.
File: libc.info, Node: Restartable multibyte conversion, Next: Non-reentrant Conversion, Prev: Charset Function Overview, Up: Character Set Handling
6.3 Restartable Multibyte Conversion Functions
==============================================
The ISO C standard defines functions to convert strings from a
multibyte representation to wide character strings. There are a number
of peculiarities:
* The character set assumed for the multibyte encoding is not
specified as an argument to the functions. Instead the character
set specified by the `LC_CTYPE' category of the current locale is
used; see *Note Locale Categories::.
* The functions handling more than one character at a time require
NUL terminated strings as the argument (i.e., converting blocks of
text does not work unless one can add a NUL byte at an appropriate
place). The GNU C library contains some extensions to the
standard that allow specifying a size, but basically they also
expect terminated strings.
Despite these limitations the ISO C functions can be used in many
contexts. In graphical user interfaces, for instance, it is not
uncommon to have functions that require text to be displayed in a wide
character string if the text is not simple ASCII. The text itself might
come from a file with translations and the user should decide about the
current locale, which determines the translation and therefore also the
external encoding used. In such a situation (and many others) the
functions described here are perfect. If more freedom while performing
the conversion is necessary take a look at the `iconv' functions (*note
Generic Charset Conversion::).
* Menu:
* Selecting the Conversion:: Selecting the conversion and its properties.
* Keeping the state:: Representing the state of the conversion.
* Converting a Character:: Converting Single Characters.
* Converting Strings:: Converting Multibyte and Wide Character
Strings.
* Multibyte Conversion Example:: A Complete Multibyte Conversion Example.
File: libc.info, Node: Selecting the Conversion, Next: Keeping the state, Up: Restartable multibyte conversion
6.3.1 Selecting the conversion and its properties
-------------------------------------------------
We already said above that the currently selected locale for the
`LC_CTYPE' category decides about the conversion that is performed by
the functions we are about to describe. Each locale uses its own
character set (given as an argument to `localedef') and this is the one
assumed as the external multibyte encoding. The wide character
character set always is UCS-4, at least on GNU systems.
A characteristic of each multibyte character set is the maximum
number of bytes that can be necessary to represent one character. This
information is quite important when writing code that uses the
conversion functions (as shown in the examples below). The ISO C
standard defines two macros that provide this information.
-- Macro: int MB_LEN_MAX
`MB_LEN_MAX' specifies the maximum number of bytes in the multibyte
sequence for a single character in any of the supported locales.
It is a compile-time constant and is defined in `limits.h'.
-- Macro: int MB_CUR_MAX
`MB_CUR_MAX' expands into a positive integer expression that is the
maximum number of bytes in a multibyte character in the current
locale. The value is never greater than `MB_LEN_MAX'. Unlike
`MB_LEN_MAX' this macro need not be a compile-time constant, and in
the GNU C library it is not.
`MB_CUR_MAX' is defined in `stdlib.h'.
Two different macros are necessary since strictly ISO C90 compilers
do not allow variable length array definitions, but still it is
desirable to avoid dynamic allocation. This incomplete piece of code
shows the problem:
{
char buf[MB_LEN_MAX];
ssize_t len = 0;
while (! feof (fp))
{
fread (&buf[len], 1, MB_CUR_MAX - len, fp);
/* ... process buf */
len -= used;
}
}
The code in the inner loop is expected to have always enough bytes in
the array BUF to convert one multibyte character. The array BUF has to
be sized statically since many compilers do not allow a variable size.
The `fread' call makes sure that `MB_CUR_MAX' bytes are always
available in BUF. Note that it isn't a problem if `MB_CUR_MAX' is not
a compile-time constant.
File: libc.info, Node: Keeping the state, Next: Converting a Character, Prev: Selecting the Conversion, Up: Restartable multibyte conversion
6.3.2 Representing the state of the conversion
----------------------------------------------
In the introduction of this chapter it was said that certain character
sets use a "stateful" encoding. That is, the encoded values depend in
some way on the previous bytes in the text.
Since the conversion functions allow converting a text in more than
one step we must have a way to pass this information from one call of
the functions to another.
-- Data type: mbstate_t
A variable of type `mbstate_t' can contain all the information
about the "shift state" needed from one call to a conversion
function to another.
`mbstate_t' is defined in `wchar.h'. It was introduced in
Amendment 1 to ISO C90.
To use objects of type `mbstate_t' the programmer has to define such
objects (normally as local variables on the stack) and pass a pointer to
the object to the conversion functions. This way the conversion
function can update the object if the current multibyte character set
is stateful.
There is no specific function or initializer to put the state object
in any specific state. The rules are that the object should always
represent the initial state before the first use, and this is achieved
by clearing the whole variable with code such as follows:
{
mbstate_t state;
memset (&state, '\0', sizeof (state));
/* from now on STATE can be used. */
...
}
When using the conversion functions to generate output it is often
necessary to test whether the current state corresponds to the initial
state. This is necessary, for example, to decide whether to emit
escape sequences to set the state to the initial state at certain
sequence points. Communication protocols often require this.
-- Function: int mbsinit (const mbstate_t *PS)
The `mbsinit' function determines whether the state object pointed
to by PS is in the initial state. If PS is a null pointer or the
object is in the initial state the return value is nonzero.
Otherwise it is zero.
`mbsinit' was introduced in Amendment 1 to ISO C90 and is declared
in `wchar.h'.
Code using `mbsinit' often looks similar to this:
{
mbstate_t state;
memset (&state, '\0', sizeof (state));
/* Use STATE. */
...
if (! mbsinit (&state))
{
/* Emit code to return to initial state. */
const wchar_t empty[] = L"";
const wchar_t *srcp = empty;
wcsrtombs (outbuf, &srcp, outbuflen, &state);
}
...
}
The code to emit the escape sequence to get back to the initial
state is interesting. The `wcsrtombs' function can be used to
determine the necessary output code (*note Converting Strings::).
Please note that on GNU systems it is not necessary to perform this
extra action for the conversion from multibyte text to wide character
text since the wide character encoding is not stateful. But there is
nothing mentioned in any standard that prohibits making `wchar_t' using
a stateful encoding.
File: libc.info, Node: Converting a Character, Next: Converting Strings, Prev: Keeping the state, Up: Restartable multibyte conversion
6.3.3 Converting Single Characters
----------------------------------
The most fundamental of the conversion functions are those dealing with
single characters. Please note that this does not always mean single
bytes. But since there is very often a subset of the multibyte
character set that consists of single byte sequences, there are
functions to help with converting bytes. Frequently, ASCII is a subpart
of the multibyte character set. In such a scenario, each ASCII
character stands for itself, and all other characters have at least a
first byte that is beyond the range 0 to 127.
-- Function: wint_t btowc (int C)
The `btowc' function ("byte to wide character") converts a valid
single byte character C in the initial shift state into the wide
character equivalent using the conversion rules from the currently
selected locale of the `LC_CTYPE' category.
If `(unsigned char) C' is no valid single byte multibyte character
or if C is `EOF', the function returns `WEOF'.
Please note the restriction of C being tested for validity only in
the initial shift state. No `mbstate_t' object is used from which
the state information is taken, and the function also does not use
any static state.
The `btowc' function was introduced in Amendment 1 to ISO C90 and
is declared in `wchar.h'.
Despite the limitation that the single byte value always is
interpreted in the initial state this function is actually useful most
of the time. Most characters are either entirely single-byte character
sets or they are extension to ASCII. But then it is possible to write
code like this (not that this specific example is very useful):
wchar_t *
itow (unsigned long int val)
{
static wchar_t buf[30];
wchar_t *wcp = &buf[29];
*wcp = L'\0';
while (val != 0)
{
*--wcp = btowc ('0' + val % 10);
val /= 10;
}
if (wcp == &buf[29])
*--wcp = L'0';
return wcp;
}
Why is it necessary to use such a complicated implementation and not
simply cast `'0' + val % 10' to a wide character? The answer is that
there is no guarantee that one can perform this kind of arithmetic on
the character of the character set used for `wchar_t' representation.
In other situations the bytes are not constant at compile time and so
the compiler cannot do the work. In situations like this it is
necessary `btowc'.
There also is a function for the conversion in the other direction.
-- Function: int wctob (wint_t C)
The `wctob' function ("wide character to byte") takes as the
parameter a valid wide character. If the multibyte representation
for this character in the initial state is exactly one byte long,
the return value of this function is this character. Otherwise
the return value is `EOF'.
`wctob' was introduced in Amendment 1 to ISO C90 and is declared
in `wchar.h'.
There are more general functions to convert single character from
multibyte representation to wide characters and vice versa. These
functions pose no limit on the length of the multibyte representation
and they also do not require it to be in the initial state.
-- Function: size_t mbrtowc (wchar_t *restrict PWC, const char
*restrict S, size_t N, mbstate_t *restrict PS)
The `mbrtowc' function ("multibyte restartable to wide character")
converts the next multibyte character in the string pointed to by
S into a wide character and stores it in the wide character string
pointed to by PWC. The conversion is performed according to the
locale currently selected for the `LC_CTYPE' category. If the
conversion for the character set used in the locale requires a
state, the multibyte string is interpreted in the state
represented by the object pointed to by PS. If PS is a null
pointer, a static, internal state variable used only by the
`mbrtowc' function is used.
If the next multibyte character corresponds to the NUL wide
character, the return value of the function is 0 and the state
object is afterwards in the initial state. If the next N or fewer
bytes form a correct multibyte character, the return value is the
number of bytes starting from S that form the multibyte character.
The conversion state is updated according to the bytes consumed
in the conversion. In both cases the wide character (either the
`L'\0'' or the one found in the conversion) is stored in the
string pointed to by PWC if PWC is not null.
If the first N bytes of the multibyte string possibly form a valid
multibyte character but there are more than N bytes needed to
complete it, the return value of the function is `(size_t) -2' and
no value is stored. Please note that this can happen even if N
has a value greater than or equal to `MB_CUR_MAX' since the input
might contain redundant shift sequences.
If the first `n' bytes of the multibyte string cannot possibly form
a valid multibyte character, no value is stored, the global
variable `errno' is set to the value `EILSEQ', and the function
returns `(size_t) -1'. The conversion state is afterwards
undefined.
`mbrtowc' was introduced in Amendment 1 to ISO C90 and is declared
in `wchar.h'.
Use of `mbrtowc' is straightforward. A function that copies a
multibyte string into a wide character string while at the same time
converting all lowercase characters into uppercase could look like this
(this is not the final version, just an example; it has no error
checking, and sometimes leaks memory):
wchar_t *
mbstouwcs (const char *s)
{
size_t len = strlen (s);
wchar_t *result = malloc ((len + 1) * sizeof (wchar_t));
wchar_t *wcp = result;
wchar_t tmp[1];
mbstate_t state;
size_t nbytes;
memset (&state, '\0', sizeof (state));
while ((nbytes = mbrtowc (tmp, s, len, &state)) > 0)
{
if (nbytes >= (size_t) -2)
/* Invalid input string. */
return NULL;
*wcp++ = towupper (tmp[0]);
len -= nbytes;
s += nbytes;
}
return result;
}
The use of `mbrtowc' should be clear. A single wide character is
stored in `TMP[0]', and the number of consumed bytes is stored in the
variable NBYTES. If the conversion is successful, the uppercase
variant of the wide character is stored in the RESULT array and the
pointer to the input string and the number of available bytes is
adjusted.
The only non-obvious thing about `mbrtowc' might be the way memory
is allocated for the result. The above code uses the fact that there
can never be more wide characters in the converted results than there
are bytes in the multibyte input string. This method yields a
pessimistic guess about the size of the result, and if many wide
character strings have to be constructed this way or if the strings are
long, the extra memory required to be allocated because the input
string contains multibyte characters might be significant. The
allocated memory block can be resized to the correct size before
returning it, but a better solution might be to allocate just the right
amount of space for the result right away. Unfortunately there is no
function to compute the length of the wide character string directly
from the multibyte string. There is, however, a function that does
part of the work.
-- Function: size_t mbrlen (const char *restrict S, size_t N,
mbstate_t *PS)
The `mbrlen' function ("multibyte restartable length") computes
the number of at most N bytes starting at S, which form the next
valid and complete multibyte character.
If the next multibyte character corresponds to the NUL wide
character, the return value is 0. If the next N bytes form a valid
multibyte character, the number of bytes belonging to this
multibyte character byte sequence is returned.
If the the first N bytes possibly form a valid multibyte character
but the character is incomplete, the return value is `(size_t)
-2'. Otherwise the multibyte character sequence is invalid and
the return value is `(size_t) -1'.
The multibyte sequence is interpreted in the state represented by
the object pointed to by PS. If PS is a null pointer, a state
object local to `mbrlen' is used.
`mbrlen' was introduced in Amendment 1 to ISO C90 and is declared
in `wchar.h'.
The attentive reader now will note that `mbrlen' can be implemented
as
mbrtowc (NULL, s, n, ps != NULL ? ps : &internal)
This is true and in fact is mentioned in the official specification.
How can this function be used to determine the length of the wide
character string created from a multibyte character string? It is not
directly usable, but we can define a function `mbslen' using it:
size_t
mbslen (const char *s)
{
mbstate_t state;
size_t result = 0;
size_t nbytes;
memset (&state, '\0', sizeof (state));
while ((nbytes = mbrlen (s, MB_LEN_MAX, &state)) > 0)
{
if (nbytes >= (size_t) -2)
/* Something is wrong. */
return (size_t) -1;
s += nbytes;
++result;
}
return result;
}
This function simply calls `mbrlen' for each multibyte character in
the string and counts the number of function calls. Please note that
we here use `MB_LEN_MAX' as the size argument in the `mbrlen' call.
This is acceptable since a) this value is larger then the length of the
longest multibyte character sequence and b) we know that the string S
ends with a NUL byte, which cannot be part of any other multibyte
character sequence but the one representing the NUL wide character.
Therefore, the `mbrlen' function will never read invalid memory.
Now that this function is available (just to make this clear, this
function is _not_ part of the GNU C library) we can compute the number
of wide character required to store the converted multibyte character
string S using
wcs_bytes = (mbslen (s) + 1) * sizeof (wchar_t);
Please note that the `mbslen' function is quite inefficient. The
implementation of `mbstouwcs' with `mbslen' would have to perform the
conversion of the multibyte character input string twice, and this
conversion might be quite expensive. So it is necessary to think about
the consequences of using the easier but imprecise method before doing
the work twice.
-- Function: size_t wcrtomb (char *restrict S, wchar_t WC, mbstate_t
*restrict PS)
The `wcrtomb' function ("wide character restartable to multibyte")
converts a single wide character into a multibyte string
corresponding to that wide character.
If S is a null pointer, the function resets the state stored in
the objects pointed to by PS (or the internal `mbstate_t' object)
to the initial state. This can also be achieved by a call like
this:
wcrtombs (temp_buf, L'\0', ps)
since, if S is a null pointer, `wcrtomb' performs as if it writes
into an internal buffer, which is guaranteed to be large enough.
If WC is the NUL wide character, `wcrtomb' emits, if necessary, a
shift sequence to get the state PS into the initial state followed
by a single NUL byte, which is stored in the string S.
Otherwise a byte sequence (possibly including shift sequences) is
written into the string S. This only happens if WC is a valid wide
character (i.e., it has a multibyte representation in the
character set selected by locale of the `LC_CTYPE' category). If
WC is no valid wide character, nothing is stored in the strings S,
`errno' is set to `EILSEQ', the conversion state in PS is
undefined and the return value is `(size_t) -1'.
If no error occurred the function returns the number of bytes
stored in the string S. This includes all bytes representing shift
sequences.
One word about the interface of the function: there is no parameter
specifying the length of the array S. Instead the function
assumes that there are at least `MB_CUR_MAX' bytes available since
this is the maximum length of any byte sequence representing a
single character. So the caller has to make sure that there is
enough space available, otherwise buffer overruns can occur.
`wcrtomb' was introduced in Amendment 1 to ISO C90 and is declared
in `wchar.h'.
Using `wcrtomb' is as easy as using `mbrtowc'. The following
example appends a wide character string to a multibyte character string.
Again, the code is not really useful (or correct), it is simply here to
demonstrate the use and some problems.
char *
mbscatwcs (char *s, size_t len, const wchar_t *ws)
{
mbstate_t state;
/* Find the end of the existing string. */
char *wp = strchr (s, '\0');
len -= wp - s;
memset (&state, '\0', sizeof (state));
do
{
size_t nbytes;
if (len < MB_CUR_LEN)
{
/* We cannot guarantee that the next
character fits into the buffer, so
return an error. */
errno = E2BIG;
return NULL;
}
nbytes = wcrtomb (wp, *ws, &state);
if (nbytes == (size_t) -1)
/* Error in the conversion. */
return NULL;
len -= nbytes;
wp += nbytes;
}
while (*ws++ != L'\0');
return s;
}
First the function has to find the end of the string currently in the
array S. The `strchr' call does this very efficiently since a
requirement for multibyte character representations is that the NUL byte
is never used except to represent itself (and in this context, the end
of the string).
After initializing the state object the loop is entered where the
first task is to make sure there is enough room in the array S. We
abort if there are not at least `MB_CUR_LEN' bytes available. This is
not always optimal but we have no other choice. We might have less
than `MB_CUR_LEN' bytes available but the next multibyte character
might also be only one byte long. At the time the `wcrtomb' call
returns it is too late to decide whether the buffer was large enough.
If this solution is unsuitable, there is a very slow but more accurate
solution.
...
if (len < MB_CUR_LEN)
{
mbstate_t temp_state;
memcpy (&temp_state, &state, sizeof (state));
if (wcrtomb (NULL, *ws, &temp_state) > len)
{
/* We cannot guarantee that the next
character fits into the buffer, so
return an error. */
errno = E2BIG;
return NULL;
}
}
...
Here we perform the conversion that might overflow the buffer so that
we are afterwards in the position to make an exact decision about the
buffer size. Please note the `NULL' argument for the destination
buffer in the new `wcrtomb' call; since we are not interested in the
converted text at this point, this is a nice way to express this. The
most unusual thing about this piece of code certainly is the duplication
of the conversion state object, but if a change of the state is
necessary to emit the next multibyte character, we want to have the
same shift state change performed in the real conversion. Therefore,
we have to preserve the initial shift state information.
There are certainly many more and even better solutions to this
problem. This example is only provided for educational purposes.
File: libc.info, Node: Converting Strings, Next: Multibyte Conversion Example, Prev: Converting a Character, Up: Restartable multibyte conversion
6.3.4 Converting Multibyte and Wide Character Strings
-----------------------------------------------------
The functions described in the previous section only convert a single
character at a time. Most operations to be performed in real-world
programs include strings and therefore the ISO C standard also defines
conversions on entire strings. However, the defined set of functions
is quite limited; therefore, the GNU C library contains a few
extensions that can help in some important situations.
-- Function: size_t mbsrtowcs (wchar_t *restrict DST, const char
**restrict SRC, size_t LEN, mbstate_t *restrict PS)
The `mbsrtowcs' function ("multibyte string restartable to wide
character string") converts an NUL-terminated multibyte character
string at `*SRC' into an equivalent wide character string,
including the NUL wide character at the end. The conversion is
started using the state information from the object pointed to by
PS or from an internal object of `mbsrtowcs' if PS is a null
pointer. Before returning, the state object is updated to match
the state after the last converted character. The state is the
initial state if the terminating NUL byte is reached and converted.
If DST is not a null pointer, the result is stored in the array
pointed to by DST; otherwise, the conversion result is not
available since it is stored in an internal buffer.
If LEN wide characters are stored in the array DST before reaching
the end of the input string, the conversion stops and LEN is
returned. If DST is a null pointer, LEN is never checked.
Another reason for a premature return from the function call is if
the input string contains an invalid multibyte sequence. In this
case the global variable `errno' is set to `EILSEQ' and the
function returns `(size_t) -1'.
In all other cases the function returns the number of wide
characters converted during this call. If DST is not null,
`mbsrtowcs' stores in the pointer pointed to by SRC either a null
pointer (if the NUL byte in the input string was reached) or the
address of the byte following the last converted multibyte
character.
`mbsrtowcs' was introduced in Amendment 1 to ISO C90 and is
declared in `wchar.h'.
The definition of the `mbsrtowcs' function has one important
limitation. The requirement that DST has to be a NUL-terminated string
provides problems if one wants to convert buffers with text. A buffer
is normally no collection of NUL-terminated strings but instead a
continuous collection of lines, separated by newline characters. Now
assume that a function to convert one line from a buffer is needed.
Since the line is not NUL-terminated, the source pointer cannot
directly point into the unmodified text buffer. This means, either one
inserts the NUL byte at the appropriate place for the time of the
`mbsrtowcs' function call (which is not doable for a read-only buffer
or in a multi-threaded application) or one copies the line in an extra
buffer where it can be terminated by a NUL byte. Note that it is not
in general possible to limit the number of characters to convert by
setting the parameter LEN to any specific value. Since it is not known
how many bytes each multibyte character sequence is in length, one can
only guess.
There is still a problem with the method of NUL-terminating a line
right after the newline character, which could lead to very strange
results. As said in the description of the `mbsrtowcs' function above
the conversion state is guaranteed to be in the initial shift state
after processing the NUL byte at the end of the input string. But this
NUL byte is not really part of the text (i.e., the conversion state
after the newline in the original text could be something different
than the initial shift state and therefore the first character of the
next line is encoded using this state). But the state in question is
never accessible to the user since the conversion stops after the NUL
byte (which resets the state). Most stateful character sets in use
today require that the shift state after a newline be the initial
state-but this is not a strict guarantee. Therefore, simply
NUL-terminating a piece of a running text is not always an adequate
solution and, therefore, should never be used in generally used code.
The generic conversion interface (*note Generic Charset Conversion::)
does not have this limitation (it simply works on buffers, not
strings), and the GNU C library contains a set of functions that take
additional parameters specifying the maximal number of bytes that are
consumed from the input string. This way the problem of `mbsrtowcs''s
example above could be solved by determining the line length and
passing this length to the function.
-- Function: size_t wcsrtombs (char *restrict DST, const wchar_t
**restrict SRC, size_t LEN, mbstate_t *restrict PS)
The `wcsrtombs' function ("wide character string restartable to
multibyte string") converts the NUL-terminated wide character
string at `*SRC' into an equivalent multibyte character string and
stores the result in the array pointed to by DST. The NUL wide
character is also converted. The conversion starts in the state
described in the object pointed to by PS or by a state object
locally to `wcsrtombs' in case PS is a null pointer. If DST is a
null pointer, the conversion is performed as usual but the result
is not available. If all characters of the input string were
successfully converted and if DST is not a null pointer, the
pointer pointed to by SRC gets assigned a null pointer.
If one of the wide characters in the input string has no valid
multibyte character equivalent, the conversion stops early, sets
the global variable `errno' to `EILSEQ', and returns `(size_t) -1'.
Another reason for a premature stop is if DST is not a null
pointer and the next converted character would require more than
LEN bytes in total to the array DST. In this case (and if DEST is
not a null pointer) the pointer pointed to by SRC is assigned a
value pointing to the wide character right after the last one
successfully converted.
Except in the case of an encoding error the return value of the
`wcsrtombs' function is the number of bytes in all the multibyte
character sequences stored in DST. Before returning the state in
the object pointed to by PS (or the internal object in case PS is
a null pointer) is updated to reflect the state after the last
conversion. The state is the initial shift state in case the
terminating NUL wide character was converted.
The `wcsrtombs' function was introduced in Amendment 1 to ISO C90
and is declared in `wchar.h'.
The restriction mentioned above for the `mbsrtowcs' function applies
here also. There is no possibility of directly controlling the number
of input characters. One has to place the NUL wide character at the
correct place or control the consumed input indirectly via the
available output array size (the LEN parameter).
-- Function: size_t mbsnrtowcs (wchar_t *restrict DST, const char
**restrict SRC, size_t NMC, size_t LEN, mbstate_t *restrict
PS)
The `mbsnrtowcs' function is very similar to the `mbsrtowcs'
function. All the parameters are the same except for NMC, which is
new. The return value is the same as for `mbsrtowcs'.
This new parameter specifies how many bytes at most can be used
from the multibyte character string. In other words, the
multibyte character string `*SRC' need not be NUL-terminated. But
if a NUL byte is found within the NMC first bytes of the string,
the conversion stops here.
This function is a GNU extension. It is meant to work around the
problems mentioned above. Now it is possible to convert a buffer
with multibyte character text piece for piece without having to
care about inserting NUL bytes and the effect of NUL bytes on the
conversion state.
A function to convert a multibyte string into a wide character string
and display it could be written like this (this is not a really useful
example):
void
showmbs (const char *src, FILE *fp)
{
mbstate_t state;
int cnt = 0;
memset (&state, '\0', sizeof (state));
while (1)
{
wchar_t linebuf[100];
const char *endp = strchr (src, '\n');
size_t n;
/* Exit if there is no more line. */
if (endp == NULL)
break;
n = mbsnrtowcs (linebuf, &src, endp - src, 99, &state);
linebuf[n] = L'\0';
fprintf (fp, "line %d: \"%S\"\n", linebuf);
}
}
There is no problem with the state after a call to `mbsnrtowcs'.
Since we don't insert characters in the strings that were not in there
right from the beginning and we use STATE only for the conversion of
the given buffer, there is no problem with altering the state.
-- Function: size_t wcsnrtombs (char *restrict DST, const wchar_t
**restrict SRC, size_t NWC, size_t LEN, mbstate_t *restrict
PS)
The `wcsnrtombs' function implements the conversion from wide
character strings to multibyte character strings. It is similar to
`wcsrtombs' but, just like `mbsnrtowcs', it takes an extra
parameter, which specifies the length of the input string.
No more than NWC wide characters from the input string `*SRC' are
converted. If the input string contains a NUL wide character in
the first NWC characters, the conversion stops at this place.
The `wcsnrtombs' function is a GNU extension and just like
`mbsnrtowcs' helps in situations where no NUL-terminated input
strings are available.
File: libc.info, Node: Multibyte Conversion Example, Prev: Converting Strings, Up: Restartable multibyte conversion
6.3.5 A Complete Multibyte Conversion Example
---------------------------------------------
The example programs given in the last sections are only brief and do
not contain all the error checking, etc. Presented here is a complete
and documented example. It features the `mbrtowc' function but it
should be easy to derive versions using the other functions.
int
file_mbsrtowcs (int input, int output)
{
/* Note the use of `MB_LEN_MAX'.
`MB_CUR_MAX' cannot portably be used here. */
char buffer[BUFSIZ + MB_LEN_MAX];
mbstate_t state;
int filled = 0;
int eof = 0;
/* Initialize the state. */
memset (&state, '\0', sizeof (state));
while (!eof)
{
ssize_t nread;
ssize_t nwrite;
char *inp = buffer;
wchar_t outbuf[BUFSIZ];
wchar_t *outp = outbuf;
/* Fill up the buffer from the input file. */
nread = read (input, buffer + filled, BUFSIZ);
if (nread < 0)
{
perror ("read");
return 0;
}
/* If we reach end of file, make a note to read no more. */
if (nread == 0)
eof = 1;
/* `filled' is now the number of bytes in `buffer'. */
filled += nread;
/* Convert those bytes to wide characters-as many as we can. */
while (1)
{
size_t thislen = mbrtowc (outp, inp, filled, &state);
/* Stop converting at invalid character;
this can mean we have read just the first part
of a valid character. */
if (thislen == (size_t) -1)
break;
/* We want to handle embedded NUL bytes
but the return value is 0. Correct this. */
if (thislen == 0)
thislen = 1;
/* Advance past this character. */
inp += thislen;
filled -= thislen;
++outp;
}
/* Write the wide characters we just made. */
nwrite = write (output, outbuf,
(outp - outbuf) * sizeof (wchar_t));
if (nwrite < 0)
{
perror ("write");
return 0;
}
/* See if we have a _real_ invalid character. */
if ((eof && filled > 0) || filled >= MB_CUR_MAX)
{
error (0, 0, "invalid multibyte character");
return 0;
}
/* If any characters must be carried forward,
put them at the beginning of `buffer'. */
if (filled > 0)
memmove (inp, buffer, filled);
}
return 1;
}
File: libc.info, Node: Non-reentrant Conversion, Next: Generic Charset Conversion, Prev: Restartable multibyte conversion, Up: Character Set Handling
6.4 Non-reentrant Conversion Function
=====================================
The functions described in the previous chapter are defined in
Amendment 1 to ISO C90, but the original ISO C90 standard also
contained functions for character set conversion. The reason that
these original functions are not described first is that they are almost
entirely useless.
The problem is that all the conversion functions described in the
original ISO C90 use a local state. Using a local state implies that
multiple conversions at the same time (not only when using threads)
cannot be done, and that you cannot first convert single characters and
then strings since you cannot tell the conversion functions which state
to use.
These original functions are therefore usable only in a very limited
set of situations. One must complete converting the entire string
before starting a new one, and each string/text must be converted with
the same function (there is no problem with the library itself; it is
guaranteed that no library function changes the state of any of these
functions). *For the above reasons it is highly requested that the
functions described in the previous section be used in place of
non-reentrant conversion functions.*
* Menu:
* Non-reentrant Character Conversion:: Non-reentrant Conversion of Single
Characters.
* Non-reentrant String Conversion:: Non-reentrant Conversion of Strings.
* Shift State:: States in Non-reentrant Functions.
File: libc.info, Node: Non-reentrant Character Conversion, Next: Non-reentrant String Conversion, Up: Non-reentrant Conversion
6.4.1 Non-reentrant Conversion of Single Characters
---------------------------------------------------
-- Function: int mbtowc (wchar_t *restrict RESULT, const char
*restrict STRING, size_t SIZE)
The `mbtowc' ("multibyte to wide character") function when called
with non-null STRING converts the first multibyte character
beginning at STRING to its corresponding wide character code. It
stores the result in `*RESULT'.
`mbtowc' never examines more than SIZE bytes. (The idea is to
supply for SIZE the number of bytes of data you have in hand.)
`mbtowc' with non-null STRING distinguishes three possibilities:
the first SIZE bytes at STRING start with valid multibyte
characters, they start with an invalid byte sequence or just part
of a character, or STRING points to an empty string (a null
character).
For a valid multibyte character, `mbtowc' converts it to a wide
character and stores that in `*RESULT', and returns the number of
bytes in that character (always at least 1 and never more than
SIZE).
For an invalid byte sequence, `mbtowc' returns -1. For an empty
string, it returns 0, also storing `'\0'' in `*RESULT'.
If the multibyte character code uses shift characters, then
`mbtowc' maintains and updates a shift state as it scans. If you
call `mbtowc' with a null pointer for STRING, that initializes the
shift state to its standard initial value. It also returns
nonzero if the multibyte character code in use actually has a
shift state. *Note Shift State::.
-- Function: int wctomb (char *STRING, wchar_t WCHAR)
The `wctomb' ("wide character to multibyte") function converts the
wide character code WCHAR to its corresponding multibyte character
sequence, and stores the result in bytes starting at STRING. At
most `MB_CUR_MAX' characters are stored.
`wctomb' with non-null STRING distinguishes three possibilities
for WCHAR: a valid wide character code (one that can be translated
to a multibyte character), an invalid code, and `L'\0''.
Given a valid code, `wctomb' converts it to a multibyte character,
storing the bytes starting at STRING. Then it returns the number
of bytes in that character (always at least 1 and never more than
`MB_CUR_MAX').
If WCHAR is an invalid wide character code, `wctomb' returns -1.
If WCHAR is `L'\0'', it returns `0', also storing `'\0'' in
`*STRING'.
If the multibyte character code uses shift characters, then
`wctomb' maintains and updates a shift state as it scans. If you
call `wctomb' with a null pointer for STRING, that initializes the
shift state to its standard initial value. It also returns
nonzero if the multibyte character code in use actually has a
shift state. *Note Shift State::.
Calling this function with a WCHAR argument of zero when STRING is
not null has the side-effect of reinitializing the stored shift
state _as well as_ storing the multibyte character `'\0'' and
returning 0.
Similar to `mbrlen' there is also a non-reentrant function that
computes the length of a multibyte character. It can be defined in
terms of `mbtowc'.
-- Function: int mblen (const char *STRING, size_t SIZE)
The `mblen' function with a non-null STRING argument returns the
number of bytes that make up the multibyte character beginning at
STRING, never examining more than SIZE bytes. (The idea is to
supply for SIZE the number of bytes of data you have in hand.)
The return value of `mblen' distinguishes three possibilities: the
first SIZE bytes at STRING start with valid multibyte characters,
they start with an invalid byte sequence or just part of a
character, or STRING points to an empty string (a null character).
For a valid multibyte character, `mblen' returns the number of
bytes in that character (always at least `1' and never more than
SIZE). For an invalid byte sequence, `mblen' returns -1. For an
empty string, it returns 0.
If the multibyte character code uses shift characters, then `mblen'
maintains and updates a shift state as it scans. If you call
`mblen' with a null pointer for STRING, that initializes the shift
state to its standard initial value. It also returns a nonzero
value if the multibyte character code in use actually has a shift
state. *Note Shift State::.
The function `mblen' is declared in `stdlib.h'.
File: libc.info, Node: Non-reentrant String Conversion, Next: Shift State, Prev: Non-reentrant Character Conversion, Up: Non-reentrant Conversion
6.4.2 Non-reentrant Conversion of Strings
-----------------------------------------
For convenience the ISO C90 standard also defines functions to convert
entire strings instead of single characters. These functions suffer
from the same problems as their reentrant counterparts from Amendment 1
to ISO C90; see *Note Converting Strings::.
-- Function: size_t mbstowcs (wchar_t *WSTRING, const char *STRING,
size_t SIZE)
The `mbstowcs' ("multibyte string to wide character string")
function converts the null-terminated string of multibyte
characters STRING to an array of wide character codes, storing not
more than SIZE wide characters into the array beginning at WSTRING.
The terminating null character counts towards the size, so if SIZE
is less than the actual number of wide characters resulting from
STRING, no terminating null character is stored.
The conversion of characters from STRING begins in the initial
shift state.
If an invalid multibyte character sequence is found, the `mbstowcs'
function returns a value of -1. Otherwise, it returns the number
of wide characters stored in the array WSTRING. This number does
not include the terminating null character, which is present if the
number is less than SIZE.
Here is an example showing how to convert a string of multibyte
characters, allocating enough space for the result.
wchar_t *
mbstowcs_alloc (const char *string)
{
size_t size = strlen (string) + 1;
wchar_t *buf = xmalloc (size * sizeof (wchar_t));
size = mbstowcs (buf, string, size);
if (size == (size_t) -1)
return NULL;
buf = xrealloc (buf, (size + 1) * sizeof (wchar_t));
return buf;
}
-- Function: size_t wcstombs (char *STRING, const wchar_t *WSTRING,
size_t SIZE)
The `wcstombs' ("wide character string to multibyte string")
function converts the null-terminated wide character array WSTRING
into a string containing multibyte characters, storing not more
than SIZE bytes starting at STRING, followed by a terminating null
character if there is room. The conversion of characters begins in
the initial shift state.
The terminating null character counts towards the size, so if SIZE
is less than or equal to the number of bytes needed in WSTRING, no
terminating null character is stored.
If a code that does not correspond to a valid multibyte character
is found, the `wcstombs' function returns a value of -1.
Otherwise, the return value is the number of bytes stored in the
array STRING. This number does not include the terminating null
character, which is present if the number is less than SIZE.
File: libc.info, Node: Shift State, Prev: Non-reentrant String Conversion, Up: Non-reentrant Conversion
6.4.3 States in Non-reentrant Functions
---------------------------------------
In some multibyte character codes, the _meaning_ of any particular byte
sequence is not fixed; it depends on what other sequences have come
earlier in the same string. Typically there are just a few sequences
that can change the meaning of other sequences; these few are called
"shift sequences" and we say that they set the "shift state" for other
sequences that follow.
To illustrate shift state and shift sequences, suppose we decide that
the sequence `0200' (just one byte) enters Japanese mode, in which
pairs of bytes in the range from `0240' to `0377' are single
characters, while `0201' enters Latin-1 mode, in which single bytes in
the range from `0240' to `0377' are characters, and interpreted
according to the ISO Latin-1 character set. This is a multibyte code
that has two alternative shift states ("Japanese mode" and "Latin-1
mode"), and two shift sequences that specify particular shift states.
When the multibyte character code in use has shift states, then
`mblen', `mbtowc', and `wctomb' must maintain and update the current
shift state as they scan the string. To make this work properly, you
must follow these rules:
* Before starting to scan a string, call the function with a null
pointer for the multibyte character address--for example, `mblen
(NULL, 0)'. This initializes the shift state to its standard
initial value.
* Scan the string one character at a time, in order. Do not "back
up" and rescan characters already scanned, and do not intersperse
the processing of different strings.
Here is an example of using `mblen' following these rules:
void
scan_string (char *s)
{
int length = strlen (s);
/* Initialize shift state. */
mblen (NULL, 0);
while (1)
{
int thischar = mblen (s, length);
/* Deal with end of string and invalid characters. */
if (thischar == 0)
break;
if (thischar == -1)
{
error ("invalid multibyte character");
break;
}
/* Advance past this character. */
s += thischar;
length -= thischar;
}
}
The functions `mblen', `mbtowc' and `wctomb' are not reentrant when
using a multibyte code that uses a shift state. However, no other
library functions call these functions, so you don't have to worry that
the shift state will be changed mysteriously.
File: libc.info, Node: Generic Charset Conversion, Prev: Non-reentrant Conversion, Up: Character Set Handling
6.5 Generic Charset Conversion
==============================
The conversion functions mentioned so far in this chapter all had in
common that they operate on character sets that are not directly
specified by the functions. The multibyte encoding used is specified by
the currently selected locale for the `LC_CTYPE' category. The wide
character set is fixed by the implementation (in the case of GNU C
library it is always UCS-4 encoded ISO 10646.
This has of course several problems when it comes to general
character conversion:
* For every conversion where neither the source nor the destination
character set is the character set of the locale for the `LC_CTYPE'
category, one has to change the `LC_CTYPE' locale using
`setlocale'.
Changing the `LC_TYPE' locale introduces major problems for the
rest of the programs since several more functions (e.g., the
character classification functions, *note Classification of
Characters::) use the `LC_CTYPE' category.
* Parallel conversions to and from different character sets are not
possible since the `LC_CTYPE' selection is global and shared by all
threads.
* If neither the source nor the destination character set is the
character set used for `wchar_t' representation, there is at least
a two-step process necessary to convert a text using the functions
above. One would have to select the source character set as the
multibyte encoding, convert the text into a `wchar_t' text, select
the destination character set as the multibyte encoding, and
convert the wide character text to the multibyte (= destination)
character set.
Even if this is possible (which is not guaranteed) it is a very
tiring work. Plus it suffers from the other two raised points
even more due to the steady changing of the locale.
The XPG2 standard defines a completely new set of functions, which
has none of these limitations. They are not at all coupled to the
selected locales, and they have no constraints on the character sets
selected for source and destination. Only the set of available
conversions limits them. The standard does not specify that any
conversion at all must be available. Such availability is a measure of
the quality of the implementation.
In the following text first the interface to `iconv' and then the
conversion function, will be described. Comparisons with other
implementations will show what obstacles stand in the way of portable
applications. Finally, the implementation is described in so far as
might interest the advanced user who wants to extend conversion
capabilities.
* Menu:
* Generic Conversion Interface:: Generic Character Set Conversion Interface.
* iconv Examples:: A complete `iconv' example.
* Other iconv Implementations:: Some Details about other `iconv'
Implementations.
* glibc iconv Implementation:: The `iconv' Implementation in the GNU C
library.
File: libc.info, Node: Generic Conversion Interface, Next: iconv Examples, Up: Generic Charset Conversion
6.5.1 Generic Character Set Conversion Interface
------------------------------------------------
This set of functions follows the traditional cycle of using a resource:
open-use-close. The interface consists of three functions, each of
which implements one step.
Before the interfaces are described it is necessary to introduce a
data type. Just like other open-use-close interfaces the functions
introduced here work using handles and the `iconv.h' header defines a
special type for the handles used.
-- Data Type: iconv_t
This data type is an abstract type defined in `iconv.h'. The user
must not assume anything about the definition of this type; it
must be completely opaque.
Objects of this type can get assigned handles for the conversions
using the `iconv' functions. The objects themselves need not be
freed, but the conversions for which the handles stand for have to.
The first step is the function to create a handle.
-- Function: iconv_t iconv_open (const char *TOCODE, const char
*FROMCODE)
The `iconv_open' function has to be used before starting a
conversion. The two parameters this function takes determine the
source and destination character set for the conversion, and if the
implementation has the possibility to perform such a conversion,
the function returns a handle.
If the wanted conversion is not available, the `iconv_open'
function returns `(iconv_t) -1'. In this case the global variable
`errno' can have the following values:
`EMFILE'
The process already has `OPEN_MAX' file descriptors open.
`ENFILE'
The system limit of open file is reached.
`ENOMEM'
Not enough memory to carry out the operation.
`EINVAL'
The conversion from FROMCODE to TOCODE is not supported.
It is not possible to use the same descriptor in different threads
to perform independent conversions. The data structures associated
with the descriptor include information about the conversion state.
This must not be messed up by using it in different conversions.
An `iconv' descriptor is like a file descriptor as for every use a
new descriptor must be created. The descriptor does not stand for
all of the conversions from FROMSET to TOSET.
The GNU C library implementation of `iconv_open' has one
significant extension to other implementations. To ease the
extension of the set of available conversions, the implementation
allows storing the necessary files with data and code in an
arbitrary number of directories. How this extension must be
written will be explained below (*note glibc iconv
Implementation::). Here it is only important to say that all
directories mentioned in the `GCONV_PATH' environment variable are
considered only if they contain a file `gconv-modules'. These
directories need not necessarily be created by the system
administrator. In fact, this extension is introduced to help users
writing and using their own, new conversions. Of course, this
does not work for security reasons in SUID binaries; in this case
only the system directory is considered and this normally is
`PREFIX/lib/gconv'. The `GCONV_PATH' environment variable is
examined exactly once at the first call of the `iconv_open'
function. Later modifications of the variable have no effect.
The `iconv_open' function was introduced early in the X/Open
Portability Guide, version 2. It is supported by all commercial
Unices as it is required for the Unix branding. However, the
quality and completeness of the implementation varies widely. The
`iconv_open' function is declared in `iconv.h'.
The `iconv' implementation can associate large data structure with
the handle returned by `iconv_open'. Therefore, it is crucial to free
all the resources once all conversions are carried out and the
conversion is not needed anymore.
-- Function: int iconv_close (iconv_t CD)
The `iconv_close' function frees all resources associated with the
handle CD, which must have been returned by a successful call to
the `iconv_open' function.
If the function call was successful the return value is 0.
Otherwise it is -1 and `errno' is set appropriately. Defined
error are:
`EBADF'
The conversion descriptor is invalid.
The `iconv_close' function was introduced together with the rest
of the `iconv' functions in XPG2 and is declared in `iconv.h'.
The standard defines only one actual conversion function. This has,
therefore, the most general interface: it allows conversion from one
buffer to another. Conversion from a file to a buffer, vice versa, or
even file to file can be implemented on top of it.
-- Function: size_t iconv (iconv_t CD, char **INBUF, size_t
*INBYTESLEFT, char **OUTBUF, size_t *OUTBYTESLEFT)
The `iconv' function converts the text in the input buffer
according to the rules associated with the descriptor CD and
stores the result in the output buffer. It is possible to call the
function for the same text several times in a row since for
stateful character sets the necessary state information is kept in
the data structures associated with the descriptor.
The input buffer is specified by `*INBUF' and it contains
`*INBYTESLEFT' bytes. The extra indirection is necessary for
communicating the used input back to the caller (see below). It is
important to note that the buffer pointer is of type `char' and the
length is measured in bytes even if the input text is encoded in
wide characters.
The output buffer is specified in a similar way. `*OUTBUF' points
to the beginning of the buffer with at least `*OUTBYTESLEFT' bytes
room for the result. The buffer pointer again is of type `char'
and the length is measured in bytes. If OUTBUF or `*OUTBUF' is a
null pointer, the conversion is performed but no output is
available.
If INBUF is a null pointer, the `iconv' function performs the
necessary action to put the state of the conversion into the
initial state. This is obviously a no-op for non-stateful
encodings, but if the encoding has a state, such a function call
might put some byte sequences in the output buffer, which perform
the necessary state changes. The next call with INBUF not being a
null pointer then simply goes on from the initial state. It is
important that the programmer never makes any assumption as to
whether the conversion has to deal with states. Even if the input
and output character sets are not stateful, the implementation
might still have to keep states. This is due to the
implementation chosen for the GNU C library as it is described
below. Therefore an `iconv' call to reset the state should always
be performed if some protocol requires this for the output text.
The conversion stops for one of three reasons. The first is that
all characters from the input buffer are converted. This actually
can mean two things: either all bytes from the input buffer are
consumed or there are some bytes at the end of the buffer that
possibly can form a complete character but the input is
incomplete. The second reason for a stop is that the output
buffer is full. And the third reason is that the input contains
invalid characters.
In all of these cases the buffer pointers after the last successful
conversion, for input and output buffer, are stored in INBUF and
OUTBUF, and the available room in each buffer is stored in
INBYTESLEFT and OUTBYTESLEFT.
Since the character sets selected in the `iconv_open' call can be
almost arbitrary, there can be situations where the input buffer
contains valid characters, which have no identical representation
in the output character set. The behavior in this situation is
undefined. The _current_ behavior of the GNU C library in this
situation is to return with an error immediately. This certainly
is not the most desirable solution; therefore, future versions
will provide better ones, but they are not yet finished.
If all input from the input buffer is successfully converted and
stored in the output buffer, the function returns the number of
non-reversible conversions performed. In all other cases the
return value is `(size_t) -1' and `errno' is set appropriately.
In such cases the value pointed to by INBYTESLEFT is nonzero.
`EILSEQ'
The conversion stopped because of an invalid byte sequence in
the input. After the call, `*INBUF' points at the first byte
of the invalid byte sequence.
`E2BIG'
The conversion stopped because it ran out of space in the
output buffer.
`EINVAL'
The conversion stopped because of an incomplete byte sequence
at the end of the input buffer.
`EBADF'
The CD argument is invalid.
The `iconv' function was introduced in the XPG2 standard and is
declared in the `iconv.h' header.
The definition of the `iconv' function is quite good overall. It
provides quite flexible functionality. The only problems lie in the
boundary cases, which are incomplete byte sequences at the end of the
input buffer and invalid input. A third problem, which is not really a
design problem, is the way conversions are selected. The standard does
not say anything about the legitimate names, a minimal set of available
conversions. We will see how this negatively impacts other
implementations, as demonstrated below.
File: libc.info, Node: iconv Examples, Next: Other iconv Implementations, Prev: Generic Conversion Interface, Up: Generic Charset Conversion
6.5.2 A complete `iconv' example
--------------------------------
The example below features a solution for a common problem. Given that
one knows the internal encoding used by the system for `wchar_t'
strings, one often is in the position to read text from a file and store
it in wide character buffers. One can do this using `mbsrtowcs', but
then we run into the problems discussed above.
int
file2wcs (int fd, const char *charset, wchar_t *outbuf, size_t avail)
{
char inbuf[BUFSIZ];
size_t insize = 0;
char *wrptr = (char *) outbuf;
int result = 0;
iconv_t cd;
cd = iconv_open ("WCHAR_T", charset);
if (cd == (iconv_t) -1)
{
/* Something went wrong. */
if (errno == EINVAL)
error (0, 0, "conversion from '%s' to wchar_t not available",
charset);
else
perror ("iconv_open");
/* Terminate the output string. */
*outbuf = L'\0';
return -1;
}
while (avail > 0)
{
size_t nread;
size_t nconv;
char *inptr = inbuf;
/* Read more input. */
nread = read (fd, inbuf + insize, sizeof (inbuf) - insize);
if (nread == 0)
{
/* When we come here the file is completely read.
This still could mean there are some unused
characters in the `inbuf'. Put them back. */
if (lseek (fd, -insize, SEEK_CUR) == -1)
result = -1;
/* Now write out the byte sequence to get into the
initial state if this is necessary. */
iconv (cd, NULL, NULL, &wrptr, &avail);
break;
}
insize += nread;
/* Do the conversion. */
nconv = iconv (cd, &inptr, &insize, &wrptr, &avail);
if (nconv == (size_t) -1)
{
/* Not everything went right. It might only be
an unfinished byte sequence at the end of the
buffer. Or it is a real problem. */
if (errno == EINVAL)
/* This is harmless. Simply move the unused
bytes to the beginning of the buffer so that
they can be used in the next round. */
memmove (inbuf, inptr, insize);
else
{
/* It is a real problem. Maybe we ran out of
space in the output buffer or we have invalid
input. In any case back the file pointer to
the position of the last processed byte. */
lseek (fd, -insize, SEEK_CUR);
result = -1;
break;
}
}
}
/* Terminate the output string. */
if (avail >= sizeof (wchar_t))
*((wchar_t *) wrptr) = L'\0';
if (iconv_close (cd) != 0)
perror ("iconv_close");
return (wchar_t *) wrptr - outbuf;
}
This example shows the most important aspects of using the `iconv'
functions. It shows how successive calls to `iconv' can be used to
convert large amounts of text. The user does not have to care about
stateful encodings as the functions take care of everything.
An interesting point is the case where `iconv' returns an error and
`errno' is set to `EINVAL'. This is not really an error in the
transformation. It can happen whenever the input character set contains
byte sequences of more than one byte for some character and texts are
not processed in one piece. In this case there is a chance that a
multibyte sequence is cut. The caller can then simply read the
remainder of the takes and feed the offending bytes together with new
character from the input to `iconv' and continue the work. The
internal state kept in the descriptor is _not_ unspecified after such
an event as is the case with the conversion functions from the ISO C
standard.
The example also shows the problem of using wide character strings
with `iconv'. As explained in the description of the `iconv' function
above, the function always takes a pointer to a `char' array and the
available space is measured in bytes. In the example, the output
buffer is a wide character buffer; therefore, we use a local variable
WRPTR of type `char *', which is used in the `iconv' calls.
This looks rather innocent but can lead to problems on platforms that
have tight restriction on alignment. Therefore the caller of `iconv'
has to make sure that the pointers passed are suitable for access of
characters from the appropriate character set. Since, in the above
case, the input parameter to the function is a `wchar_t' pointer, this
is the case (unless the user violates alignment when computing the
parameter). But in other situations, especially when writing generic
functions where one does not know what type of character set one uses
and, therefore, treats text as a sequence of bytes, it might become
tricky.
File: libc.info, Node: Other iconv Implementations, Next: glibc iconv Implementation, Prev: iconv Examples, Up: Generic Charset Conversion
6.5.3 Some Details about other `iconv' Implementations
------------------------------------------------------
This is not really the place to discuss the `iconv' implementation of
other systems but it is necessary to know a bit about them to write
portable programs. The above mentioned problems with the specification
of the `iconv' functions can lead to portability issues.
The first thing to notice is that, due to the large number of
character sets in use, it is certainly not practical to encode the
conversions directly in the C library. Therefore, the conversion
information must come from files outside the C library. This is
usually done in one or both of the following ways:
* The C library contains a set of generic conversion functions that
can read the needed conversion tables and other information from
data files. These files get loaded when necessary.
This solution is problematic as it requires a great deal of effort
to apply to all character sets (potentially an infinite set). The
differences in the structure of the different character sets is so
large that many different variants of the table-processing
functions must be developed. In addition, the generic nature of
these functions make them slower than specifically implemented
functions.
* The C library only contains a framework that can dynamically load
object files and execute the conversion functions contained
therein.
This solution provides much more flexibility. The C library itself
contains only very little code and therefore reduces the general
memory footprint. Also, with a documented interface between the C
library and the loadable modules it is possible for third parties
to extend the set of available conversion modules. A drawback of
this solution is that dynamic loading must be available.
Some implementations in commercial Unices implement a mixture of
these possibilities; the majority implement only the second solution.
Using loadable modules moves the code out of the library itself and
keeps the door open for extensions and improvements, but this design is
also limiting on some platforms since not many platforms support dynamic
loading in statically linked programs. On platforms without this
capability it is therefore not possible to use this interface in
statically linked programs. The GNU C library has, on ELF platforms, no
problems with dynamic loading in these situations; therefore, this
point is moot. The danger is that one gets acquainted with this
situation and forgets about the restrictions on other systems.
A second thing to know about other `iconv' implementations is that
the number of available conversions is often very limited. Some
implementations provide, in the standard release (not special
international or developer releases), at most 100 to 200 conversion
possibilities. This does not mean 200 different character sets are
supported; for example, conversions from one character set to a set of
10 others might count as 10 conversions. Together with the other
direction this makes 20 conversion possibilities used up by one
character set. One can imagine the thin coverage these platform
provide. Some Unix vendors even provide only a handful of conversions,
which renders them useless for almost all uses.
This directly leads to a third and probably the most problematic
point. The way the `iconv' conversion functions are implemented on all
known Unix systems and the availability of the conversion functions from
character set A to B and the conversion from B to C does _not_ imply
that the conversion from A to C is available.
This might not seem unreasonable and problematic at first, but it is
a quite big problem as one will notice shortly after hitting it. To
show the problem we assume to write a program that has to convert from
A to C. A call like
cd = iconv_open ("C", "A");
fails according to the assumption above. But what does the program do
now? The conversion is necessary; therefore, simply giving up is not
an option.
This is a nuisance. The `iconv' function should take care of this.
But how should the program proceed from here on? If it tries to convert
to character set B, first the two `iconv_open' calls
cd1 = iconv_open ("B", "A");
and
cd2 = iconv_open ("C", "B");
will succeed, but how to find B?
Unfortunately, the answer is: there is no general solution. On some
systems guessing might help. On those systems most character sets can
convert to and from UTF-8 encoded ISO 10646 or Unicode text. Beside
this only some very system-specific methods can help. Since the
conversion functions come from loadable modules and these modules must
be stored somewhere in the filesystem, one _could_ try to find them and
determine from the available file which conversions are available and
whether there is an indirect route from A to C.
This example shows one of the design errors of `iconv' mentioned
above. It should at least be possible to determine the list of
available conversion programmatically so that if `iconv_open' says
there is no such conversion, one could make sure this also is true for
indirect routes.
File: libc.info, Node: glibc iconv Implementation, Prev: Other iconv Implementations, Up: Generic Charset Conversion
6.5.4 The `iconv' Implementation in the GNU C library
-----------------------------------------------------
After reading about the problems of `iconv' implementations in the last
section it is certainly good to note that the implementation in the GNU
C library has none of the problems mentioned above. What follows is a
step-by-step analysis of the points raised above. The evaluation is
based on the current state of the development (as of January 1999).
The development of the `iconv' functions is not complete, but basic
functionality has solidified.
The GNU C library's `iconv' implementation uses shared loadable
modules to implement the conversions. A very small number of
conversions are built into the library itself but these are only rather
trivial conversions.
All the benefits of loadable modules are available in the GNU C
library implementation. This is especially appealing since the
interface is well documented (see below), and it, therefore, is easy to
write new conversion modules. The drawback of using loadable objects
is not a problem in the GNU C library, at least on ELF systems. Since
the library is able to load shared objects even in statically linked
binaries, static linking need not be forbidden in case one wants to use
`iconv'.
The second mentioned problem is the number of supported conversions.
Currently, the GNU C library supports more than 150 character sets. The
way the implementation is designed the number of supported conversions
is greater than 22350 (150 times 149). If any conversion from or to a
character set is missing, it can be added easily.
Particularly impressive as it may be, this high number is due to the
fact that the GNU C library implementation of `iconv' does not have the
third problem mentioned above (i.e., whenever there is a conversion
from a character set A to B and from B to C it is always possible to
convert from A to C directly). If the `iconv_open' returns an error
and sets `errno' to `EINVAL', there is no known way, directly or
indirectly, to perform the wanted conversion.
Triangulation is achieved by providing for each character set a
conversion from and to UCS-4 encoded ISO 10646. Using ISO 10646 as an
intermediate representation it is possible to "triangulate" (i.e.,
convert with an intermediate representation).
There is no inherent requirement to provide a conversion to
ISO 10646 for a new character set, and it is also possible to provide
other conversions where neither source nor destination character set is
ISO 10646. The existing set of conversions is simply meant to cover all
conversions that might be of interest.
All currently available conversions use the triangulation method
above, making conversion run unnecessarily slow. If, for example,
somebody often needs the conversion from ISO-2022-JP to EUC-JP, a
quicker solution would involve direct conversion between the two
character sets, skipping the input to ISO 10646 first. The two
character sets of interest are much more similar to each other than to
ISO 10646.
In such a situation one easily can write a new conversion and
provide it as a better alternative. The GNU C library `iconv'
implementation would automatically use the module implementing the
conversion if it is specified to be more efficient.
6.5.4.1 Format of `gconv-modules' files
.......................................
All information about the available conversions comes from a file named
`gconv-modules', which can be found in any of the directories along the
`GCONV_PATH'. The `gconv-modules' files are line-oriented text files,
where each of the lines has one of the following formats:
* If the first non-whitespace character is a `#' the line contains
only comments and is ignored.
* Lines starting with `alias' define an alias name for a character
set. Two more words are expected on the line. The first word
defines the alias name, and the second defines the original name
of the character set. The effect is that it is possible to use
the alias name in the FROMSET or TOSET parameters of `iconv_open'
and achieve the same result as when using the real character set
name.
This is quite important as a character set has often many different
names. There is normally an official name but this need not
correspond to the most popular name. Beside this many character
sets have special names that are somehow constructed. For
example, all character sets specified by the ISO have an alias of
the form `ISO-IR-NNN' where NNN is the registration number. This
allows programs that know about the registration number to
construct character set names and use them in `iconv_open' calls.
More on the available names and aliases follows below.
* Lines starting with `module' introduce an available conversion
module. These lines must contain three or four more words.
The first word specifies the source character set, the second word
the destination character set of conversion implemented in this
module, and the third word is the name of the loadable module.
The filename is constructed by appending the usual shared object
suffix (normally `.so') and this file is then supposed to be found
in the same directory the `gconv-modules' file is in. The last
word on the line, which is optional, is a numeric value
representing the cost of the conversion. If this word is missing,
a cost of 1 is assumed. The numeric value itself does not matter
that much; what counts are the relative values of the sums of
costs for all possible conversion paths. Below is a more precise
description of the use of the cost value.
Returning to the example above where one has written a module to
directly convert from ISO-2022-JP to EUC-JP and back. All that has to
be done is to put the new module, let its name be ISO2022JP-EUCJP.so,
in a directory and add a file `gconv-modules' with the following
content in the same directory:
module ISO-2022-JP// EUC-JP// ISO2022JP-EUCJP 1
module EUC-JP// ISO-2022-JP// ISO2022JP-EUCJP 1
To see why this is sufficient, it is necessary to understand how the
conversion used by `iconv' (and described in the descriptor) is
selected. The approach to this problem is quite simple.
At the first call of the `iconv_open' function the program reads all
available `gconv-modules' files and builds up two tables: one
containing all the known aliases and another that contains the
information about the conversions and which shared object implements
them.
6.5.4.2 Finding the conversion path in `iconv'
..............................................
The set of available conversions form a directed graph with weighted
edges. The weights on the edges are the costs specified in the
`gconv-modules' files. The `iconv_open' function uses an algorithm
suitable for search for the best path in such a graph and so constructs
a list of conversions that must be performed in succession to get the
transformation from the source to the destination character set.
Explaining why the above `gconv-modules' files allows the `iconv'
implementation to resolve the specific ISO-2022-JP to EUC-JP conversion
module instead of the conversion coming with the library itself is
straightforward. Since the latter conversion takes two steps (from
ISO-2022-JP to ISO 10646 and then from ISO 10646 to EUC-JP), the cost
is 1+1 = 2. The above `gconv-modules' file, however, specifies that
the new conversion modules can perform this conversion with only the
cost of 1.
A mysterious item about the `gconv-modules' file above (and also the
file coming with the GNU C library) are the names of the character sets
specified in the `module' lines. Why do almost all the names end in
`//'? And this is not all: the names can actually be regular
expressions. At this point in time this mystery should not be
revealed, unless you have the relevant spell-casting materials: ashes
from an original DOS 6.2 boot disk burnt in effigy, a crucifix blessed
by St. Emacs, assorted herbal roots from Central America, sand from
Cebu, etc. Sorry! *The part of the implementation where this is used
is not yet finished. For now please simply follow the existing
examples. It'll become clearer once it is. -drepper*
A last remark about the `gconv-modules' is about the names not
ending with `//'. A character set named `INTERNAL' is often mentioned.
From the discussion above and the chosen name it should have become
clear that this is the name for the representation used in the
intermediate step of the triangulation. We have said that this is UCS-4
but actually that is not quite right. The UCS-4 specification also
includes the specification of the byte ordering used. Since a UCS-4
value consists of four bytes, a stored value is effected by byte
ordering. The internal representation is _not_ the same as UCS-4 in
case the byte ordering of the processor (or at least the running
process) is not the same as the one required for UCS-4. This is done
for performance reasons as one does not want to perform unnecessary
byte-swapping operations if one is not interested in actually seeing
the result in UCS-4. To avoid trouble with endianess, the internal
representation consistently is named `INTERNAL' even on big-endian
systems where the representations are identical.
6.5.4.3 `iconv' module data structures
......................................
So far this section has described how modules are located and considered
to be used. What remains to be described is the interface of the
modules so that one can write new ones. This section describes the
interface as it is in use in January 1999. The interface will change a
bit in the future but, with luck, only in an upwardly compatible way.
The definitions necessary to write new modules are publicly available
in the non-standard header `gconv.h'. The following text, therefore,
describes the definitions from this header file. First, however, it is
necessary to get an overview.
From the perspective of the user of `iconv' the interface is quite
simple: the `iconv_open' function returns a handle that can be used in
calls to `iconv', and finally the handle is freed with a call to
`iconv_close'. The problem is that the handle has to be able to
represent the possibly long sequences of conversion steps and also the
state of each conversion since the handle is all that is passed to the
`iconv' function. Therefore, the data structures are really the
elements necessary to understanding the implementation.
We need two different kinds of data structures. The first describes
the conversion and the second describes the state etc. There are
really two type definitions like this in `gconv.h'.
-- Data type: struct __gconv_step
This data structure describes one conversion a module can perform.
For each function in a loaded module with conversion functions
there is exactly one object of this type. This object is shared
by all users of the conversion (i.e., this object does not contain
any information corresponding to an actual conversion; it only
describes the conversion itself).
`struct __gconv_loaded_object *__shlib_handle'
`const char *__modname'
`int __counter'
All these elements of the structure are used internally in
the C library to coordinate loading and unloading the shared.
One must not expect any of the other elements to be
available or initialized.
`const char *__from_name'
`const char *__to_name'
`__from_name' and `__to_name' contain the names of the source
and destination character sets. They can be used to identify
the actual conversion to be carried out since one module
might implement conversions for more than one character set
and/or direction.
`gconv_fct __fct'
`gconv_init_fct __init_fct'
`gconv_end_fct __end_fct'
These elements contain pointers to the functions in the
loadable module. The interface will be explained below.
`int __min_needed_from'
`int __max_needed_from'
`int __min_needed_to'
`int __max_needed_to;'
These values have to be supplied in the init function of the
module. The `__min_needed_from' value specifies how many
bytes a character of the source character set at least needs.
The `__max_needed_from' specifies the maximum value that
also includes possible shift sequences.
The `__min_needed_to' and `__max_needed_to' values serve the
same purpose as `__min_needed_from' and `__max_needed_from'
but this time for the destination character set.
It is crucial that these values be accurate since otherwise
the conversion functions will have problems or not work at
all.
`int __stateful'
This element must also be initialized by the init function.
`int __stateful' is nonzero if the source character set is
stateful. Otherwise it is zero.
`void *__data'
This element can be used freely by the conversion functions
in the module. `void *__data' can be used to communicate
extra information from one call to another. `void *__data'
need not be initialized if not needed at all. If `void
*__data' element is assigned a pointer to dynamically
allocated memory (presumably in the init function) it has to
be made sure that the end function deallocates the memory.
Otherwise the application will leak memory.
It is important to be aware that this data structure is
shared by all users of this specification conversion and
therefore the `__data' element must not contain data specific
to one specific use of the conversion function.
-- Data type: struct __gconv_step_data
This is the data structure that contains the information specific
to each use of the conversion functions.
`char *__outbuf'
`char *__outbufend'
These elements specify the output buffer for the conversion
step. The `__outbuf' element points to the beginning of the
buffer, and `__outbufend' points to the byte following the
last byte in the buffer. The conversion function must not
assume anything about the size of the buffer but it can be
safely assumed the there is room for at least one complete
character in the output buffer.
Once the conversion is finished, if the conversion is the
last step, the `__outbuf' element must be modified to point
after the last byte written into the buffer to signal how
much output is available. If this conversion step is not the
last one, the element must not be modified. The
`__outbufend' element must not be modified.
`int __is_last'
This element is nonzero if this conversion step is the last
one. This information is necessary for the recursion. See
the description of the conversion function internals below.
This element must never be modified.
`int __invocation_counter'
The conversion function can use this element to see how many
calls of the conversion function already happened. Some
character sets require a certain prolog when generating
output, and by comparing this value with zero, one can find
out whether it is the first call and whether, therefore, the
prolog should be emitted. This element must never be
modified.
`int __internal_use'
This element is another one rarely used but needed in certain
situations. It is assigned a nonzero value in case the
conversion functions are used to implement `mbsrtowcs' et.al.
(i.e., the function is not used directly through the `iconv'
interface).
This sometimes makes a difference as it is expected that the
`iconv' functions are used to translate entire texts while the
`mbsrtowcs' functions are normally used only to convert single
strings and might be used multiple times to convert entire
texts.
But in this situation we would have problem complying with
some rules of the character set specification. Some
character sets require a prolog, which must appear exactly
once for an entire text. If a number of `mbsrtowcs' calls
are used to convert the text, only the first call must add
the prolog. However, because there is no communication
between the different calls of `mbsrtowcs', the conversion
functions have no possibility to find this out. The
situation is different for sequences of `iconv' calls since
the handle allows access to the needed information.
The `int __internal_use' element is mostly used together with
`__invocation_counter' as follows:
if (!data->__internal_use
&& data->__invocation_counter == 0)
/* Emit prolog. */
...
This element must never be modified.
`mbstate_t *__statep'
The `__statep' element points to an object of type `mbstate_t'
(*note Keeping the state::). The conversion of a stateful
character set must use the object pointed to by `__statep' to
store information about the conversion state. The `__statep'
element itself must never be modified.
`mbstate_t __state'
This element must _never_ be used directly. It is only part
of this structure to have the needed space allocated.
6.5.4.4 `iconv' module interfaces
.................................
With the knowledge about the data structures we now can describe the
conversion function itself. To understand the interface a bit of
knowledge is necessary about the functionality in the C library that
loads the objects with the conversions.
It is often the case that one conversion is used more than once
(i.e., there are several `iconv_open' calls for the same set of
character sets during one program run). The `mbsrtowcs' et.al.
functions in the GNU C library also use the `iconv' functionality, which
increases the number of uses of the same functions even more.
Because of this multiple use of conversions, the modules do not get
loaded exclusively for one conversion. Instead a module once loaded can
be used by an arbitrary number of `iconv' or `mbsrtowcs' calls at the
same time. The splitting of the information between conversion-
function-specific information and conversion data makes this possible.
The last section showed the two data structures used to do this.
This is of course also reflected in the interface and semantics of
the functions that the modules must provide. There are three functions
that must have the following names:
`gconv_init'
The `gconv_init' function initializes the conversion function
specific data structure. This very same object is shared by all
conversions that use this conversion and, therefore, no state
information about the conversion itself must be stored in here.
If a module implements more than one conversion, the `gconv_init'
function will be called multiple times.
`gconv_end'
The `gconv_end' function is responsible for freeing all resources
allocated by the `gconv_init' function. If there is nothing to do,
this function can be missing. Special care must be taken if the
module implements more than one conversion and the `gconv_init'
function does not allocate the same resources for all conversions.
`gconv'
This is the actual conversion function. It is called to convert
one block of text. It gets passed the conversion step information
initialized by `gconv_init' and the conversion data, specific to
this use of the conversion functions.
There are three data types defined for the three module interface
functions and these define the interface.
-- Data type: int (*__gconv_init_fct) (struct __gconv_step *)
This specifies the interface of the initialization function of the
module. It is called exactly once for each conversion the module
implements.
As explained in the description of the `struct __gconv_step' data
structure above the initialization function has to initialize
parts of it.
`__min_needed_from'
`__max_needed_from'
`__min_needed_to'
`__max_needed_to'
These elements must be initialized to the exact numbers of
the minimum and maximum number of bytes used by one character
in the source and destination character sets, respectively.
If the characters all have the same size, the minimum and
maximum values are the same.
`__stateful'
This element must be initialized to an nonzero value if the
source character set is stateful. Otherwise it must be zero.
If the initialization function needs to communicate some
information to the conversion function, this communication can
happen using the `__data' element of the `__gconv_step' structure.
But since this data is shared by all the conversions, it must not
be modified by the conversion function. The example below shows
how this can be used.
#define MIN_NEEDED_FROM 1
#define MAX_NEEDED_FROM 4
#define MIN_NEEDED_TO 4
#define MAX_NEEDED_TO 4
int
gconv_init (struct __gconv_step *step)
{
/* Determine which direction. */
struct iso2022jp_data *new_data;
enum direction dir = illegal_dir;
enum variant var = illegal_var;
int result;
if (__strcasecmp (step->__from_name, "ISO-2022-JP//") == 0)
{
dir = from_iso2022jp;
var = iso2022jp;
}
else if (__strcasecmp (step->__to_name, "ISO-2022-JP//") == 0)
{
dir = to_iso2022jp;
var = iso2022jp;
}
else if (__strcasecmp (step->__from_name, "ISO-2022-JP-2//") == 0)
{
dir = from_iso2022jp;
var = iso2022jp2;
}
else if (__strcasecmp (step->__to_name, "ISO-2022-JP-2//") == 0)
{
dir = to_iso2022jp;
var = iso2022jp2;
}
result = __GCONV_NOCONV;
if (dir != illegal_dir)
{
new_data = (struct iso2022jp_data *)
malloc (sizeof (struct iso2022jp_data));
result = __GCONV_NOMEM;
if (new_data != NULL)
{
new_data->dir = dir;
new_data->var = var;
step->__data = new_data;
if (dir == from_iso2022jp)
{
step->__min_needed_from = MIN_NEEDED_FROM;
step->__max_needed_from = MAX_NEEDED_FROM;
step->__min_needed_to = MIN_NEEDED_TO;
step->__max_needed_to = MAX_NEEDED_TO;
}
else
{
step->__min_needed_from = MIN_NEEDED_TO;
step->__max_needed_from = MAX_NEEDED_TO;
step->__min_needed_to = MIN_NEEDED_FROM;
step->__max_needed_to = MAX_NEEDED_FROM + 2;
}
/* Yes, this is a stateful encoding. */
step->__stateful = 1;
result = __GCONV_OK;
}
}
return result;
}
The function first checks which conversion is wanted. The module
from which this function is taken implements four different
conversions; which one is selected can be determined by comparing
the names. The comparison should always be done without paying
attention to the case.
Next, a data structure, which contains the necessary information
about which conversion is selected, is allocated. The data
structure `struct iso2022jp_data' is locally defined since,
outside the module, this data is not used at all. Please note
that if all four conversions this modules supports are requested
there are four data blocks.
One interesting thing is the initialization of the `__min_' and
`__max_' elements of the step data object. A single ISO-2022-JP
character can consist of one to four bytes. Therefore the
`MIN_NEEDED_FROM' and `MAX_NEEDED_FROM' macros are defined this
way. The output is always the `INTERNAL' character set (aka
UCS-4) and therefore each character consists of exactly four
bytes. For the conversion from `INTERNAL' to ISO-2022-JP we have
to take into account that escape sequences might be necessary to
switch the character sets. Therefore the `__max_needed_to'
element for this direction gets assigned `MAX_NEEDED_FROM + 2'.
This takes into account the two bytes needed for the escape
sequences to single the switching. The asymmetry in the maximum
values for the two directions can be explained easily: when
reading ISO-2022-JP text, escape sequences can be handled alone
(i.e., it is not necessary to process a real character since the
effect of the escape sequence can be recorded in the state
information). The situation is different for the other direction.
Since it is in general not known which character comes next, one
cannot emit escape sequences to change the state in advance. This
means the escape sequences that have to be emitted together with
the next character. Therefore one needs more room than only for
the character itself.
The possible return values of the initialization function are:
`__GCONV_OK'
The initialization succeeded
`__GCONV_NOCONV'
The requested conversion is not supported in the module.
This can happen if the `gconv-modules' file has errors.
`__GCONV_NOMEM'
Memory required to store additional information could not be
allocated.
The function called before the module is unloaded is significantly
easier. It often has nothing at all to do; in which case it can be left
out completely.
-- Data type: void (*__gconv_end_fct) (struct gconv_step *)
The task of this function is to free all resources allocated in the
initialization function. Therefore only the `__data' element of
the object pointed to by the argument is of interest. Continuing
the example from the initialization function, the finalization
function looks like this:
void
gconv_end (struct __gconv_step *data)
{
free (data->__data);
}
The most important function is the conversion function itself, which
can get quite complicated for complex character sets. But since this
is not of interest here, we will only describe a possible skeleton for
the conversion function.
-- Data type: int (*__gconv_fct) (struct __gconv_step *, struct
__gconv_step_data *, const char **, const char *, size_t *,
int)
The conversion function can be called for two basic reason: to
convert text or to reset the state. From the description of the
`iconv' function it can be seen why the flushing mode is
necessary. What mode is selected is determined by the sixth
argument, an integer. This argument being nonzero means that
flushing is selected.
Common to both modes is where the output buffer can be found. The
information about this buffer is stored in the conversion step
data. A pointer to this information is passed as the second
argument to this function. The description of the `struct
__gconv_step_data' structure has more information on the
conversion step data.
What has to be done for flushing depends on the source character
set. If the source character set is not stateful, nothing has to
be done. Otherwise the function has to emit a byte sequence to
bring the state object into the initial state. Once this all
happened the other conversion modules in the chain of conversions
have to get the same chance. Whether another step follows can be
determined from the `__is_last' element of the step data structure
to which the first parameter points.
The more interesting mode is when actual text has to be converted.
The first step in this case is to convert as much text as
possible from the input buffer and store the result in the output
buffer. The start of the input buffer is determined by the third
argument, which is a pointer to a pointer variable referencing the
beginning of the buffer. The fourth argument is a pointer to the
byte right after the last byte in the buffer.
The conversion has to be performed according to the current state
if the character set is stateful. The state is stored in an
object pointed to by the `__statep' element of the step data
(second argument). Once either the input buffer is empty or the
output buffer is full the conversion stops. At this point, the
pointer variable referenced by the third parameter must point to
the byte following the last processed byte (i.e., if all of the
input is consumed, this pointer and the fourth parameter have the
same value).
What now happens depends on whether this step is the last one. If
it is the last step, the only thing that has to be done is to
update the `__outbuf' element of the step data structure to point
after the last written byte. This update gives the caller the
information on how much text is available in the output buffer.
In addition, the variable pointed to by the fifth parameter, which
is of type `size_t', must be incremented by the number of
characters (_not bytes_) that were converted in a non-reversible
way. Then, the function can return.
In case the step is not the last one, the later conversion
functions have to get a chance to do their work. Therefore, the
appropriate conversion function has to be called. The information
about the functions is stored in the conversion data structures,
passed as the first parameter. This information and the step data
are stored in arrays, so the next element in both cases can be
found by simple pointer arithmetic:
int
gconv (struct __gconv_step *step, struct __gconv_step_data *data,
const char **inbuf, const char *inbufend, size_t *written,
int do_flush)
{
struct __gconv_step *next_step = step + 1;
struct __gconv_step_data *next_data = data + 1;
...
The `next_step' pointer references the next step information and
`next_data' the next data record. The call of the next function
therefore will look similar to this:
next_step->__fct (next_step, next_data, &outerr, outbuf,
written, 0)
But this is not yet all. Once the function call returns the
conversion function might have some more to do. If the return
value of the function is `__GCONV_EMPTY_INPUT', more room is
available in the output buffer. Unless the input buffer is empty
the conversion, functions start all over again and process the
rest of the input buffer. If the return value is not
`__GCONV_EMPTY_INPUT', something went wrong and we have to recover
from this.
A requirement for the conversion function is that the input buffer
pointer (the third argument) always point to the last character
that was put in converted form into the output buffer. This is
trivially true after the conversion performed in the current step,
but if the conversion functions deeper downstream stop
prematurely, not all characters from the output buffer are
consumed and, therefore, the input buffer pointers must be backed
off to the right position.
Correcting the input buffers is easy to do if the input and output
character sets have a fixed width for all characters. In this
situation we can compute how many characters are left in the
output buffer and, therefore, can correct the input buffer pointer
appropriately with a similar computation. Things are getting
tricky if either character set has characters represented with
variable length byte sequences, and it gets even more complicated
if the conversion has to take care of the state. In these cases
the conversion has to be performed once again, from the known
state before the initial conversion (i.e., if necessary the state
of the conversion has to be reset and the conversion loop has to be
executed again). The difference now is that it is known how much
input must be created, and the conversion can stop before
converting the first unused character. Once this is done the
input buffer pointers must be updated again and the function can
return.
One final thing should be mentioned. If it is necessary for the
conversion to know whether it is the first invocation (in case a
prolog has to be emitted), the conversion function should
increment the `__invocation_counter' element of the step data
structure just before returning to the caller. See the
description of the `struct __gconv_step_data' structure above for
more information on how this can be used.
The return value must be one of the following values:
`__GCONV_EMPTY_INPUT'
All input was consumed and there is room left in the output
buffer.
`__GCONV_FULL_OUTPUT'
No more room in the output buffer. In case this is not the
last step this value is propagated down from the call of the
next conversion function in the chain.
`__GCONV_INCOMPLETE_INPUT'
The input buffer is not entirely empty since it contains an
incomplete character sequence.
The following example provides a framework for a conversion
function. In case a new conversion has to be written the holes in
this implementation have to be filled and that is it.
int
gconv (struct __gconv_step *step, struct __gconv_step_data *data,
const char **inbuf, const char *inbufend, size_t *written,
int do_flush)
{
struct __gconv_step *next_step = step + 1;
struct __gconv_step_data *next_data = data + 1;
gconv_fct fct = next_step->__fct;
int status;
/* If the function is called with no input this means we have
to reset to the initial state. The possibly partly
converted input is dropped. */
if (do_flush)
{
status = __GCONV_OK;
/* Possible emit a byte sequence which put the state object
into the initial state. */
/* Call the steps down the chain if there are any but only
if we successfully emitted the escape sequence. */
if (status == __GCONV_OK && ! data->__is_last)
status = fct (next_step, next_data, NULL, NULL,
written, 1);
}
else
{
/* We preserve the initial values of the pointer variables. */
const char *inptr = *inbuf;
char *outbuf = data->__outbuf;
char *outend = data->__outbufend;
char *outptr;
do
{
/* Remember the start value for this round. */
inptr = *inbuf;
/* The outbuf buffer is empty. */
outptr = outbuf;
/* For stateful encodings the state must be safe here. */
/* Run the conversion loop. `status' is set
appropriately afterwards. */
/* If this is the last step, leave the loop. There is
nothing we can do. */
if (data->__is_last)
{
/* Store information about how many bytes are
available. */
data->__outbuf = outbuf;
/* If any non-reversible conversions were performed,
add the number to `*written'. */
break;
}
/* Write out all output that was produced. */
if (outbuf > outptr)
{
const char *outerr = data->__outbuf;
int result;
result = fct (next_step, next_data, &outerr,
outbuf, written, 0);
if (result != __GCONV_EMPTY_INPUT)
{
if (outerr != outbuf)
{
/* Reset the input buffer pointer. We
document here the complex case. */
size_t nstatus;
/* Reload the pointers. */
*inbuf = inptr;
outbuf = outptr;
/* Possibly reset the state. */
/* Redo the conversion, but this time
the end of the output buffer is at
`outerr'. */
}
/* Change the status. */
status = result;
}
else
/* All the output is consumed, we can make
another run if everything was ok. */
if (status == __GCONV_FULL_OUTPUT)
status = __GCONV_OK;
}
}
while (status == __GCONV_OK);
/* We finished one use of this step. */
++data->__invocation_counter;
}
return status;
}
This information should be sufficient to write new modules. Anybody
doing so should also take a look at the available source code in the GNU
C library sources. It contains many examples of working and optimized
modules.
File: libc.info, Node: Locales, Next: Message Translation, Prev: Character Set Handling, Up: Top
7 Locales and Internationalization
**********************************
Different countries and cultures have varying conventions for how to
communicate. These conventions range from very simple ones, such as the
format for representing dates and times, to very complex ones, such as
the language spoken.
"Internationalization" of software means programming it to be able
to adapt to the user's favorite conventions. In ISO C,
internationalization works by means of "locales". Each locale
specifies a collection of conventions, one convention for each purpose.
The user chooses a set of conventions by specifying a locale (via
environment variables).
All programs inherit the chosen locale as part of their environment.
Provided the programs are written to obey the choice of locale, they
will follow the conventions preferred by the user.
* Menu:
* Effects of Locale:: Actions affected by the choice of
locale.
* Choosing Locale:: How the user specifies a locale.
* Locale Categories:: Different purposes for which you can
select a locale.
* Setting the Locale:: How a program specifies the locale
with library functions.
* Standard Locales:: Locale names available on all systems.
* Locale Information:: How to access the information for the locale.
* Formatting Numbers:: A dedicated function to format numbers.
* Yes-or-No Questions:: Check a Response against the locale.
File: libc.info, Node: Effects of Locale, Next: Choosing Locale, Up: Locales
7.1 What Effects a Locale Has
=============================
Each locale specifies conventions for several purposes, including the
following:
* What multibyte character sequences are valid, and how they are
interpreted (*note Character Set Handling::).
* Classification of which characters in the local character set are
considered alphabetic, and upper- and lower-case conversion
conventions (*note Character Handling::).
* The collating sequence for the local language and character set
(*note Collation Functions::).
* Formatting of numbers and currency amounts (*note General
Numeric::).
* Formatting of dates and times (*note Formatting Calendar Time::).
* What language to use for output, including error messages (*note
Message Translation::).
* What language to use for user answers to yes-or-no questions
(*note Yes-or-No Questions::).
* What language to use for more complex user input. (The C library
doesn't yet help you implement this.)
Some aspects of adapting to the specified locale are handled
automatically by the library subroutines. For example, all your program
needs to do in order to use the collating sequence of the chosen locale
is to use `strcoll' or `strxfrm' to compare strings.
Other aspects of locales are beyond the comprehension of the library.
For example, the library can't automatically translate your program's
output messages into other languages. The only way you can support
output in the user's favorite language is to program this more or less
by hand. The C library provides functions to handle translations for
multiple languages easily.
This chapter discusses the mechanism by which you can modify the
current locale. The effects of the current locale on specific library
functions are discussed in more detail in the descriptions of those
functions.
File: libc.info, Node: Choosing Locale, Next: Locale Categories, Prev: Effects of Locale, Up: Locales
7.2 Choosing a Locale
=====================
The simplest way for the user to choose a locale is to set the
environment variable `LANG'. This specifies a single locale to use for
all purposes. For example, a user could specify a hypothetical locale
named `espana-castellano' to use the standard conventions of most of
Spain.
The set of locales supported depends on the operating system you are
using, and so do their names. We can't make any promises about what
locales will exist, except for one standard locale called `C' or
`POSIX'. Later we will describe how to construct locales.
A user also has the option of specifying different locales for
different purposes--in effect, choosing a mixture of multiple locales.
For example, the user might specify the locale `espana-castellano'
for most purposes, but specify the locale `usa-english' for currency
formatting. This might make sense if the user is a Spanish-speaking
American, working in Spanish, but representing monetary amounts in US
dollars.
Note that both locales `espana-castellano' and `usa-english', like
all locales, would include conventions for all of the purposes to which
locales apply. However, the user can choose to use each locale for a
particular subset of those purposes.
File: libc.info, Node: Locale Categories, Next: Setting the Locale, Prev: Choosing Locale, Up: Locales
7.3 Categories of Activities that Locales Affect
================================================
The purposes that locales serve are grouped into "categories", so that
a user or a program can choose the locale for each category
independently. Here is a table of categories; each name is both an
environment variable that a user can set, and a macro name that you can
use as an argument to `setlocale'.
`LC_COLLATE'
This category applies to collation of strings (functions `strcoll'
and `strxfrm'); see *Note Collation Functions::.
`LC_CTYPE'
This category applies to classification and conversion of
characters, and to multibyte and wide characters; see *Note
Character Handling::, and *Note Character Set Handling::.
`LC_MONETARY'
This category applies to formatting monetary values; see *Note
General Numeric::.
`LC_NUMERIC'
This category applies to formatting numeric values that are not
monetary; see *Note General Numeric::.
`LC_TIME'
This category applies to formatting date and time values; see
*Note Formatting Calendar Time::.
`LC_MESSAGES'
This category applies to selecting the language used in the user
interface for message translation (*note The Uniforum approach::;
*note Message catalogs a la X/Open::) and contains regular
expressions for affirmative and negative responses.
`LC_ALL'
This is not an environment variable; it is only a macro that you
can use with `setlocale' to set a single locale for all purposes.
Setting this environment variable overwrites all selections by the
other `LC_*' variables or `LANG'.
`LANG'
If this environment variable is defined, its value specifies the
locale to use for all purposes except as overridden by the
variables above.
When developing the message translation functions it was felt that
the functionality provided by the variables above is not sufficient.
For example, it should be possible to specify more than one locale name.
Take a Swedish user who better speaks German than English, and a program
whose messages are output in English by default. It should be possible
to specify that the first choice of language is Swedish, the second
German, and if this also fails to use English. This is possible with
the variable `LANGUAGE'. For further description of this GNU extension
see *Note Using gettextized software::.
File: libc.info, Node: Setting the Locale, Next: Standard Locales, Prev: Locale Categories, Up: Locales
7.4 How Programs Set the Locale
===============================
A C program inherits its locale environment variables when it starts up.
This happens automatically. However, these variables do not
automatically control the locale used by the library functions, because
ISO C says that all programs start by default in the standard `C'
locale. To use the locales specified by the environment, you must call
`setlocale'. Call it as follows:
setlocale (LC_ALL, "");
to select a locale based on the user choice of the appropriate
environment variables.
You can also use `setlocale' to specify a particular locale, for
general use or for a specific category.
The symbols in this section are defined in the header file
`locale.h'.
-- Function: char * setlocale (int CATEGORY, const char *LOCALE)
The function `setlocale' sets the current locale for category
CATEGORY to LOCALE. A list of all the locales the system provides
can be created by running
locale -a
If CATEGORY is `LC_ALL', this specifies the locale for all
purposes. The other possible values of CATEGORY specify an single
purpose (*note Locale Categories::).
You can also use this function to find out the current locale by
passing a null pointer as the LOCALE argument. In this case,
`setlocale' returns a string that is the name of the locale
currently selected for category CATEGORY.
The string returned by `setlocale' can be overwritten by subsequent
calls, so you should make a copy of the string (*note Copying and
Concatenation::) if you want to save it past any further calls to
`setlocale'. (The standard library is guaranteed never to call
`setlocale' itself.)
You should not modify the string returned by `setlocale'. It might
be the same string that was passed as an argument in a previous
call to `setlocale'. One requirement is that the CATEGORY must be
the same in the call the string was returned and the one when the
string is passed in as LOCALE parameter.
When you read the current locale for category `LC_ALL', the value
encodes the entire combination of selected locales for all
categories. In this case, the value is not just a single locale
name. In fact, we don't make any promises about what it looks
like. But if you specify the same "locale name" with `LC_ALL' in
a subsequent call to `setlocale', it restores the same combination
of locale selections.
To be sure you can use the returned string encoding the currently
selected locale at a later time, you must make a copy of the
string. It is not guaranteed that the returned pointer remains
valid over time.
When the LOCALE argument is not a null pointer, the string returned
by `setlocale' reflects the newly-modified locale.
If you specify an empty string for LOCALE, this means to read the
appropriate environment variable and use its value to select the
locale for CATEGORY.
If a nonempty string is given for LOCALE, then the locale of that
name is used if possible.
If you specify an invalid locale name, `setlocale' returns a null
pointer and leaves the current locale unchanged.
Here is an example showing how you might use `setlocale' to
temporarily switch to a new locale.
#include <stddef.h>
#include <locale.h>
#include <stdlib.h>
#include <string.h>
void
with_other_locale (char *new_locale,
void (*subroutine) (int),
int argument)
{
char *old_locale, *saved_locale;
/* Get the name of the current locale. */
old_locale = setlocale (LC_ALL, NULL);
/* Copy the name so it won't be clobbered by `setlocale'. */
saved_locale = strdup (old_locale);
if (saved_locale == NULL)
fatal ("Out of memory");
/* Now change the locale and do some stuff with it. */
setlocale (LC_ALL, new_locale);
(*subroutine) (argument);
/* Restore the original locale. */
setlocale (LC_ALL, saved_locale);
free (saved_locale);
}
*Portability Note:* Some ISO C systems may define additional locale
categories, and future versions of the library will do so. For
portability, assume that any symbol beginning with `LC_' might be
defined in `locale.h'.
File: libc.info, Node: Standard Locales, Next: Locale Information, Prev: Setting the Locale, Up: Locales
7.5 Standard Locales
====================
The only locale names you can count on finding on all operating systems
are these three standard ones:
`"C"'
This is the standard C locale. The attributes and behavior it
provides are specified in the ISO C standard. When your program
starts up, it initially uses this locale by default.
`"POSIX"'
This is the standard POSIX locale. Currently, it is an alias for
the standard C locale.
`""'
The empty name says to select a locale based on environment
variables. *Note Locale Categories::.
Defining and installing named locales is normally a responsibility of
the system administrator at your site (or the person who installed the
GNU C library). It is also possible for the user to create private
locales. All this will be discussed later when describing the tool to
do so.
If your program needs to use something other than the `C' locale, it
will be more portable if you use whatever locale the user specifies
with the environment, rather than trying to specify some non-standard
locale explicitly by name. Remember, different machines might have
different sets of locales installed.
File: libc.info, Node: Locale Information, Next: Formatting Numbers, Prev: Standard Locales, Up: Locales
7.6 Accessing Locale Information
================================
There are several ways to access locale information. The simplest way
is to let the C library itself do the work. Several of the functions
in this library implicitly access the locale data, and use what
information is provided by the currently selected locale. This is how
the locale model is meant to work normally.
As an example take the `strftime' function, which is meant to nicely
format date and time information (*note Formatting Calendar Time::).
Part of the standard information contained in the `LC_TIME' category is
the names of the months. Instead of requiring the programmer to take
care of providing the translations the `strftime' function does this
all by itself. `%A' in the format string is replaced by the
appropriate weekday name of the locale currently selected by `LC_TIME'.
This is an easy example, and wherever possible functions do things
automatically in this way.
But there are quite often situations when there is simply no function
to perform the task, or it is simply not possible to do the work
automatically. For these cases it is necessary to access the
information in the locale directly. To do this the C library provides
two functions: `localeconv' and `nl_langinfo'. The former is part of
ISO C and therefore portable, but has a brain-damaged interface. The
second is part of the Unix interface and is portable in as far as the
system follows the Unix standards.
* Menu:
* The Lame Way to Locale Data:: ISO C's `localeconv'.
* The Elegant and Fast Way:: X/Open's `nl_langinfo'.
File: libc.info, Node: The Lame Way to Locale Data, Next: The Elegant and Fast Way, Up: Locale Information
7.6.1 `localeconv': It is portable but ...
------------------------------------------
Together with the `setlocale' function the ISO C people invented the
`localeconv' function. It is a masterpiece of poor design. It is
expensive to use, not extendable, and not generally usable as it
provides access to only `LC_MONETARY' and `LC_NUMERIC' related
information. Nevertheless, if it is applicable to a given situation it
should be used since it is very portable. The function `strfmon'
formats monetary amounts according to the selected locale using this
information.
-- Function: struct lconv * localeconv (void)
The `localeconv' function returns a pointer to a structure whose
components contain information about how numeric and monetary
values should be formatted in the current locale.
You should not modify the structure or its contents. The
structure might be overwritten by subsequent calls to
`localeconv', or by calls to `setlocale', but no other function in
the library overwrites this value.
-- Data Type: struct lconv
`localeconv''s return value is of this data type. Its elements are
described in the following subsections.
If a member of the structure `struct lconv' has type `char', and the
value is `CHAR_MAX', it means that the current locale has no value for
that parameter.
* Menu:
* General Numeric:: Parameters for formatting numbers and
currency amounts.
* Currency Symbol:: How to print the symbol that identifies an
amount of money (e.g. `$').
* Sign of Money Amount:: How to print the (positive or negative) sign
for a monetary amount, if one exists.
File: libc.info, Node: General Numeric, Next: Currency Symbol, Up: The Lame Way to Locale Data
7.6.1.1 Generic Numeric Formatting Parameters
.............................................
These are the standard members of `struct lconv'; there may be others.
`char *decimal_point'
`char *mon_decimal_point'
These are the decimal-point separators used in formatting
non-monetary and monetary quantities, respectively. In the `C'
locale, the value of `decimal_point' is `"."', and the value of
`mon_decimal_point' is `""'.
`char *thousands_sep'
`char *mon_thousands_sep'
These are the separators used to delimit groups of digits to the
left of the decimal point in formatting non-monetary and monetary
quantities, respectively. In the `C' locale, both members have a
value of `""' (the empty string).
`char *grouping'
`char *mon_grouping'
These are strings that specify how to group the digits to the left
of the decimal point. `grouping' applies to non-monetary
quantities and `mon_grouping' applies to monetary quantities. Use
either `thousands_sep' or `mon_thousands_sep' to separate the digit
groups.
Each member of these strings is to be interpreted as an integer
value of type `char'. Successive numbers (from left to right)
give the sizes of successive groups (from right to left, starting
at the decimal point.) The last member is either `0', in which
case the previous member is used over and over again for all the
remaining groups, or `CHAR_MAX', in which case there is no more
grouping--or, put another way, any remaining digits form one large
group without separators.
For example, if `grouping' is `"\04\03\02"', the correct grouping
for the number `123456787654321' is `12', `34', `56', `78', `765',
`4321'. This uses a group of 4 digits at the end, preceded by a
group of 3 digits, preceded by groups of 2 digits (as many as
needed). With a separator of `,', the number would be printed as
`12,34,56,78,765,4321'.
A value of `"\03"' indicates repeated groups of three digits, as
normally used in the U.S.
In the standard `C' locale, both `grouping' and `mon_grouping'
have a value of `""'. This value specifies no grouping at all.
`char int_frac_digits'
`char frac_digits'
These are small integers indicating how many fractional digits (to
the right of the decimal point) should be displayed in a monetary
value in international and local formats, respectively. (Most
often, both members have the same value.)
In the standard `C' locale, both of these members have the value
`CHAR_MAX', meaning "unspecified". The ISO standard doesn't say
what to do when you find this value; we recommend printing no
fractional digits. (This locale also specifies the empty string
for `mon_decimal_point', so printing any fractional digits would be
confusing!)
File: libc.info, Node: Currency Symbol, Next: Sign of Money Amount, Prev: General Numeric, Up: The Lame Way to Locale Data
7.6.1.2 Printing the Currency Symbol
....................................
These members of the `struct lconv' structure specify how to print the
symbol to identify a monetary value--the international analog of `$'
for US dollars.
Each country has two standard currency symbols. The "local currency
symbol" is used commonly within the country, while the "international
currency symbol" is used internationally to refer to that country's
currency when it is necessary to indicate the country unambiguously.
For example, many countries use the dollar as their monetary unit,
and when dealing with international currencies it's important to specify
that one is dealing with (say) Canadian dollars instead of U.S. dollars
or Australian dollars. But when the context is known to be Canada,
there is no need to make this explicit--dollar amounts are implicitly
assumed to be in Canadian dollars.
`char *currency_symbol'
The local currency symbol for the selected locale.
In the standard `C' locale, this member has a value of `""' (the
empty string), meaning "unspecified". The ISO standard doesn't
say what to do when you find this value; we recommend you simply
print the empty string as you would print any other string pointed
to by this variable.
`char *int_curr_symbol'
The international currency symbol for the selected locale.
The value of `int_curr_symbol' should normally consist of a
three-letter abbreviation determined by the international standard
`ISO 4217 Codes for the Representation of Currency and Funds',
followed by a one-character separator (often a space).
In the standard `C' locale, this member has a value of `""' (the
empty string), meaning "unspecified". We recommend you simply
print the empty string as you would print any other string pointed
to by this variable.
`char p_cs_precedes'
`char n_cs_precedes'
`char int_p_cs_precedes'
`char int_n_cs_precedes'
These members are `1' if the `currency_symbol' or
`int_curr_symbol' strings should precede the value of a monetary
amount, or `0' if the strings should follow the value. The
`p_cs_precedes' and `int_p_cs_precedes' members apply to positive
amounts (or zero), and the `n_cs_precedes' and `int_n_cs_precedes'
members apply to negative amounts.
In the standard `C' locale, all of these members have a value of
`CHAR_MAX', meaning "unspecified". The ISO standard doesn't say
what to do when you find this value. We recommend printing the
currency symbol before the amount, which is right for most
countries. In other words, treat all nonzero values alike in
these members.
The members with the `int_' prefix apply to the `int_curr_symbol'
while the other two apply to `currency_symbol'.
`char p_sep_by_space'
`char n_sep_by_space'
`char int_p_sep_by_space'
`char int_n_sep_by_space'
These members are `1' if a space should appear between the
`currency_symbol' or `int_curr_symbol' strings and the amount, or
`0' if no space should appear. The `p_sep_by_space' and
`int_p_sep_by_space' members apply to positive amounts (or zero),
and the `n_sep_by_space' and `int_n_sep_by_space' members apply to
negative amounts.
In the standard `C' locale, all of these members have a value of
`CHAR_MAX', meaning "unspecified". The ISO standard doesn't say
what you should do when you find this value; we suggest you treat
it as 1 (print a space). In other words, treat all nonzero values
alike in these members.
The members with the `int_' prefix apply to the `int_curr_symbol'
while the other two apply to `currency_symbol'. There is one
specialty with the `int_curr_symbol', though. Since all legal
values contain a space at the end the string one either printf
this space (if the currency symbol must appear in front and must
be separated) or one has to avoid printing this character at all
(especially when at the end of the string).
File: libc.info, Node: Sign of Money Amount, Prev: Currency Symbol, Up: The Lame Way to Locale Data
7.6.1.3 Printing the Sign of a Monetary Amount
..............................................
These members of the `struct lconv' structure specify how to print the
sign (if any) of a monetary value.
`char *positive_sign'
`char *negative_sign'
These are strings used to indicate positive (or zero) and negative
monetary quantities, respectively.
In the standard `C' locale, both of these members have a value of
`""' (the empty string), meaning "unspecified".
The ISO standard doesn't say what to do when you find this value;
we recommend printing `positive_sign' as you find it, even if it is
empty. For a negative value, print `negative_sign' as you find it
unless both it and `positive_sign' are empty, in which case print
`-' instead. (Failing to indicate the sign at all seems rather
unreasonable.)
`char p_sign_posn'
`char n_sign_posn'
`char int_p_sign_posn'
`char int_n_sign_posn'
These members are small integers that indicate how to position the
sign for nonnegative and negative monetary quantities,
respectively. (The string used by the sign is what was specified
with `positive_sign' or `negative_sign'.) The possible values are
as follows:
`0'
The currency symbol and quantity should be surrounded by
parentheses.
`1'
Print the sign string before the quantity and currency symbol.
`2'
Print the sign string after the quantity and currency symbol.
`3'
Print the sign string right before the currency symbol.
`4'
Print the sign string right after the currency symbol.
`CHAR_MAX'
"Unspecified". Both members have this value in the standard
`C' locale.
The ISO standard doesn't say what you should do when the value is
`CHAR_MAX'. We recommend you print the sign after the currency
symbol.
The members with the `int_' prefix apply to the `int_curr_symbol'
while the other two apply to `currency_symbol'.
File: libc.info, Node: The Elegant and Fast Way, Prev: The Lame Way to Locale Data, Up: Locale Information
7.6.2 Pinpoint Access to Locale Data
------------------------------------
When writing the X/Open Portability Guide the authors realized that the
`localeconv' function is not enough to provide reasonable access to
locale information. The information which was meant to be available in
the locale (as later specified in the POSIX.1 standard) requires more
ways to access it. Therefore the `nl_langinfo' function was introduced.
-- Function: char * nl_langinfo (nl_item ITEM)
The `nl_langinfo' function can be used to access individual
elements of the locale categories. Unlike the `localeconv'
function, which returns all the information, `nl_langinfo' lets
the caller select what information it requires. This is very fast
and it is not a problem to call this function multiple times.
A second advantage is that in addition to the numeric and monetary
formatting information, information from the `LC_TIME' and
`LC_MESSAGES' categories is available.
The type `nl_type' is defined in `nl_types.h'. The argument ITEM
is a numeric value defined in the header `langinfo.h'. The X/Open
standard defines the following values:
`CODESET'
`nl_langinfo' returns a string with the name of the coded
character set used in the selected locale.
`ABDAY_1'
`ABDAY_2'
`ABDAY_3'
`ABDAY_4'
`ABDAY_5'
`ABDAY_6'
`ABDAY_7'
`nl_langinfo' returns the abbreviated weekday name. `ABDAY_1'
corresponds to Sunday.
`DAY_1'
`DAY_2'
`DAY_3'
`DAY_4'
`DAY_5'
`DAY_6'
`DAY_7'
Similar to `ABDAY_1' etc., but here the return value is the
unabbreviated weekday name.
`ABMON_1'
`ABMON_2'
`ABMON_3'
`ABMON_4'
`ABMON_5'
`ABMON_6'
`ABMON_7'
`ABMON_8'
`ABMON_9'
`ABMON_10'
`ABMON_11'
`ABMON_12'
The return value is abbreviated name of the month. `ABMON_1'
corresponds to January.
`MON_1'
`MON_2'
`MON_3'
`MON_4'
`MON_5'
`MON_6'
`MON_7'
`MON_8'
`MON_9'
`MON_10'
`MON_11'
`MON_12'
Similar to `ABMON_1' etc., but here the month names are not
abbreviated. Here the first value `MON_1' also corresponds
to January.
`AM_STR'
`PM_STR'
The return values are strings which can be used in the
representation of time as an hour from 1 to 12 plus an am/pm
specifier.
Note that in locales which do not use this time representation
these strings might be empty, in which case the am/pm format
cannot be used at all.
`D_T_FMT'
The return value can be used as a format string for
`strftime' to represent time and date in a locale-specific
way.
`D_FMT'
The return value can be used as a format string for
`strftime' to represent a date in a locale-specific way.
`T_FMT'
The return value can be used as a format string for
`strftime' to represent time in a locale-specific way.
`T_FMT_AMPM'
The return value can be used as a format string for
`strftime' to represent time in the am/pm format.
Note that if the am/pm format does not make any sense for the
selected locale, the return value might be the same as the
one for `T_FMT'.
`ERA'
The return value represents the era used in the current
locale.
Most locales do not define this value. An example of a
locale which does define this value is the Japanese one. In
Japan, the traditional representation of dates includes the
name of the era corresponding to the then-emperor's reign.
Normally it should not be necessary to use this value
directly. Specifying the `E' modifier in their format
strings causes the `strftime' func |