The R Project SVN R

Rev

Rev 73489 | Rev 79719 | Go to most recent revision | Show entire file | Ignore whitespace | Details | Blame | Last modification | View Log | RSS feed

Rev 73489 Rev 73491
Line 32... Line 32...
32 
  Unicode defines a name and a number of all of the glyphs it
 32 
  Unicode defines a name and a number of all of the glyphs it
33 
  encompasses: the numbers are called \emph{code points}: since RFC3629
 33 
  encompasses: the numbers are called \emph{code points}: since RFC3629
34 
  they run from \code{0} to \code{0x10FFFF} (with about 12\% being
 34 
  they run from \code{0} to \code{0x10FFFF} (with about 12\% being
35 
  assigned by version 10.0 of the Unicode standard).
 35 
  assigned by version 10.0 of the Unicode standard).
36
36
37 
  \code{intToUtf8} does not by default handle surrogate pairs (which
 37 
  \code{intToUtf8} does not by default handle surrogate pairs: inputs in
- 
 
 38 
  the surrogate ranges are mapped to \code{NA}.  They might occur if a
- 
 
 39 
  UTF-16 byte stream has been read as 2-byte integers (in the correct
- 
 
 40 
  byte order), in which case \code{allow_surrogate_pairs = TRUE} will
38 
  should not occur in UTF-32): inputs in the surrogate ranges are mapped
 41 
  try to interpret them (with unmatched surrogate values still treated
- 
 
 42 
  as \code{NA}).
- 
 
 43 
}
- 
 
 44 
\section{Validity}{
- 
 
 45 
  Which code points are regarded as valid has changed over the lifetime
- 
 
 46 
  of UTF-8.  Originally all 32-bit unsigned integers were potentially
- 
 
 47 
  valid and could be converted to up to 6 bytes in UTF-8.  Since 2003 it
- 
 
 48 
  has been stated that there will never be valid code points larger than
39 
  to \code{NA}.  They might occur if a UTF-16 byte stream has been read
 49 
  \code{0x10FFFF}, and so valid UTF-8 encodings are never more than 4
- 
 
 50 
  bytes.
- 
 
 51 
 
40 
  as 2-byte integers (in the correct byte order), in which case
 52 
  The code points in the surrogate-pair range \code{0xD000} to
41 
  \code{allow_surrogate_pairs = TRUE} will try to interpret them (with
 53 
  \code{0xDFFF} are prohibited in UTF-8 and so are regarded as invalid
42 
  unmatched surrogate values still treated as \code{NA}).
 54 
  by \code{utf8ToInt} and by default by \code{intToUtf8}.
- 
 
 55 
 
- 
 
 56 
  The position of \sQuote{noncharacters} (notably \code{0xFFFE} and
- 
 
 57 
  \code{0xFFFF}) was clarified by \sQuote{Corrigendum 9} in 2013.  These
- 
 
 58 
  are valid but will never be given an official interpretation.  (In some
- 
 
 59 
  earlier versions of \R \code{utf8ToInt} treated them as invalid.)
43 
}
 60 
}
44 
\value{
 61 
\value{
45 
  \code{utf8ToInt} converts a length-one character string encoded in
 62 
  \code{utf8ToInt} converts a length-one character string encoded in
46 
  UTF-8 to an integer vector of Unicode code points.  It checks validity
 63 
  UTF-8 to an integer vector of Unicode code points.
47 
  of the input and returns \code{NA} if it is invalid.  (Currently it
 - 
 
48 
  accepts UTF-8 encodings of code points greater than \code{0x10FFFF}:
 - 
 
49 
  these are no longer regarded as valid by the UTF-8 RFC and will in
 - 
 
50 
  future be mapped to \code{NA}.)
- 
 
51 
 
 64 
 
52 
  \code{intToUtf8} converts a numeric vector of Unicode code points
 65 
  \code{intToUtf8} converts a numeric vector of Unicode code points
53 
  either (default) to a single character string or a character vector of
 66 
  either (default) to a single character string or a character vector of
54 
  single characters. Non-integral numeric values are truncated to
 67 
  single characters. Non-integral numeric values are truncated to
55 
  integers: values above the maximum are mapped to \code{NA}.  For a
 68 
  integers.  For output to a single character string \code{0} is
56 
  single character string \code{0} is silently omitted: otherwise
 69 
  silently omitted: otherwise \code{0} is mapped to \code{""}.  The
57 
  \code{0} is mapped to \code{""}.  The \code{\link{Encoding}} of a
 70 
  \code{\link{Encoding}} of a non-\code{NA} return value is declared as
58 
  non-\code{NA} return value is declared as \code{"UTF-8"}.
 71 
  \code{"UTF-8"}.
59 
 
 72 
 
60 
  \code{NA} inputs are mapped to \code{NA} output.
 73 
  Invalid and \code{NA} inputs are mapped to \code{NA} output.
61 
}
 74 
}
62 
\references{
 75 
\references{
63 
  \url{https://tools.ietf.org/html/rfc3629}, the current standard for UTF-8.
 76 
  \url{https://tools.ietf.org/html/rfc3629}, the current standard for UTF-8.
- 
 
 77
- 
 
 78 
  \url{http://www.unicode.org/versions/corrigendum9.html} for non-characters.
64 
}
 79 
}
65 
\examples{\donttest{
 80 
\examples{\donttest{
66 
## will only display in some locales and fonts
 81 
## will only display in some locales and fonts
67 
intToUtf8(0x03B2L) # Greek beta
 82 
intToUtf8(0x03B2L) # Greek beta
68 
}
 83 
}