| Tristan Matthews | 0461646 | 2013-11-14 16:09:34 -0500 | [diff] [blame] | 1 | <html> |
| 2 | <head> |
| 3 | <title>pcreunicode specification</title> |
| 4 | </head> |
| 5 | <body bgcolor="#FFFFFF" text="#00005A" link="#0066FF" alink="#3399FF" vlink="#2222BB"> |
| 6 | <h1>pcreunicode man page</h1> |
| 7 | <p> |
| 8 | Return to the <a href="index.html">PCRE index page</a>. |
| 9 | </p> |
| 10 | <p> |
| 11 | This page is part of the PCRE HTML documentation. It was generated automatically |
| 12 | from the original man page. If there is any nonsense in it, please consult the |
| 13 | man page, in case the conversion went wrong. |
| 14 | <br> |
| 15 | <br><b> |
| 16 | UTF-8 AND UNICODE PROPERTY SUPPORT |
| 17 | </b><br> |
| 18 | <P> |
| 19 | In order process UTF-8 strings, you must build PCRE to include UTF-8 support in |
| 20 | the code, and, in addition, you must call |
| 21 | <a href="pcre_compile.html"><b>pcre_compile()</b></a> |
| 22 | with the PCRE_UTF8 option flag, or the pattern must start with the sequence |
| 23 | (*UTF8). When either of these is the case, both the pattern and any subject |
| 24 | strings that are matched against it are treated as UTF-8 strings instead of |
| 25 | strings of 1-byte characters. PCRE does not support any other formats (in |
| 26 | particular, it does not support UTF-16). |
| 27 | </P> |
| 28 | <P> |
| 29 | If you compile PCRE with UTF-8 support, but do not use it at run time, the |
| 30 | library will be a bit bigger, but the additional run time overhead is limited |
| 31 | to testing the PCRE_UTF8 flag occasionally, so should not be very big. |
| 32 | </P> |
| 33 | <P> |
| 34 | If PCRE is built with Unicode character property support (which implies UTF-8 |
| 35 | support), the escape sequences \p{..}, \P{..}, and \X are supported. |
| 36 | The available properties that can be tested are limited to the general |
| 37 | category properties such as Lu for an upper case letter or Nd for a decimal |
| 38 | number, the Unicode script names such as Arabic or Han, and the derived |
| 39 | properties Any and L&. A full list is given in the |
| 40 | <a href="pcrepattern.html"><b>pcrepattern</b></a> |
| 41 | documentation. Only the short names for properties are supported. For example, |
| 42 | \p{L} matches a letter. Its Perl synonym, \p{Letter}, is not supported. |
| 43 | Furthermore, in Perl, many properties may optionally be prefixed by "Is", for |
| 44 | compatibility with Perl 5.6. PCRE does not support this. |
| 45 | <a name="utf8strings"></a></P> |
| 46 | <br><b> |
| 47 | Validity of UTF-8 strings |
| 48 | </b><br> |
| 49 | <P> |
| 50 | When you set the PCRE_UTF8 flag, the strings passed as patterns and subjects |
| 51 | are (by default) checked for validity on entry to the relevant functions. From |
| 52 | release 7.3 of PCRE, the check is according the rules of RFC 3629, which are |
| 53 | themselves derived from the Unicode specification. Earlier releases of PCRE |
| 54 | followed the rules of RFC 2279, which allows the full range of 31-bit values (0 |
| 55 | to 0x7FFFFFFF). The current check allows only values in the range U+0 to |
| 56 | U+10FFFF, excluding U+D800 to U+DFFF. |
| 57 | </P> |
| 58 | <P> |
| 59 | The excluded code points are the "Low Surrogate Area" of Unicode, of which the |
| 60 | Unicode Standard says this: "The Low Surrogate Area does not contain any |
| 61 | character assignments, consequently no character code charts or namelists are |
| 62 | provided for this area. Surrogates are reserved for use with UTF-16 and then |
| 63 | must be used in pairs." The code points that are encoded by UTF-16 pairs are |
| 64 | available as independent code points in the UTF-8 encoding. (In other words, |
| 65 | the whole surrogate thing is a fudge for UTF-16 which unfortunately messes up |
| 66 | UTF-8.) |
| 67 | </P> |
| 68 | <P> |
| 69 | If an invalid UTF-8 string is passed to PCRE, an error return is given. At |
| 70 | compile time, the only additional information is the offset to the first byte |
| 71 | of the failing character. The runtime functions <b>pcre_exec()</b> and |
| 72 | <b>pcre_dfa_exec()</b> also pass back this information, as well as a more |
| 73 | detailed reason code if the caller has provided memory in which to do this. |
| 74 | </P> |
| 75 | <P> |
| 76 | In some situations, you may already know that your strings are valid, and |
| 77 | therefore want to skip these checks in order to improve performance. If you set |
| 78 | the PCRE_NO_UTF8_CHECK flag at compile time or at run time, PCRE assumes that |
| 79 | the pattern or subject it is given (respectively) contains only valid UTF-8 |
| 80 | codes. In this case, it does not diagnose an invalid UTF-8 string. |
| 81 | </P> |
| 82 | <P> |
| 83 | If you pass an invalid UTF-8 string when PCRE_NO_UTF8_CHECK is set, what |
| 84 | happens depends on why the string is invalid. If the string conforms to the |
| 85 | "old" definition of UTF-8 (RFC 2279), it is processed as a string of characters |
| 86 | in the range 0 to 0x7FFFFFFF by <b>pcre_dfa_exec()</b> and the interpreted |
| 87 | version of <b>pcre_exec()</b>. In other words, apart from the initial validity |
| 88 | test, these functions (when in UTF-8 mode) handle strings according to the more |
| 89 | liberal rules of RFC 2279. However, the just-in-time (JIT) optimization for |
| 90 | <b>pcre_exec()</b> supports only RFC 3629. If you are using JIT optimization, or |
| 91 | if the string does not even conform to RFC 2279, the result is undefined. Your |
| 92 | program may crash. |
| 93 | </P> |
| 94 | <P> |
| 95 | If you want to process strings of values in the full range 0 to 0x7FFFFFFF, |
| 96 | encoded in a UTF-8-like manner as per the old RFC, you can set |
| 97 | PCRE_NO_UTF8_CHECK to bypass the more restrictive test. However, in this |
| 98 | situation, you will have to apply your own validity check, and avoid the use of |
| 99 | JIT optimization. |
| 100 | </P> |
| 101 | <br><b> |
| 102 | General comments about UTF-8 mode |
| 103 | </b><br> |
| 104 | <P> |
| 105 | 1. An unbraced hexadecimal escape sequence (such as \xb3) matches a two-byte |
| 106 | UTF-8 character if the value is greater than 127. |
| 107 | </P> |
| 108 | <P> |
| 109 | 2. Octal numbers up to \777 are recognized, and match two-byte UTF-8 |
| 110 | characters for values greater than \177. |
| 111 | </P> |
| 112 | <P> |
| 113 | 3. Repeat quantifiers apply to complete UTF-8 characters, not to individual |
| 114 | bytes, for example: \x{100}{3}. |
| 115 | </P> |
| 116 | <P> |
| 117 | 4. The dot metacharacter matches one UTF-8 character instead of a single byte. |
| 118 | </P> |
| 119 | <P> |
| 120 | 5. The escape sequence \C can be used to match a single byte in UTF-8 mode, |
| 121 | but its use can lead to some strange effects because it breaks up multibyte |
| 122 | characters (see the description of \C in the |
| 123 | <a href="pcrepattern.html"><b>pcrepattern</b></a> |
| 124 | documentation). The use of \C is not supported in the alternative matching |
| 125 | function <b>pcre_dfa_exec()</b>, nor is it supported in UTF-8 mode by the JIT |
| 126 | optimization of <b>pcre_exec()</b>. If JIT optimization is requested for a UTF-8 |
| 127 | pattern that contains \C, it will not succeed, and so the matching will be |
| 128 | carried out by the normal interpretive function. |
| 129 | </P> |
| 130 | <P> |
| 131 | 6. The character escapes \b, \B, \d, \D, \s, \S, \w, and \W correctly |
| 132 | test characters of any code value, but, by default, the characters that PCRE |
| 133 | recognizes as digits, spaces, or word characters remain the same set as before, |
| 134 | all with values less than 256. This remains true even when PCRE is built to |
| 135 | include Unicode property support, because to do otherwise would slow down PCRE |
| 136 | in many common cases. Note in particular that this applies to \b and \B, |
| 137 | because they are defined in terms of \w and \W. If you really want to test |
| 138 | for a wider sense of, say, "digit", you can use explicit Unicode property tests |
| 139 | such as \p{Nd}. Alternatively, if you set the PCRE_UCP option, the way that |
| 140 | the character escapes work is changed so that Unicode properties are used to |
| 141 | determine which characters match. There are more details in the section on |
| 142 | <a href="pcrepattern.html#genericchartypes">generic character types</a> |
| 143 | in the |
| 144 | <a href="pcrepattern.html"><b>pcrepattern</b></a> |
| 145 | documentation. |
| 146 | </P> |
| 147 | <P> |
| 148 | 7. Similarly, characters that match the POSIX named character classes are all |
| 149 | low-valued characters, unless the PCRE_UCP option is set. |
| 150 | </P> |
| 151 | <P> |
| 152 | 8. However, the horizontal and vertical whitespace matching escapes (\h, \H, |
| 153 | \v, and \V) do match all the appropriate Unicode characters, whether or not |
| 154 | PCRE_UCP is set. |
| 155 | </P> |
| 156 | <P> |
| 157 | 9. Case-insensitive matching applies only to characters whose values are less |
| 158 | than 128, unless PCRE is built with Unicode property support. Even when Unicode |
| 159 | property support is available, PCRE still uses its own character tables when |
| 160 | checking the case of low-valued characters, so as not to degrade performance. |
| 161 | The Unicode property information is used only for characters with higher |
| 162 | values. Furthermore, PCRE supports case-insensitive matching only when there is |
| 163 | a one-to-one mapping between a letter's cases. There are a small number of |
| 164 | many-to-one mappings in Unicode; these are not supported by PCRE. |
| 165 | </P> |
| 166 | <br><b> |
| 167 | AUTHOR |
| 168 | </b><br> |
| 169 | <P> |
| 170 | Philip Hazel |
| 171 | <br> |
| 172 | University Computing Service |
| 173 | <br> |
| 174 | Cambridge CB2 3QH, England. |
| 175 | <br> |
| 176 | </P> |
| 177 | <br><b> |
| 178 | REVISION |
| 179 | </b><br> |
| 180 | <P> |
| 181 | Last updated: 19 October 2011 |
| 182 | <br> |
| 183 | Copyright © 1997-2011 University of Cambridge. |
| 184 | <br> |
| 185 | <p> |
| 186 | Return to the <a href="index.html">PCRE index page</a>. |
| 187 | </p> |