Tristan Matthews | 0461646 | 2013-11-14 16:09:34 -0500 | [diff] [blame] | 1 | .TH PCREPARTIAL 3 |
| 2 | .SH NAME |
| 3 | PCRE - Perl-compatible regular expressions |
| 4 | .SH "PARTIAL MATCHING IN PCRE" |
| 5 | .rs |
| 6 | .sp |
| 7 | In normal use of PCRE, if the subject string that is passed to |
| 8 | \fBpcre_exec()\fP or \fBpcre_dfa_exec()\fP matches as far as it goes, but is |
| 9 | too short to match the entire pattern, PCRE_ERROR_NOMATCH is returned. There |
| 10 | are circumstances where it might be helpful to distinguish this case from other |
| 11 | cases in which there is no match. |
| 12 | .P |
| 13 | Consider, for example, an application where a human is required to type in data |
| 14 | for a field with specific formatting requirements. An example might be a date |
| 15 | in the form \fIddmmmyy\fP, defined by this pattern: |
| 16 | .sp |
| 17 | ^\ed?\ed(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\ed\ed$ |
| 18 | .sp |
| 19 | If the application sees the user's keystrokes one by one, and can check that |
| 20 | what has been typed so far is potentially valid, it is able to raise an error |
| 21 | as soon as a mistake is made, by beeping and not reflecting the character that |
| 22 | has been typed, for example. This immediate feedback is likely to be a better |
| 23 | user interface than a check that is delayed until the entire string has been |
| 24 | entered. Partial matching can also be useful when the subject string is very |
| 25 | long and is not all available at once. |
| 26 | .P |
| 27 | PCRE supports partial matching by means of the PCRE_PARTIAL_SOFT and |
| 28 | PCRE_PARTIAL_HARD options, which can be set when calling \fBpcre_exec()\fP or |
| 29 | \fBpcre_dfa_exec()\fP. For backwards compatibility, PCRE_PARTIAL is a synonym |
| 30 | for PCRE_PARTIAL_SOFT. The essential difference between the two options is |
| 31 | whether or not a partial match is preferred to an alternative complete match, |
| 32 | though the details differ between the two matching functions. If both options |
| 33 | are set, PCRE_PARTIAL_HARD takes precedence. |
| 34 | .P |
| 35 | Setting a partial matching option for \fBpcre_exec()\fP disables the use of any |
| 36 | just-in-time code that was set up by calling \fBpcre_study()\fP with the |
| 37 | PCRE_STUDY_JIT_COMPILE option. It also disables two of PCRE's standard |
| 38 | optimizations. PCRE remembers the last literal byte in a pattern, and abandons |
| 39 | matching immediately if such a byte is not present in the subject string. This |
| 40 | optimization cannot be used for a subject string that might match only |
| 41 | partially. If the pattern was studied, PCRE knows the minimum length of a |
| 42 | matching string, and does not bother to run the matching function on shorter |
| 43 | strings. This optimization is also disabled for partial matching. |
| 44 | . |
| 45 | . |
| 46 | .SH "PARTIAL MATCHING USING pcre_exec()" |
| 47 | .rs |
| 48 | .sp |
| 49 | A partial match occurs during a call to \fBpcre_exec()\fP when the end of the |
| 50 | subject string is reached successfully, but matching cannot continue because |
| 51 | more characters are needed. However, at least one character in the subject must |
| 52 | have been inspected. This character need not form part of the final matched |
| 53 | string; lookbehind assertions and the \eK escape sequence provide ways of |
| 54 | inspecting characters before the start of a matched substring. The requirement |
| 55 | for inspecting at least one character exists because an empty string can always |
| 56 | be matched; without such a restriction there would always be a partial match of |
| 57 | an empty string at the end of the subject. |
| 58 | .P |
| 59 | If there are at least two slots in the offsets vector when \fBpcre_exec()\fP |
| 60 | returns with a partial match, the first slot is set to the offset of the |
| 61 | earliest character that was inspected when the partial match was found. For |
| 62 | convenience, the second offset points to the end of the subject so that a |
| 63 | substring can easily be identified. |
| 64 | .P |
| 65 | For the majority of patterns, the first offset identifies the start of the |
| 66 | partially matched string. However, for patterns that contain lookbehind |
| 67 | assertions, or \eK, or begin with \eb or \eB, earlier characters have been |
| 68 | inspected while carrying out the match. For example: |
| 69 | .sp |
| 70 | /(?<=abc)123/ |
| 71 | .sp |
| 72 | This pattern matches "123", but only if it is preceded by "abc". If the subject |
| 73 | string is "xyzabc12", the offsets after a partial match are for the substring |
| 74 | "abc12", because all these characters are needed if another match is tried |
| 75 | with extra characters added to the subject. |
| 76 | .P |
| 77 | What happens when a partial match is identified depends on which of the two |
| 78 | partial matching options are set. |
| 79 | . |
| 80 | . |
| 81 | .SS "PCRE_PARTIAL_SOFT with pcre_exec()" |
| 82 | .rs |
| 83 | .sp |
| 84 | If PCRE_PARTIAL_SOFT is set when \fBpcre_exec()\fP identifies a partial match, |
| 85 | the partial match is remembered, but matching continues as normal, and other |
| 86 | alternatives in the pattern are tried. If no complete match can be found, |
| 87 | \fBpcre_exec()\fP returns PCRE_ERROR_PARTIAL instead of PCRE_ERROR_NOMATCH. |
| 88 | .P |
| 89 | This option is "soft" because it prefers a complete match over a partial match. |
| 90 | All the various matching items in a pattern behave as if the subject string is |
| 91 | potentially complete. For example, \ez, \eZ, and $ match at the end of the |
| 92 | subject, as normal, and for \eb and \eB the end of the subject is treated as a |
| 93 | non-alphanumeric. |
| 94 | .P |
| 95 | If there is more than one partial match, the first one that was found provides |
| 96 | the data that is returned. Consider this pattern: |
| 97 | .sp |
| 98 | /123\ew+X|dogY/ |
| 99 | .sp |
| 100 | If this is matched against the subject string "abc123dog", both |
| 101 | alternatives fail to match, but the end of the subject is reached during |
| 102 | matching, so PCRE_ERROR_PARTIAL is returned. The offsets are set to 3 and 9, |
| 103 | identifying "123dog" as the first partial match that was found. (In this |
| 104 | example, there are two partial matches, because "dog" on its own partially |
| 105 | matches the second alternative.) |
| 106 | . |
| 107 | . |
| 108 | .SS "PCRE_PARTIAL_HARD with pcre_exec()" |
| 109 | .rs |
| 110 | .sp |
| 111 | If PCRE_PARTIAL_HARD is set for \fBpcre_exec()\fP, it returns |
| 112 | PCRE_ERROR_PARTIAL as soon as a partial match is found, without continuing to |
| 113 | search for possible complete matches. This option is "hard" because it prefers |
| 114 | an earlier partial match over a later complete match. For this reason, the |
| 115 | assumption is made that the end of the supplied subject string may not be the |
| 116 | true end of the available data, and so, if \ez, \eZ, \eb, \eB, or $ are |
| 117 | encountered at the end of the subject, the result is PCRE_ERROR_PARTIAL. |
| 118 | .P |
| 119 | Setting PCRE_PARTIAL_HARD also affects the way \fBpcre_exec()\fP checks UTF-8 |
| 120 | subject strings for validity. Normally, an invalid UTF-8 sequence causes the |
| 121 | error PCRE_ERROR_BADUTF8. However, in the special case of a truncated UTF-8 |
| 122 | character at the end of the subject, PCRE_ERROR_SHORTUTF8 is returned when |
| 123 | PCRE_PARTIAL_HARD is set. |
| 124 | . |
| 125 | . |
| 126 | .SS "Comparing hard and soft partial matching" |
| 127 | .rs |
| 128 | .sp |
| 129 | The difference between the two partial matching options can be illustrated by a |
| 130 | pattern such as: |
| 131 | .sp |
| 132 | /dog(sbody)?/ |
| 133 | .sp |
| 134 | This matches either "dog" or "dogsbody", greedily (that is, it prefers the |
| 135 | longer string if possible). If it is matched against the string "dog" with |
| 136 | PCRE_PARTIAL_SOFT, it yields a complete match for "dog". However, if |
| 137 | PCRE_PARTIAL_HARD is set, the result is PCRE_ERROR_PARTIAL. On the other hand, |
| 138 | if the pattern is made ungreedy the result is different: |
| 139 | .sp |
| 140 | /dog(sbody)??/ |
| 141 | .sp |
| 142 | In this case the result is always a complete match because \fBpcre_exec()\fP |
| 143 | finds that first, and it never continues after finding a match. It might be |
| 144 | easier to follow this explanation by thinking of the two patterns like this: |
| 145 | .sp |
| 146 | /dog(sbody)?/ is the same as /dogsbody|dog/ |
| 147 | /dog(sbody)??/ is the same as /dog|dogsbody/ |
| 148 | .sp |
| 149 | The second pattern will never match "dogsbody" when \fBpcre_exec()\fP is |
| 150 | used, because it will always find the shorter match first. |
| 151 | . |
| 152 | . |
| 153 | .SH "PARTIAL MATCHING USING pcre_dfa_exec()" |
| 154 | .rs |
| 155 | .sp |
| 156 | The \fBpcre_dfa_exec()\fP function moves along the subject string character by |
| 157 | character, without backtracking, searching for all possible matches |
| 158 | simultaneously. If the end of the subject is reached before the end of the |
| 159 | pattern, there is the possibility of a partial match, again provided that at |
| 160 | least one character has been inspected. |
| 161 | .P |
| 162 | When PCRE_PARTIAL_SOFT is set, PCRE_ERROR_PARTIAL is returned only if there |
| 163 | have been no complete matches. Otherwise, the complete matches are returned. |
| 164 | However, if PCRE_PARTIAL_HARD is set, a partial match takes precedence over any |
| 165 | complete matches. The portion of the string that was inspected when the longest |
| 166 | partial match was found is set as the first matching string, provided there are |
| 167 | at least two slots in the offsets vector. |
| 168 | .P |
| 169 | Because \fBpcre_dfa_exec()\fP always searches for all possible matches, and |
| 170 | there is no difference between greedy and ungreedy repetition, its behaviour is |
| 171 | different from \fBpcre_exec\fP when PCRE_PARTIAL_HARD is set. Consider the |
| 172 | string "dog" matched against the ungreedy pattern shown above: |
| 173 | .sp |
| 174 | /dog(sbody)??/ |
| 175 | .sp |
| 176 | Whereas \fBpcre_exec()\fP stops as soon as it finds the complete match for |
| 177 | "dog", \fBpcre_dfa_exec()\fP also finds the partial match for "dogsbody", and |
| 178 | so returns that when PCRE_PARTIAL_HARD is set. |
| 179 | . |
| 180 | . |
| 181 | .SH "PARTIAL MATCHING AND WORD BOUNDARIES" |
| 182 | .rs |
| 183 | .sp |
| 184 | If a pattern ends with one of sequences \eb or \eB, which test for word |
| 185 | boundaries, partial matching with PCRE_PARTIAL_SOFT can give counter-intuitive |
| 186 | results. Consider this pattern: |
| 187 | .sp |
| 188 | /\ebcat\eb/ |
| 189 | .sp |
| 190 | This matches "cat", provided there is a word boundary at either end. If the |
| 191 | subject string is "the cat", the comparison of the final "t" with a following |
| 192 | character cannot take place, so a partial match is found. However, |
| 193 | \fBpcre_exec()\fP carries on with normal matching, which matches \eb at the end |
| 194 | of the subject when the last character is a letter, thus finding a complete |
| 195 | match. The result, therefore, is \fInot\fP PCRE_ERROR_PARTIAL. The same thing |
| 196 | happens with \fBpcre_dfa_exec()\fP, because it also finds the complete match. |
| 197 | .P |
| 198 | Using PCRE_PARTIAL_HARD in this case does yield PCRE_ERROR_PARTIAL, because |
| 199 | then the partial match takes precedence. |
| 200 | . |
| 201 | . |
| 202 | .SH "FORMERLY RESTRICTED PATTERNS" |
| 203 | .rs |
| 204 | .sp |
| 205 | For releases of PCRE prior to 8.00, because of the way certain internal |
| 206 | optimizations were implemented in the \fBpcre_exec()\fP function, the |
| 207 | PCRE_PARTIAL option (predecessor of PCRE_PARTIAL_SOFT) could not be used with |
| 208 | all patterns. From release 8.00 onwards, the restrictions no longer apply, and |
| 209 | partial matching with \fBpcre_exec()\fP can be requested for any pattern. |
| 210 | .P |
| 211 | Items that were formerly restricted were repeated single characters and |
| 212 | repeated metasequences. If PCRE_PARTIAL was set for a pattern that did not |
| 213 | conform to the restrictions, \fBpcre_exec()\fP returned the error code |
| 214 | PCRE_ERROR_BADPARTIAL (-13). This error code is no longer in use. The |
| 215 | PCRE_INFO_OKPARTIAL call to \fBpcre_fullinfo()\fP to find out if a compiled |
| 216 | pattern can be used for partial matching now always returns 1. |
| 217 | . |
| 218 | . |
| 219 | .SH "EXAMPLE OF PARTIAL MATCHING USING PCRETEST" |
| 220 | .rs |
| 221 | .sp |
| 222 | If the escape sequence \eP is present in a \fBpcretest\fP data line, the |
| 223 | PCRE_PARTIAL_SOFT option is used for the match. Here is a run of \fBpcretest\fP |
| 224 | that uses the date example quoted above: |
| 225 | .sp |
| 226 | re> /^\ed?\ed(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\ed\ed$/ |
| 227 | data> 25jun04\eP |
| 228 | 0: 25jun04 |
| 229 | 1: jun |
| 230 | data> 25dec3\eP |
| 231 | Partial match: 23dec3 |
| 232 | data> 3ju\eP |
| 233 | Partial match: 3ju |
| 234 | data> 3juj\eP |
| 235 | No match |
| 236 | data> j\eP |
| 237 | No match |
| 238 | .sp |
| 239 | The first data string is matched completely, so \fBpcretest\fP shows the |
| 240 | matched substrings. The remaining four strings do not match the complete |
| 241 | pattern, but the first two are partial matches. Similar output is obtained |
| 242 | when \fBpcre_dfa_exec()\fP is used. |
| 243 | .P |
| 244 | If the escape sequence \eP is present more than once in a \fBpcretest\fP data |
| 245 | line, the PCRE_PARTIAL_HARD option is set for the match. |
| 246 | . |
| 247 | . |
| 248 | .SH "MULTI-SEGMENT MATCHING WITH pcre_dfa_exec()" |
| 249 | .rs |
| 250 | .sp |
| 251 | When a partial match has been found using \fBpcre_dfa_exec()\fP, it is possible |
| 252 | to continue the match by providing additional subject data and calling |
| 253 | \fBpcre_dfa_exec()\fP again with the same compiled regular expression, this |
| 254 | time setting the PCRE_DFA_RESTART option. You must pass the same working |
| 255 | space as before, because this is where details of the previous partial match |
| 256 | are stored. Here is an example using \fBpcretest\fP, using the \eR escape |
| 257 | sequence to set the PCRE_DFA_RESTART option (\eD specifies the use of |
| 258 | \fBpcre_dfa_exec()\fP): |
| 259 | .sp |
| 260 | re> /^\ed?\ed(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\ed\ed$/ |
| 261 | data> 23ja\eP\eD |
| 262 | Partial match: 23ja |
| 263 | data> n05\eR\eD |
| 264 | 0: n05 |
| 265 | .sp |
| 266 | The first call has "23ja" as the subject, and requests partial matching; the |
| 267 | second call has "n05" as the subject for the continued (restarted) match. |
| 268 | Notice that when the match is complete, only the last part is shown; PCRE does |
| 269 | not retain the previously partially-matched string. It is up to the calling |
| 270 | program to do that if it needs to. |
| 271 | .P |
| 272 | You can set the PCRE_PARTIAL_SOFT or PCRE_PARTIAL_HARD options with |
| 273 | PCRE_DFA_RESTART to continue partial matching over multiple segments. This |
| 274 | facility can be used to pass very long subject strings to |
| 275 | \fBpcre_dfa_exec()\fP. |
| 276 | . |
| 277 | . |
| 278 | .SH "MULTI-SEGMENT MATCHING WITH pcre_exec()" |
| 279 | .rs |
| 280 | .sp |
| 281 | From release 8.00, \fBpcre_exec()\fP can also be used to do multi-segment |
| 282 | matching. Unlike \fBpcre_dfa_exec()\fP, it is not possible to restart the |
| 283 | previous match with a new segment of data. Instead, new data must be added to |
| 284 | the previous subject string, and the entire match re-run, starting from the |
| 285 | point where the partial match occurred. Earlier data can be discarded. It is |
| 286 | best to use PCRE_PARTIAL_HARD in this situation, because it does not treat the |
| 287 | end of a segment as the end of the subject when matching \ez, \eZ, \eb, \eB, |
| 288 | and $. Consider an unanchored pattern that matches dates: |
| 289 | .sp |
| 290 | re> /\ed?\ed(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\ed\ed/ |
| 291 | data> The date is 23ja\eP\eP |
| 292 | Partial match: 23ja |
| 293 | .sp |
| 294 | At this stage, an application could discard the text preceding "23ja", add on |
| 295 | text from the next segment, and call \fBpcre_exec()\fP again. Unlike |
| 296 | \fBpcre_dfa_exec()\fP, the entire matching string must always be available, and |
| 297 | the complete matching process occurs for each call, so more memory and more |
| 298 | processing time is needed. |
| 299 | .P |
| 300 | \fBNote:\fP If the pattern contains lookbehind assertions, or \eK, or starts |
| 301 | with \eb or \eB, the string that is returned for a partial match will include |
| 302 | characters that precede the partially matched string itself, because these must |
| 303 | be retained when adding on more characters for a subsequent matching attempt. |
| 304 | . |
| 305 | . |
| 306 | .SH "ISSUES WITH MULTI-SEGMENT MATCHING" |
| 307 | .rs |
| 308 | .sp |
| 309 | Certain types of pattern may give problems with multi-segment matching, |
| 310 | whichever matching function is used. |
| 311 | .P |
| 312 | 1. If the pattern contains a test for the beginning of a line, you need to pass |
| 313 | the PCRE_NOTBOL option when the subject string for any call does start at the |
| 314 | beginning of a line. There is also a PCRE_NOTEOL option, but in practice when |
| 315 | doing multi-segment matching you should be using PCRE_PARTIAL_HARD, which |
| 316 | includes the effect of PCRE_NOTEOL. |
| 317 | .P |
| 318 | 2. Lookbehind assertions at the start of a pattern are catered for in the |
| 319 | offsets that are returned for a partial match. However, in theory, a lookbehind |
| 320 | assertion later in the pattern could require even earlier characters to be |
| 321 | inspected, and it might not have been reached when a partial match occurs. This |
| 322 | is probably an extremely unlikely case; you could guard against it to a certain |
| 323 | extent by always including extra characters at the start. |
| 324 | .P |
| 325 | 3. Matching a subject string that is split into multiple segments may not |
| 326 | always produce exactly the same result as matching over one single long string, |
| 327 | especially when PCRE_PARTIAL_SOFT is used. The section "Partial Matching and |
| 328 | Word Boundaries" above describes an issue that arises if the pattern ends with |
| 329 | \eb or \eB. Another kind of difference may occur when there are multiple |
| 330 | matching possibilities, because (for PCRE_PARTIAL_SOFT) a partial match result |
| 331 | is given only when there are no completed matches. This means that as soon as |
| 332 | the shortest match has been found, continuation to a new subject segment is no |
| 333 | longer possible. Consider again this \fBpcretest\fP example: |
| 334 | .sp |
| 335 | re> /dog(sbody)?/ |
| 336 | data> dogsb\eP |
| 337 | 0: dog |
| 338 | data> do\eP\eD |
| 339 | Partial match: do |
| 340 | data> gsb\eR\eP\eD |
| 341 | 0: g |
| 342 | data> dogsbody\eD |
| 343 | 0: dogsbody |
| 344 | 1: dog |
| 345 | .sp |
| 346 | The first data line passes the string "dogsb" to \fBpcre_exec()\fP, setting the |
| 347 | PCRE_PARTIAL_SOFT option. Although the string is a partial match for |
| 348 | "dogsbody", the result is not PCRE_ERROR_PARTIAL, because the shorter string |
| 349 | "dog" is a complete match. Similarly, when the subject is presented to |
| 350 | \fBpcre_dfa_exec()\fP in several parts ("do" and "gsb" being the first two) the |
| 351 | match stops when "dog" has been found, and it is not possible to continue. On |
| 352 | the other hand, if "dogsbody" is presented as a single string, |
| 353 | \fBpcre_dfa_exec()\fP finds both matches. |
| 354 | .P |
| 355 | Because of these problems, it is best to use PCRE_PARTIAL_HARD when matching |
| 356 | multi-segment data. The example above then behaves differently: |
| 357 | .sp |
| 358 | re> /dog(sbody)?/ |
| 359 | data> dogsb\eP\eP |
| 360 | Partial match: dogsb |
| 361 | data> do\eP\eD |
| 362 | Partial match: do |
| 363 | data> gsb\eR\eP\eP\eD |
| 364 | Partial match: gsb |
| 365 | .sp |
| 366 | 4. Patterns that contain alternatives at the top level which do not all |
| 367 | start with the same pattern item may not work as expected when |
| 368 | PCRE_DFA_RESTART is used with \fBpcre_dfa_exec()\fP. For example, consider this |
| 369 | pattern: |
| 370 | .sp |
| 371 | 1234|3789 |
| 372 | .sp |
| 373 | If the first part of the subject is "ABC123", a partial match of the first |
| 374 | alternative is found at offset 3. There is no partial match for the second |
| 375 | alternative, because such a match does not start at the same point in the |
| 376 | subject string. Attempting to continue with the string "7890" does not yield a |
| 377 | match because only those alternatives that match at one point in the subject |
| 378 | are remembered. The problem arises because the start of the second alternative |
| 379 | matches within the first alternative. There is no problem with anchored |
| 380 | patterns or patterns such as: |
| 381 | .sp |
| 382 | 1234|ABCD |
| 383 | .sp |
| 384 | where no string can be a partial match for both alternatives. This is not a |
| 385 | problem if \fBpcre_exec()\fP is used, because the entire match has to be rerun |
| 386 | each time: |
| 387 | .sp |
| 388 | re> /1234|3789/ |
| 389 | data> ABC123\eP\eP |
| 390 | Partial match: 123 |
| 391 | data> 1237890 |
| 392 | 0: 3789 |
| 393 | .sp |
| 394 | Of course, instead of using PCRE_DFA_RESTART, the same technique of re-running |
| 395 | the entire match can also be used with \fBpcre_dfa_exec()\fP. Another |
| 396 | possibility is to work with two buffers. If a partial match at offset \fIn\fP |
| 397 | in the first buffer is followed by "no match" when PCRE_DFA_RESTART is used on |
| 398 | the second buffer, you can then try a new match starting at offset \fIn+1\fP in |
| 399 | the first buffer. |
| 400 | . |
| 401 | . |
| 402 | .SH AUTHOR |
| 403 | .rs |
| 404 | .sp |
| 405 | .nf |
| 406 | Philip Hazel |
| 407 | University Computing Service |
| 408 | Cambridge CB2 3QH, England. |
| 409 | .fi |
| 410 | . |
| 411 | . |
| 412 | .SH REVISION |
| 413 | .rs |
| 414 | .sp |
| 415 | .nf |
| 416 | Last updated: 26 August 2011 |
| 417 | Copyright (c) 1997-2011 University of Cambridge. |
| 418 | .fi |