Tristan Matthews | 0461646 | 2013-11-14 16:09:34 -0500 | [diff] [blame] | 1 | README file for PCRE (Perl-compatible regular expression library) |
| 2 | ----------------------------------------------------------------- |
| 3 | |
| 4 | The latest release of PCRE is always available in three alternative formats |
| 5 | from: |
| 6 | |
| 7 | ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/pcre-xxx.tar.gz |
| 8 | ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/pcre-xxx.tar.bz2 |
| 9 | ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/pcre-xxx.zip |
| 10 | |
| 11 | There is a mailing list for discussion about the development of PCRE at |
| 12 | |
| 13 | pcre-dev@exim.org |
| 14 | |
| 15 | Please read the NEWS file if you are upgrading from a previous release. |
| 16 | The contents of this README file are: |
| 17 | |
| 18 | The PCRE APIs |
| 19 | Documentation for PCRE |
| 20 | Contributions by users of PCRE |
| 21 | Building PCRE on non-Unix systems |
| 22 | Building PCRE on Unix-like systems |
| 23 | Retrieving configuration information on Unix-like systems |
| 24 | Shared libraries on Unix-like systems |
| 25 | Cross-compiling on Unix-like systems |
| 26 | Using HP's ANSI C++ compiler (aCC) |
| 27 | Using PCRE from MySQL |
| 28 | Making new tarballs |
| 29 | Testing PCRE |
| 30 | Character tables |
| 31 | File manifest |
| 32 | |
| 33 | |
| 34 | The PCRE APIs |
| 35 | ------------- |
| 36 | |
| 37 | PCRE is written in C, and it has its own API. The distribution also includes a |
| 38 | set of C++ wrapper functions (see the pcrecpp man page for details), courtesy |
| 39 | of Google Inc. |
| 40 | |
| 41 | In addition, there is a set of C wrapper functions that are based on the POSIX |
| 42 | regular expression API (see the pcreposix man page). These end up in the |
| 43 | library called libpcreposix. Note that this just provides a POSIX calling |
| 44 | interface to PCRE; the regular expressions themselves still follow Perl syntax |
| 45 | and semantics. The POSIX API is restricted, and does not give full access to |
| 46 | all of PCRE's facilities. |
| 47 | |
| 48 | The header file for the POSIX-style functions is called pcreposix.h. The |
| 49 | official POSIX name is regex.h, but I did not want to risk possible problems |
| 50 | with existing files of that name by distributing it that way. To use PCRE with |
| 51 | an existing program that uses the POSIX API, pcreposix.h will have to be |
| 52 | renamed or pointed at by a link. |
| 53 | |
| 54 | If you are using the POSIX interface to PCRE and there is already a POSIX regex |
| 55 | library installed on your system, as well as worrying about the regex.h header |
| 56 | file (as mentioned above), you must also take care when linking programs to |
| 57 | ensure that they link with PCRE's libpcreposix library. Otherwise they may pick |
| 58 | up the POSIX functions of the same name from the other library. |
| 59 | |
| 60 | One way of avoiding this confusion is to compile PCRE with the addition of |
| 61 | -Dregcomp=PCREregcomp (and similarly for the other POSIX functions) to the |
| 62 | compiler flags (CFLAGS if you are using "configure" -- see below). This has the |
| 63 | effect of renaming the functions so that the names no longer clash. Of course, |
| 64 | you have to do the same thing for your applications, or write them using the |
| 65 | new names. |
| 66 | |
| 67 | |
| 68 | Documentation for PCRE |
| 69 | ---------------------- |
| 70 | |
| 71 | If you install PCRE in the normal way on a Unix-like system, you will end up |
| 72 | with a set of man pages whose names all start with "pcre". The one that is just |
| 73 | called "pcre" lists all the others. In addition to these man pages, the PCRE |
| 74 | documentation is supplied in two other forms: |
| 75 | |
| 76 | 1. There are files called doc/pcre.txt, doc/pcregrep.txt, and |
| 77 | doc/pcretest.txt in the source distribution. The first of these is a |
| 78 | concatenation of the text forms of all the section 3 man pages except |
| 79 | those that summarize individual functions. The other two are the text |
| 80 | forms of the section 1 man pages for the pcregrep and pcretest commands. |
| 81 | These text forms are provided for ease of scanning with text editors or |
| 82 | similar tools. They are installed in <prefix>/share/doc/pcre, where |
| 83 | <prefix> is the installation prefix (defaulting to /usr/local). |
| 84 | |
| 85 | 2. A set of files containing all the documentation in HTML form, hyperlinked |
| 86 | in various ways, and rooted in a file called index.html, is distributed in |
| 87 | doc/html and installed in <prefix>/share/doc/pcre/html. |
| 88 | |
| 89 | Users of PCRE have contributed files containing the documentation for various |
| 90 | releases in CHM format. These can be found in the Contrib directory of the FTP |
| 91 | site (see next section). |
| 92 | |
| 93 | |
| 94 | Contributions by users of PCRE |
| 95 | ------------------------------ |
| 96 | |
| 97 | You can find contributions from PCRE users in the directory |
| 98 | |
| 99 | ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/Contrib |
| 100 | |
| 101 | There is a README file giving brief descriptions of what they are. Some are |
| 102 | complete in themselves; others are pointers to URLs containing relevant files. |
| 103 | Some of this material is likely to be well out-of-date. Several of the earlier |
| 104 | contributions provided support for compiling PCRE on various flavours of |
| 105 | Windows (I myself do not use Windows). Nowadays there is more Windows support |
| 106 | in the standard distribution, so these contibutions have been archived. |
| 107 | |
| 108 | |
| 109 | Building PCRE on non-Unix systems |
| 110 | --------------------------------- |
| 111 | |
| 112 | For a non-Unix system, please read the comments in the file NON-UNIX-USE, |
| 113 | though if your system supports the use of "configure" and "make" you may be |
| 114 | able to build PCRE in the same way as for Unix-like systems. PCRE can also be |
| 115 | configured in many platform environments using the GUI facility provided by |
| 116 | CMake's cmake-gui command. This creates Makefiles, solution files, etc. |
| 117 | |
| 118 | PCRE has been compiled on many different operating systems. It should be |
| 119 | straightforward to build PCRE on any system that has a Standard C compiler and |
| 120 | library, because it uses only Standard C functions. |
| 121 | |
| 122 | |
| 123 | Building PCRE on Unix-like systems |
| 124 | ---------------------------------- |
| 125 | |
| 126 | If you are using HP's ANSI C++ compiler (aCC), please see the special note |
| 127 | in the section entitled "Using HP's ANSI C++ compiler (aCC)" below. |
| 128 | |
| 129 | The following instructions assume the use of the widely used "configure, make, |
| 130 | make install" process. There is also support for CMake in the PCRE |
| 131 | distribution; there are some comments about using CMake in the NON-UNIX-USE |
| 132 | file, though it can also be used in Unix-like systems. |
| 133 | |
| 134 | To build PCRE on a Unix-like system, first run the "configure" command from the |
| 135 | PCRE distribution directory, with your current directory set to the directory |
| 136 | where you want the files to be created. This command is a standard GNU |
| 137 | "autoconf" configuration script, for which generic instructions are supplied in |
| 138 | the file INSTALL. |
| 139 | |
| 140 | Most commonly, people build PCRE within its own distribution directory, and in |
| 141 | this case, on many systems, just running "./configure" is sufficient. However, |
| 142 | the usual methods of changing standard defaults are available. For example: |
| 143 | |
| 144 | CFLAGS='-O2 -Wall' ./configure --prefix=/opt/local |
| 145 | |
| 146 | specifies that the C compiler should be run with the flags '-O2 -Wall' instead |
| 147 | of the default, and that "make install" should install PCRE under /opt/local |
| 148 | instead of the default /usr/local. |
| 149 | |
| 150 | If you want to build in a different directory, just run "configure" with that |
| 151 | directory as current. For example, suppose you have unpacked the PCRE source |
| 152 | into /source/pcre/pcre-xxx, but you want to build it in /build/pcre/pcre-xxx: |
| 153 | |
| 154 | cd /build/pcre/pcre-xxx |
| 155 | /source/pcre/pcre-xxx/configure |
| 156 | |
| 157 | PCRE is written in C and is normally compiled as a C library. However, it is |
| 158 | possible to build it as a C++ library, though the provided building apparatus |
| 159 | does not have any features to support this. |
| 160 | |
| 161 | There are some optional features that can be included or omitted from the PCRE |
| 162 | library. They are also documented in the pcrebuild man page. |
| 163 | |
| 164 | . By default, both shared and static libraries are built. You can change this |
| 165 | by adding one of these options to the "configure" command: |
| 166 | |
| 167 | --disable-shared |
| 168 | --disable-static |
| 169 | |
| 170 | (See also "Shared libraries on Unix-like systems" below.) |
| 171 | |
| 172 | . If you want to suppress the building of the C++ wrapper library, you can add |
| 173 | --disable-cpp to the "configure" command. Otherwise, when "configure" is run, |
| 174 | it will try to find a C++ compiler and C++ header files, and if it succeeds, |
| 175 | it will try to build the C++ wrapper. |
| 176 | |
| 177 | . If you want to include support for just-in-time compiling, which can give |
| 178 | large performance improvements on certain platforms, add --enable-jit to the |
| 179 | "configure" command. This support is available only for certain hardware |
| 180 | architectures. If you try to enable it on an unsupported architecture, there |
| 181 | will be a compile time error. |
| 182 | |
| 183 | . When JIT support is enabled, pcregrep automatically makes use of it, unless |
| 184 | you add --disable-pcregrep-jit to the "configure" command. |
| 185 | |
| 186 | . If you want to make use of the support for UTF-8 Unicode character strings in |
| 187 | PCRE, you must add --enable-utf8 to the "configure" command. Without it, the |
| 188 | code for handling UTF-8 is not included in the library. Even when included, |
| 189 | it still has to be enabled by an option at run time. When PCRE is compiled |
| 190 | with this option, its input can only either be ASCII or UTF-8, even when |
| 191 | running on EBCDIC platforms. It is not possible to use both --enable-utf8 and |
| 192 | --enable-ebcdic at the same time. |
| 193 | |
| 194 | . If, in addition to support for UTF-8 character strings, you want to include |
| 195 | support for the \P, \p, and \X sequences that recognize Unicode character |
| 196 | properties, you must add --enable-unicode-properties to the "configure" |
| 197 | command. This adds about 30K to the size of the library (in the form of a |
| 198 | property table); only the basic two-letter properties such as Lu are |
| 199 | supported. |
| 200 | |
| 201 | . You can build PCRE to recognize either CR or LF or the sequence CRLF or any |
| 202 | of the preceding, or any of the Unicode newline sequences as indicating the |
| 203 | end of a line. Whatever you specify at build time is the default; the caller |
| 204 | of PCRE can change the selection at run time. The default newline indicator |
| 205 | is a single LF character (the Unix standard). You can specify the default |
| 206 | newline indicator by adding --enable-newline-is-cr or --enable-newline-is-lf |
| 207 | or --enable-newline-is-crlf or --enable-newline-is-anycrlf or |
| 208 | --enable-newline-is-any to the "configure" command, respectively. |
| 209 | |
| 210 | If you specify --enable-newline-is-cr or --enable-newline-is-crlf, some of |
| 211 | the standard tests will fail, because the lines in the test files end with |
| 212 | LF. Even if the files are edited to change the line endings, there are likely |
| 213 | to be some failures. With --enable-newline-is-anycrlf or |
| 214 | --enable-newline-is-any, many tests should succeed, but there may be some |
| 215 | failures. |
| 216 | |
| 217 | . By default, the sequence \R in a pattern matches any Unicode line ending |
| 218 | sequence. This is independent of the option specifying what PCRE considers to |
| 219 | be the end of a line (see above). However, the caller of PCRE can restrict \R |
| 220 | to match only CR, LF, or CRLF. You can make this the default by adding |
| 221 | --enable-bsr-anycrlf to the "configure" command (bsr = "backslash R"). |
| 222 | |
| 223 | . When called via the POSIX interface, PCRE uses malloc() to get additional |
| 224 | storage for processing capturing parentheses if there are more than 10 of |
| 225 | them in a pattern. You can increase this threshold by setting, for example, |
| 226 | |
| 227 | --with-posix-malloc-threshold=20 |
| 228 | |
| 229 | on the "configure" command. |
| 230 | |
| 231 | . PCRE has a counter that can be set to limit the amount of resources it uses. |
| 232 | If the limit is exceeded during a match, the match fails. The default is ten |
| 233 | million. You can change the default by setting, for example, |
| 234 | |
| 235 | --with-match-limit=500000 |
| 236 | |
| 237 | on the "configure" command. This is just the default; individual calls to |
| 238 | pcre_exec() can supply their own value. There is more discussion on the |
| 239 | pcreapi man page. |
| 240 | |
| 241 | . There is a separate counter that limits the depth of recursive function calls |
| 242 | during a matching process. This also has a default of ten million, which is |
| 243 | essentially "unlimited". You can change the default by setting, for example, |
| 244 | |
| 245 | --with-match-limit-recursion=500000 |
| 246 | |
| 247 | Recursive function calls use up the runtime stack; running out of stack can |
| 248 | cause programs to crash in strange ways. There is a discussion about stack |
| 249 | sizes in the pcrestack man page. |
| 250 | |
| 251 | . The default maximum compiled pattern size is around 64K. You can increase |
| 252 | this by adding --with-link-size=3 to the "configure" command. You can |
| 253 | increase it even more by setting --with-link-size=4, but this is unlikely |
| 254 | ever to be necessary. Increasing the internal link size will reduce |
| 255 | performance. |
| 256 | |
| 257 | . You can build PCRE so that its internal match() function that is called from |
| 258 | pcre_exec() does not call itself recursively. Instead, it uses memory blocks |
| 259 | obtained from the heap via the special functions pcre_stack_malloc() and |
| 260 | pcre_stack_free() to save data that would otherwise be saved on the stack. To |
| 261 | build PCRE like this, use |
| 262 | |
| 263 | --disable-stack-for-recursion |
| 264 | |
| 265 | on the "configure" command. PCRE runs more slowly in this mode, but it may be |
| 266 | necessary in environments with limited stack sizes. This applies only to the |
| 267 | normal execution of the pcre_exec() function; if JIT support is being |
| 268 | successfully used, it is not relevant. Equally, it does not apply to |
| 269 | pcre_dfa_exec(), which does not use deeply nested recursion. There is a |
| 270 | discussion about stack sizes in the pcrestack man page. |
| 271 | |
| 272 | . For speed, PCRE uses four tables for manipulating and identifying characters |
| 273 | whose code point values are less than 256. By default, it uses a set of |
| 274 | tables for ASCII encoding that is part of the distribution. If you specify |
| 275 | |
| 276 | --enable-rebuild-chartables |
| 277 | |
| 278 | a program called dftables is compiled and run in the default C locale when |
| 279 | you obey "make". It builds a source file called pcre_chartables.c. If you do |
| 280 | not specify this option, pcre_chartables.c is created as a copy of |
| 281 | pcre_chartables.c.dist. See "Character tables" below for further information. |
| 282 | |
| 283 | . It is possible to compile PCRE for use on systems that use EBCDIC as their |
| 284 | character code (as opposed to ASCII) by specifying |
| 285 | |
| 286 | --enable-ebcdic |
| 287 | |
| 288 | This automatically implies --enable-rebuild-chartables (see above). However, |
| 289 | when PCRE is built this way, it always operates in EBCDIC. It cannot support |
| 290 | both EBCDIC and UTF-8. |
| 291 | |
| 292 | . It is possible to compile pcregrep to use libz and/or libbz2, in order to |
| 293 | read .gz and .bz2 files (respectively), by specifying one or both of |
| 294 | |
| 295 | --enable-pcregrep-libz |
| 296 | --enable-pcregrep-libbz2 |
| 297 | |
| 298 | Of course, the relevant libraries must be installed on your system. |
| 299 | |
| 300 | . The default size of internal buffer used by pcregrep can be set by, for |
| 301 | example: |
| 302 | |
| 303 | --with-pcregrep-bufsize=50K |
| 304 | |
| 305 | The default value is 20K. |
| 306 | |
| 307 | . It is possible to compile pcretest so that it links with the libreadline |
| 308 | library, by specifying |
| 309 | |
| 310 | --enable-pcretest-libreadline |
| 311 | |
| 312 | If this is done, when pcretest's input is from a terminal, it reads it using |
| 313 | the readline() function. This provides line-editing and history facilities. |
| 314 | Note that libreadline is GPL-licenced, so if you distribute a binary of |
| 315 | pcretest linked in this way, there may be licensing issues. |
| 316 | |
| 317 | Setting this option causes the -lreadline option to be added to the pcretest |
| 318 | build. In many operating environments with a sytem-installed readline |
| 319 | library this is sufficient. However, in some environments (e.g. if an |
| 320 | unmodified distribution version of readline is in use), it may be necessary |
| 321 | to specify something like LIBS="-lncurses" as well. This is because, to quote |
| 322 | the readline INSTALL, "Readline uses the termcap functions, but does not link |
| 323 | with the termcap or curses library itself, allowing applications which link |
| 324 | with readline the to choose an appropriate library." If you get error |
| 325 | messages about missing functions tgetstr, tgetent, tputs, tgetflag, or tgoto, |
| 326 | this is the problem, and linking with the ncurses library should fix it. |
| 327 | |
| 328 | The "configure" script builds the following files for the basic C library: |
| 329 | |
| 330 | . Makefile the makefile that builds the library |
| 331 | . config.h build-time configuration options for the library |
| 332 | . pcre.h the public PCRE header file |
| 333 | . pcre-config script that shows the building settings such as CFLAGS |
| 334 | that were set for "configure" |
| 335 | . libpcre.pc ) data for the pkg-config command |
| 336 | . libpcreposix.pc ) |
| 337 | . libtool script that builds shared and/or static libraries |
| 338 | . RunTest script for running tests on the basic C library |
| 339 | . RunGrepTest script for running tests on the pcregrep command |
| 340 | |
| 341 | Versions of config.h and pcre.h are distributed in the PCRE tarballs under the |
| 342 | names config.h.generic and pcre.h.generic. These are provided for those who |
| 343 | have to built PCRE without using "configure" or CMake. If you use "configure" |
| 344 | or CMake, the .generic versions are not used. |
| 345 | |
| 346 | If a C++ compiler is found, the following files are also built: |
| 347 | |
| 348 | . libpcrecpp.pc data for the pkg-config command |
| 349 | . pcrecpparg.h header file for calling PCRE via the C++ wrapper |
| 350 | . pcre_stringpiece.h header for the C++ "stringpiece" functions |
| 351 | |
| 352 | The "configure" script also creates config.status, which is an executable |
| 353 | script that can be run to recreate the configuration, and config.log, which |
| 354 | contains compiler output from tests that "configure" runs. |
| 355 | |
| 356 | Once "configure" has run, you can run "make". It builds two libraries, called |
| 357 | libpcre and libpcreposix, a test program called pcretest, and the pcregrep |
| 358 | command. If a C++ compiler was found on your system, and you did not disable it |
| 359 | with --disable-cpp, "make" also builds the C++ wrapper library, which is called |
| 360 | libpcrecpp, and some test programs called pcrecpp_unittest, |
| 361 | pcre_scanner_unittest, and pcre_stringpiece_unittest. If you enabled JIT |
| 362 | support with --enable-jit, a test program called pcre_jit_test is also built. |
| 363 | |
| 364 | The command "make check" runs all the appropriate tests. Details of the PCRE |
| 365 | tests are given below in a separate section of this document. |
| 366 | |
| 367 | You can use "make install" to install PCRE into live directories on your |
| 368 | system. The following are installed (file names are all relative to the |
| 369 | <prefix> that is set when "configure" is run): |
| 370 | |
| 371 | Commands (bin): |
| 372 | pcretest |
| 373 | pcregrep |
| 374 | pcre-config |
| 375 | |
| 376 | Libraries (lib): |
| 377 | libpcre |
| 378 | libpcreposix |
| 379 | libpcrecpp (if C++ support is enabled) |
| 380 | |
| 381 | Configuration information (lib/pkgconfig): |
| 382 | libpcre.pc |
| 383 | libpcreposix.pc |
| 384 | libpcrecpp.pc (if C++ support is enabled) |
| 385 | |
| 386 | Header files (include): |
| 387 | pcre.h |
| 388 | pcreposix.h |
| 389 | pcre_scanner.h ) |
| 390 | pcre_stringpiece.h ) if C++ support is enabled |
| 391 | pcrecpp.h ) |
| 392 | pcrecpparg.h ) |
| 393 | |
| 394 | Man pages (share/man/man{1,3}): |
| 395 | pcregrep.1 |
| 396 | pcretest.1 |
| 397 | pcre-config.1 |
| 398 | pcre.3 |
| 399 | pcre*.3 (lots more pages, all starting "pcre") |
| 400 | |
| 401 | HTML documentation (share/doc/pcre/html): |
| 402 | index.html |
| 403 | *.html (lots more pages, hyperlinked from index.html) |
| 404 | |
| 405 | Text file documentation (share/doc/pcre): |
| 406 | AUTHORS |
| 407 | COPYING |
| 408 | ChangeLog |
| 409 | LICENCE |
| 410 | NEWS |
| 411 | README |
| 412 | pcre.txt (a concatenation of the man(3) pages) |
| 413 | pcretest.txt the pcretest man page |
| 414 | pcregrep.txt the pcregrep man page |
| 415 | pcre-config.txt the pcre-config man page |
| 416 | |
| 417 | If you want to remove PCRE from your system, you can run "make uninstall". |
| 418 | This removes all the files that "make install" installed. However, it does not |
| 419 | remove any directories, because these are often shared with other programs. |
| 420 | |
| 421 | |
| 422 | Retrieving configuration information on Unix-like systems |
| 423 | --------------------------------------------------------- |
| 424 | |
| 425 | Running "make install" installs the command pcre-config, which can be used to |
| 426 | recall information about the PCRE configuration and installation. For example: |
| 427 | |
| 428 | pcre-config --version |
| 429 | |
| 430 | prints the version number, and |
| 431 | |
| 432 | pcre-config --libs |
| 433 | |
| 434 | outputs information about where the library is installed. This command can be |
| 435 | included in makefiles for programs that use PCRE, saving the programmer from |
| 436 | having to remember too many details. |
| 437 | |
| 438 | The pkg-config command is another system for saving and retrieving information |
| 439 | about installed libraries. Instead of separate commands for each library, a |
| 440 | single command is used. For example: |
| 441 | |
| 442 | pkg-config --cflags pcre |
| 443 | |
| 444 | The data is held in *.pc files that are installed in a directory called |
| 445 | <prefix>/lib/pkgconfig. |
| 446 | |
| 447 | |
| 448 | Shared libraries on Unix-like systems |
| 449 | ------------------------------------- |
| 450 | |
| 451 | The default distribution builds PCRE as shared libraries and static libraries, |
| 452 | as long as the operating system supports shared libraries. Shared library |
| 453 | support relies on the "libtool" script which is built as part of the |
| 454 | "configure" process. |
| 455 | |
| 456 | The libtool script is used to compile and link both shared and static |
| 457 | libraries. They are placed in a subdirectory called .libs when they are newly |
| 458 | built. The programs pcretest and pcregrep are built to use these uninstalled |
| 459 | libraries (by means of wrapper scripts in the case of shared libraries). When |
| 460 | you use "make install" to install shared libraries, pcregrep and pcretest are |
| 461 | automatically re-built to use the newly installed shared libraries before being |
| 462 | installed themselves. However, the versions left in the build directory still |
| 463 | use the uninstalled libraries. |
| 464 | |
| 465 | To build PCRE using static libraries only you must use --disable-shared when |
| 466 | configuring it. For example: |
| 467 | |
| 468 | ./configure --prefix=/usr/gnu --disable-shared |
| 469 | |
| 470 | Then run "make" in the usual way. Similarly, you can use --disable-static to |
| 471 | build only shared libraries. |
| 472 | |
| 473 | |
| 474 | Cross-compiling on Unix-like systems |
| 475 | ------------------------------------ |
| 476 | |
| 477 | You can specify CC and CFLAGS in the normal way to the "configure" command, in |
| 478 | order to cross-compile PCRE for some other host. However, you should NOT |
| 479 | specify --enable-rebuild-chartables, because if you do, the dftables.c source |
| 480 | file is compiled and run on the local host, in order to generate the inbuilt |
| 481 | character tables (the pcre_chartables.c file). This will probably not work, |
| 482 | because dftables.c needs to be compiled with the local compiler, not the cross |
| 483 | compiler. |
| 484 | |
| 485 | When --enable-rebuild-chartables is not specified, pcre_chartables.c is created |
| 486 | by making a copy of pcre_chartables.c.dist, which is a default set of tables |
| 487 | that assumes ASCII code. Cross-compiling with the default tables should not be |
| 488 | a problem. |
| 489 | |
| 490 | If you need to modify the character tables when cross-compiling, you should |
| 491 | move pcre_chartables.c.dist out of the way, then compile dftables.c by hand and |
| 492 | run it on the local host to make a new version of pcre_chartables.c.dist. |
| 493 | Then when you cross-compile PCRE this new version of the tables will be used. |
| 494 | |
| 495 | |
| 496 | Using HP's ANSI C++ compiler (aCC) |
| 497 | ---------------------------------- |
| 498 | |
| 499 | Unless C++ support is disabled by specifying the "--disable-cpp" option of the |
| 500 | "configure" script, you must include the "-AA" option in the CXXFLAGS |
| 501 | environment variable in order for the C++ components to compile correctly. |
| 502 | |
| 503 | Also, note that the aCC compiler on PA-RISC platforms may have a defect whereby |
| 504 | needed libraries fail to get included when specifying the "-AA" compiler |
| 505 | option. If you experience unresolved symbols when linking the C++ programs, |
| 506 | use the workaround of specifying the following environment variable prior to |
| 507 | running the "configure" script: |
| 508 | |
| 509 | CXXLDFLAGS="-lstd_v2 -lCsup_v2" |
| 510 | |
| 511 | |
| 512 | Using Sun's compilers for Solaris |
| 513 | --------------------------------- |
| 514 | |
| 515 | A user reports that the following configurations work on Solaris 9 sparcv9 and |
| 516 | Solaris 9 x86 (32-bit): |
| 517 | |
| 518 | Solaris 9 sparcv9: ./configure --disable-cpp CC=/bin/cc CFLAGS="-m64 -g" |
| 519 | Solaris 9 x86: ./configure --disable-cpp CC=/bin/cc CFLAGS="-g" |
| 520 | |
| 521 | |
| 522 | Using PCRE from MySQL |
| 523 | --------------------- |
| 524 | |
| 525 | On systems where both PCRE and MySQL are installed, it is possible to make use |
| 526 | of PCRE from within MySQL, as an alternative to the built-in pattern matching. |
| 527 | There is a web page that tells you how to do this: |
| 528 | |
| 529 | http://www.mysqludf.org/lib_mysqludf_preg/index.php |
| 530 | |
| 531 | |
| 532 | Making new tarballs |
| 533 | ------------------- |
| 534 | |
| 535 | The command "make dist" creates three PCRE tarballs, in tar.gz, tar.bz2, and |
| 536 | zip formats. The command "make distcheck" does the same, but then does a trial |
| 537 | build of the new distribution to ensure that it works. |
| 538 | |
| 539 | If you have modified any of the man page sources in the doc directory, you |
| 540 | should first run the PrepareRelease script before making a distribution. This |
| 541 | script creates the .txt and HTML forms of the documentation from the man pages. |
| 542 | |
| 543 | |
| 544 | Testing PCRE |
| 545 | ------------ |
| 546 | |
| 547 | To test the basic PCRE library on a Unix system, run the RunTest script that is |
| 548 | created by the configuring process. There is also a script called RunGrepTest |
| 549 | that tests the options of the pcregrep command. If the C++ wrapper library is |
| 550 | built, three test programs called pcrecpp_unittest, pcre_scanner_unittest, and |
| 551 | pcre_stringpiece_unittest are also built. When JIT support is enabled, another |
| 552 | test program called pcre_jit_test is built. |
| 553 | |
| 554 | Both the scripts and all the program tests are run if you obey "make check" or |
| 555 | "make test". For other systems, see the instructions in NON-UNIX-USE. |
| 556 | |
| 557 | The RunTest script runs the pcretest test program (which is documented in its |
| 558 | own man page) on each of the relevant testinput files in the testdata |
| 559 | directory, and compares the output with the contents of the corresponding |
| 560 | testoutput files. Some tests are relevant only when certain build-time options |
| 561 | were selected. For example, the tests for UTF-8 support are run only if |
| 562 | --enable-utf8 was used. RunTest outputs a comment when it skips a test. |
| 563 | |
| 564 | Many of the tests that are not skipped are run up to three times. The second |
| 565 | run forces pcre_study() to be called for all patterns except for a few in some |
| 566 | tests that are marked "never study" (see the pcretest program for how this is |
| 567 | done). If JIT support is available, the non-DFA tests are run a third time, |
| 568 | this time with a forced pcre_study() with the PCRE_STUDY_JIT_COMPILE option. |
| 569 | |
| 570 | RunTest uses a file called testtry to hold the main output from pcretest |
| 571 | (testsavedregex is also used as a working file). To run pcretest on just one of |
| 572 | the test files, give its number as an argument to RunTest, for example: |
| 573 | |
| 574 | RunTest 2 |
| 575 | |
| 576 | The first test file can be fed directly into the perltest.pl script to check |
| 577 | that Perl gives the same results. The only difference you should see is in the |
| 578 | first few lines, where the Perl version is given instead of the PCRE version. |
| 579 | |
| 580 | The second set of tests check pcre_fullinfo(), pcre_info(), pcre_study(), |
| 581 | pcre_copy_substring(), pcre_get_substring(), pcre_get_substring_list(), error |
| 582 | detection, and run-time flags that are specific to PCRE, as well as the POSIX |
| 583 | wrapper API. It also uses the debugging flags to check some of the internals of |
| 584 | pcre_compile(). |
| 585 | |
| 586 | If you build PCRE with a locale setting that is not the standard C locale, the |
| 587 | character tables may be different (see next paragraph). In some cases, this may |
| 588 | cause failures in the second set of tests. For example, in a locale where the |
| 589 | isprint() function yields TRUE for characters in the range 128-255, the use of |
| 590 | [:isascii:] inside a character class defines a different set of characters, and |
| 591 | this shows up in this test as a difference in the compiled code, which is being |
| 592 | listed for checking. Where the comparison test output contains [\x00-\x7f] the |
| 593 | test will contain [\x00-\xff], and similarly in some other cases. This is not a |
| 594 | bug in PCRE. |
| 595 | |
| 596 | The third set of tests checks pcre_maketables(), the facility for building a |
| 597 | set of character tables for a specific locale and using them instead of the |
| 598 | default tables. The tests make use of the "fr_FR" (French) locale. Before |
| 599 | running the test, the script checks for the presence of this locale by running |
| 600 | the "locale" command. If that command fails, or if it doesn't include "fr_FR" |
| 601 | in the list of available locales, the third test cannot be run, and a comment |
| 602 | is output to say why. If running this test produces instances of the error |
| 603 | |
| 604 | ** Failed to set locale "fr_FR" |
| 605 | |
| 606 | in the comparison output, it means that locale is not available on your system, |
| 607 | despite being listed by "locale". This does not mean that PCRE is broken. |
| 608 | |
| 609 | [If you are trying to run this test on Windows, you may be able to get it to |
| 610 | work by changing "fr_FR" to "french" everywhere it occurs. Alternatively, use |
| 611 | RunTest.bat. The version of RunTest.bat included with PCRE 7.4 and above uses |
| 612 | Windows versions of test 2. More info on using RunTest.bat is included in the |
| 613 | document entitled NON-UNIX-USE.] |
| 614 | |
| 615 | The fourth test checks the UTF-8 support. This file can be also fed directly to |
| 616 | the perltest.pl script, provided you are running Perl 5.8 or higher. |
| 617 | |
| 618 | The fifth test checks error handling with UTF-8 encoding, and internal UTF-8 |
| 619 | features of PCRE that are not relevant to Perl. |
| 620 | |
| 621 | The sixth test (which is Perl-5.10 compatible) checks the support for Unicode |
| 622 | character properties. This file can be also fed directly to the perltest.pl |
| 623 | script, provided you are running Perl 5.10 or higher. |
| 624 | |
| 625 | The seventh, eighth, and ninth tests check the pcre_dfa_exec() alternative |
| 626 | matching function, in non-UTF-8 mode, UTF-8 mode, and UTF-8 mode with Unicode |
| 627 | property support, respectively. |
| 628 | |
| 629 | The tenth test checks some internal offsets and code size features; it is run |
| 630 | only when the default "link size" of 2 is set (in other cases the sizes |
| 631 | change) and when Unicode property support is enabled. |
| 632 | |
| 633 | The eleventh and twelfth tests check out features that are new in Perl 5.10, |
| 634 | without and with UTF-8 support, respectively. This file can be also fed |
| 635 | directly to the perltest.pl script, provided you are running Perl 5.10 or |
| 636 | higher. |
| 637 | |
| 638 | The thirteenth test checks a number internals and non-Perl features concerned |
| 639 | with Unicode property support. |
| 640 | |
| 641 | The fourteenth test is run only when JIT support is available, and the |
| 642 | fifteenth test is run only when JIT support is not available. They test some |
| 643 | JIT-specific features such as information output from pcretest about JIT |
| 644 | compilation. |
| 645 | |
| 646 | |
| 647 | Character tables |
| 648 | ---------------- |
| 649 | |
| 650 | For speed, PCRE uses four tables for manipulating and identifying characters |
| 651 | whose code point values are less than 256. The final argument of the |
| 652 | pcre_compile() function is a pointer to a block of memory containing the |
| 653 | concatenated tables. A call to pcre_maketables() can be used to generate a set |
| 654 | of tables in the current locale. If the final argument for pcre_compile() is |
| 655 | passed as NULL, a set of default tables that is built into the binary is used. |
| 656 | |
| 657 | The source file called pcre_chartables.c contains the default set of tables. By |
| 658 | default, this is created as a copy of pcre_chartables.c.dist, which contains |
| 659 | tables for ASCII coding. However, if --enable-rebuild-chartables is specified |
| 660 | for ./configure, a different version of pcre_chartables.c is built by the |
| 661 | program dftables (compiled from dftables.c), which uses the ANSI C character |
| 662 | handling functions such as isalnum(), isalpha(), isupper(), islower(), etc. to |
| 663 | build the table sources. This means that the default C locale which is set for |
| 664 | your system will control the contents of these default tables. You can change |
| 665 | the default tables by editing pcre_chartables.c and then re-building PCRE. If |
| 666 | you do this, you should take care to ensure that the file does not get |
| 667 | automatically re-generated. The best way to do this is to move |
| 668 | pcre_chartables.c.dist out of the way and replace it with your customized |
| 669 | tables. |
| 670 | |
| 671 | When the dftables program is run as a result of --enable-rebuild-chartables, |
| 672 | it uses the default C locale that is set on your system. It does not pay |
| 673 | attention to the LC_xxx environment variables. In other words, it uses the |
| 674 | system's default locale rather than whatever the compiling user happens to have |
| 675 | set. If you really do want to build a source set of character tables in a |
| 676 | locale that is specified by the LC_xxx variables, you can run the dftables |
| 677 | program by hand with the -L option. For example: |
| 678 | |
| 679 | ./dftables -L pcre_chartables.c.special |
| 680 | |
| 681 | The first two 256-byte tables provide lower casing and case flipping functions, |
| 682 | respectively. The next table consists of three 32-byte bit maps which identify |
| 683 | digits, "word" characters, and white space, respectively. These are used when |
| 684 | building 32-byte bit maps that represent character classes for code points less |
| 685 | than 256. |
| 686 | |
| 687 | The final 256-byte table has bits indicating various character types, as |
| 688 | follows: |
| 689 | |
| 690 | 1 white space character |
| 691 | 2 letter |
| 692 | 4 decimal digit |
| 693 | 8 hexadecimal digit |
| 694 | 16 alphanumeric or '_' |
| 695 | 128 regular expression metacharacter or binary zero |
| 696 | |
| 697 | You should not alter the set of characters that contain the 128 bit, as that |
| 698 | will cause PCRE to malfunction. |
| 699 | |
| 700 | |
| 701 | File manifest |
| 702 | ------------- |
| 703 | |
| 704 | The distribution should contain the following files: |
| 705 | |
| 706 | (A) Source files of the PCRE library functions and their headers: |
| 707 | |
| 708 | dftables.c auxiliary program for building pcre_chartables.c |
| 709 | when --enable-rebuild-chartables is specified |
| 710 | |
| 711 | pcre_chartables.c.dist a default set of character tables that assume ASCII |
| 712 | coding; used, unless --enable-rebuild-chartables is |
| 713 | specified, by copying to pcre_chartables.c |
| 714 | |
| 715 | pcreposix.c ) |
| 716 | pcre_compile.c ) |
| 717 | pcre_config.c ) |
| 718 | pcre_dfa_exec.c ) |
| 719 | pcre_exec.c ) |
| 720 | pcre_fullinfo.c ) |
| 721 | pcre_get.c ) sources for the functions in the library, |
| 722 | pcre_globals.c ) and some internal functions that they use |
| 723 | pcre_info.c ) |
| 724 | pcre_jit_compile.c ) |
| 725 | pcre_maketables.c ) |
| 726 | pcre_newline.c ) |
| 727 | pcre_ord2utf8.c ) |
| 728 | pcre_refcount.c ) |
| 729 | pcre_study.c ) |
| 730 | pcre_tables.c ) |
| 731 | pcre_try_flipped.c ) |
| 732 | pcre_ucd.c ) |
| 733 | pcre_valid_utf8.c ) |
| 734 | pcre_version.c ) |
| 735 | pcre_xclass.c ) |
| 736 | pcre_printint.src ) debugging function that is #included in pcretest, |
| 737 | ) and can also be #included in pcre_compile() |
| 738 | pcre.h.in template for pcre.h when built by "configure" |
| 739 | pcreposix.h header for the external POSIX wrapper API |
| 740 | pcre_internal.h header for internal use |
| 741 | sljit/* 16 files that make up the JIT compiler |
| 742 | ucp.h header for Unicode property handling |
| 743 | |
| 744 | config.h.in template for config.h, which is built by "configure" |
| 745 | |
| 746 | pcrecpp.h public header file for the C++ wrapper |
| 747 | pcrecpparg.h.in template for another C++ header file |
| 748 | pcre_scanner.h public header file for C++ scanner functions |
| 749 | pcrecpp.cc ) |
| 750 | pcre_scanner.cc ) source for the C++ wrapper library |
| 751 | |
| 752 | pcre_stringpiece.h.in template for pcre_stringpiece.h, the header for the |
| 753 | C++ stringpiece functions |
| 754 | pcre_stringpiece.cc source for the C++ stringpiece functions |
| 755 | |
| 756 | (B) Source files for programs that use PCRE: |
| 757 | |
| 758 | pcredemo.c simple demonstration of coding calls to PCRE |
| 759 | pcregrep.c source of a grep utility that uses PCRE |
| 760 | pcretest.c comprehensive test program |
| 761 | |
| 762 | (C) Auxiliary files: |
| 763 | |
| 764 | 132html script to turn "man" pages into HTML |
| 765 | AUTHORS information about the author of PCRE |
| 766 | ChangeLog log of changes to the code |
| 767 | CleanTxt script to clean nroff output for txt man pages |
| 768 | Detrail script to remove trailing spaces |
| 769 | HACKING some notes about the internals of PCRE |
| 770 | INSTALL generic installation instructions |
| 771 | LICENCE conditions for the use of PCRE |
| 772 | COPYING the same, using GNU's standard name |
| 773 | Makefile.in ) template for Unix Makefile, which is built by |
| 774 | ) "configure" |
| 775 | Makefile.am ) the automake input that was used to create |
| 776 | ) Makefile.in |
| 777 | NEWS important changes in this release |
| 778 | NON-UNIX-USE notes on building PCRE on non-Unix systems |
| 779 | PrepareRelease script to make preparations for "make dist" |
| 780 | README this file |
| 781 | RunTest a Unix shell script for running tests |
| 782 | RunGrepTest a Unix shell script for pcregrep tests |
| 783 | aclocal.m4 m4 macros (generated by "aclocal") |
| 784 | config.guess ) files used by libtool, |
| 785 | config.sub ) used only when building a shared library |
| 786 | configure a configuring shell script (built by autoconf) |
| 787 | configure.ac ) the autoconf input that was used to build |
| 788 | ) "configure" and config.h |
| 789 | depcomp ) script to find program dependencies, generated by |
| 790 | ) automake |
| 791 | doc/*.3 man page sources for PCRE |
| 792 | doc/*.1 man page sources for pcregrep and pcretest |
| 793 | doc/index.html.src the base HTML page |
| 794 | doc/html/* HTML documentation |
| 795 | doc/pcre.txt plain text version of the man pages |
| 796 | doc/pcretest.txt plain text documentation of test program |
| 797 | doc/perltest.txt plain text documentation of Perl test program |
| 798 | install-sh a shell script for installing files |
| 799 | libpcre.pc.in template for libpcre.pc for pkg-config |
| 800 | libpcreposix.pc.in template for libpcreposix.pc for pkg-config |
| 801 | libpcrecpp.pc.in template for libpcrecpp.pc for pkg-config |
| 802 | ltmain.sh file used to build a libtool script |
| 803 | missing ) common stub for a few missing GNU programs while |
| 804 | ) installing, generated by automake |
| 805 | mkinstalldirs script for making install directories |
| 806 | perltest.pl Perl test program |
| 807 | pcre-config.in source of script which retains PCRE information |
| 808 | pcre_jit_test.c test program for the JIT compiler |
| 809 | pcrecpp_unittest.cc ) |
| 810 | pcre_scanner_unittest.cc ) test programs for the C++ wrapper |
| 811 | pcre_stringpiece_unittest.cc ) |
| 812 | testdata/testinput* test data for main library tests |
| 813 | testdata/testoutput* expected test results |
| 814 | testdata/grep* input and output for pcregrep tests |
| 815 | |
| 816 | (D) Auxiliary files for cmake support |
| 817 | |
| 818 | cmake/COPYING-CMAKE-SCRIPTS |
| 819 | cmake/FindPackageHandleStandardArgs.cmake |
| 820 | cmake/FindReadline.cmake |
| 821 | CMakeLists.txt |
| 822 | config-cmake.h.in |
| 823 | |
| 824 | (E) Auxiliary files for VPASCAL |
| 825 | |
| 826 | makevp.bat |
| 827 | makevp_c.txt |
| 828 | makevp_l.txt |
| 829 | pcregexp.pas |
| 830 | |
| 831 | (F) Auxiliary files for building PCRE "by hand" |
| 832 | |
| 833 | pcre.h.generic ) a version of the public PCRE header file |
| 834 | ) for use in non-"configure" environments |
| 835 | config.h.generic ) a version of config.h for use in non-"configure" |
| 836 | ) environments |
| 837 | |
| 838 | (F) Miscellaneous |
| 839 | |
| 840 | RunTest.bat a script for running tests under Windows |
| 841 | |
| 842 | Philip Hazel |
| 843 | Email local part: ph10 |
| 844 | Email domain: cam.ac.uk |
| 845 | Last updated: 06 September 2011 |