Blame - jni/libpcre/sources/doc/html/pcrejit.html - jami-client-android

blob: 7411ecf2e8ce06df6154bafd53b6345bf7717171 [file] [log] [blame]

Tristan Matthews	0461646	2013-11-14 16:09:34 -0500	[diff] [blame]	1	<html>
				2	<head>
				3	<title>pcrejit specification</title>
				4	</head>
				5	<body bgcolor="#FFFFFF" text="#00005A" link="#0066FF" alink="#3399FF" vlink="#2222BB">
				6	<h1>pcrejit man page</h1>
				7	<p>
				8	Return to the <a href="index.html">PCRE index page</a>.
				9	</p>
				10	<p>
				11	This page is part of the PCRE HTML documentation. It was generated automatically
				12	from the original man page. If there is any nonsense in it, please consult the
				13	man page, in case the conversion went wrong.
				14	<br>
				15	<ul>
				16	<li><a name="TOC1" href="#SEC1">PCRE JUST-IN-TIME COMPILER SUPPORT</a>
				17	<li><a name="TOC2" href="#SEC2">AVAILABILITY OF JIT SUPPORT</a>
				18	<li><a name="TOC3" href="#SEC3">SIMPLE USE OF JIT</a>
				19	<li><a name="TOC4" href="#SEC4">UNSUPPORTED OPTIONS AND PATTERN ITEMS</a>
				20	<li><a name="TOC5" href="#SEC5">RETURN VALUES FROM JIT EXECUTION</a>
				21	<li><a name="TOC6" href="#SEC6">SAVING AND RESTORING COMPILED PATTERNS</a>
				22	<li><a name="TOC7" href="#SEC7">CONTROLLING THE JIT STACK</a>
				23	<li><a name="TOC8" href="#SEC8">JIT STACK FAQ</a>
				24	<li><a name="TOC9" href="#SEC9">EXAMPLE CODE</a>
				25	<li><a name="TOC10" href="#SEC10">SEE ALSO</a>
				26	<li><a name="TOC11" href="#SEC11">AUTHOR</a>
				27	<li><a name="TOC12" href="#SEC12">REVISION</a>
				28	</ul>
				29	<br><a name="SEC1" href="#TOC1">PCRE JUST-IN-TIME COMPILER SUPPORT</a><br>
				30	<P>
				31	Just-in-time compiling is a heavyweight optimization that can greatly speed up
				32	pattern matching. However, it comes at the cost of extra processing before the
				33	match is performed. Therefore, it is of most benefit when the same pattern is
				34	going to be matched many times. This does not necessarily mean many calls of
				35	\fPpcre_exec()\fP; if the pattern is not anchored, matching attempts may take
				36	place many times at various positions in the subject, even for a single call to
				37	<b>pcre_exec()</b>. If the subject string is very long, it may still pay to use
				38	JIT for one-off matches.
				39	</P>
				40	<P>
				41	JIT support applies only to the traditional matching function,
				42	<b>pcre_exec()</b>. It does not apply when <b>pcre_dfa_exec()</b> is being used.
				43	The code for this support was written by Zoltan Herczeg.
				44	</P>
				45	<br><a name="SEC2" href="#TOC1">AVAILABILITY OF JIT SUPPORT</a><br>
				46	<P>
				47	JIT support is an optional feature of PCRE. The "configure" option --enable-jit
				48	(or equivalent CMake option) must be set when PCRE is built if you want to use
				49	JIT. The support is limited to the following hardware platforms:
				50	<pre>
				51	ARM v5, v7, and Thumb2
				52	Intel x86 32-bit and 64-bit
				53	MIPS 32-bit
				54	Power PC 32-bit and 64-bit (experimental)
				55	</pre>
				56	The Power PC support is designated as experimental because it has not been
				57	fully tested. If --enable-jit is set on an unsupported platform, compilation
				58	fails.
				59	</P>
				60	<P>
				61	A program that is linked with PCRE 8.20 or later can tell if JIT support is
				62	available by calling <b>pcre_config()</b> with the PCRE_CONFIG_JIT option. The
				63	result is 1 when JIT is available, and 0 otherwise. However, a simple program
				64	does not need to check this in order to use JIT. The API is implemented in a
				65	way that falls back to the ordinary PCRE code if JIT is not available.
				66	</P>
				67	<P>
				68	If your program may sometimes be linked with versions of PCRE that are older
				69	than 8.20, but you want to use JIT when it is available, you can test
				70	the values of PCRE_MAJOR and PCRE_MINOR, or the existence of a JIT macro such
				71	as PCRE_CONFIG_JIT, for compile-time control of your code.
				72	</P>
				73	<br><a name="SEC3" href="#TOC1">SIMPLE USE OF JIT</a><br>
				74	<P>
				75	You have to do two things to make use of the JIT support in the simplest way:
				76	<pre>
				77	(1) Call <b>pcre_study()</b> with the PCRE_STUDY_JIT_COMPILE option for
				78	each compiled pattern, and pass the resulting <b>pcre_extra</b> block to
				79	<b>pcre_exec()</b>.
				80
				81	(2) Use <b>pcre_free_study()</b> to free the <b>pcre_extra</b> block when it is
				82	no longer needed instead of just freeing it yourself. This
				83	ensures that any JIT data is also freed.
				84	</pre>
				85	For a program that may be linked with pre-8.20 versions of PCRE, you can insert
				86	<pre>
				87	#ifndef PCRE_STUDY_JIT_COMPILE
				88	#define PCRE_STUDY_JIT_COMPILE 0
				89	#endif
				90	</pre>
				91	so that no option is passed to <b>pcre_study()</b>, and then use something like
				92	this to free the study data:
				93	<pre>
				94	#ifdef PCRE_CONFIG_JIT
				95	pcre_free_study(study_ptr);
				96	#else
				97	pcre_free(study_ptr);
				98	#endif
				99	</pre>
				100	In some circumstances you may need to call additional functions. These are
				101	described in the section entitled
				102	<a href="#stackcontrol">"Controlling the JIT stack"</a>
				103	below.
				104	</P>
				105	<P>
				106	If JIT support is not available, PCRE_STUDY_JIT_COMPILE is ignored, and no JIT
				107	data is set up. Otherwise, the compiled pattern is passed to the JIT compiler,
				108	which turns it into machine code that executes much faster than the normal
				109	interpretive code. When <b>pcre_exec()</b> is passed a <b>pcre_extra</b> block
				110	containing a pointer to JIT code, it obeys that instead of the normal code. The
				111	result is identical, but the code runs much faster.
				112	</P>
				113	<P>
				114	There are some <b>pcre_exec()</b> options that are not supported for JIT
				115	execution. There are also some pattern items that JIT cannot handle. Details
				116	are given below. In both cases, execution automatically falls back to the
				117	interpretive code.
				118	</P>
				119	<P>
				120	If the JIT compiler finds an unsupported item, no JIT data is generated. You
				121	can find out if JIT execution is available after studying a pattern by calling
				122	<b>pcre_fullinfo()</b> with the PCRE_INFO_JIT option. A result of 1 means that
				123	JIT compilation was successful. A result of 0 means that JIT support is not
				124	available, or the pattern was not studied with PCRE_STUDY_JIT_COMPILE, or the
				125	JIT compiler was not able to handle the pattern.
				126	</P>
				127	<P>
				128	Once a pattern has been studied, with or without JIT, it can be used as many
				129	times as you like for matching different subject strings.
				130	</P>
				131	<br><a name="SEC4" href="#TOC1">UNSUPPORTED OPTIONS AND PATTERN ITEMS</a><br>
				132	<P>
				133	The only <b>pcre_exec()</b> options that are supported for JIT execution are
				134	PCRE_NO_UTF8_CHECK, PCRE_NOTBOL, PCRE_NOTEOL, PCRE_NOTEMPTY, and
				135	PCRE_NOTEMPTY_ATSTART. Note in particular that partial matching is not
				136	supported.
				137	</P>
				138	<P>
				139	The unsupported pattern items are:
				140	<pre>
				141	\C match a single byte; not supported in UTF-8 mode
				142	(?Cn) callouts
				143	(*COMMIT) )
				144	(*MARK) )
				145	(*PRUNE) ) the backtracking control verbs
				146	(*SKIP) )
				147	(*THEN) )
				148	</pre>
				149	Support for some of these may be added in future.
				150	</P>
				151	<br><a name="SEC5" href="#TOC1">RETURN VALUES FROM JIT EXECUTION</a><br>
				152	<P>
				153	When a pattern is matched using JIT execution, the return values are the same
				154	as those given by the interpretive <b>pcre_exec()</b> code, with the addition of
				155	one new error code: PCRE_ERROR_JIT_STACKLIMIT. This means that the memory used
				156	for the JIT stack was insufficient. See
				157	<a href="#stackcontrol">"Controlling the JIT stack"</a>
				158	below for a discussion of JIT stack usage. For compatibility with the
				159	interpretive <b>pcre_exec()</b> code, no more than two-thirds of the
				160	<i>ovector</i> argument is used for passing back captured substrings.
				161	</P>
				162	<P>
				163	The error code PCRE_ERROR_MATCHLIMIT is returned by the JIT code if searching a
				164	very large pattern tree goes on for too long, as it is in the same circumstance
				165	when JIT is not used, but the details of exactly what is counted are not the
				166	same. The PCRE_ERROR_RECURSIONLIMIT error code is never returned by JIT
				167	execution.
				168	</P>
				169	<br><a name="SEC6" href="#TOC1">SAVING AND RESTORING COMPILED PATTERNS</a><br>
				170	<P>
				171	The code that is generated by the JIT compiler is architecture-specific, and is
				172	also position dependent. For those reasons it cannot be saved (in a file or
				173	database) and restored later like the bytecode and other data of a compiled
				174	pattern. Saving and restoring compiled patterns is not something many people
				175	do. More detail about this facility is given in the
				176	<a href="pcreprecompile.html"><b>pcreprecompile</b></a>
				177	documentation. It should be possible to run <b>pcre_study()</b> on a saved and
				178	restored pattern, and thereby recreate the JIT data, but because JIT
				179	compilation uses significant resources, it is probably not worth doing this;
				180	you might as well recompile the original pattern.
				181	<a name="stackcontrol"></a></P>
				182	<br><a name="SEC7" href="#TOC1">CONTROLLING THE JIT STACK</a><br>
				183	<P>
				184	When the compiled JIT code runs, it needs a block of memory to use as a stack.
				185	By default, it uses 32K on the machine stack. However, some large or
				186	complicated patterns need more than this. The error PCRE_ERROR_JIT_STACKLIMIT
				187	is given when there is not enough stack. Three functions are provided for
				188	managing blocks of memory for use as JIT stacks. There is further discussion
				189	about the use of JIT stacks in the section entitled
				190	<a href="#stackcontrol">"JIT stack FAQ"</a>
				191	below.
				192	</P>
				193	<P>
				194	The <b>pcre_jit_stack_alloc()</b> function creates a JIT stack. Its arguments
				195	are a starting size and a maximum size, and it returns a pointer to an opaque
				196	structure of type <b>pcre_jit_stack</b>, or NULL if there is an error. The
				197	<b>pcre_jit_stack_free()</b> function can be used to free a stack that is no
				198	longer needed. (For the technically minded: the address space is allocated by
				199	mmap or VirtualAlloc.)
				200	</P>
				201	<P>
				202	JIT uses far less memory for recursion than the interpretive code,
				203	and a maximum stack size of 512K to 1M should be more than enough for any
				204	pattern.
				205	</P>
				206	<P>
				207	The <b>pcre_assign_jit_stack()</b> function specifies which stack JIT code
				208	should use. Its arguments are as follows:
				209	<pre>
				210	pcre_extra *extra
				211	pcre_jit_callback callback
				212	void *data
				213	</pre>
				214	The <i>extra</i> argument must be the result of studying a pattern with
				215	PCRE_STUDY_JIT_COMPILE. There are three cases for the values of the other two
				216	options:
				217	<pre>
				218	(1) If <i>callback</i> is NULL and <i>data</i> is NULL, an internal 32K block
				219	on the machine stack is used.
				220
				221	(2) If <i>callback</i> is NULL and <i>data</i> is not NULL, <i>data</i> must be
				222	a valid JIT stack, the result of calling <b>pcre_jit_stack_alloc()</b>.
				223
				224	(3) If <i>callback</i> not NULL, it must point to a function that is called
				225	with <i>data</i> as an argument at the start of matching, in order to
				226	set up a JIT stack. If the result is NULL, the internal 32K stack
				227	is used; otherwise the return value must be a valid JIT stack,
				228	the result of calling <b>pcre_jit_stack_alloc()</b>.
				229	</pre>
				230	You may safely assign the same JIT stack to more than one pattern, as long as
				231	they are all matched sequentially in the same thread. In a multithread
				232	application, each thread must use its own JIT stack.
				233	</P>
				234	<P>
				235	Strictly speaking, even more is allowed. You can assign the same stack to any
				236	number of patterns as long as they are not used for matching by multiple
				237	threads at the same time. For example, you can assign the same stack to all
				238	compiled patterns, and use a global mutex in the callback to wait until the
				239	stack is available for use. However, this is an inefficient solution, and
				240	not recommended.
				241	</P>
				242	<P>
				243	This is a suggestion for how a typical multithreaded program might operate:
				244	<pre>
				245	During thread initalization
				246	thread_local_var = pcre_jit_stack_alloc(...)
				247
				248	During thread exit
				249	pcre_jit_stack_free(thread_local_var)
				250
				251	Use a one-line callback function
				252	return thread_local_var
				253	</pre>
				254	All the functions described in this section do nothing if JIT is not available,
				255	and <b>pcre_assign_jit_stack()</b> does nothing unless the <b>extra</b> argument
				256	is non-NULL and points to a <b>pcre_extra</b> block that is the result of a
				257	successful study with PCRE_STUDY_JIT_COMPILE.
				258	<a name="stackfaq"></a></P>
				259	<br><a name="SEC8" href="#TOC1">JIT STACK FAQ</a><br>
				260	<P>
				261	(1) Why do we need JIT stacks?
				262	<br>
				263	<br>
				264	PCRE (and JIT) is a recursive, depth-first engine, so it needs a stack where
				265	the local data of the current node is pushed before checking its child nodes.
				266	Allocating real machine stack on some platforms is difficult. For example, the
				267	stack chain needs to be updated every time if we extend the stack on PowerPC.
				268	Although it is possible, its updating time overhead decreases performance. So
				269	we do the recursion in memory.
				270	</P>
				271	<P>
				272	(2) Why don't we simply allocate blocks of memory with <b>malloc()</b>?
				273	<br>
				274	<br>
				275	Modern operating systems have a nice feature: they can reserve an address space
				276	instead of allocating memory. We can safely allocate memory pages inside this
				277	address space, so the stack could grow without moving memory data (this is
				278	important because of pointers). Thus we can allocate 1M address space, and use
				279	only a single memory page (usually 4K) if that is enough. However, we can still
				280	grow up to 1M anytime if needed.
				281	</P>
				282	<P>
				283	(3) Who "owns" a JIT stack?
				284	<br>
				285	<br>
				286	The owner of the stack is the user program, not the JIT studied pattern or
				287	anything else. The user program must ensure that if a stack is used by
				288	<b>pcre_exec()</b>, (that is, it is assigned to the pattern currently running),
				289	that stack must not be used by any other threads (to avoid overwriting the same
				290	memory area). The best practice for multithreaded programs is to allocate a
				291	stack for each thread, and return this stack through the JIT callback function.
				292	</P>
				293	<P>
				294	(4) When should a JIT stack be freed?
				295	<br>
				296	<br>
				297	You can free a JIT stack at any time, as long as it will not be used by
				298	<b>pcre_exec()</b> again. When you assign the stack to a pattern, only a pointer
				299	is set. There is no reference counting or any other magic. You can free the
				300	patterns and stacks in any order, anytime. Just <i>do not</i> call
				301	<b>pcre_exec()</b> with a pattern pointing to an already freed stack, as that
				302	will cause SEGFAULT. (Also, do not free a stack currently used by
				303	<b>pcre_exec()</b> in another thread). You can also replace the stack for a
				304	pattern at any time. You can even free the previous stack before assigning a
				305	replacement.
				306	</P>
				307	<P>
				308	(5) Should I allocate/free a stack every time before/after calling
				309	<b>pcre_exec()</b>?
				310	<br>
				311	<br>
				312	No, because this is too costly in terms of resources. However, you could
				313	implement some clever idea which release the stack if it is not used in let's
				314	say two minutes. The JIT callback can help to achive this without keeping a
				315	list of the currently JIT studied patterns.
				316	</P>
				317	<P>
				318	(6) OK, the stack is for long term memory allocation. But what happens if a
				319	pattern causes stack overflow with a stack of 1M? Is that 1M kept until the
				320	stack is freed?
				321	<br>
				322	<br>
				323	Especially on embedded sytems, it might be a good idea to release
				324	memory sometimes without freeing the stack. There is no API for this at the
				325	moment. Probably a function call which returns with the currently allocated
				326	memory for any stack and another which allows releasing memory (shrinking the
				327	stack) would be a good idea if someone needs this.
				328	</P>
				329	<P>
				330	(7) This is too much of a headache. Isn't there any better solution for JIT
				331	stack handling?
				332	<br>
				333	<br>
				334	No, thanks to Windows. If POSIX threads were used everywhere, we could throw
				335	out this complicated API.
				336	</P>
				337	<br><a name="SEC9" href="#TOC1">EXAMPLE CODE</a><br>
				338	<P>
				339	This is a single-threaded example that specifies a JIT stack without using a
				340	callback.
				341	<pre>
				342	int rc;
				343	int ovector[30];
				344	pcre *re;
				345	pcre_extra *extra;
				346	pcre_jit_stack *jit_stack;
				347
				348	re = pcre_compile(pattern, 0, &error, &erroffset, NULL);
				349	/* Check for errors */
				350	extra = pcre_study(re, PCRE_STUDY_JIT_COMPILE, &error);
				351	jit_stack = pcre_jit_stack_alloc(321024, 5121024);
				352	/* Check for error (NULL) */
				353	pcre_assign_jit_stack(extra, NULL, jit_stack);
				354	rc = pcre_exec(re, extra, subject, length, 0, 0, ovector, 30);
				355	/* Check results */
				356	pcre_free(re);
				357	pcre_free_study(extra);
				358	pcre_jit_stack_free(jit_stack);
				359
				360	</PRE>
				361	</P>
				362	<br><a name="SEC10" href="#TOC1">SEE ALSO</a><br>
				363	<P>
				364	<b>pcreapi</b>(3)
				365	</P>
				366	<br><a name="SEC11" href="#TOC1">AUTHOR</a><br>
				367	<P>
				368	Philip Hazel (FAQ by Zoltan Herczeg)
				369	<br>
				370	University Computing Service
				371	<br>
				372	Cambridge CB2 3QH, England.
				373	<br>
				374	</P>
				375	<br><a name="SEC12" href="#TOC1">REVISION</a><br>
				376	<P>
				377	Last updated: 26 November 2011
				378	<br>
				379	Copyright © 1997-2011 University of Cambridge.
				380	<br>
				381	<p>
				382	Return to the <a href="index.html">PCRE index page</a>.
				383	</p>