As an XML library, lxml.etree is very fast. It is also slow. As with all software, it depends on what you do with it. Rest assured that lxml is fast enough for most applications, so lxml is probably somewhere between 'fast enough' and 'the best choice' for yours.
This text describes where lxml.etree (lxe) excels, gives hints on some performance traps and compares the overall performance to the original ElementTree (ET) and cElementTree (cET) libraries by Fredrik Lundh. The cElementTree library is a fast C-implementation of the original ElementTree.
The statements made here are backed by the benchmark script bench.py that comes with the lxml source distribution. The timings cited below compare lxml 1.0 (with libxml2 2.6.24), ElementTree 1.2.6 and cElementTree 1.0.5 under CPython 2.4.2 on a 1.6GHz AMD64 machine.
The bench.py script runs a number of simple tests on the different libraries, using different XML tree configurations: different tree sizes, with or without attributes (-/A) and with or without ASCII or unicode text (-/S/U). In the result extracts cited below, T1 refers to a 3-level tree with many children at the third level, T2 is swapped around to have many children at the root element, T3 is a deep tree with few children at each level and T4 is a small tree, slightly broader than deep. Most benchmarks run in a loop over all children of the tree root.
First thing to say: there is an overhead involved in having a C library mimic the ElementTree API. As opposed to ElementTree, lxml has to generate Python objects on the fly when asked for them. What this means is: the more of your code runs in Python, the slower your application gets. Note, however, that this is true for most performance critical Python applications.
These are areas where lxml excels. The reason is that both parts are executed entirely at the C level, without major interaction with Python code. The results are rather impressive. Compared to cElementTree, lxml is about 20 to 40 times faster on serialisation:
lxe: tostring_utf16 (SA T2) 30.9846 msec/pass cET: tostring_utf16 (SA T2) 715.5002 msec/pass ET : tostring_utf16 (SA T2) 758.5271 msec/pass lxe: tostring_utf16 (U- T3) 3.0509 msec/pass cET: tostring_utf16 (U- T3) 72.4721 msec/pass ET : tostring_utf16 (U- T3) 87.0735 msec/pass lxe: tostring_utf8 (UA T2) 26.8996 msec/pass cET: tostring_utf8 (UA T2) 700.4889 msec/pass ET : tostring_utf8 (UA T2) 745.3317 msec/pass lxe: tostring_utf8 (S- T3) 2.1876 msec/pass cET: tostring_utf8 (S- T3) 71.1290 msec/pass ET : tostring_utf8 (S- T3) 87.1525 msec/pass
For parsing, the difference between the libraries is smaller. The (c)ET libraries use the expat parser, which is known to be extremely fast:
lxe: parse_stringIO (SA T2) 197.7678 msec/pass cET: parse_stringIO (SA T2) 38.9390 msec/pass ET : parse_stringIO (SA T2) 364.3468 msec/pass lxe: parse_stringIO (UA T3) 48.6735 msec/pass cET: parse_stringIO (UA T3) 39.7455 msec/pass ET : parse_stringIO (UA T3) 237.9971 msec/pass
The expat parser allows cET to be up to 80% faster than lxml on plain parser performance. The same applies to the iterparse() function. However, if you take a complete serialize-parse cycle, the numbers will look similar to these:
lxe: write_utf8_parse_stringIO (S- T1) 187.0444 msec/pass cET: write_utf8_parse_stringIO (S- T1) 828.4068 msec/pass ET : write_utf8_parse_stringIO (S- T1) 1181.0658 msec/pass lxe: write_utf8_parse_stringIO (UA T2) 213.6599 msec/pass cET: write_utf8_parse_stringIO (UA T2) 927.2374 msec/pass ET : write_utf8_parse_stringIO (UA T2) 1297.9678 msec/pass
For applications that require a high parser throughput and do little serialization, cET is the best choice. Also for iterparse applications that extract small amounts of data from large XML data sets. If it comes to round-trip performance, however, lxml tends to be 3-4 times faster in total.
Since all three libraries implement the same API, their performance is easy to compare in this area. A major disadvantage for lxml's performance is the different tree model that underlies libxml2. It allows lxml to provide parent pointers for elements, but also increases the overhead of tree building and restructuring. This can be seen from the tree setup times of the benchmark (given in seconds):
lxe: -- S- U- -A SA UA
T1: 0.1360 0.1214 0.1214 0.1217 0.1232 0.1226
T2: 0.1258 0.1257 0.1250 0.1348 0.1359 0.1358
T3: 0.0354 0.0282 0.0288 0.0850 0.0860 0.0862
T4: 0.0006 0.0006 0.0006 0.0019 0.0018 0.0019
cET: -- S- U- -A SA UA
T1: 0.0417 0.0409 0.0403 0.0410 0.0410 0.0415
T2: 0.0413 0.0414 0.0413 0.0417 0.0411 0.0417
T3: 0.0097 0.0100 0.0099 0.0187 0.0142 0.0146
T4: 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001
ET : -- S- U- -A SA UA
T1: 0.2189 0.2832 0.2210 0.2646 0.2905 0.2214
T2: 0.3022 0.2322 0.2868 0.3192 0.2290 0.3075
T3: 0.0519 0.0553 0.0527 0.0601 0.0572 0.0911
T4: 0.0009 0.0008 0.0008 0.0008 0.0009 0.0009
While lxml is still faster than ET in most cases (30-60%), cET can be up to three times faster than lxml here. One of the reasons is that lxml must additionally discard the created Python elements after their use, when they are no longer referenced. ET and cET represent the tree itself through these objects, which reduces the overhead in creating them.
The same reason makes operations like getchildren() more costly in lxml. Where ET and cET can quickly create a shallow copy of their list of children, lxml has to create a Python object for each child and collect them in a list:
lxe: root_getchildren (-- T2 ) 6.3981 msec/pass cET: root_getchildren (-- T2 ) 0.0651 msec/pass ET : root_getchildren (-- T2 ) 0.0224 msec/pass
As opposed to ET, libxml2 has a notion of documents that each element must be in. This results in a major performance difference for creating independent Elements that end up in independently created documents:
lxe: create_elements (-- T2 ) 22.0083 msec/pass cET: create_elements (-- T2 ) 0.3920 msec/pass ET : create_elements (-- T2 ) 3.0865 msec/pass
Therefore, it is always preferable to create Elements for the document they are supposed to end up in, either as SubElements of an Element or using the explicit Element.makeelement() call:
lxe: makeelement (-- T2 ) 4.2658 msec/pass cET: makeelement (-- T2 ) 0.5658 msec/pass ET : makeelement (-- T2 ) 3.7136 msec/pass lxe: create_subelements (-- T2 ) 3.7640 msec/pass cET: create_subelements (-- T2 ) 0.5332 msec/pass ET : create_subelements (-- T2 ) 6.5937 msec/pass
So, if the main performance bottleneck of an application is creating large XML trees in memory through calls to Element and SubElement, cET is the best choice. Note, however, that the serialisation performance may even out this advantage, especially for smaller trees and trees with many attributes.
A critical action for lxml is moving elements between document contexts. It requires lxml to do recursive adaptations throughout the moved tree structure.
The following benchmark appends all root children of the second tree to the root of the first tree:
lxe: append_from_document (-- T1,T2) 11.7905 msec/pass cET: append_from_document (-- T1,T2) 0.4673 msec/pass ET : append_from_document (-- T1,T2) 2.0460 msec/pass lxe: append_from_document (-- T3,T4) 0.1582 msec/pass cET: append_from_document (-- T3,T4) 0.0224 msec/pass ET : append_from_document (-- T3,T4) 0.1618 msec/pass
Although these are fairly small numbers compared to parsing, this easily shows the different performance classes for lxml and (c)ET. Where the latter do not have to care about parent pointers and tree structures, lxml has to deep traverse the appended tree. The performance difference therefore increases with the size of the tree that is moved.
This difference is not always as visible, but applies to most parts of the API, like inserting newly created elements:
lxe: insert_from_document (-- T1,T2) 16.2342 msec/pass cET: insert_from_document (-- T1,T2) 1.1786 msec/pass ET : insert_from_document (-- T1,T2) 3.6107 msec/pass
Or replacing the child slice by a new element:
lxe: replace_children_element (-- T1 ) 9.1834 msec/pass cET: replace_children_element (-- T1 ) 0.9731 msec/pass ET : replace_children_element (-- T1 ) 14.8213 msec/pass
You should keep this difference in mind when you merge very large trees. On the other hand, deep copying a tree is fast in lxml:
lxe: deepcopy (-- T1 ) 24.7359 msec/pass cET: deepcopy (-- T1 ) 450.5479 msec/pass ET : deepcopy (-- T1 ) 717.8308 msec/pass lxe: deepcopy (-- T3 ) 2.1182 msec/pass cET: deepcopy (-- T3 ) 107.2124 msec/pass ET : deepcopy (-- T3 ) 173.9782 msec/pass
So, for example, if you often need to create independent subtrees from a large tree that you have parsed in, lxml is by far the best choice here.
Another area where lxml is very fast is iteration for tree traversal. If your algorithms can benefit from step-by-step traversal of the XML tree and especially if few elements are of interest or the element tag name is known, lxml is a good choice:
lxe: getiterator_all (-- T2 ) 22.5847 msec/pass cET: getiterator_all (-- T2 ) 36.8212 msec/pass ET : getiterator_all (-- T2 ) 46.2846 msec/pass lxe: getiterator_islice (-- T2 ) 2.0421 msec/pass cET: getiterator_islice (-- T2 ) 0.3343 msec/pass ET : getiterator_islice (-- T2 ) 44.5898 msec/pass lxe: getiterator_tag (-- T2 ) 1.9593 msec/pass cET: getiterator_tag (-- T2 ) 11.7767 msec/pass ET : getiterator_tag (-- T2 ) 37.5661 msec/pass lxe: getiterator_tag_all (-- T2 ) 4.5667 msec/pass cET: getiterator_tag_all (-- T2 ) 33.5681 msec/pass ET : getiterator_tag_all (-- T2 ) 37.6200 msec/pass
This similarly shows in Element.findall():
lxe: findall (-- T2 ) 26.9907 msec/pass cET: findall (-- T2 ) 39.1728 msec/pass ET : findall (-- T2 ) 50.9692 msec/pass lxe: findall (-- T3 ) 3.6452 msec/pass cET: findall (-- T3 ) 12.0210 msec/pass ET : findall (-- T3 ) 11.2570 msec/pass lxe: findall_tag (-- T2 ) 4.6065 msec/pass cET: findall_tag (-- T2 ) 34.0267 msec/pass ET : findall_tag (-- T2 ) 36.7813 msec/pass lxe: findall_tag (-- T3 ) 0.5884 msec/pass cET: findall_tag (-- T3 ) 7.6307 msec/pass ET : findall_tag (-- T3 ) 9.2943 msec/pass
Note that all three libraries currently use the same Python implementation for findall(), except for their native tree iterator.
This part of lxml does not have an equivalent in ElementTree. However, lxml provides more than one way of accessing it and you should take care which part of the lxml API you use. The most straight forward way is to call the xpath() method on an Element or ElementTree:
lxe: xpath_method (-- T1) 9.9304 msec/pass lxe: xpath_method (-- T2) 29.3595 msec/pass lxe: xpath_method (-- T3) 0.2791 msec/pass lxe: xpath_method (-- T4) 0.9906 msec/pass
This is well suited for testing and when the XPath expressions are as diverse as the trees they are called on. However, if you have a single XPath expression that you want to apply to a larger number of different elements, the XPath class is the most efficient way to do it:
lxe: xpath_class (-- T1) 4.7921 msec/pass lxe: xpath_class (-- T2) 9.6187 msec/pass lxe: xpath_class (-- T3) 0.2215 msec/pass lxe: xpath_class (-- T4) 0.2697 msec/pass
Note that this still allows you to use variables in the expression, so you can parse it once and then adapt it through variables at call time. In other cases, where you have a fixed Element or ElementTree and want to run different expressions on it, you should consider the XPathEvaluator:
lxe: xpath_element (-- T1) 5.3826 msec/pass lxe: xpath_element (-- T2) 11.3929 msec/pass lxe: xpath_element (-- T3) 0.2514 msec/pass lxe: xpath_element (-- T4) 0.3038 msec/pass
While it looks slightly slower, creating an XPath object for each of the expressions generates a much higher overhead here:
lxe: xpath_class_repeat (-- T1) 6.8099 msec/pass lxe: xpath_class_repeat (-- T2) 26.7462 msec/pass lxe: xpath_class_repeat (-- T3) 0.3126 msec/pass lxe: xpath_class_repeat (-- T4) 1.1111 msec/pass
Objectify is a data-binding API for XML based on lxml.etree, that was added in version 1.1. It uses standard Python attribute access to traverse the XML tree. It also features ObjectPath, a fast path language based on the same meme.
Just like lxml.etree, lxml.objectify creates Python representations of elements on the fly. To save memory, the normal Python garbage collection mechanisms will discard them when their last reference is gone. In cases where deeply nested elements are frequently accessed through the objectify API, the create-discard cycles can become a bottleneck, as elements have to be instantiated over and over again.
ObjectPath can be used to speed up the access to elements that are deep in the tree. It avoids step-by-step Python element instantiations along the path, which can substantially improve the access time:
lxe: attribute (--T T1) 14.8621 msec/pass lxe: attribute (--T T2) 61.8820 msec/pass lxe: attribute (--T T4) 14.9317 msec/pass lxe: objectpath (--T T1) 13.7311 msec/pass lxe: objectpath (--T T2) 58.5930 msec/pass lxe: objectpath (--T T4) 8.0961 msec/pass lxe: attributes_deep (--T T1) 81.4488 msec/pass lxe: attributes_deep (--T T2) 77.0266 msec/pass lxe: attributes_deep (--T T4) 27.1226 msec/pass lxe: objectpath_deep (--T T1) 63.1915 msec/pass lxe: objectpath_deep (--T T2) 65.2469 msec/pass lxe: objectpath_deep (--T T4) 11.0138 msec/pass
Note, however, that parsing ObjectPath expressions is not for free either, so this is most effective for frequently accessing the same element.
A way to improve the normal attribute access time is static instantiation of the Python objects, thus trading memory for speed. Just create a cache dictionary and run:
cache[root] = list(root.getiterator())
after parsing and:
del cache[root]
when you are done with the tree. This will keep the Python element representations of all elements alive and thus avoid the overhead of repeated Python object creation. You can also consider using filters or generator expressions to be more selective. By choosing the right trees (or even subtrees and elements) to cache, you can trade memory usage against access speed:
lxe: attribute_cached (--T T1) 10.8343 msec/pass lxe: attribute_cached (--T T2) 55.5890 msec/pass lxe: attribute_cached (--T T4) 10.9514 msec/pass lxe: attributes_deep_cached (--T T1) 63.7080 msec/pass lxe: attributes_deep_cached (--T T2) 65.6838 msec/pass lxe: attributes_deep_cached (--T T4) 15.4514 msec/pass
Things to note: you cannot currently use weakref.WeakKeyDictionary objects for this as lxml's element objects do not support weak references (which are costly in terms of memory). Also note that new element objects that you add to these trees will not turn up in the cache automatically and will therefore still be garbage collected when all their Python references are gone, so this is most effective for largely immutable trees. You should consider using a set instead of a list in this case and add new elements by hand.
Here are some more things to try if optimisation is required:
Note that none of these measures is guaranteed to speed up your application. As usual, you should prefer readable code over premature optimisations and profile your expected use cases before bothering to apply optimisations at random.