The Wendelin project started in the beginning of 2015 with Nexedi as consortium
leader in charge of managing the development of a big data solution "made in France".
Wendelin is based on open source software components and combines a number of
widely used libraries for data analysis, modification and visualization.
The project also includes the development of prototype applications in the
automotive and green energy sectors underlining its purpose of being immediately
applicable for the development of industrial solutions.
Wendelin was bumped to version 0.4 alpha recently with bugfixes
and faster installation. The Wendelin core itself has just been upgraded to version 0.5
bringing some interesting new features. It now supports two different
formats to store data: ZBlock0 and ZBlock1. The major difference
between these two formats is that ZBlock0 stores just one object in the database,
which makes it faster at the price of using a lot of storage space, while ZBlock1
stores many objects, but this causes it to lose in speed when compared to the
other format. As one of the main features of Wendelin are its out-of-core capabilities,
which allow it to easily extend computation capacity beyond the limits of available
hardware in a cluster, the time has come to run some performance tests and see
how it compares to a well known and widely used competitor: MariaDB.
We ran tests to compare the read performance of Wendelin, pure NumPy (both in-memory
and using memory maps) and MariaDB's SQL functions. Write speed was also
measured for Wendelin. These were the different tool sets used and how they will
be later referenced:
All those tool sets were tested in a environment with cold and warm cache. Cold
and warm cache have a analogy with cold and warm engine of a car. A cold cache
does not help performance, it is an empty cache. While the warm cache has some
values and can give a speedup.
A virtual machine provided by Vifib
was used to run the tests. It has the following hardware: 4 cores, 4GB of
RAM (swap was disabled), 25GB of SSD storage and runs
Debian 8.2 Jessie. An all-in-one Wendelin instance of ERP5
was used and ZEO
was chosen as storage backend for ZODB.
This instance was installed using
our "how to get started with Wendelin" tutorial. The test data
consisted of 1.430.394 stock objects.
The test's code for Wendelin and NumPy can be found in the repository at
It is splitted in server and client codes. The server code is responsible for
running the test for a specific tool set according to the parameters it receives.
The client code is in charge of providing these parameters to trigger each tool
set's test and also for cleaning the server's cache and restarting services when
it's running with cold cache. The schema used for the data in the Python part can
be found in the repository code and was as close as possible to MariaDB's schema.
In the wendelin.pandas test a simple numpy.ndarray was
used as index and Pandas was monkey-patched to avoid data copy in the
DataFrame constructor by changing the _consolidate
function at https://github.com/pydata/pandas/blob/master/pandas/core/internals.py#L4074 to
this more simple version:
MariaDB was tested with the query "SELECT SUM(quantity) FROM stock;" after a cache flush and reset. System cache were properly cleaned and the service restarted for cold cache runs. The table schema was as follows:
CREATE TABLE `stock` (
`uid` bigint(20) unsigned NOT NULL,
`order_id` bigint(20) unsigned NOT NULL,
`explanation_uid` bigint(20) unsigned DEFAULT NULL,
`node_uid` bigint(20) unsigned DEFAULT NULL,
`section_uid` bigint(20) unsigned DEFAULT NULL,
`payment_uid` bigint(20) unsigned DEFAULT NULL,
`function_uid` bigint(20) unsigned DEFAULT NULL,
`project_uid` bigint(20) unsigned DEFAULT NULL,
`funding_uid` bigint(20) unsigned DEFAULT NULL,
`payment_request_uid` bigint(20) unsigned DEFAULT NULL,
`mirror_section_uid` bigint(20) unsigned DEFAULT NULL,
`mirror_node_uid` bigint(20) unsigned DEFAULT NULL,
`resource_uid` bigint(20) unsigned DEFAULT NULL,
`quantity` double DEFAULT NULL,
`is_cancellation` tinyint(1) DEFAULT NULL,
`is_accountable` tinyint(1) DEFAULT NULL,
`date` datetime DEFAULT NULL,
`mirror_date` datetime DEFAULT NULL,
`total_price` double DEFAULT NULL,
`portal_type` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`simulation_state` varchar(255) COLLATE utf8_unicode_ci DEFAULT '',
`variation_text` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`sub_variation_text` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
After 5 samples of each test were taken, the average and standard deviation in
percentage were calculated. All data is measured in seconds. MariaDB's data is
provided by MariaDB itself and this is the reason behind lesser precision numbers
than our code's measurements. Lower numbers are always better.
First, with regards to writes, format ZBlock0 is much faster than ZBlock1. This
is because of the nature of ZBlock0, which uses just one object in the database,
while ZBlock1 uses many objects in the database. This was an expected result. But
writes will often be executed with a big frequency and small amounts of data each
time, so this is not a big problem. Also write speed can be improved by a better
Now about tools that are not dependent on the Wendelin block format usage: NumPy
with memmap shows itself 10 times faster than MariaDB with
cold cache, which is very interesting. NumPy in-memory crushes everything, as
you would expect, and takes only 0.013 seconds to process the whole data. This
means we can get significant speed improvements by modifying our most time-consuming
reports to take advantage of NumPy when data fits in memory, and/or its memory map
function when data doesn't fit entirely in the memory.
The Wendelin tests (either with NumPy or Pandas) with format ZBlock0 shows that
it can read very fast with cold cache - almost close to MariaDB - which is a good
and motivating result for such a new library. It also shows that Wendelin with
NumPy is 20 times slower than pure NumPy's memmaps. This means that by using
Wendelin and a distributed storage backend for ZODB, like NEO,
we can achieve performance close to NumPy's memmap. Hot cache result is 10
times faster for wendelin.numpy and about the same
speed as MariaDB for wendelin.pandas.
The read test results with Wendelin using format ZBlock1 shows that it is much
slower than ZBlock0 with cold cache, but this was already known and explained
earlier in the write tests and in the motivation section. With hot cache almost
all tool sets behaved the same, except for wendelin.pandas,
which still has a tiny overhead introduced by Pandas processing data in its block
During tests execution the data size was measured for each tool set. For
Wendelin-based tool sets, wendelin.pandas and wendelin.numpy,
the data size was 354 megabytes, measured with a ls -la
before and after data was written. For MariaDB the data size was
342 megabytes, as provided by the column data_length
of the query SHOW TABLE STATUS WHERE NAME == 'stock';.The data
size for pure NumPy tool sets, numpy.memory and numpy.memmap,
was exactly 250 megabytes.
But there is still room for improvements. Our team is currently very active and
performance gains are expected in the next versions. Optimizations are not only
coming to Wendelin though, but also to all tools involved in the storage process.
Stay tuned for more information about Wendelin Exanalytics platform and its evolution!
We are working hard to also upgrade the whole Wendelin platform to the next release, so stay tuned.