Return unique elements within each chunk by filtering repeated values.
Usage: cu(A)
Filter chunk-wise unique elements where:
- A is an array with one string attribute
cu(A) returns a sparse array with schema identical to A whose entries contain
unique elements within each chunk by marking repeated entries empty. The array
A may be multi-dimensional, dense or sparse, and chunked arbitrarily. The cu
filter is applied independently within each chunk. It is intended to help speed
up the SciDB uniq and sort operators by cutting down on the amount of data to
be sorted. Note that the order of returned elements may differ from the input
array although the schema will be the same.
iquery -aq "load_library('cu')"
iquery -aq "cu(build(<s:string>[i=1:4,4,0],'{1}[(x),(y),(x),(a)]',true))"
{i} s
{1} 'x'
{2} 'y'
{3} 'a'
{4} 'x'
iquery -aq "cu(build(<s:string>[i=1:4,4,0],\'{1}[(x),(y),(x),(a)]\',true))"
{i} s
{1} 'a'
{2} 'x'
{4} 'y'
The cu operator can help speed up the SciDB uniq operator workflow quite a bit.
Consider the following timing results run on a 64-instance SciDB cluster comparing
the usual uniq(sort(...)) approach with and without cu:
time iquery -aq "op_count(uniq(sort(build(<s:string>[i=1:100000000,1000000,0],i % 5000))))"
{i} count
{0} 5000
real 2m34.963s
user 0m0.017s
sys 0m0.005s
time iquery -aq "op_count(uniq(sort(cu(build(<s:string>[i=1:100000000,1000000,0],i % 5000)))))"
{i} count
{0} 5000
real 0m10.777s
user 0m0.013s
sys 0m0.008s
You'll need SciDB installed, along with the SciDB development header packages. The names vary depending on your operating system type, but they are the package that have "-dev" in the name. You don't need the SciDB source code to compile and install this.
Run make and copy the *.so plugin to the lib/scidb/plugins
directory on each of your SciDB cluster nodes. Here is an example:
cd chunk_unique
make
cp *.so /opt/scidb/14.8/lib/scidb/plugins
iquery -aq "load_library('cu')"
Remember to copy the plugin to all your SciDB cluster nodes.