From Fedora Project Wiki
Line 1: Line 1:
 
* '''[https://symas.com/products/lightning-memory-mapped-database/ LMDB]'''
 
* '''[https://symas.com/products/lightning-memory-mapped-database/ LMDB]'''
 
** '''Written in:''' C
 
** '''Written in:''' C
** '''Wrappers for:''' C++, Java, Python, Perl, Ruby, Go, Erlang, PHP, Lua
+
** '''Wrappers for:''' C++, Java, Python, Perl, Ruby, Go, Erlang, PHP, Lua, .NET, Common Lisp, node.js, Haskell, Tcl
 
** '''Packages available for:''' Fedora, EPEL6+
 
** '''Packages available for:''' Fedora, EPEL6+
 
** '''Latest release:''' 0.9.18 / 2016-02-05
 
** '''Latest release:''' 0.9.18 / 2016-02-05
Line 9: Line 9:
 
** '''Supported architectures:''' i386/x86-64/PowerPC/ARM
 
** '''Supported architectures:''' i386/x86-64/PowerPC/ARM
 
** '''Query keys larger than X:''' Using cursors: [http://lmdb.tech/doc/group__mdb.html#gga1206b2af8b95e7f6b0ef6b28708c9127af9feb0557c2954dbf7732eee5e1b59e7 MDB_SET_RANGE] - Position at first key greater than or equal to specified key.
 
** '''Query keys larger than X:''' Using cursors: [http://lmdb.tech/doc/group__mdb.html#gga1206b2af8b95e7f6b0ef6b28708c9127af9feb0557c2954dbf7732eee5e1b59e7 MDB_SET_RANGE] - Position at first key greater than or equal to specified key.
** '''Notes:''' Developed as a replacement for Berkeley DB in the OpenLDAP project. Optimized for read performance. BTree based.
+
** '''Notes:''' Developed as a replacement for Berkeley DB in the OpenLDAP project. Optimized for read performance. B+Tree based.
  
 
* '''[http://fallabs.com/kyotocabinet/ Kyoto Cabinet]''' (replacing Tokyo Cabinet)
 
* '''[http://fallabs.com/kyotocabinet/ Kyoto Cabinet]''' (replacing Tokyo Cabinet)
Line 33: Line 33:
 
** '''Supported architectures:''' i386/x86-64/ARM
 
** '''Supported architectures:''' i386/x86-64/ARM
 
** '''Query keys larger than X:'''
 
** '''Query keys larger than X:'''
 +
** '''Notes:'''  Records are organized in hash table.
  
 
* '''[http://www.gnu.org.ua/software/gdbm/ GDBM]'''
 
* '''[http://www.gnu.org.ua/software/gdbm/ GDBM]'''
Line 44: Line 45:
 
** '''Supported architectures:''' i386/x86-64/ARM
 
** '''Supported architectures:''' i386/x86-64/ARM
 
** '''Query keys larger than X:'''
 
** '''Query keys larger than X:'''
 +
** '''Notes:'''  Records are organized in hash table.
  
 
* '''[http://leveldb.org/ LevelDB]'''
 
* '''[http://leveldb.org/ LevelDB]'''
Line 55: Line 57:
 
** '''Supported architectures:''' i386/x86-64/PowerPC/ARM
 
** '''Supported architectures:''' i386/x86-64/PowerPC/ARM
 
** '''Query keys larger than X:''' Native library's Iterator can Seek (Position at the first key in the source that at or past target) + reverse, node.js wrapper is able to use basic filters (greater than, lower than)
 
** '''Query keys larger than X:''' Native library's Iterator can Seek (Position at the first key in the source that at or past target) + reverse, node.js wrapper is able to use basic filters (greater than, lower than)
** '''Notes:''' Does not support transactions
+
** '''Notes:''' Does not support transactions. Records are organized in an LSM tree.
  
 
* '''[http://rocksdb.org/ RocksDB]''' (Built on LevelDB)
 
* '''[http://rocksdb.org/ RocksDB]''' (Built on LevelDB)
Line 67: Line 69:
 
** '''Supported architectures:''' x86-64/PowerPC/ARM
 
** '''Supported architectures:''' x86-64/PowerPC/ARM
 
** '''Query keys larger than X:''' Iterator can Seek (Position at the first key in the source that at or past target) + reverse
 
** '''Query keys larger than X:''' Iterator can Seek (Position at the first key in the source that at or past target) + reverse
** '''Notes:'''
+
** '''Notes:''' Records are organized in an LSM tree.
  
 
* '''[http://unqlite.org/ UnQlite]'''
 
* '''[http://unqlite.org/ UnQlite]'''
Line 79: Line 81:
 
** '''Supported architectures:'''
 
** '''Supported architectures:'''
 
** '''Query keys larger than X:''' UNQLITE_CURSOR_MATCH_LE and UNQLITE_CURSOR_MATCH_GE when using [https://unqlite.org/c_api/unqlite_kv_cursor.html cursors]. This options have sense only if the underlying key/value storage subsystem support range search (i.e: B+Tree, R+Tree, etc.).
 
** '''Query keys larger than X:''' UNQLITE_CURSOR_MATCH_LE and UNQLITE_CURSOR_MATCH_GE when using [https://unqlite.org/c_api/unqlite_kv_cursor.html cursors]. This options have sense only if the underlying key/value storage subsystem support range search (i.e: B+Tree, R+Tree, etc.).
 +
** '''Notes:''' UnQLite works with run-time interchangeable storage engines (i.e. Hash, B+Tree, R+Tree, LSM, etc.).
  
 
== Older (probably unmantained) projects ==
 
== Older (probably unmantained) projects ==
Line 93: Line 96:
 
** '''Supported architectures:''' Linux 2.4 and later (x86-32/x86-64/PowerPC/Alpha/SPARC)
 
** '''Supported architectures:''' Linux 2.4 and later (x86-32/x86-64/PowerPC/Alpha/SPARC)
 
** '''Query keys larger than X:''' It is possible to access each record with the cursor in ascending or descending order. According to this mechanism, forward matching search for strings and range search for integers are realized.
 
** '''Query keys larger than X:''' It is possible to access each record with the cursor in ascending or descending order. According to this mechanism, forward matching search for strings and range search for integers are realized.
 +
** '''Notes:''' Records are organized in hash table, B+ tree, or fixed-length array.
  
 
* '''[http://fallabs.com/qdbm/ QDBM]'''
 
* '''[http://fallabs.com/qdbm/ QDBM]'''
Line 104: Line 108:
 
** '''Supported architectures:''' Linux (2.2, 2.4, 2.6) (IA32, IA64, AMD64, PA-RISC, Alpha, PowerPC, M68000, ARM)
 
** '''Supported architectures:''' Linux (2.2, 2.4, 2.6) (IA32, IA64, AMD64, PA-RISC, Alpha, PowerPC, M68000, ARM)
 
** '''Query keys larger than X:''' It is possible to access each record with the cursor in ascending or descending order. According to this mechanism, forward matching search for strings and range search for integers are realized.
 
** '''Query keys larger than X:''' It is possible to access each record with the cursor in ascending or descending order. According to this mechanism, forward matching search for strings and range search for integers are realized.
 +
** '''Notes:''' Records are organized in hash table or B+ tree.

Revision as of 09:30, 28 July 2016

  • LMDB
    • Written in: C
    • Wrappers for: C++, Java, Python, Perl, Ruby, Go, Erlang, PHP, Lua, .NET, Common Lisp, node.js, Haskell, Tcl
    • Packages available for: Fedora, EPEL6+
    • Latest release: 0.9.18 / 2016-02-05
    • License: OpenLDAP Public License
    • Search key length: Maximum length of 1/3 of a page (~1365B)
    • Read-only access to DB: The library is fully thread-aware and supports concurrent read/write access from multiple processes and threads. (docs)
    • Supported architectures: i386/x86-64/PowerPC/ARM
    • Query keys larger than X: Using cursors: MDB_SET_RANGE - Position at first key greater than or equal to specified key.
    • Notes: Developed as a replacement for Berkeley DB in the OpenLDAP project. Optimized for read performance. B+Tree based.
  • Kyoto Cabinet (replacing Tokyo Cabinet)
    • Written in: C++
    • Wrappers for: C, Java, Python, Ruby, Perl, and Lua
    • Packages available for: Fedora, EPEL5+
    • Latest release: 1.2.76 / 2012-05-24
    • License: GNU GPL
    • Search key length: No limit from database itself (uses c++ std::string or combination of c-style string and its length).
    • Read-only access to DB: See http://fallabs.com/kyotocabinet/spex.html, part Sharing One database by Multiple Processes. TL;DR: One DB cannot be accessed by multiple processes, DB file is protected by reader-writer lock (which means it will probably require write access even for read-only operation).
    • Supported architectures: Linux 2.6 and later (i386/x86-64/PowerPC/Alpha/SPARC)
    • Query keys larger than X: DB supports matching by regex, which should be possible to use for this kind of query. Other possibilities are match by prefix and match by similarity.
    • Notes: Records are organized in hash table or B+ tree.
  • TDB
    • Written in: C
    • Wrappers for: Python
    • Packages available for: Fedora, RHEL6+
    • Latest release: 1.3.10 2016-07-28
    • License: GNU LGPL
    • Search key length: No limit apart from the data type used for holding the key and data lengths (int).
    • Read-only access to DB: Multiple processes can read a single database. Exclusion control is managed by a reader/writer lock. Internal locks in place to provide the ability for mutliple writer access (on different parts of the DB).
    • Supported architectures: i386/x86-64/ARM
    • Query keys larger than X:
    • Notes: Records are organized in hash table.
  • GDBM
    • Written in: C
    • Wrappers for:
    • Packages available for: Fedora, RHEL6+
    • Latest release: 1.12 / 2016-05-16
    • License: GNU GPLv3
    • Search key length: No limit apart from the data type used for holding the key and data lengths (int).
    • Read-only access to DB: Multiple processes can read a single database. Exclusion control is managed by a reader/writer lock. Can be disabled by the user to provide his own way of exclusion control.
    • Supported architectures: i386/x86-64/ARM
    • Query keys larger than X:
    • Notes: Records are organized in hash table.
  • LevelDB
    • Written in: C++
    • Wrappers for: Java, Go, Erlang, node.js
    • Packages available for: Fedora, EPEL6+
    • Latest release: 1.18 / 2014-09-16
    • License: 3-Clause BSD
    • Search key length: No limit apart from the data type used for holding the key length (size_t)
    • Read-only access to DB: A database may only be opened by one process at a time. The leveldb implementation acquires a lock from the operating system to prevent misuse. Within a single process, the same leveldb::DB object may be safely shared by multiple concurrent threads. (docs)
    • Supported architectures: i386/x86-64/PowerPC/ARM
    • Query keys larger than X: Native library's Iterator can Seek (Position at the first key in the source that at or past target) + reverse, node.js wrapper is able to use basic filters (greater than, lower than)
    • Notes: Does not support transactions. Records are organized in an LSM tree.
  • RocksDB (Built on LevelDB)
    • Written in: C++
    • Wrappers for: Java (In development)
    • Packages available for: Not yet available in Fedora
    • Latest release: 4.9.0 / 2016-06-09
    • License: 3-Clause BSD
    • Search key length: No limit apart from the data type used for holding the key length (size_t)
    • Read-only access to DB: A database may only be opened by one process at a time. The rocksdb implementation acquires a lock from the operating system to prevent misuse. Within a single process, the same rocksdb::DB object may be safely shared by multiple concurrent threads. (docs)
    • Supported architectures: x86-64/PowerPC/ARM
    • Query keys larger than X: Iterator can Seek (Position at the first key in the source that at or past target) + reverse
    • Notes: Records are organized in an LSM tree.
  • UnQlite
    • Written in: C
    • Wrappers for:
    • Packages available for: Not yet available in Fedora
    • Latest release: 1.1.6 / 2015-08-26
    • License: 2-Clause BSD
    • Search key length: No limit apart from the data type used for holding the key length (int). Data length limited by the data type unqlite_int64 (64b int).
    • Read-only access to DB: Supports concurrent readers. In order to be thread-safe, UnQLite must be compiled with the UNQLITE_ENABLE_THREADS directive defined.
    • Supported architectures:
    • Query keys larger than X: UNQLITE_CURSOR_MATCH_LE and UNQLITE_CURSOR_MATCH_GE when using cursors. This options have sense only if the underlying key/value storage subsystem support range search (i.e: B+Tree, R+Tree, etc.).
    • Notes: UnQLite works with run-time interchangeable storage engines (i.e. Hash, B+Tree, R+Tree, LSM, etc.).

Older (probably unmantained) projects

  • TokyoCabinet (replacing QDBM)
    • Written in: C
    • Wrappers for: Perl, Ruby, Java, and Lua
    • Packages available for: Fedora, EPEL5+, RHEL6+
    • Latest release: 1.4.48 / 2012-08-18
    • License: GNU LGPL
    • Search key length: No limit apart from the data type used for holding the key and data lengths (int).
    • Read-only access to DB: Multiple processes can read a single database. Exclusion control is managed by a reader/writer lock.
    • Supported architectures: Linux 2.4 and later (x86-32/x86-64/PowerPC/Alpha/SPARC)
    • Query keys larger than X: It is possible to access each record with the cursor in ascending or descending order. According to this mechanism, forward matching search for strings and range search for integers are realized.
    • Notes: Records are organized in hash table, B+ tree, or fixed-length array.
  • QDBM
    • Written in: C
    • Wrappers for: C++, Java, Perl, and Ruby
    • Packages available for: Fedora
    • Latest release: 1.8.78 / 2007-03-07
    • License: GNU LGPL
    • Search key length: No limit apart from the data type used for holding the key and data lengths (int).
    • Read-only access to DB: Multiple processes can read a single database. Exclusion control is managed by a reader/writer lock.
    • Supported architectures: Linux (2.2, 2.4, 2.6) (IA32, IA64, AMD64, PA-RISC, Alpha, PowerPC, M68000, ARM)
    • Query keys larger than X: It is possible to access each record with the cursor in ascending or descending order. According to this mechanism, forward matching search for strings and range search for integers are realized.
    • Notes: Records are organized in hash table or B+ tree.