Diskstore and a new b-tree implementation

A few weeks ago I presented a new development direction for Redis, called diskstore.
The original announcement is in this Redis Group post.

Diskstore is working very well in our initial tests, and we are working to improve it.
Diskstore is basically a smart caching layer on top of Redis backed by an on disk plain key value store.
The current implementation of the disk store is file system based: every key is represented on disk by a single file. This setup is interesting in many ways as it is pretty hard to corrupt, and very transparent (you can see the individual files, rsync them into another server, and even joining two databases just merging all the files), but hardly the fastest.

So the idea is to provide two disk backends built-in into Redis for different needs. One is the current file system based implementation, and one is an ongoing effort, based on a b-tree. If you follow me on twitter you probably noticed that many of my recent tweets are about a b-tree implementation: it is the one that is going to join the current backend into diskstore.

So diskstore users can select via redis.conf if they want to use the diskstore-fs or diskstore-btree backend, at least initially. It is possible that in the long run one solution will show to be superior in almost every way to the other and we may deprecate it. But current indications and experiences with other NoSQL databases are suggesting that there is some space for a plain filesystem based backend, especially if you want to use Redis to store very very large values.

Unfortunately I was not able to find a BSD implementation of a btree that satisfied my requirements and that was self-contained enough, so I’m writing one from scratch. The good news is that the new implementation is written as a stand alone library and completely decoupled from Redis itself, so it will be possible to use it for other projects as well.

I’ll take you posted on twitter about progresses. I hope to have something working in a few weeks at max.