Database theory

Erasure codes are one of those seemingly magical mathematical creations that with the developments described in the paper XORing Elephants: Novel Erasure Codes for Big Data, are set to replace triple replication as the data storage protection mechanism of choice.

The result says Robin Harris (StorageMojo) in an excellent article, Facebook’s advanced erasure codes: "WebCos will be able to store massive amounts of data more efficiently than ever before. Bad news: so will anyone else."

Robin says with cheap disks triple replication made sense and was economical. With ever bigger BigData the overhead has become costly. But erasure codes have always suffered from unacceptably long time to repair times. This paper describes new Locally Repairable Codes (LRCs) that are efficiently repairable in disk I/O and bandwidth requirements:

These systems are now designed to survive the loss of up to four storage elements – disks, servers, nodes or even entire data centers – without losing any data. What is even more remarkable is that, as this paper demonstrates, these codes achieve this reliability with a capacity overhead of only 60%.

They examined a large Facebook analytics Hadoop cluster of 3000 nodes with about 45 PB of raw capacity. On average about 22 nodes a day fail, but some days failures could spike to more than 100.

LRC test results found several key results.

• Disk I/O and network traffic were reduced by half compared to RS codes.
• The LRC required 14% more storage than RS, information theoretically optimal for the obtained locality.
• Repairs times were much lower thanks to the local repair codes.
• Much greater reliability thanks to fast repairs.
• Reduced network traffic makes them suitable for geographic distribution.
• LRC test results found several key results.
• Disk I/O and network traffic were reduced by half compared to RS codes.

I wonder if we'll see a change in NoSQL database systems as well? 

Related Articles

• Erasure Coding vs. Replication: A Quantitative Comparison
• Ceph - a distributed object store.

Stack Exchange

This Q&A is part of a weekly series of posts highlighting common questions encountered by technophiles and answered by users at Stack Exchange, a free, community-powered network of 100+ Q&A sites.

Dokkat appears to think that databases are overused. "Instead of a database, I just serialize my data to JSON, saving and loading it to disk when necessary," he writes. "All the data management is made on the program itself, which is faster AND easier than using SQL queries." What is missing here? Why should a developer use a database when saving data to a disk might work just as well?

See the original question here.

Read 18 remaining paragraphs | Comments

Comments

Comments

Comments

Hi, I'm just starting with trying to build web applications and I'm not really familiar with how to store data to make sure something like this scales well.

Consider this pseudo-data, in JSON: { unique id, [array of child ids], [auxiliary data] }

Right now, every request I get requires me to get auxiliary info given an ID. About 3/4 of the time I need to update some of the auxiliary data. About half of the time, I need to query again for the auxiliary data of one of the "child" id's from the array of ids. This makes me want to just use a relational database and generate another query based on the child IDs if I need to traverse the natural graph structure of the data (do very "shallow" searches).

I'm wondering how well this would continue to work if I suddenly decided to do a lot more "depth-first" query patterns (that is, every query would likely be followed by a query to it's child, which has an unpredictable ID), and whether specialized graph databases (not SQL) would give me more scalability in this case. I don't actually know much about how they work but I imagine if there's any reason they exist it's for stuff like this.

Can anyone point me in the right direction? If a single request generates a chain of sequential SELECTs to traverse a graph am I doing it wrong?

submitted by ReallyGoodAdvice
[link] [3 comments]

Comments

New submitter rescrv writes "Key-value stores (like Cassandra, Redis and DynamoDB) have been replacing traditional databases in many demanding web applications (e.g. Twitter, Google, Facebook, LinkedIn, and others). But for the most part, the differences between existing NoSQL systems come down to the choice of well-studied implementation techniques; in particular, they all provide a similar API that achieves high performance and scalability by limiting applications to simple operations like GET and PUT.

HyperDex, a new key-value store developed at Cornell, stands out in the NoSQL spectrum with its unique design. HyperDex employs a unique multi-dimensional hash function to enable efficient search operations — that is, objects may be retrieved without using the key (PDF) under which they are stored. Other systems employ indexing techniques to enable search, or enumerate all objects in the system. In contrast, HyperDex's design enables applications to retrieve search results directly from servers in the system. The results are impressive. Preliminary benchmark results on the project website show that HyperDex provides significant performance improvements over Cassandra and MongoDB. With its unique design, and impressive performance, it seems fittng to ask: Is HyperDex the start of NoSQL 2.0?"

Read more of this story at Slashdot.

Views Explain is a very simple module that runs EXPLAIN on the query generated by views, and appends that information to the views preview table.