Skip to content

Storage Tips from the Kings of Big Data

by on November 4, 2011

Big Data might be the next big thing in terms of helping companies better understand and meet customer needs, but it’s far from easy to manage. As organizations everywhere struggle with the challenges of storing huge files and creating long-term archives—all while making the data accessible—some of the companies dealing with the largest data stores (numbers like 450 billion database objects and 40 TB per week) recently shared their best practices in “Storage Tips From Heavy-Duty Users” on

Here are some of the tips offered by the Library of Congress, and Mazda, which might provide some thought-starters for how your own organization can better manage Big Data storage.

  1. To keep scale under control, consider having infrastructure in a primary location that handles most of the data and a second location for long term archiving.
  2. Contain costs by storing seldom accessed content offline and on tape.
  3. Use parallelization to split files into smaller, more manageable pieces.
  4. Manage file corruption diligently, as even a low failure rate can mean lots of errors to manage when data stores are so massive.
  5. Use virtualization to reduce hardware costs, stop server sprawl and streamline disaster recovery.
  6. Implement a tiered storage strategy, with inactive data quickly migrating to low-cost, reliable, secure, environmentally friendly tape.

What about Google, MasterCard, the federal government and other big data “kings” who are managing some of the most massive data stores in the universe? As these and other leaders continue to get their arms around Big Data, there might well be some lessons from their experience for those of us managing smaller—yet no less critical—Big Data stores.

How is your organization coping with Big Data storage? What have you learned along the way—on your own or from another organization—that had helped you in this challenge?

Leave a Comment

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: