Sun Microsystems Chosen by Internet Archive
Sun Microsystems announced that the Internet Archive, one of the fastest growing digital libraries in the world, has migrated its digital archival service onto Sun's open hardware and software platforms and established a new primary datacenter that will be housed at Sun's Santa Clara, California, campus. The Archive chose Sun to help move away from its customized storage architecture into a flexible and open Sun Modular Datacenter (Sun MD) or "Black Box" comprised of 60 Sun Fire X4500 (Thumper) Open Storage Systems totaling 2880TB of storage.
The Internet Archive, a San Francisco, California based organization, maintains what is probably the world’s
largest free digital archive. Since 1996, IA has begun collecting “ snapshots” of the Web every two months. Today, it also records content from TV channels worldwide, as well as movies, music, and books. With every Web snapshot alone, the archive collects about 100 TB of data - approximately 4 billion Web pages. At the end of 2008, the Internet Archive housed over three petabtyes of information , which is roughly equivalent to about 150 times the information contained in the Library of Congress, and receives up to 500 user queries per second . Going forward, the Archive is expected to grow at approximately 100 terabytes a month.
"We've worked closely with the Internet Archive to ensure the right technology platform is in place to handle and manage growing amounts of the world's most valuable data and that it lives on for future generations," said Dave Douglas, Chief Sustainability Officer and senior vice-president of Cloud Computing, Sun Microsystems, Inc. "The combination of open storage technology innovation in a Sun Modular Datacenter is a perfect match for this organization's mission and gives them the storage performance they need in a smaller power envelope - all at a cost-effective price point."
Sun's Open Storage Innovation as Backbone of Archive
The Internet Archive looked to Sun to help with two important technological challenges: storing massive amounts of data and ensuring this data will be preserved in the future. The non-profit organization needed a way to parse, index and physically encode exponentially greater amounts of raw data; while, at the same time protect stored resources from damage or destruction. Data degradation and maintaining accessibility of the data in unknown future formatting were among the many challenges facing this project.
Sun worked closely with the Internet Archive to design an open and cost-effective solution that met their needs. Using the Sun MD platform as the basis of their datacenter, Internet Archive made the switch from their customized hardware and storage to industry standard Sun Fire x4500 systems running Solaris 10 and Solaris ZFS. Sun also agreed to host and manage the datacenter on Sun's Santa Clara campus and will provide the power, cooling and networking capabilities.
In the past, if a disk failed, Internet Archive could lose 1 TB of data. “By using the Sun Fire X4500 and Solaris 10 with ZFS, we can have two disk failures and not lose any data,” says Brewster Kahle, founder and digital librarian, Internet Archive. Solaris ZFS performs constant checks of the data to help protect against data loss that is often undetected in normal use.
"No company can match the storage innovation that is coming out of Sun right now," said Kahle. "Putting Sun's highly dense storage technology into a modular datacenter gives us the performance and efficiency we need at a low price point."
;) |
;) |
;) |
| The Sun Modular Datacenter | Power Distribution |
Cooling System |
The Internet Archive, a San Francisco, California based organization, maintains what is probably the world’s
largest free digital archive. Since 1996, IA has begun collecting “ snapshots” of the Web every two months. Today, it also records content from TV channels worldwide, as well as movies, music, and books. With every Web snapshot alone, the archive collects about 100 TB of data - approximately 4 billion Web pages. At the end of 2008, the Internet Archive housed over three petabtyes of information , which is roughly equivalent to about 150 times the information contained in the Library of Congress, and receives up to 500 user queries per second . Going forward, the Archive is expected to grow at approximately 100 terabytes a month. "We've worked closely with the Internet Archive to ensure the right technology platform is in place to handle and manage growing amounts of the world's most valuable data and that it lives on for future generations," said Dave Douglas, Chief Sustainability Officer and senior vice-president of Cloud Computing, Sun Microsystems, Inc. "The combination of open storage technology innovation in a Sun Modular Datacenter is a perfect match for this organization's mission and gives them the storage performance they need in a smaller power envelope - all at a cost-effective price point."
Sun's Open Storage Innovation as Backbone of Archive
The Internet Archive looked to Sun to help with two important technological challenges: storing massive amounts of data and ensuring this data will be preserved in the future. The non-profit organization needed a way to parse, index and physically encode exponentially greater amounts of raw data; while, at the same time protect stored resources from damage or destruction. Data degradation and maintaining accessibility of the data in unknown future formatting were among the many challenges facing this project.
;) |
;) |
;) |
| Server Rack |
Servers Under Plexiglass |
Slide Out Server Racks |
Sun worked closely with the Internet Archive to design an open and cost-effective solution that met their needs. Using the Sun MD platform as the basis of their datacenter, Internet Archive made the switch from their customized hardware and storage to industry standard Sun Fire x4500 systems running Solaris 10 and Solaris ZFS. Sun also agreed to host and manage the datacenter on Sun's Santa Clara campus and will provide the power, cooling and networking capabilities.
In the past, if a disk failed, Internet Archive could lose 1 TB of data. “By using the Sun Fire X4500 and Solaris 10 with ZFS, we can have two disk failures and not lose any data,” says Brewster Kahle, founder and digital librarian, Internet Archive. Solaris ZFS performs constant checks of the data to help protect against data loss that is often undetected in normal use.
"No company can match the storage innovation that is coming out of Sun right now," said Kahle. "Putting Sun's highly dense storage technology into a modular datacenter gives us the performance and efficiency we need at a low price point."
