Large Datasets: Kernelogic
Filecoin currently offers over 18 EiB of decentralized storage available thanks to a strong community of Storage Providers, developers, and ecosystem partners. This island economy is being leveraged by clients like Kernelogic who are interested in preserving large datasets of useful information.
Kernelogic started as an individual Storage Provider effort that evolved into the role of a large dataset client via the Slingshot competition. The project focuses on scripts that prepare open datasets into fixed-sized sectors that Filecoin can store. It then transfers these datasets to SPs who are interested in storing them.
Kernelogic’s Data
Over the course of its operations, Kernelogic has onboarded over 10 PiB of useful and verified datasets by making storage deals with over 50 different SPs from across the globe. It follows Slingshot specifications and stores royalty-free open datasets across different fields. Some of these include:
- NEXRAD: Real-time and archival data from the Next Generation Weather Radar (NEXRAD) network.
- NASA NEX: A collection of Earth science datasets maintained by NASA, including climate change projections and satellite images of the Earth’s surface.
- Brazil Data Cubes: Earth observation (EO) data cubes produced from analysis-ready data (ARD) of CBERS-4, Sentinel-2 A/B and Landsat-8 satellite images for Brazil.
- Terra Fusion Data Sampler: The Terra Basic Fusion dataset is a fused dataset of the original Level 1 radiances from the five Terra instruments.
- Fly Brain Anatomy: FlyLight Gen1 and Split-GAL4 imagery.
- 3000 Rice Genomes: An international effort to sequence the genomes of 3,024 rice varieties from 89 countries.
Kernelogic’s Use of Filecoin
Having these useful scientific research data replicated in many different locations and entities increases the chance of data preservation in the event of a single point of failure or natural disaster. As web3’s largest decentralized storage system, Filecoin provides a viable alternative to AWS where this kind of important data would be dependent on one company and usually hosted in a single geographic location.
Kernelogic has plans to continue onboarding data to the Filecoin network through the Slingshot program and cites constant improvements in Lotus, Indexer, and Boost software regarding retrieval. The project has also open sourced a toolset called Singularity to ease the data onboarding for large datasets.
There are also incentives in the Filecoin ecosystem that are designed to accommodate the use of the network’s capacity for meaningful data such as Kernelogic’s. Slingshot and the broader Filecoin+ program are great ways to take advantage of decentralized storage for datasets. If you have over 500 TiB, apply for Filecoin+ DataCap today and get better deals with Storage Providers.
