Open Scientific Data Compression

开放科学数据压缩社区

An open, neutral community for high-performance scientific data compression. We develop, maintain, and benchmark error-bounded lossy and lossless compressors for HPC and AI-for-Science, portable across CPUs, GPUs, and accelerators.

一个开放、中立的高性能科学数据压缩社区。我们研发、维护和评测面向高性能计算与科学智能(AI4S)的误差可控有损压缩与无损压缩软件,支持多种 CPU、GPU 与加速器平台。

Error-bounded lossy误差可控有损压缩 Lossless无损压缩 CPU · GPU · Accelerators加速器 BSD-3-Clause / MIT

Why OpenSDC

为什么需要 OpenSDC

Simulations, large instruments, and AI models now produce data far faster than storage, I/O, and networks can absorb. OpenSDC aims to make compression a dependable building block of scientific data management: high ratio, high throughput, strict error control, and portability across platforms.

科学模拟、大科学装置和 AI 模型产生数据的速度,已远超存储、I/O 与网络的承载能力。OpenSDC 希望让压缩成为科学数据管理中可靠的基础组件:高压缩比、高吞吐、严格误差控制、跨平台可移植。

O

Open开放

Permissive licenses (BSD-3-Clause / MIT). Anyone can use, modify, and contribute.

采用 BSD-3-Clause / MIT 等宽松许可证,任何人均可使用、修改和贡献。

N

Neutral中立

Not tied to any single team or vendor. Every compressor joins on equal footing.

不绑定任何特定团队或厂商,各家压缩软件平等加入。

P

Portable可移植

Designed to run efficiently on diverse hardware, including domestic platforms.

面向多样化硬件(含国产平台)高效运行。

R

Reproducible可复现

Every claim backed by public datasets, benchmarks, and scripts.

所有性能结论均有公开数据集、基准与脚本支撑。

Projects

项目

General-purpose and domain-specific compressors hosted by OpenSDC. New projects are welcome.

OpenSDC 托管的通用型与领域专用压缩软件,欢迎更多项目加入。

General-purpose compressors

通用压缩器

Domain-specific compressors

领域专用压缩器

Roadmap

路线图

What the community is building together in its early stage.

社区初期正在共同推进的工作。

Software onboarding压缩软件入驻

Bring members' open-source compressors into OpenSDC with shared maintenance, review, and maintainers from multiple institutions.

各单位开源压缩软件入驻联盟,日常化管理与评审,欢迎多单位共同担任 maintainer。

In progress进行中

Open benchmark datasets开源基准数据集

Collect contributed scientific datasets (climate, cosmology, light sources, solar physics, materials, ...) into a common benchmark suite.

汇集各单位贡献的科学数据集(气象气候、宇宙学、光源、太阳物理、材料等),形成统一基准数据集。

Coming soon即将发布

Benchmark system & leaderboard评测系统与榜单

A reproducible evaluation platform with periodically published results on ratio, throughput, and error quality.

搭建可复现的评测平台,定期发布压缩比、吞吐与误差质量等榜单。

Coming soon即将发布

Community meetings定期研讨

Regular member meetings and workshops, in collaboration with CCF and related open-source organizations.

与 CCF 及相关开源组织合作,定期举办成员见面会和技术研讨。

Planned规划中

Standardization标准化

Work with partner institutions toward group and national standards for scientific data compression.

联合相关单位推动科学数据压缩团体标准,进而推动国家标准。

Planned规划中

Get involved

参与方式

Researchers, students, and industry partners are all welcome.

欢迎高校、科研院所的老师同学以及企业伙伴共同参与。

Contribute software贡献软件

Host your compressor under OpenSDC, or open issues and pull requests in any project.

将您的压缩软件加入 OpenSDC,或在任意项目中提交 Issue 与 PR。

Contributing guide →贡献指南 →

Contribute datasets贡献数据集

Share open scientific datasets for the benchmark. Others optimize for your data, and your applications benefit.

为基准贡献开源科学数据集,其他团队据此优化压缩软件,最终反哺您的应用。

Contact us →联系我们 →

Industry partnership企业合作

Support platform adaptation and real-world deployment of open compression software.

支持压缩软件的国产化平台适配与落地应用。

Contact us →联系我们 →

Contact:联系方式: taodingwen@ict.ac.cn · jinyang.liu@sjtu.edu.cn