Who is using Apache Celeborn?

Question

Who is using Apache Celeborn?

SteNicholas opened this issue a year ago · 30 comments

Search before asking

I searched in the issues and found nothing similar.

Motivation

Who's using Apache Celeborn?

The Apache Celeborn community grows with the combined efforts of every user and developer and appreciates every user and developer who uses and cares about Celeborn. We are committed to listening to each one of you to make the community more prosperous.

Therefore, the purpose of this topic is to encourage you, as a user or developer, to leave your real-world scenarios so that we can subsequently explore new and better solutions, schedule the releases, and solve real-world problems based on these scenarios.

We look forward to your comments in the comments space, including but not limited to the following areas.

Organization: e.g. Alibaba Corp.
Region: e.g. Hangzhou, China
Contact: e.g. zhouky@apache.org
Usage Scenario: e.g., we use Apache Celeborn to improve the efficiency and elasticity of different map-reduce engines and provides an elastic, high-efficient management service for intermediate data including shuffle data, spilled data, result data, etc.

Regards,

Apache Celeborn Community

Solution

No response

Anything else?

No response

Are you willing to submit a PR?

I'm willing to submit a PR!

Answer 1 · 2023-12-11T03:57:44.000Z

Organization: New Baatur
Region: Shen zhen, China
Contact information: 1623735386@qq.com
Usage scenario: Use Apache Celeborn to improve the efficiency and resilience of the Spark/MapReduce engine, providing elastic and efficient management services for intermediate data such as Shuffle data, overflow data, and result data.
Location: Currently, 10+locations have been deployed, with significant stability improvement and performance improvement of over 30%.

Answer 2 · 2023-12-11T05:15:27.000Z

Region: Xi'an, China Contact information: 1623735386@qq.com Usage scenario: Use Apache Celeborn to improve the efficiency and resilience of the Spark/MapReduce engine, providing elastic and efficient management services for intermediate data such as Shuffle data, overflow data, and result data. Location: Currently, 10+locations have been deployed, with significant stability improvement and performance improvement of over 30%.

@gaochao0509 Thanks for sharing. Mind also share the company/organization? :-)

Answer 3 · 2023-12-11T07:06:56.000Z

Organization: Trip.com Group
Region: Shanghai, China
Contact: sychen@trip.com
Usage Scenario: We use Apache Celeborn to improve the stability and efficiency of Spark shuffle.

Answer 4 · 2023-12-11T08:40:09.000Z

Organization: inmobi Group
Region: Beijing, China
Contact: alex.song@inmobi.com
Usage Scenario: We use Apache Celeborn to improve the stability and efficiency of Spark shuffle.

Answer 5 · 2023-12-11T09:13:54.000Z

Organization: zto.com Group
Region: Shanghai, China
Contact: qr7972@gmail.com
Usage Scenario: We use Apache Celeborn to improve the stability and efficiency of Spark shuffle

Answer 6 · 2023-12-11T09:37:46.000Z

Organization: shuyun Inc
Region: hangzhou, China
Contact: lisoda@yeah.net
Usage Scenario: By using apache celeborn, We realised the separation of storage and compute in HIVE-MR3/SPARK-KYUUBI, which effectively reduces the cost of the server room.

Answer 7 · 2023-12-11T12:29:46.000Z

Organization: Bilibili Inc
Region: Shanghai, China
Contact: jiangxiaofeng@bilibili.com
Usage Scenario: We use Apache Celeborn to improve the stability and efficiency of Spark and Flink(Batch) engine shuffle.

Answer 8 · 2023-12-12T02:21:59.000Z

Organization: AsiaInfo
Region: Beijing, China
Contact: jiaoqb@asiainfo.com
Usage Scenario: We use Apache Celeborn to improve the stability and efficiency of Spark shuffle.

Answer 9 · 2023-12-12T05:36:15.000Z

Organization: Zhihu Inc
Region: Beijing, China
Contact: lifulong@zhihu.com
Usage Scenario: We use Apache Celeborn to improve the stability and efficiency of Spark engine shuffle.

Answer 10 · 2023-12-12T07:21:19.000Z

Organization: Xiaohongshu Inc
Region: Shanghai, China
Contact: paen@xiaohongshu.com
Usage Scenario: We use Apache Celeborn to improve the stability and efficiency of Spark and Flink(Batch) engine shuffle.

Answer 11 · 2023-12-12T08:03:18.000Z

Organization: BIGO
Region: Guangzhou, China
Contact: liyihe@bigo.sg
Usage Scenario: We use Apache Celeborn to improve the stability and efficiency of Spark shuffle.

Answer 12 · 2023-12-12T09:12:13.000Z

Organization: SHOPEE
Region: Shenzhen, China
Contact: anger.zhu@gmail.com
Usage Scenario: We use Apache Celeborn to improve the stability and efficiency of Spark shuffle.

Answer 13 · 2023-12-21T02:07:15.000Z

Organization: DMALL
Region: Beijing, China
Contact: gang.huang@dmall.com, ming.li@dmall.com, ming.li2@dmall.com
Usage Scenario: We use Apache Celeborn to improve the stability and efficiency of Spark shuffle.

Answer 14 · 2024-01-16T10:31:30.000Z

Organization: Douban
Region: Beijing, China
Contact: caofengyu@douban.com
Usage Scenario: we use Apache Celeborn to support better Spark dynamic resource allocation in Kubernetes.

Answer 15 · 2024-02-27T02:48:05.000Z

Organization: WanFang Data
Region: Beijing, China
Contact: xuxinhao@wanfangdata.com.cn
Usage Scenario: We use Apache Celeborn to improve the stability and efficiency of Spark shuffle on k8s

Answer 16 · 2024-03-01T03:39:34.000Z

Organization: Tongcheng Travel（同程旅行）.
Region: Jiangsu, China
Contact: www.ly.com
Usage Scenario: we use Apache Celeborn to improve the stability and efficiency of spark shuffle.

Answer 17 · 2024-03-08T15:21:28.000Z

Organization: e.g. Pinterest
Region: US
Contact: @CodingCat
Usage Scenario: we use Apache Celeborn to improve the efficiency and elasticity of Spark@K8S

Answer 18 · 2024-03-12T06:15:39.000Z

Organization: LinkedIn
Region: US
Contact: @akpatnam25
Usage Scenario: we use Apache Celeborn to improve the stability and efficiency of spark shuffle.

Answer 19 · 2024-03-20T09:40:15.000Z

Organization: BOSS Zhipin
Region: Beijing, China
Contact: @liugs0213
Usage Scenario: we use Apache Celeborn to improve the efficiency and elasticity of Spark@K8S

Answer 20 · 2024-08-06T03:52:33.000Z

Organization: Stripe
Region: US
Contact: @lohitvijayarenu
Usage Scenario: Apache Celeborn helps Spark workloads with efficient and elastic shuffle. Celeborn will also help in Storage and compute disaggregation for efficient use of resources.

Answer 21 · 2024-08-09T10:44:02.000Z

Organization: Xiaomi
Region: Beijing, China
Contact: wangshengjie3@xiaomi.com
Usage Scenario: We use Celeborn to improve the stability and efficiency of shuffle.

Answer 22 · 2024-08-12T07:57:48.000Z

Organization: YY
Region: GuangZhou, China
Contact: https://www.yy.com/web/jinhong/#/connect
Usage Scenario: We use Celeborn to improve the stability and efficiency of shuffle.

Answer 23 · 2024-08-15T09:57:58.000Z

Organization: WuXianShengHuo
Region: HangZhou, China
Contact: https://www.weidian.com
Usage Scenario: We use Celeborn to improve the stability and efficiency of shuffle.

Answer 24 · 2024-10-09T03:35:58.000Z

Organization: 竞技世界(jjworld)
Region: Beijing, China
Contact: https://www.jj.cn/
Usage Scenario: we use Apache Celeborn to improve the efficiency and elasticity of Spark@K8S

Answer 25 · 2024-10-10T11:59:17.000Z

Organization: 蔚来（www.nio.com）
Region: Shanghai, China
Contact: @LennonChin
Usage Scenario: we use Apache Celeborn to improve the efficiency and elasticity of Spark on K8S.