18 lines
785 B
Markdown
18 lines
785 B
Markdown
|
## checkpoint
|
||
|
|
|
||
|
|
分布式快照,应对失败后重新开始
|
||
|
|
|
||
|
|
1. 批计算是否需要checkpoint?否
|
||
|
|
2. 流计算
|
||
|
|
1. 任务7*24小时运行,重启能够无缝重启
|
||
|
|
2. exactly-once 恰好一次,at-most-once至多一次
|
||
|
|
3. source读取来的数据,必须让整个链路执行成功处理完成才会将state持久化到外部存储(checkpoint)
|
||
|
|
1. source数据源必须支持再次消费
|
||
|
|
2. 必须等到所有的算子处理完才checkpoint,否则at-most-once
|
||
|
|
4. 吞吐量不高,battier(广播分发)栅栏,按批次checkpoint,可能引起重复数据落盘
|
||
|
|
1. 幂等处理id
|
||
|
|
2. checkpoint和mysql事物结合在一起
|
||
|
|
3. source允许再消费,sink幂等或者事务
|
||
|
|
5. 失败后恢复checkpoint快照.
|
||
|
|
6. savepoint->手动checkpoint
|
||
|
|
7. 设置uid,方便代码升级s
|