[Cleanup] 框架历史遗留 API 清理计划（一期）
gouzil opened this issue · 9 comments
gouzil commented
背景
众所周知，Paddle 是一个历史悠久的框架，使得 Paddle 能够久经考验，应对各种场景稳定运行。但历史的沉淀同样带来一个严重的问题，就是框架内 API 语义不清晰，多种 API 能够做同样或者类似的事情。得益于我们的公开 API 审查机制和 fluid 清理，公开 API 中类似问题较少，但框架内部仍存在大量历史遗留的内部 API 的使用，这些 API 的存在导致框架内部需要一些兼容逻辑，使得框架难以维护和升级。
为了使得框架架构更加清晰，我们希望对框架内部的历史遗留 API 进行清理，以使得框架架构更加清晰，更易于维护。为了迎接 3.0 的新时代，希望大家能够一起清理这些「旧时代的残党」～
提交规范介绍
⭐️ 提交PR 模版 ⭐️：
- // ------- PR 标题 --------
[Cleanup][A-1] clean some `VarType` for test
⭐️ 认领方式 ⭐️：
请大家以 comment 的形式认领任务，如：
【报名】：1、3、12-13
状态介绍：
✅：已经完全迁移，所有单测都OK！
🟢：审核完毕待合入，合入之后完全迁移！
🔵：可认领！
🟡：当前阶段不需要人力继续跟进，下阶段推进（大部分是精度问题）
🚧：迁移中，单测还没有过，还没有审核完。
大致正常流程为:
🔵 -> 🚧 -> 🟢 -> ✅
异常流程为:
🔵 -> 🚧 -> 🟡
⭐️ 提交 PR 注意事项 ⭐️：
- 单个 PR 不要超过 15 个文件
- 请在 PR 描述里
@gouzil，并链接本 issue#61385，会由任务 Leader @gouzil 进行任务一审，一审通过后由 @SigureMo 进行二审
任务详情
残党 A：VarType
VarType 是定义在 framework.proto 的一个枚举字段，用于表示 VarDesc 的类型，其定义如下：
Paddle/paddle/fluid/framework/framework.proto
Lines 142 to 186 in b88fd78
VarType 是同时表示 dtype 和 Tensor type 等多种信息的，语义不清晰，我们现在 paddle.dtype（含 paddle.int32、paddle.float32 等）就是 VarType 相应字段的 re-export。
众所周知，Paddle 是先有静态图的，而后动态图直接复用了静态图的 VarType，而如今我们 PIR 下直接使用了语义更加清晰的 DataType，但框架现存代码中大量 VarType 的使用阻碍了我们将 VarType 替换为 DataType 的进程。
因此我们希望先对框架内的 VarType 使用进行清理，逐步隐藏 Python 端对 VarType 的使用，初步希望替换 dtype == VarType.FP32 的 case 为 dtype == paddle.float32
-if p.dtype == core.VarDesc.VarType.FP32:
+if p.dtype == paddle.float32:
-self.assertEqual(p.dtype, core.VarDesc.VarType.BOOL)
+self.assertEqual(p.dtype, paddle.bool)Warning
- 注意，非 dtype 的不要替换，比如
VarType.SELECTED_ROWS VarType.FP32等 dtype 和paddle.float32完全等价，可以放心替换～- 目前仅仅推荐替换上述两种 pattern，其它 pattern 暂时不要替换～（可分别使用
==.+VarType\.([A-Z]+\d+|BOOL)和assertEqual.+VarType\.([A-Z]+\d+|BOOL)正则来查找）
| 序号 | 文件位置 | 认领人 | PR |
|---|---|---|---|
| ✅A-1 | amp_o2_pass.py gradient_checker.py op_test.py test_batch_norm_op.py test_coalesce_tensor_op.py |
✅@ooooo-create | #61466 |
| ✅A-2 | test_cummax_op.py test_cummin_op.py test_cumsum_op.py test_custom_conj.py test_elementwise_add_op.py |
✅@ooooo-create | #61466 |
| ✅A-3 | test_gather_op.py test_imperative_auto_mixed_precision_for_eager.py test_initializer.py test_initializer_nn.py test_ir_fusion_group_pass.py |
✅@ooooo-create | #61466 |
| ✅A-4 | test_logcumsumexp_op.py test_pass_bf16.py test_to_tensor.py |
✅@ooooo-create | #61466 |
| ✅A-5 | test/collective/fleet/test_fleet_amp_meta_optimizer.py test/collective/fleet/test_fleet_gradient_merge_meta_optimizer.py test/legacy_test/test_data.py test/legacy_test/test_egr_python_api.py test/legacy_test/test_executor_feed_non_tensor.py |
✅@co63oc | #61549 |
| ✅A-6 | test/legacy_test/test_gaussian_random_op.py test/legacy_test/test_lod_tensor.py test/legacy_test/test_math_op_patch_var_base.py test/legacy_test/test_math_op_patch.py test/legacy_test/test_parameter.py |
✅@co63oc | #61550 |
| ✅A-7 | test/legacy_test/test_protobuf_descs.py test/legacy_test/test_rand_op.py test/legacy_test/test_tensor.py test/legacy_test/test_uniform_random_op.py test/legacy_test/test_var_base.py |
✅@enkilee | #61548 |
| ✅A-8 | test/legacy_test/test_variable.py test/sequence/test_sequence_pad_op.py test/xpu/test_gaussian_random_op_xpu.py |
✅@co63oc | #61553 |
| ✅A-9 | python/paddle/amp/grad_scaler.py python/paddle/base/dygraph/math_op_patch.py python/paddle/base/dataset.py python/paddle/base/framework.py python/paddle/distributed/auto_parallel/static/cost/base_cost.py |
✅@co63oc | #61562 |
| ✅A-10 | python/paddle/distributed/auto_parallel/static/engine.py python/paddle/distributed/fleet/dataset/dataset.py python/paddle/distributed/fleet/meta_optimizers/sharding/offload_helper.py python/paddle/distributed/fleet/utils/mix_precision_utils.py python/paddle/distributed/passes/auto_parallel_amp.py |
✅@co63oc | #61563 |
| ✅A-11 | python/paddle/distributed/passes/auto_parallel_fp16.py python/paddle/distributed/passes/auto_parallel_master_grad.py python/paddle/hapi/model.py python/paddle/incubate/distributed/models/moe/grad_clip.py python/paddle/incubate/optimizer/distributed_fused_lamb.py |
✅@co63oc | #61564 |
| ✅A-12 | python/paddle/jit/translated_layer.py python/paddle/optimizer/optimizer.py python/paddle/static/amp/bf16/amp_utils.py python/paddle/static/amp/amp_nn.py python/paddle/static/amp/debugging.py |
✅@co63oc | #61565 |
| ✅A-13 | python/paddle/static/amp/decorator.py python/paddle/static/amp/fp16_lists.py python/paddle/static/amp/fp16_utils.py python/paddle/static/nn/common.py python/paddle/static/quantization/quantization_pass.py |
✅@co63oc | #61566 |
| ✅A-14 | python/paddle/tensor/attribute.py python/paddle/tensor/creation.py python/paddle/tensor/to_string.py |
✅@co63oc | #61567 |
残党 B：paddle.base.dygraph.to_variable
to_variable 是早期动态图下用来创建 Tensor 的 API，现在有一个语义更加清晰的 paddle.to_tensor API，但 Paddle 框架中遗留的 to_variable 使用特别多，为了能够让 paddle.to_tensor 替换掉 to_variable，我们需要将框架中现有使用处全部替换掉：
-x = paddle.base.dygraph.to_variable(x)
+x = paddle.to_tensor(x)| 序号 | 文件位置 | 认领人 | PR |
|---|---|---|---|
| ✅B-1 | check_flags_mkldnn_ops_on_off.py check_flags_use_mkldnn.py op_test.py parallel_dygraph_control_flow_same.py parallel_dygraph_mnist.py parallel_dygraph_se_resnext.py parallel_dygraph_shared_unused_var.py parallel_dygraph_sparse_embedding.py parallel_dygraph_sync_batch_norm.py parallel_dygraph_transformer.py |
✅@enkilee | #61468 |
| ✅B-2 | seq2seq_dygraph_model.py simnet_dygraph_model.py test_activation_op.py test_adam_op.py test_adaptive_avg_pool1d.py test_adaptive_max_pool1d.py test_addmm_op.py test_affine_grid_function.py test_array_read_write_op.py test_basic_api_transformation.py |
✅@PommesPeter | #61904 |
| ✅B-3 | test_batch_norm_op.py test_bicubic_interp_op.py test_bicubic_interp_v2_op.py test_bincount_op.py test_bmm_op.py test_bmn.py test_cast.py test_cholesky_op.py test_chunk_op.py test_coalesce_tensor_op.py |
✅@PommesPeter | #61905 |
| ✅B-4 | test_complex_elementwise_layers.py test_complex_getitem.py test_complex_kron.py test_complex_matmul.py test_complex_reshape.py test_complex_sum_layer.py test_complex_trace_layer.py test_complex_transpose.py test_complex_variable.py test_conv2d_api.py |
✅@flying-forever | #61793 |
| ✅B-5 | test_conv2d_transpose_layer.py test_conv3d_transpose_layer.py test_correlation.py test_cross_op.py test_cycle_gan.py test_declarative.py test_detach.py test_detection.py test_dict.py test_directory_migration.py |
✅@NKNaN | #61530 |
| ✅B-6 | test_dot_op.py test_dropout_op.py test_dropout_op_xpu.py test_dygraph_mnist_fp16.py test_dygraph_multi_forward.py test_dygraph_weight_norm.py test_egr_python_api.py test_eig_op.py test_elementwise_add_op.py test_elementwise_add_op_xpu.py |
✅@megemini | #61476 |
| ✅B-7 | test_elementwise_add_op_xpu_kp.py test_elementwise_pow_op.py test_elementwise_sub_op.py test_erf_op.py test_exception.py test_fill_constant_op.py test_fleet_base.py test_flip.py test_for_enumerate.py test_functional_conv1d.py |
✅@Liyulingyue | #61511 |
| ✅B-8 | test_functional_conv1d_transpose.py test_functional_conv2d.py test_functional_conv2d_transpose.py test_functional_conv3d.py test_functional_conv3d_transpose.py test_gather_nd_op.py test_gelu_op.py test_grad_clip_minimize.py test_gradient_clip.py test_group_norm_op.py |
✅@zade23 | #61581 |
| ✅B-9 | test_histogram_op.py test_imperative_auto_mixed_precision_for_eager.py test_imperative_container_layerlist.py test_imperative_container_parameterlist.py test_imperative_container_sequential.py test_imperative_deepcf.py test_imperative_double_grad.py test_imperative_framework.py test_imperative_gan.py test_imperative_gnn.py |
✅@zade23 | #61585 |
| ✅B-10 | test_imperative_hook_for_layer.py test_imperative_layer_trainable.py test_imperative_lod_tensor_to_selected_rows.py test_imperative_mnist_sorted_gradient.py test_imperative_named_members.py test_imperative_ocr_attention_model.py test_imperative_optimizer.py test_imperative_optimizer_v2.py test_imperative_parallel_coalesce_split.py test_imperative_partitial_backward.py |
✅@co63oc | #61531 |
| ✅B-11 | test_imperative_ptb_rnn.py test_imperative_ptb_rnn_sorted_gradient.py test_imperative_recurrent_usage.py test_imperative_reinforcement.py test_imperative_resnet.py test_imperative_resnet_sorted_gradient.py test_imperative_save_load_v2.py test_imperative_selected_rows.py test_imperative_selected_rows_to_lod_tensor.py test_imperative_star_gan_with_gradient_penalty.py |
✅@co63oc | #61532 |
| ✅B-12 | test_imperative_trace_non_persistable_inputs.py test_imperative_transformer_sorted_gradient.py test_imperative_triple_grad.py test_imperative_using_non_zero_gpu.py test_index_select_op.py test_index_select_op_xpu.py test_instance_norm_op.py test_instance_norm_op_v2.py test_inverse_op.py test_inverse_op_xpu.py |
✅@co63oc | #61533 |
| ✅B-13 | test_jit_save_load.py test_kron_op.py test_label_smooth_functional.py test_lac.py test_learning_rate_scheduler.py test_linear_interp_op.py test_linear_interp_v2_op.py test_matmul_op.py test_matmul_op_xpu.py test_matrix_power_op.py |
✅@co63oc | #61603 #61534 |
| ✅B-14 | test_mean_op.py test_merged_adam_op.py test_merged_adam_op_xpu.py test_meshgrid_op.py test_mnist.py test_mobile_net.py test_mse_loss.py test_nansum_api.py test_nn_functional_hot_op.py test_nonzero_api.py |
✅@zade23 | #61574 |
| ✅B-15 | test_one_hot_v2_op.py test_one_hot_v2_op_xpu.py test_op_function_generator.py test_optimizer_in_control_flow.py test_paddle_imperative_double_grad.py test_partial_program.py test_pylayer.py test_reduce_op.py test_regularizer.py test_regularizer_api.py |
✅@co63oc | #61535 |
| ✅B-16 | test_reinforcement_learning.py test_repeat_interleave_op.py test_roll_op.py test_rot90_op.py test_save_load.py test_scatter_nd_op.py test_scatter_op.py test_se_resnet.py test_sentiment.py test_slice_op.py |
✅@co63oc | #61538 |
| ✅B-17 | test_smooth_l1_loss.py test_softmax_mask_fuse_op.py test_softmax_mask_fuse_upper_triangle_op.py test_solve_op.py test_stack_op.py test_traced_layer_err_msg.py test_transpose_op.py test_tril_triu_op.py test_unfold_op.py test_unfold_op_xpu.py |
✅@co63oc | #61539 |
| ✅B-18 | test_unstack_op.py test_where_op.py test_where_op_xpu.py test_word2vec.py transformer_dygraph_model.py |
✅@ooooo-create | #61503 |
| ✅B-19 | python/paddle/amp/grad_scaler.py python/paddle/base/dygraph/tensor_patch_methods.py python/paddle/base/layers/math_op_patch.py python/paddle/base/framework.py python/paddle/distributed/fleet/meta_parallel/sharding/group_sharded_utils.py python/paddle/distributed/fleet/utils/mix_precision_utils.py |
✅@co63oc | #61545 |
| ✅B-20 | python/paddle/distributed/fleet/scaler.py python/paddle/hapi/model.py python/paddle/nn/functional/loss.py python/paddle/nn/layer/layers.py python/paddle/nn/layer/norm.py python/paddle/pir/math_op_patch.py |
✅@co63oc | #61546 |
残党 C：Program.random_seed
Program.random_seed 是 Paddle 早期静态图下用来设置随机种子的方式，之后动态图也直接复用了 Program.random_seed，并有一些 random_seed 的处理逻辑。但我们现在有另一个更常用的 paddle.seed API，更加符合动态图的惯用范式，因此希望完全将 Program.random_seed 的用法用 paddle.seed 来替代，同样，我们需要先清理掉框架内已有的用法，比如：
-paddle.static.default_main_program().random_seed = 2023
-paddle.static.default_startup_program().random_seed = 2023
+paddle.seed(2023)Warning
如果遇到修改后发生精度对不上的问题，直接 revert 掉相关文件的修改，并在 PR 中说明问题即可～
| 序号 | 文件位置 | 认领人 | PR |
|---|---|---|---|
| ✅C-1 | dist_allreduce_op.py dist_ctr.py dist_fleet_ctr.py dist_fleet_ctr_ps_gpu.py dist_fleet_heter_pipeline_ctr.py |
✅@PommesPeter | #61902 |
| ✅C-2 | dist_fleet_raw_program_optimizer.py dist_fleet_simnet_bow.py dist_fleet_sync_batch_norm.py dist_mnist.py dist_mnist_dgc.py |
✅@enkilee | #61470 |
| ✅C-3 | dist_mnist_fp16_allreduce.py dist_mnist_lars.py dist_se_resnext.py dist_sharding_save.py dist_word2vec.py |
✅@co63oc | #61524 |
| ✅C-4 | fused_attention_pass_with_mp.py ir_memory_optimize_net_base.py op_test_ipu.py test_cond.py test_custom_leaky_relu_ipu.py |
✅@co63oc | #61523 |
| ✅C-5 | test_desc_clone.py test_detection.py test_dist_base.py test_dist_data_parallel_ipu.py test_dist_pod128_sample.py |
✅@co63oc | #61521 |
| ✅C-6 | test_dist_sample.py test_dist_transpiler.py test_dropout_nd_op.py test_eager_deletion_dynamic_rnn_base.py test_eager_deletion_padding_rnn.py |
✅@co63oc | #61520 |
| ✅C-7 | test_eval_model_ipu.py test_fused_attention_op.py test_fused_attention_op_xpu.py test_fused_attention_pass.py test_fused_bias_dropout_residual_layer_norm_op.py |
✅@co63oc | #61519 |
| ✅C-8 | test_fused_ec_moe_op.py test_fused_feedforward_op.py test_fused_feedforward_op_xpu.py test_fused_feedforward_pass.py test_fused_multi_transformer_int8_op.py |
✅@co63oc | #61518 |
| ✅C-9 | test_fused_multi_transformer_op.py test_fused_resnet_basic_block_op_xpu.py test_fused_transformer_encoder_layer.py test_identity_loss_ipu.py test_imperative_deepcf.py |
✅@co63oc | #61517 |
| ✅C-10 | test_imperative_hook_for_layer.py test_imperative_mnist.py test_imperative_mnist_sorted_gradient.py test_imperative_out_scale.py test_imperative_ptq.py |
✅@co63oc | #61512 |
| ✅C-11 | test_imperative_qat.py test_imperative_qat_amp.py test_imperative_qat_lsq.py test_imperative_qat_matmul.py test_imperative_qat_user_defined.py |
✅@co63oc | #61513 |
| ✅C-12 | test_imperative_recurrent_usage.py test_inference_model_io_ipu.py test_initializer.py test_initializer_nn.py test_lambv2_op.py |
✅@co63oc | #61497 |
| ✅C-13 | test_layers.py test_llm_int8_linear.py test_metrics.py test_model_parallel_ipu.py test_modelruntime_ipu.py |
✅@co63oc | #61525 |
| ✅C-14 | test_multiprocess_dataloader_dataset.py test_multiprocess_dataloader_dynamic.py test_multiprocess_dataloader_iterable_dataset_dynamic.py test_multiprocess_dataloader_iterable_dataset_static.py test_multiprocess_dataloader_static.py |
✅@co63oc | #61526 |
| ✅C-15 | test_optimizer_in_control_flow.py test_optimizer_ipu.py test_program.py test_quantization_mkldnn_pass.py test_quantization_pass.py |
✅@co63oc | #61527 |
| ✅C-16 | test_quantization_scale_pass.py test_rnn_decode_api.py test_save_inference_model.py test_seq2seq.py test_static_save_load.py |
✅@Liyulingyue | #61510 |
| ✅C-17 | test_static_save_load_bf16.py test_sync_batch_norm_op.py test_trt_conv_quant_dequant_pass.py test_trt_fc_fuse_quant_dequant_pass.py test_trt_matmul_quant_dequant.py |
✅@ooooo-create | #61504 |
| ✅C-18 | test_user_defined_quantization.py test_weight_decay.py test_weight_only_linear.py test_word2vec.py test_yolov3.py |
✅@GreatV | #61484 |
看板信息
| 任务数量 | 🔵可认领 | 🚧迁移中 | 🟢待合入 | ✅完成 | 🟡下阶段推进 | 🏁完成率 |
|---|---|---|---|---|---|---|
| 52 | 0 | 0 | 0 | 52 | 0 | 100.0% |
排名不分先后 @ooooo-create(6) @co63oc(31) @enkilee(3) @PommesPeter(3) @flying-forever(1) @NKNaN(1) @megemini(1) @Liyulingyue(2) @zade23(3) @GreatV(1)
enkilee commented
【报名】：B-1
PommesPeter commented
【报名】：B-2, B-3, C-1
enkilee commented
【报名】：C-2
flying-forever commented
【报名】：B-4
zade23 commented
【报名】：B-8, B-9, B-14
NKNaN commented
【报名】：B-5
megemini commented
【报名】：B-6
GreatV commented
【报名】：C-18
luotao1 commented
[Cleanup] 框架历史遗留 API 清理计划（一期）已全部完成，感谢参与的小伙伴们！
排名不分先后 @ooooo-create(6) @co63oc(31) @enkilee(3) @PommesPeter(3) @flying-forever(1) @NKNaN(1) @megemini(1) @Liyulingyue(2) @zade23(3) @GreatV(1)
欢迎继续参与快乐开源的其他任务！