Document mora_type

[winglian]

Can you add more flavor about what each mora_type value means or refers to in the paper? The is pretty opaque currently and might be best handled by using enums for mora_type.

[winglian]

1 & 6 are defined in the readme as

    # type 1 (Sharing) for large lora ranks, Eq. 6 in paper
    # type 6 (RoPE based) for small lora ranks, Eq. 9 in paper

but there are several places in the code where it is compared to values 2, 3, or 4.

[kongds]

Thanks for your advice.

Types 2, 3, and 4 use sharing like Type 1, but with different share groups. They are used for ReMoRA, which merges MoRA and changes its types to increase rank of delta w by different share groups.

MoRA/train.py

Lines 64 to 70 in 0ff64b1

	if linear.use_mora['default']:
	#print('mora change type', linear.mora_type['default'], end='->')
	if self.remora_types == 4:
	# 1->2->3->4
	mora_type_map = {1:2, 2:3, 3:4, 4:1}
	else:
	mora_type_map = {1:2, 2:1}