Bug: `optimize_sa` outputs wrong starting layout
Closed this issue · 3 comments
Glitchy-Tozier commented
Not sure what to do about this, but layout.resolve_modifiers(&k) produces (from time to time) different results.
I discovered this when I noticed that layouts weren't always evaluated the same way. I dug deeper and discovered that even the total combined weight after n-gram-expansion differed each time OnDemandBigramMapper (and the other n-gram mappers) are used.
There may be other issues at play as well, but what certainly is happening is that layout.resolve_modifiers(&k) produces inconsistent results. You can test it this way:
Anywhere in the Bigram-mapper, for example at the start of process_hold_layers, paste this code:
for i in 0..50 {
let (_, m) = layout.resolve_modifiers(&i);
println!("{}: {:?}", i, layout.resolve_modifiers(&i));
let good_count = m.layerkey_indices().len();
for _ in 0..10000 {
let (k, mods) = layout.resolve_modifiers(&i);
if mods.layerkey_indices().len() != good_count {
println!("Wrong: {:?}", (k, mods));
}
}
}Then run the optimizer.
You'll see that within each call of BigramMapper, the function behaves consistently. However, between different calls of BigramMapper, the printed list looks differently.
Here's my test-results. You can use tools like https://text-compare.com/ to compare the different calls. At the end of each call, I've attached the total no. of modifiers found while looping through all ngrams in process_hold_layers().
0: (0, Hold([]))
1: (0, Hold([211]))
2: (0, Hold([212]))
3: (3, Hold([]))
4: (3, Hold([211]))
5: (3, Hold([212]))
6: (6, Hold([]))
7: (6, Hold([211]))
8: (6, Hold([212]))
9: (9, Hold([]))
10: (9, Hold([211]))
11: (9, Hold([212]))
12: (12, Hold([]))
13: (12, Hold([211]))
14: (12, Hold([212]))
15: (15, Hold([]))
16: (15, Hold([211]))
17: (15, Hold([212]))
18: (18, Hold([]))
19: (18, Hold([210]))
20: (18, Hold([213]))
21: (21, Hold([]))
22: (21, Hold([210]))
23: (21, Hold([213]))
24: (21, Lock([214, 215]))
25: (25, Hold([]))
26: (25, Hold([210]))
27: (25, Hold([213]))
28: (25, Lock([214, 215]))
29: (29, Hold([]))
30: (29, Hold([210]))
31: (29, Hold([213]))
32: (29, Lock([214, 215]))
33: (33, Hold([]))
34: (33, Hold([210]))
35: (35, Hold([]))
36: (35, Hold([210]))
37: (37, Hold([]))
38: (38, Hold([]))
39: (39, Hold([]))
40: (39, Hold([211]))
41: (41, Hold([]))
42: (41, Hold([211]))
43: (41, Hold([212]))
44: (41, Lock([216, 216]))
45: (41, Hold([217]))
46: (46, Hold([]))
47: (46, Hold([211]))
48: (46, Hold([212]))
49: (46, Lock([216, 216]))
16327
0: (0, Hold([]))
1: (0, Hold([210]))
2: (0, Hold([211]))
3: (3, Hold([]))
4: (3, Hold([210]))
5: (3, Hold([211]))
6: (6, Hold([]))
7: (6, Hold([210]))
8: (6, Hold([211]))
9: (9, Hold([]))
10: (9, Hold([210]))
11: (9, Hold([211]))
12: (12, Hold([]))
13: (12, Hold([210]))
14: (12, Hold([211]))
15: (15, Hold([]))
16: (15, Hold([210]))
17: (15, Hold([211]))
18: (18, Hold([]))
19: (18, Hold([209]))
20: (18, Hold([212]))
21: (21, Hold([]))
22: (21, Hold([209]))
23: (21, Hold([212]))
24: (21, Lock([213, 214]))
25: (25, Hold([]))
26: (25, Hold([209]))
27: (25, Hold([212]))
28: (25, Lock([213, 214]))
29: (29, Hold([]))
30: (29, Hold([209]))
31: (29, Hold([212]))
32: (29, Lock([213, 214]))
33: (33, Hold([]))
34: (33, Hold([209]))
35: (35, Hold([]))
36: (35, Hold([209]))
37: (35, Hold([212]))
38: (35, Lock([213, 214]))
39: (39, Hold([]))
40: (40, Hold([]))
41: (41, Hold([]))
42: (41, Hold([210]))
43: (43, Hold([]))
44: (43, Hold([210]))
45: (43, Hold([211]))
46: (43, Lock([215, 215]))
47: (43, Hold([216]))
48: (48, Hold([]))
49: (48, Hold([210]))
16312
0: (0, Hold([]))
1: (0, Hold([211]))
2: (0, Hold([212]))
3: (3, Hold([]))
4: (3, Hold([211]))
5: (3, Hold([212]))
6: (6, Hold([]))
7: (6, Hold([211]))
8: (6, Hold([212]))
9: (9, Hold([]))
10: (9, Hold([211]))
11: (9, Hold([212]))
12: (12, Hold([]))
13: (12, Hold([211]))
14: (12, Hold([212]))
15: (15, Hold([]))
16: (15, Hold([211]))
17: (15, Hold([212]))
18: (18, Hold([]))
19: (18, Hold([210]))
20: (18, Hold([213]))
21: (21, Hold([]))
22: (21, Hold([210]))
23: (21, Hold([213]))
24: (21, Lock([214, 215]))
25: (25, Hold([]))
26: (25, Hold([210]))
27: (25, Hold([213]))
28: (25, Lock([214, 215]))
29: (29, Hold([]))
30: (29, Hold([210]))
31: (29, Hold([213]))
32: (29, Lock([214, 215]))
33: (33, Hold([]))
34: (33, Hold([210]))
35: (35, Hold([]))
36: (35, Hold([210]))
37: (37, Hold([]))
38: (38, Hold([]))
39: (39, Hold([]))
40: (39, Hold([211]))
41: (41, Hold([]))
42: (41, Hold([211]))
43: (41, Hold([212]))
44: (41, Lock([216, 216]))
45: (41, Hold([217]))
46: (46, Hold([]))
47: (46, Hold([211]))
48: (46, Hold([212]))
49: (46, Lock([216, 216]))
16327
0: (0, Hold([]))
1: (0, Hold([211]))
2: (0, Hold([212]))
3: (3, Hold([]))
4: (3, Hold([211]))
5: (3, Hold([212]))
6: (6, Hold([]))
7: (6, Hold([211]))
8: (6, Hold([212]))
9: (9, Hold([]))
10: (9, Hold([211]))
11: (9, Hold([212]))
12: (12, Hold([]))
13: (12, Hold([211]))
14: (12, Hold([212]))
15: (15, Hold([]))
16: (15, Hold([211]))
17: (15, Hold([212]))
18: (18, Hold([]))
19: (18, Hold([210]))
20: (18, Hold([213]))
21: (21, Hold([]))
22: (21, Hold([210]))
23: (21, Hold([213]))
24: (21, Lock([214, 215]))
25: (25, Hold([]))
26: (25, Hold([210]))
27: (25, Hold([213]))
28: (25, Lock([214, 215]))
29: (29, Hold([]))
30: (29, Hold([210]))
31: (29, Hold([213]))
32: (29, Lock([214, 215]))
33: (33, Hold([]))
34: (33, Hold([210]))
35: (33, Hold([213]))
36: (33, Lock([214, 215]))
37: (37, Hold([]))
38: (37, Hold([210]))
39: (39, Hold([]))
40: (40, Hold([]))
41: (41, Hold([]))
42: (41, Hold([211]))
43: (43, Hold([]))
44: (43, Hold([211]))
45: (43, Hold([212]))
46: (43, Lock([216, 216]))
47: (43, Hold([217]))
48: (48, Hold([]))
49: (48, Hold([211]))
16345
0: (0, Hold([]))
1: (0, Hold([211]))
2: (0, Hold([212]))
3: (3, Hold([]))
4: (3, Hold([211]))
5: (3, Hold([212]))
6: (6, Hold([]))
7: (6, Hold([211]))
8: (6, Hold([212]))
9: (9, Hold([]))
10: (9, Hold([211]))
11: (9, Hold([212]))
12: (12, Hold([]))
13: (12, Hold([211]))
14: (12, Hold([212]))
15: (15, Hold([]))
16: (15, Hold([211]))
17: (15, Hold([212]))
18: (18, Hold([]))
19: (18, Hold([210]))
20: (18, Hold([213]))
21: (21, Hold([]))
22: (21, Hold([210]))
23: (21, Hold([213]))
24: (21, Lock([214, 215]))
25: (25, Hold([]))
26: (25, Hold([210]))
27: (25, Hold([213]))
28: (25, Lock([214, 215]))
29: (29, Hold([]))
30: (29, Hold([210]))
31: (29, Hold([213]))
32: (29, Lock([214, 215]))
33: (33, Hold([]))
34: (33, Hold([210]))
35: (35, Hold([]))
36: (35, Hold([210]))
37: (37, Hold([]))
38: (38, Hold([]))
39: (39, Hold([]))
40: (39, Hold([211]))
41: (41, Hold([]))
42: (41, Hold([211]))
43: (41, Hold([212]))
44: (41, Lock([216, 216]))
45: (41, Hold([217]))
46: (46, Hold([]))
47: (46, Hold([211]))
48: (46, Hold([212]))
49: (46, Lock([216, 216]))
16327
0: (0, Hold([]))
1: (0, Hold([211]))
2: (0, Hold([212]))
3: (3, Hold([]))
4: (3, Hold([211]))
5: (3, Hold([212]))
6: (6, Hold([]))
7: (6, Hold([211]))
8: (6, Hold([212]))
9: (9, Hold([]))
10: (9, Hold([211]))
11: (9, Hold([212]))
12: (12, Hold([]))
13: (12, Hold([211]))
14: (12, Hold([212]))
15: (15, Hold([]))
16: (15, Hold([211]))
17: (15, Hold([212]))
18: (18, Hold([]))
19: (18, Hold([210]))
20: (18, Hold([213]))
21: (21, Hold([]))
22: (21, Hold([210]))
23: (21, Hold([213]))
24: (21, Lock([214, 215]))
25: (25, Hold([]))
26: (25, Hold([210]))
27: (25, Hold([213]))
28: (25, Lock([214, 215]))
29: (29, Hold([]))
30: (29, Hold([210]))
31: (29, Hold([213]))
32: (29, Lock([214, 215]))
33: (33, Hold([]))
34: (33, Hold([210]))
35: (35, Hold([]))
36: (35, Hold([210]))
37: (37, Hold([]))
38: (38, Hold([]))
39: (39, Hold([]))
40: (39, Hold([211]))
41: (41, Hold([]))
42: (41, Hold([211]))
43: (41, Hold([212]))
44: (41, Lock([216, 216]))
45: (41, Hold([217]))
46: (46, Hold([]))
47: (46, Hold([211]))
48: (46, Hold([212]))
49: (46, Lock([216, 216]))
16327
0: (0, Hold([]))
1: (0, Hold([211]))
2: (0, Hold([212]))
3: (3, Hold([]))
4: (3, Hold([211]))
5: (3, Hold([212]))
6: (6, Hold([]))
7: (6, Hold([211]))
8: (6, Hold([212]))
9: (9, Hold([]))
10: (9, Hold([211]))
11: (9, Hold([212]))
12: (12, Hold([]))
13: (12, Hold([211]))
14: (12, Hold([212]))
15: (15, Hold([]))
16: (15, Hold([211]))
17: (15, Hold([212]))
18: (18, Hold([]))
19: (18, Hold([210]))
20: (18, Hold([213]))
21: (21, Hold([]))
22: (21, Hold([210]))
23: (21, Hold([213]))
24: (21, Lock([214, 215]))
25: (25, Hold([]))
26: (25, Hold([210]))
27: (25, Hold([213]))
28: (25, Lock([214, 215]))
29: (29, Hold([]))
30: (29, Hold([210]))
31: (29, Hold([213]))
32: (29, Lock([214, 215]))
33: (33, Hold([]))
34: (33, Hold([210]))
35: (35, Hold([]))
36: (35, Hold([210]))
37: (37, Hold([]))
38: (38, Hold([]))
39: (39, Hold([]))
40: (39, Hold([211]))
41: (41, Hold([]))
42: (41, Hold([211]))
43: (41, Hold([212]))
44: (41, Lock([216, 216]))
45: (41, Hold([217]))
46: (46, Hold([]))
47: (46, Hold([211]))
48: (46, Hold([212]))
49: (46, Lock([216, 216]))
16327
0: (0, Hold([]))
1: (0, Hold([211]))
2: (0, Hold([212]))
3: (3, Hold([]))
4: (3, Hold([211]))
5: (3, Hold([212]))
6: (6, Hold([]))
7: (6, Hold([211]))
8: (6, Hold([212]))
9: (9, Hold([]))
10: (9, Hold([211]))
11: (9, Hold([212]))
12: (12, Hold([]))
13: (12, Hold([211]))
14: (12, Hold([212]))
15: (15, Hold([]))
16: (15, Hold([211]))
17: (15, Hold([212]))
18: (18, Hold([]))
19: (18, Hold([210]))
20: (18, Hold([213]))
21: (21, Hold([]))
22: (21, Hold([210]))
23: (21, Hold([213]))
24: (21, Lock([214, 215]))
25: (25, Hold([]))
26: (25, Hold([210]))
27: (25, Hold([213]))
28: (25, Lock([214, 215]))
29: (29, Hold([]))
30: (29, Hold([210]))
31: (29, Hold([213]))
32: (29, Lock([214, 215]))
33: (33, Hold([]))
34: (33, Hold([210]))
35: (35, Hold([]))
36: (35, Hold([210]))
37: (37, Hold([]))
38: (38, Hold([]))
39: (39, Hold([]))
40: (39, Hold([211]))
41: (41, Hold([]))
42: (41, Hold([211]))
43: (41, Hold([212]))
44: (41, Lock([216, 216]))
45: (41, Hold([217]))
46: (46, Hold([]))
47: (46, Hold([211]))
48: (46, Hold([212]))
49: (46, Lock([216, 216]))
16327
dariogoetz commented
I currently don't have the time to run the tests myself. I may be able to find some time in a couple of weeks.
I'm not sure that I understand what you are saying. When you say "then run the optimizer", do you mean that the various calls to bigram mapper happen for different layouts? In that case, it would not be surprising that the modifier expansion in the bigram mapper ends up with different total numbers of modifiers.
Example: If the two symbols in a bigram come from the same upper layer, it makes a difference if they belong to the same hand. On the same hand, the modifiers are only "held" for both symbols. On different hands, multiple key presses for the modifiers are involved. Maybe this is what you are observing.
Glitchy-Tozier commented
- You're absolutely correct, my testing was flawed.
- Evaluation might work just fine. The thing that confused me initially was that when starting a
--run-foreversimmulated-annealing optimization with a starting layout, that the starting-layout had different evaluations. However, now that I'm looking more closely, I can see that the optimizer sometimes shows mutated starting-layouts (see below). I'll have to check that out.
cargo build --release && RUST_LOG=INFO ./target/release/optimize_sa -s k.o,yvgclßzhaeiudtrnsfxqäüöbpwmj --run-foreverOutput (the last 8 lines are the important part):
Finished release [optimized] target(s) in 0.16s
[2023-07-09T14:57:20Z INFO keyboard_layout_optimizer::common] A-priori estimations from key_costs:
[2023-07-09T14:57:20Z INFO keyboard_layout_optimizer::common] Finger loads (thumbs set to 0.00): 0.13 0.10 0.10 0.14 0.00 - 0.00 0.15 0.12 0.10 0.17
[2023-07-09T14:57:20Z INFO keyboard_layout_optimizer::common] Row loads: Row 0: 0.05 Row 1: 0.21 Row 2: 0.48 Row 3: 0.14 Row 4: 0.12
[2023-07-09T14:57:20Z INFO keyboard_layout_optimizer::common] Reading unigram file: '"ngrams/deu_mixed_wiki_web_0.6_eng_news_typical_wiki_web_0.4/1-grams.txt"'
[2023-07-09T14:57:20Z INFO keyboard_layout_optimizer::common] Reading bigram file: '"ngrams/deu_mixed_wiki_web_0.6_eng_news_typical_wiki_web_0.4/2-grams.txt"'
[2023-07-09T14:57:20Z INFO keyboard_layout_optimizer::common] Reading trigram file: '"ngrams/deu_mixed_wiki_web_0.6_eng_news_typical_wiki_web_0.4/3-grams.txt"'
[2023-07-09T14:57:20Z INFO optimize_sa] Process 4: Starting optimization from k.o,yvgclßzhaeiudtrnsfxqäüöbpwmj
[2023-07-09T14:57:20Z INFO optimize_sa] Process 2: Starting optimization from k.o,yvgclßzhaeiudtrnsfxqäüöbpwmj
[2023-07-09T14:57:20Z INFO optimize_sa] Process 1: Starting optimization from k.o,yvgclßzhaeiudtrnsfxqäüöbpwmj
[2023-07-09T14:57:20Z INFO optimize_sa] Process 5: Starting optimization from k.o,yvgclßzhaeiudtrnsfxqäüöbpwmj
[2023-07-09T14:57:20Z INFO optimize_sa] Process 3: Starting optimization from k.o,yvgclßzhaeiudtrnsfxqäüöbpwmj
[2023-07-09T14:57:20Z INFO optimize_sa] Process 6: Starting optimization from k.o,yvgclßzhaeiudtrnsfxqäüöbpwmj
[2023-07-09T14:57:20Z INFO optimize_sa] Process 0: Starting optimization from k.o,yvgclßzhaeiudtrnsfxqäüöbpwmj
[2023-07-09T14:57:20Z INFO optimize_sa] Process 7: Starting optimization from k.o,yvgclßzhaeiudtrnsfxqäüöbpwmj
[2023-07-09T14:57:20Z INFO layout_optimization_sa::optimization] Process 0: Starting optimization with: initial_temperature: 150.00°, Parameters { init_temp: Some(150.0), key_switches: 1, stall_accepted: 5000, max_iters: 100000 }
[2023-07-09T14:57:20Z INFO layout_optimization_sa::optimization] Process 1: Starting optimization with: initial_temperature: 150.00°, Parameters { init_temp: Some(150.0), key_switches: 1, stall_accepted: 5000, max_iters: 100000 }
[2023-07-09T14:57:20Z INFO layout_optimization_sa::optimization] Process 4: Starting optimization with: initial_temperature: 150.00°, Parameters { init_temp: Some(150.0), key_switches: 1, stall_accepted: 5000, max_iters: 100000 }
[2023-07-09T14:57:20Z INFO layout_optimization_sa::optimization] Process 7: Starting optimization with: initial_temperature: 150.00°, Parameters { init_temp: Some(150.0), key_switches: 1, stall_accepted: 5000, max_iters: 100000 }
[2023-07-09T14:57:20Z INFO layout_optimization_sa::optimization] Process 5: Starting optimization with: initial_temperature: 150.00°, Parameters { init_temp: Some(150.0), key_switches: 1, stall_accepted: 5000, max_iters: 100000 }
[2023-07-09T14:57:20Z INFO layout_optimization_sa::optimization] Process 6: Starting optimization with: initial_temperature: 150.00°, Parameters { init_temp: Some(150.0), key_switches: 1, stall_accepted: 5000, max_iters: 100000 }
[2023-07-09T14:57:20Z INFO layout_optimization_sa::optimization] Process 2: Starting optimization with: initial_temperature: 150.00°, Parameters { init_temp: Some(150.0), key_switches: 1, stall_accepted: 5000, max_iters: 100000 }
[2023-07-09T14:57:20Z INFO layout_optimization_sa::optimization] Process 3: Starting optimization with: initial_temperature: 150.00°, Parameters { init_temp: Some(150.0), key_switches: 1, stall_accepted: 5000, max_iters: 100000 }
[2023-07-09T14:57:21Z INFO layout_optimization_sa::optimization] Process 5: Starting layout: k.o,yvgclßzhueiadtrnsfxqäüöbpwmj ( 398.9)
[2023-07-09T14:57:21Z INFO layout_optimization_sa::optimization] Process 1: Starting layout: k.o,yvgclßzhaeiudtrnsfxqäüwbpömj ( 409.9)
[2023-07-09T14:57:21Z INFO layout_optimization_sa::optimization] Process 3: Starting layout: k.o,yvgclßzhaeiudtrnsmxqäüöbpwfj ( 414.6)
[2023-07-09T14:57:21Z INFO layout_optimization_sa::optimization] Process 6: Starting layout: k.o,yvgclßzhaeiudtrnsfxqäüöbpwmj ( 417.6)
[2023-07-09T14:57:21Z INFO layout_optimization_sa::optimization] Process 0: Starting layout: k.o,yvgclßzhaeiudtrnsfxqäüöbpwmj ( 417.6)
[2023-07-09T14:57:21Z INFO layout_optimization_sa::optimization] Process 2: Starting layout: k.o,yvgclßzhaeiudtrnsfxqäüöbpwmj ( 417.6)
[2023-07-09T14:57:21Z INFO layout_optimization_sa::optimization] Process 4: Starting layout: k.omyvgclßzhaeiudtrnsfxqäüöbpw,j ( 406.5)
[2023-07-09T14:57:21Z INFO layout_optimization_sa::optimization] Process 7: Starting layout: k.o,yvgclßzhaeiudtrnsfxqäüöbpwmj ( 417.6)
Glitchy-Tozier commented
It seems this probably was a weird quirk in the way Argmin-observers work.
Once we're done with #62, I can create a PR that clears up the output. It's really simple.