Continuation lines are not correctly emitted in manually set multiline values

[eviltak]

If a field added manually (i.e. via Paragraph::insert) has a multiline value, it does not add the continuation indent (whitespace) to the emitted text. This produces a malformed DEB822 file. As a simple example,

let entry = Entry::new("foo", "bar\nbaz");
println!("{:?}", entry.to_string());
// prints foo: bar\nbaz\n - note the lack of whitespace after the newline between `bar` and `baz`

A straightforward fix would be to add a new VALUE token preceded by whitespace and succeeded by a newline for every line in the value of an Entry (which is exactly what the lexer does - see the parsed tokens of the Depends field in lex::tests::test_simple):

impl Entry {
    pub fn new(key: &str, value: &str) -> Entry {
        let mut builder = GreenNodeBuilder::new();

        builder.start_node(ENTRY.into());
        builder.token(KEY.into(), key);
        builder.token(COLON.into(), ":");
        for line in value.split("\n") {
            // Semantically this probably should be an INDENT token for all the continuation lines
            // but using WHITESPACE here keeps this logic simple
            builder.token(WHITESPACE.into(), " ");
            builder.token(VALUE.into(), line);
            builder.token(NEWLINE.into(), "\n");
        }
        builder.finish_node();
        Entry(SyntaxNode::new_root(builder.finish()))
    }
}

This change lets the following test pass:

#[test]
fn test_multiline_entry() {
    let entry = Entry::new("foo", "bar\nbaz");
    assert_eq!("foo: bar\n baz\n", entry.to_string());
    assert_eq!("bar\nbaz", entry.value());
}

[jelmer]

Thanks for the bug report and the suggested fix. Can you create a PR?

[eviltak]

Yes I can. Will my proposed fix (with the likely incorrect usage of the WHITESPACE token) suffice, or should I try and implement something that uses INDENT for continuation lines?

[eviltak]

Thanks for merging my PR! Do you know when you will be able to publish a new version of the crate with this fix to crates.io? I'm currently using a local clone with my changes in my workspace, but would like to move back to using the crates.io version soon, if possible.

[jelmer]

Thanks for merging my PR! Do you know when you will be able to publish a new version of the crate with this fix to crates.io? I'm currently using a local clone with my changes in my workspace, but would like to move back to using the crates.io version soon, if possible.

I've just created a new release.