Use bss for zero-init data

[jyn514]

Right now, calling Artifact.define requires allocating memory greedily, which means that any file that has more than a GB or so of zero-init data takes a very long time to create. Using a .bss section instead will only allocate memory when the executable is actually loaded into memory.

I know this is a little bit of an edge case, since not many people are loading 1 GB arrays into memory, but I do think it's an important improvement.

[m4b]

Hi, thanks for the issue ! I don't think it's an edge case and I agree it's definitely an important improvement! If you have any time, perhaps you might be interested in writing a PR for the fix ? :)

[jyn514]

I guess I could give it a shot! Where would I start, I assume I need to implement it for every backend?

[jyn514]

It looks like I'd have to change Artifact.define, Artifact.define_with_symbols, and InternalDefinition to allow zero-init data. In particular, the Data type alias in artifact.rs needs to change to be an enum, I'm imagining something like this:

enum Data {
    Blob(Vec<u8>),
    ZeroInit(usize)
}

[jyn514]

Is there a reason that InternalDefinition and Reloc are Ord? I'm not sure how much sense it makes to compare them or Data. Ignore me, it's so they can go in a BTreeSet

[jyn514]

Noob question: What's the difference between faerie and goblin? Is faerie for writing object files and goblin for reading them?

[bjorn3]

Goblin is used for raw reading and writing object files. For example https://docs.rs/goblin/0.0.24/goblin/elf/section_header/section_header32/struct.SectionHeader.html allows you to read or write an elf section header.

Faerie is used for writing object files in a file format independent way without having to worry about the exact way object files are structured.

[m4b]

Yes making data an enum seems right ? It would be a backwards incompatible change but getting bss data is very important so if have to, it’s acceptable to me.

I wonder if there’s an alternative? Perhaps empty vec (don’t like it, just thinning out loud) + flag?

Enum probably best approach and clearer to user :)

[jyn514]

I kept the interface for define the same, so only define_with_symbols will be backwards incompatible. Not ideal, but I bet most users are using define, not define_with_symbols.

[jyn514]

Where are DataDecls created? I want to set the DataType to a new ZeroInit variant but I'm not sure where it's getting called

[bjorn3]

faerie/src/artifact/decl.rs

Lines 246 to 248 in d1bf163

	pub fn data() -> DataDecl {
	DataDecl::default()
	}

faerie/src/artifact/decl.rs

Lines 449 to 476 in d1bf163

	impl DataDecl {
	scope_methods!();
	visibility_methods!();
	datatype_methods!();
	align_methods!();
	/// Builder for writability
	pub fn with_writable(mut self, writable: bool) -> Self {
	self.writable = writable;
	self
	}
	/// Set mutability to writable
	pub fn writable(self) -> Self {
	self.with_writable(true)
	}
	/// Set mutability to read-only
	pub fn read_only(self) -> Self {
	self.with_writable(false)
	}
	/// Setter for mutability
	pub fn set_writable(&mut self, writable: bool) {
	self.writable = writable;
	}
	/// Accessor for mutability
	pub fn is_writable(&self) -> bool {
	self.writable
	}
	}

[jyn514]

.bss only makes sense for data, right? Or can you create a custom section that starts with .bss?

Related, what does it mean to define a section with data? Is that used just for emitting .text or can it do something else?

[bjorn3]

.bss only makes sense for data, right?

I thinks so.

Related, what does it mean to define a section with data? Is that used just for emitting .text or can it do something else?

A custom section allows you to define a section with any name you want. For example in cg_clif a custom section (named .rustc) is used to store metadata generated by rustc in a dylib.

[jyn514]

I think I have this working for MachO, but goblin is returning 'Err(Scroll(TooBig { size: 100000000000000, len: 27 }))' when I parse it and I don't have any other Mach-O readers on hand. Anyone on Mac able to help out? If you run cargo test bss it will generate a file called mach.o in the root folder and you can look from there.

In particular, I'm interested to see the output of the following command (it should segfault before hitting main): cc mach.o -x c - <<< 'extern int my_data[]; int main() {}' && ./a.out

(use branch bss on https://github.com/jyn514/faerie/tree/bss)

[jyn514]

Is there a way to emit a warning from SectionBuilder::create? See https://github.com/jyn514/faerie/blob/bss/src/elf.rs#L323

[m4b]

I think I have this working for MachO, but goblin is returning 'Err(Scroll(TooBig { size: 100000000000000, len: 27 }))' when I parse it and I don't have any other Mach-O readers on hand. Anyone on Mac able to help out? If you run cargo test bss it will generate a file called mach.o in the root folder and you can look from there.

In particular, I'm interested to see the output of the following command (it should segfault before hitting main): cc mach.o -x c - <<< 'extern int my_data[]; int main() {}' && ./a.out

(use branch bss on https://github.com/jyn514/faerie/tree/bss)

Ok I ran this from latest PR as of 2 minutes ago, and it doesn't segfault:

[ ~/projects/faerie ] cc mach.o -x c - <<< 'extern int my_data[]; int main() {}' && ./a.out
[ ~/projects/faerie ] echo $?
0

is this not expected?

[jyn514]

That seems odd, maybe Mac behaves different from Linux here? What should be going on is that it tries to load an array of 10,000 gigabytes into memory and crashes. Maybe it's lazily loaded? Try returning my_data[100000000000000 - 1] and see what that does.

This would be an even better test to make sure _bss is writable:

extern int my_data[];

#include<stdint.h>
#define MAX SIZE_MAX

int main() {
    for (int i = 0; i < MAX; ++i) {
        my_data[i] = 1;
    }
    return my_data[MAX - 1];
}

That should segfault or start swapping like crazy as is, and should return 1 if you change max to something smaller like 10.