gopep

gopep (Go Lang Portable Executable Parser) is Python script for extracting attributes from PE executables compiled in Go. This repo is part of a personal project for learning about executables compiled Go. For anyone else wanting to learn about executable compiled in Go, I would recommend reading through the links in the resources section.

gopep can be viewed for viewing contents of fields/structures extracted from Go's ModuleData.

functab

functab contains function symbol names and their start offset.

instance.functab
instance.symbols (easier to remember)

Example Output

(4198400, b'go.buildid'), (4198512, b'internal/cpu.Initialize')

filetab

filetab contains all of the file paths for the source code (path/foo.go).

instance.filetab

Example Output

[b'<autogenerated>', b'/usr/local/go/src/runtime/internal/sys/intrinsics_386.s', b'/usr/local/go/test/helloworld.go']

The last file path is commonly a path that is not associated with Go libraries but the path of the developers source code. For example, /usr/local/go/test/helloworld.go is the source path for a "hello, world" Go binary. Other source files likely from the same source path are found using the difflib library and the SequenceMatcher function. These can be found by using the following.

instance.go_base_paths

Example Output

[b'/usr/local/go/test/helloworld.go']

itabs

instance.itab_sym

Not present if binary is stripped.

Example Output

[b'go.itab.*os.File,io.Writer', b'go.itab.*errors.errorString,error', b'go.itab.*reflect.rtype,reflect.Type', b'go.itab.*fmt.pp,fmt.State', b'go.itab.*internal/reflectlite.rtype,internal/reflectlite.Type', b'go.itab.*internal/fmtsort.SortedMap,sort.Interface', b'go.itab.*os.PathError,error', b'go.itab.syscall.Errno,error', b'go.itab.runtime.errorString,error', b'go.itab.*syscall.DLLError,error', b'go.itab.*internal/poll.TimeoutError,error']

.symtab symbols

instance.symtab_symbols

Example Output

[b'runtime.text', b'runtime.etext', b'main..inittask', b'internal/cpu.X86',

stripped

instance.stripped
True or False

packed

instance.packeed
True or False
Only checks for UPX by comparing section names.
yeah, it sucks but I'm not caring much about packers detection right now.

Hashes

instance.hash_sys_all
instance.hash_sys_main
instance.hash_sys_nomain
instance.hash_itabs
instance.hash_file_paths

Clustering

Attributes from Go binaries can be hashed and used for clustering. This technique is the same as Imphash (Import Hashing), which hashes blocks of string using MD5. This form of hashing is trivial to break but is still interesting to experiment with. GoPeP hashes based off the following

all symbol/function names
main symbols/functions
non-main symbols/functions
source code paths (functab)
/usr/local/go/test/helloworld.go
itabs (if present)

Command line

gopep can be used as a class of executed as a script. The following command line options are available.

python gopep.py -h
usage: gopep.py [-h] [-c C_DIR] [-e E_FILE] [-x EA_DIR] [-v IN_FILE] [-m MD_FILE] [-t T_FILE] [-ev ET_FILE]

gopep Go Portable Executable Parser

optional arguments:
  -h, --help            show this help message and exit
  -c C_DIR, --cluster C_DIR
                        cluster directory of files
  -e E_FILE, --export E_FILE
                        export results of file to JSON
  -x EA_DIR, --export_all EA_DIR
                        export results of directory to JSONs
  -v IN_FILE, --version IN_FILE
                        print version
  -m MD_FILE, --module-data MD_FILE
                        print module data details
  -t T_FILE, --triage T_FILE
                        triage file, print interesting attributes
  -ev ET_FILE, --everything ET_FILE
                        print EVERYTHING!

go_logger.py

Go logger is a simple function hooker for Go executables. Using libptrace (thank you 0vercl0k for the recommendation), it sets a breakpoint on the main.main then sets a breakpoint on all functions. The function address and name is extracted from functab. The API addresses and function names are extracted using gopep.py with the -st argument. This arugment creates a json with the string .json appended to the executable file name.

Command line arguments %python3.7%\python.exe go_logger.py go.exe go.json. An example command line can be seen below.

C:\Users\null\AppData\Local\Programs\Python\Python37\python.exe go_hook.py C:\Use
rs\null\Documents\repo\gopep\test\go-hello-stripped.exe C:\Users\null\Documents\repo\gopep\test\go-hello-stripped.exe.js
on

Log Ouput

C:\Users\null\Documents\repo\libptrace>C:\Users\null\AppData\Local\Programs\Python\Python37\python.exe go_hook.py C:\Use
rs\null\Documents\repo\gopep\test\go-hello-stripped.exe C:\Users\null\Documents\repo\gopep\test\go-hello-stripped.exe.js
on
[15256] Module ntdll loaded at 0x7ffb99da0000
[15256] Module ntdll loaded at 0x77a30000
[15256] Created thread with tid 11528
[15256] Module wow64 loaded at 0x7ffb98080000
[15256] Module wow64win loaded at 0x7ffb992a0000
[15256] Attached
[15256] BreakPoint Set at 0x00451840
[15256] Module wow64cpu loaded at 0x77a20000
[15256] Breakpoint
[15256] Thread with tid 11528 exited
[15256] Module kernel32 loaded at 0x75c50000
[15256] Module KernelBase loaded at 0x76b80000
[15256] Breakpoint
[15256] Module advapi32 loaded at 0x75b80000
[15256] Module msvcrt loaded at 0x76a60000
[15256] Created thread with tid 6972
[15256] Module sechost loaded at 0x752b0000
[15256] Module rpcrt4 loaded at 0x76f00000
[15256] Module sspicli loaded at 0x751f0000
[15256] Module cryptbase loaded at 0x751e0000
[15256] Module bcryptprimitives loaded at 0x77110000
[15256] Created thread with tid 15136
[15256] Created thread with tid 14576
[15256] Module winmm loaded at 0x74f10000
[15256] Module winmmbase loaded at 0x006a0000
[15256] Module winmmbase loaded at 0x749d0000
[15256] Module winmmbase unloaded
[15256] Module winmmbase loaded at 0x00860000
[15256] Module winmmbase unloaded
[15256] Module cfgmgr32 loaded at 0x75d50000
[15256] Module ucrtbase loaded at 0x768e0000
[15256] Module ws2_32 loaded at 0x76b20000
[15256] Module powrprof loaded at 0x773e0000
[15256] Module umpdc loaded at 0x77a10000
[15256] Created thread with tid 14972
[15256] Created thread with tid 14472
[15256] Created thread with tid 13372
[15256] Created thread with tid 2728
[15256] Func=runtime.printlock, IP=0x0042a390, Frame=0x11028fb8, Base=0x00000038, Return=0x0045185a
[15256] Func=runtime.lock, IP=0x00407250, Frame=0x11028fac, Base=0x00000038, Return=0x0042a3e5
[15256] Func=runtime/internal/atomic.Casuintptr, IP=0x00401fd0, Frame=0x11028f84, Base=0x00000038, Return=0x004072ac
[15256] Func=runtime/internal/atomic.Cas, IP=0x00401fb0, Frame=0x11028f84, Base=0x00000038, Return=0x004072ac
[15256] Func=runtime.printstring, IP=0x0042ac30, Frame=0x11028fb8, Base=0x00000038, Return=0x00451870
[15256] Func=runtime.gwrite, IP=0x0042a460, Frame=0x11028f94, Base=0x00000038, Return=0x0042ac99
[15256] Func=runtime.recordForPanic, IP=0x0042a270, Frame=0x11028f7c, Base=0x00000038, Return=0x0042a49d
[15256] Func=runtime.printlock, IP=0x0042a390, Frame=0x11028f64, Base=0x00000038, Return=0x0042a28e
[15256] Func=runtime/internal/atomic.Load, IP=0x00401f70, Frame=0x11028f64, Base=0x00000038, Return=0x0042a29c
[15256] Func=runtime.memmove, IP=0x0044d2c0, Frame=0x11028f64, Base=0x0000000d, Return=0x0042a35f
[15256] Func=runtime.printunlock, IP=0x0042a400, Frame=0x11028f64, Base=0x0000000d, Return=0x0042a36d
[15256] Func=runtime.writeErr, IP=0x00448160, Frame=0x11028f7c, Base=0x00000000, Return=0x0042a4ea
[15256] Func=runtime.write, IP=0x00440680, Frame=0x11028f68, Base=0x00000000, Return=0x00448195
[15256] Func=runtime.write1, IP=0x00425d10, Frame=0x11028f54, Base=0x00000000, Return=0x0044069f
[15256] Func=runtime.stdcall1, IP=0x00426930, Frame=0x11028f28, Base=0x00000000, Return=0x00425e09
[15256] Func=runtime.stdcall, IP=0x00426850, Frame=0x11028f1c, Base=0x00000000, Return=0x00426963
[15256] Func=runtime.asmcgocall, IP=0x0044c400, Frame=0x11028f04, Base=0x00000000, Return=0x004268c1
[15256] Func=gosave, IP=0x0044c3c0, Frame=0x11028f00, Base=0x004b62a0, Return=0x0044c436
[15256] Func=runtime.asmstdcall, IP=0x0044d560, Frame=0x0019feac, Base=0x004b62a0, Return=0x0044c45e
[15256] Func=runtime.stdcall5, IP=0x00426a30, Frame=0x11028f28, Base=0x0000000a, Return=0x00425dbf
[15256] Func=runtime.stdcall, IP=0x00426850, Frame=0x11028f1c, Base=0x0000000a, Return=0x00426a63
[15256] Func=runtime.asmcgocall, IP=0x0044c400, Frame=0x11028f04, Base=0x0000000a, Return=0x004268c1
[15256] Func=gosave, IP=0x0044c3c0, Frame=0x11028f00, Base=0x004b62a0, Return=0x0044c436
[15256] Func=runtime.asmstdcall, IP=0x0044d560, Frame=0x0019feac, Base=0x004b62a0, Return=0x0044c45e
[15256] Func=runtime.printunlock, IP=0x0042a400, Frame=0x11028fb8, Base=0x0019feac, Return=0x00451875
[15256] Func=runtime.unlock, IP=0x00407430, Frame=0x11028fb0, Base=0x0019feac, Return=0x0042a44c
[15256] Func=runtime/internal/atomic.Loaduintptr, IP=0x00401ff0, Frame=0x11028f94, Base=0x0019feac, Return=0x00407468
[15256] Func=runtime/internal/atomic.Load, IP=0x00401f70, Frame=0x11028f94, Base=0x0019feac, Return=0x00407468
[15256] Func=runtime/internal/atomic.Casuintptr, IP=0x00401fd0, Frame=0x11028f94, Base=0x0019feac, Return=0x0040748d
[15256] Func=runtime/internal/atomic.Cas, IP=0x00401fb0, Frame=0x11028f94, Base=0x0019feac, Return=0x0040748d
[15256] Func=runtime/internal/atomic.Load, IP=0x00401f70, Frame=0x11028fc4, Base=0x0019feac, Return=0x0042b1ff
[15256] Func=runtime/internal/atomic.Load, IP=0x00401f70, Frame=0x11028fc4, Base=0x0019feac, Return=0x0042b24e
[15256] Func=runtime.exit, IP=0x00425cc0, Frame=0x11028fc4, Base=0x0019feac, Return=0x0042b262
[15256] Func=runtime.lock, IP=0x00407250, Frame=0x11028fb4, Base=0x0019feac, Return=0x00425cd1
[15256] Func=runtime/internal/atomic.Casuintptr, IP=0x00401fd0, Frame=0x11028f8c, Base=0x0019feac, Return=0x004072ac
[15256] Func=runtime/internal/atomic.Cas, IP=0x00401fb0, Frame=0x11028f8c, Base=0x0019feac, Return=0x004072ac
[15256] Func=runtime/internal/atomic.Store, IP=0x00402150, Frame=0x11028fb4, Base=0x0019feac, Return=0x00425ce7
[15256] Func=runtime.stdcall1, IP=0x00426930, Frame=0x11028fb4, Base=0x0019feac, Return=0x00425cfd
[15256] Func=runtime.stdcall, IP=0x00426850, Frame=0x11028fa8, Base=0x0019feac, Return=0x00426963
[15256] Func=runtime.asmcgocall, IP=0x0044c400, Frame=0x11028f90, Base=0x0019feac, Return=0x004268c1
[15256] Func=gosave, IP=0x0044c3c0, Frame=0x11028f8c, Base=0x004b62a0, Return=0x0044c436
[15256] Func=runtime.asmstdcall, IP=0x0044d560, Frame=0x0019feac, Base=0x004b62a0, Return=0x0044c45e
[15256] Thread with tid 14576 exited
[15256] Thread with tid 15136 exited
[15256] Thread with tid 6972 exited
[15256] Thread with tid 2728 exited
[15256] Func=runtime.nanotime1, IP=0x0044d930, Frame=0x10f0fee0, Base=0x1102c1e0, Return=0x0043407d
[15256] Thread with tid 13372 exited
[15256] Thread with tid 14472 exited
[15256] Thread with tid 14972 exited
[15256] exited

Notes

Go Versions

Compile Executables

Hello World

package main
import "fmt"
func main() {
    fmt.Println("Hello, World!")
}

Compile 64 Bit

OOS=windows GOARCH=amd64 go build -o ~/hello-stripped-64.exe helloworld.go

Compile 32 Bit

OOS=windows GOARCH=386 go build -o ~/hello-stripped-64.exe helloworld.go

Compile 64 Bit Stripped

OOS=windows GOARCH=amd64 go build -o ~/hello-stripped-64.exe -ldflags="-s -w" helloworld.go

Compile 32 Bit Stripped

OOS=windows GOARCH=386 go build -o ~/hello-stripped-64.exe -ldflags="-s -w" helloworld.go

Moduledata

Via Say Hello to Moduledata Note

moduledata can be located in the .text or .data sections
Is present in stripped binaries. As long as "-s -w" is correct for compiling stripped binaries.
Defined in symtab.go
pclntable a table that holds mappings between source code line numbers and the program counter.
Used for stack traces. Not removed if stripped.
Includes the full file path and name of the source file at compile time and the name of the function
ftab or functab short for function tab. Used by the runtime function FuncForPC
PTR size is stored in the binary.

func moduledataverify1(datap *moduledata) {
	// See golang.org/s/go12symtab for header: 0xfffffffb,
	// two zero bytes, a byte giving the PC quantum,
	// and a byte giving the pointer width in bytes.

Don't need to check the bit version.

.text:00520D60 runtime_pclntab dd 0FFFFFFFBh           ; DATA XREF: .data:runtime_firstmoduledata↓o
.text:00520D64         db 0
.text:00520D65         db 0
.text:00520D66         db    1
.text:00520D67         db    4

.symtab

Symbol table information about.
Functions and global variables.
Regardless of -g compile switch.
Every relocatable object file.
Has a symbol table in .symtab.
The symbol table in .symtab.
No entries for local variables.
The symbol table inside a compiler.
Does have entries for local variables.

Parsing Coff from symtab

n_name has another check. If the first four bytes are null (00 00 00 00) then the last four byte are an offset into the string table.
The start of the string table can be found by FileHeader.PointeToSybolTable + (FileHeader.NumberOfSymbols * 18).
In ELF executables there is another section named .strtab that appears to be similar.

stucture of coff table, well kind of
{
    char		n_name[8];	/* Symbol Name */
    long		n_value;	/* Value of Symbol */
    short		n_scnum;	/* Section Number */
    unsigned short	n_type;		/* Symbol Type */
    char		n_sclass;	/* Storage Class */
    char		n_numaux;	/* Auxiliary Count */
}

source: https://wiki.osdev.org/COFF#String_Table

Stripped Binaries

Stripped binaries for Go Windows executables can be identified by traversing the section names.
If "/" is present in a section name then the binary is not stripped.
If the section .symtab is filled with null (\x00) bytes the binary is stripped.

alexander-hanel/gopep

gopep

functab

filetab

itabs

.symtab symbols

stripped

packed

Hashes

Clustering

Command line

go_logger.py

Log Ouput

Notes

Go Versions

Compile Executables

Compile 64 Bit

Compile 32 Bit

Compile 64 Bit Stripped

Compile 32 Bit Stripped

Moduledata

.symtab

Parsing Coff from symtab

Stripped Binaries

Resources