SRD | XNU Build Pipeline | ./example-cryptex/ | Makefile | make | m4 | configure | conftest | X86_64 | arm64e | rule debugging

Question

SRD | XNU Build Pipeline | ./example-cryptex/ | Makefile | make | m4 | configure | conftest | X86_64 | arm64e | rule debugging

xsscx opened this issue 3 years ago · 11 comments

Discussion & Analysis

Build Pipeline when using Makefile on arm64e when compared to X86_64 for the Apple SRD ./example-cryptex/ as provided at URL https://github.com/apple/security-research-device are not the same Result.

With reference to Issue #36 there are Build Pipeline anomalies between X86_64 and arm64e when using same OS and XNU.

As of June 1, 2022, the Net Result is that toybox is not properly groomed for installation to the SRD's, resulting in a UX of ssh login failure as shown in Issue https://github.com/apple/security-research-device/issues/59.

With respect to the Build Pipeline for the Apple Security Research Device and the Makefile in the provided ./example-cryptex/, there are Rule issues preventing the proper Build and Codesigning of toybox, and possibly other example Source, such as the XNU sdk-graft rules.

Additionally, Gatekeeper, CoreTrust & AMFI Endpoints must be reachable, and therefore need the typical: if exist, if success, on error Rules aded to the Makefile to write to stdout any errors with Reachability that would automagically impact the Build Pipeline.

The debugging exercise begins by comparing the top-level Makefile as provided by Apple with that of a normal Build for toybox, then backing out the net additions, like sed and the goodies section.

The recent tip-off that there was a Build Pipeline Issue was the time-to-build metric which added many seconds to the process and the visual graph of buildtime showed toybox constantly Re-building, always following a:

make all

when CWD == ./example-cryptex/.

Additionally, the second tip-off to a Build Pipeline error was the Codesigning error:

cryptexctl: mach-o is not signed: /private/var/folders/.../usr/bin/toybox

Workaround

https://github.com/xsscx/srd/blob/main/srd_tools-24.100.3/example-cryptex/june_1_2022-daily-build-fixup.sh

REMOVING the Apple injected top-level Makefile rule for toybox-bin and correctly hand-rolling the toybox unstripped Build process provides continuity & stability to the Example DMG.

Debugging the Makefile, testing the potential modifications and validating with unit tests are in proccess.

Answer 1 · 2022-06-03T14:38:01.000Z

X86_64 Profile for Build Tools

Fri Jun 3 10:35:29 EDT 2022

kern.version: Darwin Kernel Version 21.5.0: Tue Apr 26 21:08:22 PDT 2022; root:xnu-8020.121.3~4/RELEASE_X86_64
machdep.cpu.brand_string: Intel(R) Core(TM) i7-8700B CPU @ 3.20GHz
...
which make
/usr/bin/make
file /usr/bin/make
/usr/bin/make: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit executable x86_64
- Mach-O 64-bit executable x86_64] [arm64e:Mach-O 64-bit executable arm64e
- Mach-O 64-bit executable arm64e]
/usr/bin/make (for architecture x86_64):	Mach-O 64-bit executable x86_64
/usr/bin/make (for architecture arm64e):	Mach-O 64-bit executable arm64e
...
codesign -dvvv /usr/bin/make
Executable=/usr/bin/make
Identifier=com.apple.dt.xcode_select.tool-shim
Format=Mach-O universal (x86_64 arm64e)
CodeDirectory v=20400 size=604 flags=0x0(none) hashes=13+2 location=embedded
Platform identifier=13
Hash type=sha256 size=32
CandidateCDHash sha256=d064ec2c7e89520f4a389422a6d5027308cbe749
CandidateCDHashFull sha256=d064ec2c7e89520f4a389422a6d5027308cbe749fa2fadc8f56ebb0e76fb290d
Hash choices=sha256
CMSDigest=d064ec2c7e89520f4a389422a6d5027308cbe749fa2fadc8f56ebb0e76fb290d
CMSDigestType=2
CDHash=d064ec2c7e89520f4a389422a6d5027308cbe749
Signature size=4442
Authority=Software Signing
Authority=Apple Code Signing Certification Authority
Authority=Apple Root CA
Signed Time=Apr 20, 2022 at 04:12:36
Info.plist entries=17
TeamIdentifier=not set
Sealed Resources=none
Internal requirements count=1 size=84
...
which m4
/usr/bin/m4
file /usr/bin/m4
/usr/bin/m4: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit executable x86_64
- Mach-O 64-bit executable x86_64] [arm64e:Mach-O 64-bit executable arm64e
- Mach-O 64-bit executable arm64e]
/usr/bin/m4 (for architecture x86_64):	Mach-O 64-bit executable x86_64
/usr/bin/m4 (for architecture arm64e):	Mach-O 64-bit executable arm64e
codesign -dvvv /usr/bin/m4
Executable=/usr/bin/m4
Identifier=com.apple.dt.xcode_select.tool-shim
Format=Mach-O universal (x86_64 arm64e)
CodeDirectory v=20400 size=604 flags=0x0(none) hashes=13+2 location=embedded
Platform identifier=13
Hash type=sha256 size=32
CandidateCDHash sha256=d064ec2c7e89520f4a389422a6d5027308cbe749
CandidateCDHashFull sha256=d064ec2c7e89520f4a389422a6d5027308cbe749fa2fadc8f56ebb0e76fb290d
Hash choices=sha256
CMSDigest=d064ec2c7e89520f4a389422a6d5027308cbe749fa2fadc8f56ebb0e76fb290d
CMSDigestType=2
CDHash=d064ec2c7e89520f4a389422a6d5027308cbe749
Signature size=4442
Authority=Software Signing
Authority=Apple Code Signing Certification Authority
Authority=Apple Root CA
Signed Time=Apr 20, 2022 at 04:12:36
Info.plist entries=17
TeamIdentifier=not set
Sealed Resources=none
Internal requirements count=1 size=84

Answer 2 · 2022-06-03T14:46:31.000Z

FUZZING

Using typical Fuzzing Harnesses to find ultra-low-hanging-fruit in the Build System.

With Bugs Already Reported in Xcode, the Instruments & Crash Reporting, this disk-image-helper Crash is just another extension of touching everything known inside the envelope to generate a Corpus of Public Domain Crash Reports.

Enjoy this piece of the puzzle being shown:

-------------------------------------
Translated Report (Full Report Below)
-------------------------------------

Process:               diskimages-helper [53014]
Path:                  /System/Library/PrivateFrameworks/DiskImages.framework/Versions/A/Resources/diskimages-helper
Identifier:            diskimages-helper
Version:               ???
Code Type:             X86-64 (Native)
Parent Process:        Exited process [53013]
Responsible:           iTerm2 [462]
User ID:               501

Date/Time:             2022-05-31 17:57:13.5179 -0400
OS Version:            macOS 12.4 (21F79)
Report Version:        12
Bridge OS Version:     6.5 (19P5071)

System Integrity Protection: enabled

Crashed Thread:        2

Exception Type:        EXC_BAD_ACCESS (SIGSEGV)
Exception Codes:       KERN_INVALID_ADDRESS at 0x00006a0997d28350
Exception Codes:       0x0000000000000001, 0x00006a0997d28350
Exception Note:        EXC_CORPSE_NOTIFY

Termination Reason:    Namespace SIGNAL, Code 11 Segmentation fault: 11
Terminating Process:   exc handler [53014]

VM Region Info: 0x6a0997d28350 is not in any region.  Bytes after previous region: 11035781268305  Bytes before following region: 6556000046256
      REGION TYPE                    START - END         [ VSIZE] PRT/MAX SHRMOD  REGION DETAIL
      MALLOC_NANO (reserved)   600018000000-600020000000 [128.0M] rw-/rwx SM=NUL  ...(unallocated)
--->  GAP OF 0xfffe7e05000 BYTES
      Stack Guard              700007e05000-700007e06000 [    4K] ---/rwx SM=NUL  

Thread 0::  Dispatch queue: com.apple.main-thread
0   libsystem_malloc.dylib        	    0x7ff80ab3dcda tiny_free_list_add_ptr + 135
1   libsystem_malloc.dylib        	    0x7ff80ab3b477 tiny_malloc_from_free_list + 1671
2   libsystem_malloc.dylib        	    0x7ff80ab3a8ad tiny_malloc_should_clear + 255
3   libsystem_malloc.dylib        	    0x7ff80ab397d2 szone_malloc_should_clear + 66
4   libsystem_malloc.dylib        	    0x7ff80ab54b75 _malloc_zone_calloc + 60
5   libobjc.A.dylib               	    0x7ff80abb01dd class_createInstance + 64
6   libdispatch.dylib             	    0x7ff80ab65a31 _os_object_alloc_realized + 25
7   libdispatch.dylib             	    0x7ff80ab90ffe dispatch_data_create_alloc + 34
8   libxpc.dylib                  	    0x7ff80aa52b3c xpc_data_create + 45
9   Foundation                    	    0x7ff80bc04a72 _NSXPCSerializationCreateWriteData + 170
10  Foundation                    	    0x7ff80bc095ab -[NSXPCEncoder _encodeInvocationObjectArgumentsOnly:count:typeString:selector:isReply:into:] + 223
11  Foundation                    	    0x7ff80bc02943 -[NSXPCConnection _sendInvocation:orArguments:count:methodSignature:selector:withProxy:] + 1864
12  Foundation                    	    0x7ff80bc0949d -[NSXPCConnection _sendSelector:withProxy:arg1:arg2:] + 132
13  Foundation                    	    0x7ff80bc093ca _NSXPCDistantObjectSimpleMessageSend2 + 63
14  diskimages-helper             	       0x1057db21b 0x1057c9000 + 74267
15  Foundation                    	    0x7ff80bc67557 __NSThreadPerformPerform + 179
16  CoreFoundation                	    0x7ff80ade419b __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ + 17
17  CoreFoundation                	    0x7ff80ade4103 __CFRunLoopDoSource0 + 180
18  CoreFoundation                	    0x7ff80ade3e7d __CFRunLoopDoSources0 + 242
19  CoreFoundation                	    0x7ff80ade2898 __CFRunLoopRun + 892
20  CoreFoundation                	    0x7ff80ade1e5c CFRunLoopRunSpecific + 562
21  Foundation                    	    0x7ff80bc45d6a -[NSRunLoop(NSRunLoop) runMode:beforeDate:] + 216
22  diskimages-helper             	       0x1057d08bb 0x1057c9000 + 30907
23  dyld                          	       0x11146d51e start + 462

Thread 1:
0   libsystem_pthread.dylib       	    0x7ff80ad18f48 start_wqthread + 0

Thread 2 Crashed:
0   libobjc.A.dylib               	    0x7ff80abb301f objc_release + 31
1   libobjc.A.dylib               	    0x7ff80abb70da AutoreleasePoolPage::releaseUntil(objc_object**) + 164
2   libobjc.A.dylib               	    0x7ff80abb3f5f objc_autoreleasePoolPop + 168
3   diskimages-helper             	       0x1057d143e 0x1057c9000 + 33854
4   Foundation                    	    0x7ff80bc3e9c4 __NSThread__start__ + 1009
5   libsystem_pthread.dylib       	    0x7ff80ad1d4e1 _pthread_start + 125
6   libsystem_pthread.dylib       	    0x7ff80ad18f6b thread_start + 15

Thread 3:
0   libsystem_pthread.dylib       	    0x7ff80ad18f48 start_wqthread + 0

Thread 4:
0   libsystem_pthread.dylib       	    0x7ff80ad18f48 start_wqthread + 0


Thread 2 crashed with X86 Thread State (64-bit):
  rax: 0x0000ea0997d28330  rbx: 0x0000000000000001  rcx: 0x00006a0997d28330  rdx: 0x00000000b42da02d
  rdi: 0x00006000013d8330  rsi: 0x0000600001dd8580  rbp: 0x0000700007f08cf0  rsp: 0x0000700007f08cb8
   r8: 0x0000000000000580   r9: 0x0000000000000020  r10: 0x00000000000007fb  r11: 0x00000000000001ff
  r12: 0x00006000013d8330  r13: 0xa3a3a3a3a3a3a3a3  r14: 0x00007f897b00b038  r15: 0x00007f897b00b000
  rip: 0x00007ff80abb301f  rfl: 0x0000000000010206  cr2: 0x00006a0997d28350
  
Logical CPU:     10
Error Code:      0x00000004 (no mapping for user data read)
Trap Number:     14

Thread 2 instruction stream:
  48 85 ff 74 76 89 f8 83-e0 01 48 85 c0 75 6c 48  H..tv.....H..ulH
  8b 07 48 b9 f8 ff ff ff-ff 7f 00 00 48 21 c1[48] ..H.........H!.H	<==
  8b 51 20 f6 c2 04 74 54-a8 01 74 72 48 b9 00 00  .Q ...tT..trH...
  00 00 00 00 80 00 48 ba-00 00 00 00 00 00 00 01  ......H.........
  48 39 c8 72 36 48 89 c6-48 29 d6 72 17 f0 48 0f  H9.r6H..H).r..H.
  b1 37 74 22 a8 01 75 e8-31 f6 ba 01 00 00 00 e9  .7t"..u.1.......
  46 cd 01 00 48 85 c8 75-3d 48 8b 35 18 91 8d 41  F...H..u=H.5...A
  ff 25 da 70 e6 3f 48 39-ce 72 ee c3 66 83 79 1c  .%.p.?H9.r..f.y.
  00 79 0e 83 e2 02 48 85-d2 74 06 f6 41 28 02 75  .y....H..t..A(.u
  1f 48 8b 35 40 90 8d 41-ff 25 b2 70 e6 3f f6 41  .H.5@..A.%.p.?.A
  1c 01 74 b4 eb d5 be 01-00 00 00 e9 4c a3 00 00  ..t.........L...
  48 8b 05 81 4c 89 41 ff-e0 90 90 90 90 90 90 90  H...L.A.........

Binary Images:
    0x7ff80ab37000 -     0x7ff80ab62fff libsystem_malloc.dylib (*) <109a9983-dd1f-3e38-aa9b-2c4da8f3529d> /usr/lib/system/libsystem_malloc.dylib
    0x7ff80abaa000 -     0x7ff80abe3fff libobjc.A.dylib (*) <a02a893a-79ff-39f2-a3f2-796b4d877b68> /usr/lib/libobjc.A.dylib
    0x7ff80ab63000 -     0x7ff80aba9fff libdispatch.dylib (*) <534511b9-b3b0-33a7-b1ea-402595d28bda> /usr/lib/system/libdispatch.dylib
    0x7ff80aa50000 -     0x7ff80aa8bfff libxpc.dylib (*) <a675716f-f789-3d56-bc4f-8ff2c94e1080> /usr/lib/system/libxpc.dylib
    0x7ff80bbe6000 -     0x7ff80bfa2fff com.apple.Foundation (6.9) <ceb9e591-a1ad-3ebc-ab8d-410f4ff96307> /System/Library/Frameworks/Foundation.framework/Versions/C/Foundation
       0x1057c9000 -        0x105814fff diskimages-helper (*) <e10d6e74-b346-3ce4-a7a5-158e0a65101d> /System/Library/PrivateFrameworks/DiskImages.framework/Versions/A/Resources/diskimages-helper
    0x7ff80ad64000 -     0x7ff80b266fff com.apple.CoreFoundation (6.9) <f8e45ef9-9fd2-3331-bb1b-703d5dacdaf1> /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
       0x111468000 -        0x1114d3fff dyld (*) <b70ce1ec-b902-3852-8268-05de00bfa8d5> /usr/lib/dyld
    0x7ff80ad17000 -     0x7ff80ad22fff libsystem_pthread.dylib (*) <bc574849-1aae-31e7-b350-916dda999d97> /usr/lib/system/libsystem_pthread.dylib
               0x0 - 0xffffffffffffffff ??? (*) <00000000-0000-0000-0000-000000000000> ???

Answer 3 · 2022-06-03T15:53:50.000Z

X86_64 Sample Build Log for toybox on FRI 3 JUN 2022

Makefile Log for Default ./example-cryptex/ at toybox rule

[toybox] - Checking for macOS SDK at /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk
[toybox] - Checking for iOS SDK at /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS15.5.sdk
[toybox] - Creating SDK header graft...
[toybox] - [%] Building toybox
[toybox] - Checking for GNU sed
// Result of Build Failure, UX == no ssh login or other undefined behavior

because of:

cryptexctl: mach-o is not signed: /private/var/folders/.../usr/bin/toybox

Confirmation of no toybox unstripped in default Build

ls -la src/toybox/toybox-src/generated/unstripped/toybox
ls: src/toybox/toybox-src/generated/unstripped/toybox: No such file or directory

Workaround

Manual Compilation of Toybox and toybox-unstripped for DMG success

cd ./example-cryptex/
make clean
make all
cd src/toybox
make clean
make all
cd ../../
make install

Build Log

[toybox] - Checking for macOS SDK at /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk
[toybox] - Checking for iOS SDK at /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS15.5.sdk
[toybox] - [%] Building toybox
[toybox] - Creating SDK header graft...
[toybox] - [%] Building toybox
[toybox] - Checking for GNU sed
// result is successful toybox build with UX == ssh login success

Confirmation of toybox unstripped build

 ls -la toybox-src/generated/unstripped/toybox
-rwxr-xr-x  1 xss  staff  398128 Jun  3 11:47 toybox-src/generated/unstripped/toybox

nm -a  toybox-src/generated/unstripped/toybox | wc -l
896

Further top-level Makefile and Rule debugging is necessary to isolate the Build Failure.

Answer 4 · 2022-06-04T11:59:16.000Z

Build Troubleshooting Overview

The SRD Build Pipeline is robust, but like all other 'live build' systems there can be transitory Network issues that prevent a successful Build & Installation.

Take for example that over US Memorial Day Weekend Holiday in 2021 there were points in time where GateKeeper, CoreTrust. AMFI and the Tatsu Signing Server were unavailable.

The UX was that the SRD ./example-cryptex/ build would fail and the Researcher was left to Diagnose the Issues.

This Troubleshooting Document is a work in progress and contributions are welcome.

The Cohort typically has ./example-crytpex/ on their local Desktop and the cryptex build experience should be scoped to:

make clean
make install
...
make clean
make install

Instead, a Review of the Closed Issues in Hoyt's SRD Repo at URL https://github.com/xsscx/srd/issues?q=is%3Aissue+is%3Aclosed would indicate there is the need for a SRD Dashboard

As an Example SRD Cryptex Build & Installation Dashboard, the "SRD Dashboard", see URL https://github.com/xsscx/srd#srd-example-dmg-build--installation-status-w--xnu-8019415-or-xnu-6153815

SRD Example DMG, Build & Installation Status w/ + XNU-8019.41.5 OR XNU 6153.81.5

Build OS & Device Info	Example DMG	debugserver DMG	ASAN DMG	UBSAN DMG
macOS 12.4 (21F79) X86_64	PASS	PASS	PASS	PASS
macOS 12.4 (21F79) T8101	PASS	PASS	PASS	PASS
X86_64 Install to iPhone 11 15.6_19G5037d	PASS	PASS	PASS	PASS
T8101 Install to iPhone 12 15.6_19G5027e	PASS	PASS	PASS	PASS

X86_64 Install with CryptexManager

The Dashboard needs to be Modified to better show the X86_64 & arm64e Build & Installation Pipelines

As an Example, this past US Memorial Day Weekend Holiday 2022 there were points in time where GateKeeper, CoreTrust. AMFI and the Tatsu Signing Server were unavailable. The considered SRD Dashboard would need real-time monitoring, the 'rtm', to Poll Apple Endpoints required for a successful Cryptex Build, and if not Available, then Indicate via SRD Dashboard.

This past Memorial Day Weekend, Hoyt indicated Build Pipeline Issues with internal MRTG Graphs showing that Toybox had long-cycle Build Time compared to its Historic Build Time. Toybox was constantly Rebuilding itself due to a Failed Build, that Toybox Unstripped was not Built, and that toybox was not Codesigned. These datapoints should be Published in Real Time for the benefit of SRDC. The top-level Makefile in ./example-cryptex/ needs additional debugging. Any contemplated Modifications to the Makefile and supporting Build & Make Files should be done by consensus. It has been shown that arbitrary injection of ungroomed & untested rules and macros by Apple are resulting in failed Builds and a UX with no ssh login to SRD.

cd ./example-cryptex/
make install

End Result was failed Toybox Build and ungroomed cryptex installation. UX may have also been .. 'failed ssh login to srd'.

Suggested Fix

Add Commercial Build Tools, Dashboard, Real Time Monitoring & Reporting so SRDC can have a visual representation of the underlaying Apple Infra required for a successful Cryptex Build & Installation.

Based on my own experience, any existing Apple System Status Dashboards that would be for Retail, or Developers, does not adequately capture the real-time status of the Build Pipeline that is required to support the build & instllation of a crytpex.

Answer 5 · 2022-06-06T10:32:39.000Z

Issue with make clean of the sdk-graft

When using arm64e or X86_64 with no specific Operating System Version, the make clean bug in the sdk-graft section of the Apple default ./example-cryptex/ issue can be seen as:

make clean
rm -rf /Users/xss/Downloads/srdm-jun5/example-cryptex/sdk-graft
rm: /Users/xss/Downloads/srdm-jun5/example-cryptex/sdk-graft: Permission denied
make: *** [sdk-graft-clean] Error 1

This 'feature" works on arm64e or X86_64 randomly.

A Directory Listing indicates the Quarantine Bit Set, and the macro does correctly delete the sub-directories, but not the top-level sdk-graft as shown below:

ls -lat sdk-graft
total 16
drwxr-xr-x   3 xss  staff    96 Jun  6 06:28 .
-rw-r--r--@  1 xss  staff  6148 Jun  6 06:28 .DS_Store
drwxrwxr-x@ 16 xss  staff   512 Jun  6 06:26 ..

This Issue prevents the Automagic Build Pipeline from function properly and additional debugging for new Rules in process.

Answer 6 · 2022-06-12T22:45:11.000Z

This issue #37 and #38 will be track seperately..

The Makefile for iOS14 is distinctly different with XNU Version(s).
The Makefile for iOS15 is distinctly different with XNU Version(s).
The Makefile for iOS16 will likely be distinct with different with XNU Version(s).

Answer 7 · 2022-07-19T00:47:52.000Z

https://github.com/apple/security-research-device/pull/67

Answer 8 · 2022-07-19T11:02:37.000Z

Note that PR 67 from Apple still doesn't Build correctly.

Continue the Toybox Fixup Script from June 1, 2022 for Build Pipeline......

Answer 9 · 2022-07-19T11:05:02.000Z

Proof:

the Fixup build toybox with this many:

example-cryptex % nm -a com.example.cryptex.dstroot/usr/bin/toybox | wc -l
932

Continue: https://github.com/xsscx/srd/blob/main/srd_tools-24.100.3/example-cryptex/june_1_2022-daily-build-fixup.sh

Answer 10 · 2022-07-20T22:54:17.000Z

The obvious tip-off that the recent PR 67 from Apple is defective.. toybox continues to rebuild itself.. over and over..
Repro:

make clean
make all
make install

watch toybox rebuild itself, and not have unstripped

Answer 11 · 2022-11-05T01:19:26.000Z

Circling back to this issue, it looks like a re-write of the Makefile stub for SRD to better integrate with the Research Build would benefit the Community. Closing out this Issue as the ToyboxUnstripped Build Pipleline is unchange for this Repo, which is still ahead of Apple.