shalzuth/ff7ecapi

Datamining ff7 evercrisis - clientless api & dumper

ff7ecapi

ff7ecapi is an educational repository for teaching how to statically dump assets and to communicate over the network of an example mobile app. This isn't for beginners. Nor a tutorial. But rather an explanation of my thinking process.

High level process

To understand how to retrieve assets and communicate over the network, the first step is always obtaining the app binary code, and then start sniffing the requests if possible, as well as examining the app code. I have broken out the solution into multiple projects to better segregate each step to reversing.

1. Download the APK.
   1. I do this in the ApkDumper project. I use qoo, you'll have to fill in the blanks for the token/etc on the download part.
   2. The app is a Unity app after quick inspection (it has libil2cpp.so in the apk).
   3. Use il2cpp dumper to dump out the metadata (or cpp2il to dump out regenerated psuedocode).
   4. In this case, the global-metadata.bin is encrypted. It's a simple xor as seen in the libil2cpp.so, so we can statically dump it out from the .so

      ff7ecapi/ApkDumper/Unity.cs

      Lines 28 to 41 in 64d1a65

      	public static byte[] ExtractXor()
      	{
      	var disassembler = CapstoneDisassembler.CreateArm64Disassembler(Gee.External.Capstone.Arm64.Arm64DisassembleMode.Arm);
      	var armCode = File.ReadAllBytes("libil2cpp.so");
      	var sigscan = new SigScanner(armCode);
      	var addr = sigscan.FindPattern("0A 19 40 92 6B 6A 68 38"); // search "Metadata" in the file
      	var xorLocs = disassembler.Disassemble(armCode.Skip(addr - 12).Take(12).ToArray());
      	var pageAddress = ((addr - 12 + xorLocs[0].Address - 0x1000) & 0xfffff000) - 0x1000;
      	var pageOffset = int.Parse(xorLocs[0].Operand.Split("0x")[1], NumberStyles.HexNumber);
      	var absOffsetl = int.Parse(xorLocs[2].Operand.Split("#").Last().Replace("0x", ""), NumberStyles.HexNumber);
      	var s = (int)pageAddress + pageOffset + absOffsetl;
      	var xor = armCode.Skip(s).Take(128).ToArray();
      	return xor;
      	}

2. Inspect the network & file structures. Usually, these are very closely linked together. Opening up the dummy dll's from il2cpp, we can usually see what frameworks they use - I use dnSpy generally. Protobuf is very common, and used here. MessagePack and FlatBuffer are also common ones, and MessagePack is used here for the files.
   1. Dump out Protobufs. I have been using SteamKit's ProtobufDumper recently, it works the best. There are many ways to skin a cat, but I was able to dump out the raw protobins by finding the static constructor address with the dummy dll, then going to that location in the binary to dump out the base64 encoded raw protobin. See https://github.com/shalzuth/ff7ecapi/blob/main/PbDumper/Dumper.cs
   2. Dump out MessagePack. I ended up just going through each class in the dummy dll's, and recreating the classes that way. https://github.com/shalzuth/ff7ecapi/blob/main/MpDumper/Dumper.cs . Initially, I used that methodology for the protobufs, but it wasn't nearly as accurate as dumping out the raw protobins.
   3. The files using MessagePack have some protection. After trying multiple routes, the route that proved successful was checking out other games made by the same developer to see if anyone else has done something. Nier is made by the same devs Applibot, and there's a massive project in reversing it, so I compared and found that it's mostly the same as https://github.com/NieR-Rein-Guide/nier-rein-apps/blob/6d1e6f717b5a5ea90e3e5583184d2b2fb8766d19/src/NierReincarnation.Datamine/Command/Data/ExportAssetsCommand.cs, but the maskLen was changed from 0xbb to 0xab (found via inspection in IDA via the dummy dll address).
3. Inspect the network traffic to figure out how to use the above structures
   1. I usually create a proxy or some hook to watch the traffic live, to inspect what the payload looks like and how to recreate it. In this case, it was a simple REST API over TCP. Fiddler is usually my goto tool for this, but I have been using Titanium more recently.
   2. Copy out the headers - note, different stages of the auth and general state machine use different headers.

      ff7ecapi/Utils/HttpRequest.cs

      Lines 90 to 115 in 64d1a65

      	requestMessage.Headers.Add("Accept", "application/protobuf");
      	requestMessage.Headers.Add("Accept-Encoding", "lz4");
      	requestMessage.Headers.Add("Accept-Language", "en-US,en;q=0.9");
      	requestMessage.Headers.Add("User-Agent", "FF7EC.96/32 CFNetwork/1410.0.3 Darwin/22.6.0");
      	if (data.GetType().Name.EndsWith("BattleEndRequest"))
      	{
      	throw new Exception(":)");
      	}
      	requestMessage.Headers.Add("x-country-code", "US");
      	requestMessage.Headers.Add("x-battle-server-area-id", "1");
      	requestMessage.Headers.Add("x-content-encoding-secure", "1");
      	requestMessage.Headers.Add("x-device-name", "iPhone12,1");
      	requestMessage.Headers.Add("x-advertising-id", "00000000-0000-0000-0000-000000000000");
      	requestMessage.Headers.Add("x-platform-type", "1");
      	//requestMessage.Headers.Add("x-adjust-id", ""); // snip
      	requestMessage.Headers.Add("x-device-id", DeviceId.ToString().ToUpper());
      	requestMessage.Headers.Add("x-terminal-id", DeviceId.ToString().ToUpper());
      	requestMessage.Headers.Add("x-os-version", "iOS 16.6.1");
      	requestMessage.Headers.Add("X-Unity-Version", "2021.3.16f1");
      	requestMessage.Headers.Add("x-app-version", "1.3.20");
      	requestMessage.Headers.Add("x-language", "en");
      	requestMessage.Headers.Add("x-server-master-version", "1696816297");
      	requestMessage.Headers.Add("x-accept-encoding-secure", "1");
      	requestMessage.Headers.Add("x-keychain-user-id", "123456789123456789");
      	if (ResponseHeaders.ContainsKey("X-Master-Path")) xmasterpath = ResponseHeaders["X-Master-Path"].First();
      	requestMessage.Headers.Add("x-master-path", xmasterpath);

   3. The payload is compressed and encrypted (this is usually the case these days). Sift through the dummy dll's to see if you can find any compress/crypt functions, then look at the base address. If you used Cpp2il, there's a chance you can read what the code does - but right now, it's still not advanced enough. So open up IDA/Ghidra/etc., go to the base address of the crypto/compession functions, and start having fun. See https://github.com/shalzuth/ff7ecapi/blob/main/Utils/Compresso.cs from compression, and https://github.com/shalzuth/ff7ecapi/blob/main/Utils/Crypto.cs from encryption.
   4. There's also a ton of API endpoints - we definitely do not want to write them all out, so parse out all of the endpoints. I used a combination of the dummy dll's to get addresses via Mono.Cecil, and Gee to dump out the strings

      ff7ecapi/PbDumper/Dumper.cs

      Lines 121 to 150 in 64d1a65

      	foreach (var a in apiClasses)
      	{
      	var cctor = a.GetConstructors().First();
      	var cctorOffset = int.Parse(cctor.CustomAttributes.First(a => a.AttributeType.Name == "AddressAttribute").Fields.First(f => f.Name == "Offset").Argument.Value.ToString().Substring(2), System.Globalization.NumberStyles.AllowHexSpecifier);
      	var func = disassembler.Disassemble(armCode.Skip(cctorOffset).Take(0x1000).ToArray());
      	var area = func.SkipWhile(a => a.Mnemonic != "strb").ToArray();
      	var b64 = "";
      	var reqType = a.Properties.First(p => p.Name == "RequestParameter").PropertyType.Name;
      	var respType = ((GenericInstanceType)a.Methods.First(m => m.Name == "RequestAsync").ReturnType).GenericArguments.First().Name;
      	for (var i = 0; i < area.Length; i++)
      	{
      	if (area[i].Mnemonic == "adrp" && area[i + 1].Mnemonic == "adrp" && area[i + 2].Mnemonic == "ldr" && area[i + 1 + 2].Mnemonic == "ldr")
      	{
      	var pageAddress = ((cctorOffset + area[i].Address - 0x1000) & 0xfffff000) - 0x1000;
      	var p2 = disassembler.Disassemble(armCode.Skip(cctorOffset + (int)area[i + 0].Address - 0x1000).Take(0x20).ToArray())[0];
      	var p3 = disassembler.Disassemble(armCode.Skip(cctorOffset + (int)area[i + 2].Address - 0x1000).Take(0x20).ToArray())[0];
      	//var pageOffset = int.Parse(area[i].Operand.Split("0x")[1], System.Globalization.NumberStyles.HexNumber);
      	var pageOffset = int.Parse(p2.Operand.Split("0x")[1], System.Globalization.NumberStyles.HexNumber);
      	var absOffsetl = int.Parse(p3.Operand.Split("#").Last().Replace("0x", "").Replace("]", ""), System.Globalization.NumberStyles.HexNumber);
      	var s2 = (int)pageAddress + pageOffset + absOffsetl;
      	var s = BitConverter.ToInt32(armCode, s2 - 0x1000);
      	var url = stringLiterals[s];
      	apiSb.AppendLine(" public async Task<" + respType + "> " + reqType.Replace("Request", "") + "(" + reqType + " request)");
      	apiSb.AppendLine(" {");
      	apiSb.AppendLine(" return await http.ApiRequestAsync<" + respType + ">(apiUrl + \"" + url + "\".Replace(\"{0}\",userId.ToString()), request);");
      	apiSb.AppendLine(" }");
      	break;
      	}
      	}
      	}

4. Putting it all together.
   1. Download assets over the web. This makes it so you don't have to download a new APK or wait for the app to be updated on the app store/etc. You can use the same in-app content upgrade process. See https://github.com/shalzuth/ff7ecapi/blob/main/NetDumper/Dumper.cs
   2. Dump game images. I gutted out this part, but I used AssetsTools to open the Unity files to dump out the downloaded files from DumpOcto
   3. Login to your account and dump out your account info

      ff7ecapi/ff7ecapi/Program.cs

      Lines 43 to 45 in 64d1a65

      	var login = await api.PostAuthSession(new PostAuthSessionRequest { DeviceUuid = accountInfo.DeviceUuid.ToLower(), LoginToken = accountInfo.LoginToken });
      	var title = await api.PostPvtUserTitle(new PostPvtUserTitleRequest { });
      	Console.WriteLine(api.http.UserData);

   4. Search other players

      ff7ecapi/ff7ecapi/Program.cs

      Lines 52 to 53 in 64d1a65

      	var profileResult = await api.PostPvtProfileGet(new PostPvtProfileGetRequest { DisplayUserId = userId });
      	Console.WriteLine(profileResult);

   5. Dump out announcements

      ff7ecapi/ff7ecapi/Program.cs

      Lines 59 to 71 in 64d1a65

      	var announcements = await api.PostAnnouncementList(new PostAnnouncementListRequest
      	{
      	AnnouncementTypeList = { new List<AnnouncementType> { AnnouncementType.Event, AnnouncementType.Gacha, AnnouncementType.Campaign, AnnouncementType.Sale, AnnouncementType.Info,
      	AnnouncementType.Bug, AnnouncementType.Update, AnnouncementType.Maintenance, AnnouncementType.DataUpdate, AnnouncementType.ImportantInfo, AnnouncementType.Topic} },
      	Limit = 2,
      	Offset = 0
      	});
      	foreach (var announcement in announcements.AnnouncementSummaryList)
      	{
      	var detailedInfo = await api.PostAnnouncementDetail(new PostAnnouncementDetailRequest { Id = announcement.AnnouncementId });
      	Console.WriteLine(announcement);
      	Console.WriteLine(detailedInfo);
      	}

5. The gray area
   1. I have intentionally excluded how to do any in-game automation that could have a negative impact on the game, which requires some additional encrypted steps, and different network protocols. No asking about this.

Support me

Currently under employed - hire me! Email me at githubusername @gmail.com, or discord eth0logy

Legal stuff

MIT License

Shalzuth isn't endorsed by SQUARE ENIX CO., LTD. or Applibot Inc. and doesn’t reflect the views or opinions of SQUARE ENIX CO., LTD. or Applibot Inc. or anyone officially involved in producing or managing SQUARE ENIX CO., LTD. or Applibot Inc. Game content and materials are trademarks and copyrights of SQUARE ENIX CO., LTD. or Applibot Inc.

https://github.com/SteamRE/SteamKit/blob/master/LICENSE - I use ProtobufDumper