Decreased performance on large JSON object in Swift 2.0
Closed this issue · 28 comments
The app that I'm working on consumes very large (~1 MB) JSON objects. In Swift 1.2, ObjectMapper was able to map it in < 5s. In Swift 2.0, it is taking > 50s.
This occurs in older versions of ObjectMapper and the latest. The difference seems to be Swift 1.2 versus Swift 2.0.
Hey @jrpdrummer, this is concerning. Off the top of my head I'm not sure what could be causing this slow down.
In your build settings, have you tried setting the "Optimization Level" to "Fast, Whole Module Optimization"?
Hi @tristanhimmelman, I have tried Whole Module Optimization. That reduced the time to ~35s, but it is still far from the 5s we were getting before.
I'm experiencing something similar I think the ArrayTransform is taking up a lot of time.
@wieseljonas Our problems had to do with mapping arrays, too. Ultimately, we resorted to converting the NSDictionary directly to our models to get the performance we needed. Using [AnyObject?] is slower than [MyModel]. I wonder what changed between Swift 1.2 and Swift 2.0 that caused the performance issue.
So basically you have a transform for each model type? Thanks for the feedback
I'm interested in more clarification as well... If have a minute could you post an example of what you are explaining to provide more insight. Thanks!
@wieseljonas @tristanhimmelman No problem. We're not using ObjectMapper at the moment; we wrote our own mappers to deal with the performance issue. Perhaps seeing what we did will give you some ideas for improving performance and we can switch back.
We got the idea that the mapping to arrays of [AnyObject?] was a problem from the Instruments Time profiler. I would definitely give Time profiler a shot to try and locate any hotspots.
Here is a stripped down version of two of our mappers (one containing an array of the other):
final class TocJSON {
private(set) var TocTree: [TocSectionJSON] = [TocSectionJSON]()
private init() { /* Force use of create() */ }
static func create(json: NSDictionary) -> TocJSON {
let tocJSON = TocJSON()
if let TocTree = json["TocTree"] as? NSArray {
for tocTree in TocTree {
if let tocTreeJson = tocTree as? NSDictionary {
tocJSON.TocTree.append(TocSectionJSON.create(tocTreeJson))
}
}
}
return tocJSON
}
}
final class TocSectionJSON {
private(set) var Title: String = ""
/* ... */
private init() { /* Force use of create() */ }
static func create(json: NSDictionary) -> TocSectionJSON {
let tocSectionJSON = TocSectionJSON()
if let Title = json["Title"] as? String {
tocSectionJSON.Title = Title
}
/* ... */
return tocSectionJSON
}
}
Hi sorry for the delay here is the array mapper I use.
class ArrayTransform<T:RealmSwift.Object where T:Mappable> : TransformType {
typealias Object = List<T>
typealias JSON = Array<AnyObject>
func transformFromJSON(value: AnyObject?) -> List<T>? {
let result = List<T>()
if let tempArr = value as! Array<AnyObject>? {
for entry in tempArr {
let mapper = Mapper<T>()
if let model = mapper.map(entry) {
result.append(model)
}
}
}
return result
}
func transformToJSON(value: List<T>?) -> Array<AnyObject>? {
if (value?.count > 0)
{
var result = Array<AnyObject>()
for entry in value! {
result.append(Mapper().toJSON(entry))
}
return result
}
return nil
}
}
I committed a performance fix yesterday, please pull the latest. Hopefully this helps y'all
@tristanhimmelman gave it a few tries. Didn't seem to improve much. I will investigate
@tristanhimmelman Any update(s) on this issue? I'm also facing the similar perf issue when I'm upgraded to Swift 2.x .
@manojmahapatra, unfortunately no, I have not made any progress on this issue. I personally did not experience the change in performance as most of the JSON files that I parse are relatively small. If someone could post a JSON file and the corresponding swift models that would help me in recreating the issue and finding performance issues.
@tristanhimmelman, I wish I could post the JSON which I'm retrieving but due to privacy I can't and to mock such a large (~1.5MB) JSON object will take time.
In the mean time, I did debug few classes and in Mapper.swift class, I found something interesting.
public func map(JSONDictionary: [String : AnyObject]) -> N? {
let map = Map(mappingType: .FromJSON, JSONDictionary: JSONDictionary)
// check if N is of type MappableCluster
if let klass = N.self as? MappableCluster.Type {
if var object = klass.objectForMapping(map) as? N {
object.mapping(map)
return object
}
}
if var object = N(map) {
object.mapping(map)
return object
}
return nil
}
If I change the above MappableCluster check with this,
public func map(JSONDictionary: [String : AnyObject]) -> N? {
let map = Map(mappingType: .FromJSON, JSONDictionary: JSONDictionary)
let object = N(map)
return object
}
It did the trick and started working as expected and this time map took only ~2-3 seconds as against >16seconds. I also tried w/ even larger JSON objects (>2MB) and the map function still clocks around ~2-3 seconds.
The changes what I did, seems to start woking again for large JSON objects(did some benchmarking on Swift 1.2 vs Swift 2.x and I'm seeing the same response time and performance). If you could shed some lights that'll be helpful.
Let me know if it helps.
I'll attach an object for 20 000 elements later today. I have to parse it one by one because client needs a progress bar. It takes about 20 minutes to complete mapping & save the object in Realm.
Here you go.
json.txt
Thanks @maksTheAwesome
Interesting @manojmahapatra. Are you calling func mapping from within your init function? I'm guessing you are since in your example the mapping function is no longer present. This would mean that all mappings were being triggered twice which would certainly decrease performance. The init? function is mainly intended for validation of the incoming map object, not to perform mapping.
When I added MappableCluster support with the following snippet
if let klass = N.self as? MappableCluster.Type {
if var object = klass.objectForMapping(map) as? N {
object.mapping(map)
return object
}
}
I tested performance and I didn't see any significant changes. Also, I believe that this ticket was opened before I even added support for MappableCluster, so I'm not sure whether this is the culprit.
I will do some more testing though to see if I can figure anything else out.
I have just pushed a fix to master which has given me a 40-50% speed increase.
After some investigation, I found a major performance issue at the following line:
https://github.com/Hearst-DD/ObjectMapper/blob/master/ObjectMapper/Core/Map.swift#L70
For every single mapping, we were creating an ArraySlice object with the results of key.componentsSeparatedByString("."). This was evidently a big bottleneck and definitely an unnecessary operation.
I have now added a simple check in the subscript function which greatly improved performance.
Furthermore, I have another potential change in mind which could improve performance another ~10%. Instead of checking all mapping keys for a . to determine if they are nested, I may force developers to pass in a flag to let ObjectMapper know the mapping in question is nested.
So,
name <- map["user.name"]
would become
name <- map["user.name", nested: true]
Since this is a breaking change, I will hold off on pushing this for the moment.
Please test and let me know if this helps in your situation.
@tristanhimmelman, I can confirm there is a performance surge (~ 1 seconds) as per my benchmarks w/ the new version. Thank you for the changes.
To my original question, I was calling mapping function inside init? when my code was in Swift 1.2 and with Swift migration to 2.x and Pod update, I totally forgot to remove that function call in my model objects. My bad!! Thank you again for pointing to that.
@manojmahapatra thats great. Just to be clear, is there an improvement of ~1 second, or does the test which you performed above now take ~1 second?
@tristanhimmelman I see an improvement of ~1 second. Sorry for any confusion 🐼
Hi there, I have the same issue. I have a huge Json object coming from my server, and it takes like 10 seconds to parse it.
When will this fix be released ? (I'm on 1.1.2)
BTW : Thanks for this lib, making my code pretty clean ;)
@mathieudebrito I just released 1.1.3 with the fix mentioned above. Please let me know if you see an improvement
@tristanhimmelman Sorry for the delay, got a lot of work to do.
So here are the results of my tests :
v1.1.2
large : (7min 20sec)
semi-large (4min 20 sec)
short (40sec)
v1.1.5
large : (6min 36sec)
semi-large (3min 40 sec)
short (28sec)
This proves that the job you've done paid : each time has decreased.
But this is way not enough for me, I will be working on it from tomorrow.
I will let you know my findings !
PS : if you have a version ready with nested keyword, I would be happy to test !
Thank you again
@tristanhimmelman Ok, turns out the problem was not in ObjectMapper but with a third-part extension.
The results are now :
large : (4 sec)
semi-large : (2 sec)
short: (0.7 sec)
As I tested with pretty about 1Mo of data for the large, it appears to me that the lib has no more efficiency issue, so I propose to close this issue.
@mathieudebrito thanks a lot for the testing 👍
@mathieudebrito I am curious, which third-party library caused this issue. Would you mind to name & blame? Before other developers have the same problem, good to know what combinations to avoid..
@winkelsdorf it actually had nothing to do with ObjectMapper, it was a third-party extension I was using to decode some encoded strings sent by our server. The extension was calling a specific function way more than necessary : this was spending a lot of time initializing and was causing big memory consumption. I sent them a message to fix it and they did. So I don't want to name & blame anyone here : this is part of open-sources projects ;)
But FYI, I used XCode Developper Tools -> Instruments -> Time Profiler to track times spent in functions
@mathieudebrito Thank you for the clarification! Good to see that this issue was resolved for you 👍