Out of Memory
Closed this issue · 22 comments
I have a large test lab of over 100,000 users. I ran this on a VM with 8 GB RAM and got the following:
Array dimensions exceeded supported range.
At line:578 char:17
+ $domainUserList += $domainUsers.name
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : OperationStopped: (:) [], OutOfMemoryException
+ FullyQualifiedErrorId : System.OutOfMemoryException`
This was after the CSV was created. and after the following output:
All the processes completed. Now, starting Accounts analysis..
So your tool never finishes as far I can tell. I have left it going for nearly 24 hours now.
Indeed you have a very large lab :)
How many trusted domains do you have in your scanned forest?
The ACLight is automatically scan all the domains, and therefore it make it tougher from the performance aspect.
The problem you have encountered is indeed lack of free RAM memory.
AClight scan wasn't intended to be performance efficient (it started with a POC project) - run with PowerShell and all the data is analyzed in the same time in the memory with no complicated DB in its backhand.
However, I can say that in one of our previous tests we ran the ACLight on a full forest network with 15 domains and 25,000 and it ended successfully (but we also used a 16GB RAM machine).
My recommendation to you is to try dividing the scan to individual/small domain groups, is it possible in your lab structure?
You can change the domains that will be scanned inside the code in the loop at line 565, or just close the pop up window domain scans that you don't want to scan in that specific execution.
Appreciate you tried the tool, and thanks for your feedback!
Keep me posted if you still encountered errors.
Thanks,
Asaf Hecht (@Hechtov)
@Hechtov I have two domains, sittingduck.info and research.sittingduck.info, only a few shared users between them, and research only has 30 users total.
@mubix for start could you run the scan only on "research" domain? (it should be quick because it's small domain), run the ACLight and close the pop-up window of the "sittingduck.info".
Then, if I got you right, you have the "sittingduck.info" with almost 100,000, I'm sorry but I think it will get again to memory space problem. Maybe you could add more RAM to your machine?
Another good option for you is to run the scan on less amount of "sensitive AD objects" - look at the script from line 150 - try to run the scan with only one query (for example you can remove all the lines between 160-215), and after this if you want you can scan the other sensitive AD object as well.
Same issue as above. 83k lines in the initial output file. Machine it is running on has 16gb of memory, about 12gb free to this process. I am already limiting this to one domain.
Gave a new VM 22 GB of RAM and ran out again. With no incremental or usable output, this just doesn't work. :(
We had the same OutOfMemoryException problem with ~43.000 users in the main domain and 3 trusted domains. The trusted domains have ~61.000 users in total.
Eine Ausnahme vom Typ "System.OutOfMemoryException" wurde ausgelöst.
In D:\Benutzer\XXXXX\ACLight.ps1:601 Zeichen:17
+ $domainUserList += $domainUsers.name
(Sorry for the german exception message)
@Hechtov @mubix https://msdn.microsoft.com/powershell/reference/5.1/Microsoft.PowerShell.Core/about/about_Arrays
Note: When you use the += operator, PowerShell actually creates a new array with the values of the original array and the added value. This might cause performance issues if the operation is repeated several times or the size of the array is too big.
So maybe we should use a C# list here? Here are also some performance tests:
https://powershell.org/2013/09/16/powershell-performance-the-operator-and-when-to-avoid-it/
@firefart I think you are right, thank you for mentioning this with the given links.
Sadly I used "+=" a lot.. I'm also not sure if it will be enough to just change those big arrays implementations.
I think that for running the scan in very large network I will need to consider to change the basic architecture of the scan (use organized/temporal DB or split the scanning for smaller scopes and in the end unite them together).
So, my memory never spiked over 8 GB and I was still getting the same errors. I think it's more of an issue of array size Array dimensions exceeded supported range. than memory now
@mubix so a list should handle this error too I think.
Maybe you are getting the out of memory exception because when powershell tries to add a new element to an 8GB array in memory it has to copy the array over so it would need 16GB of memory. But this highly depends on the memory management of powershell which I have no plan about :)
Will the script ever end with this error? I have one system that has been now running for > 30 hours. Need to figure out where to handle this
I don't know if it will be finished.. I think it will try to use the hardisk to extend the memory (pagefiles and so on) and it's much more slower.. I don't currently have a domain with 100,000 users, but next week I will try to build something like this and will perform more testings.
Meanwhile -
I have two quick suggestions for you to make the script much more lightweight in its memory usages:
-
The scan is querying the ACLs of multiple sensitive objects in the Active Directory (those are the multiple "Invoke-ACLScanner" calls from line 150).
Just for testing the script and reduce the memory usage you can try running the scan only on the "domain root object" - it will also give interesting result on your domain - the privileged accounts with sensitive power over the domain root + include shadow admins that might have the specific sensitive "ds-replication-get-changes-*" permissions that give the ability to perform DCSync attack.
For that, you just need convert the lines 159 until 215 to a comment section (<#.....#> will do the trick..). -
Edited: On the 18.10.17, this specific tip with the large domain users list in the end of the scan was solved and implemented into the main master version of ACLight.
Previously written: The ACLight in the end after discovering all the privileged entities from the sensitive ACEs - it queries all the users in the domain to make sure the entities it discovered are still existing in the domain and they are indeed users accounts (it reduces some false-positives).
In your cases when there are 50,000+ users it's a heavy test, try to remove this test in the end, you just need to convet the following command to "comments": lines 597, 600, 601, 602, 657 and 664.
I hope those two reductions will be sufficient to run the scan in your domains.
@Hechtov I tried the second suggestion you had and it worked! I did this on a HUGE environment. Seems to be some of the issues listed above with the maximum size of the structures you are using to do your analysis. Great tool and I look forward to incorporating this in to a few operational processes.
Edit: there was one weird issue. When it finished (the process wasn't using any memory or cpu) I had to press "enter" and then it wrapped up and output the results and gave it's "closing dialog"
Thank you!
@nwdean Thanks for the feedback! I'm happy it worked better.
The next tool will be more memory efficient :)
@Hechtov
I might be mistaken but shouldn't it be $domainUserList += $userAccount.name instead of $domainUserList += $domainUsers.name ?
The problem is that you are loading all of the names at each loop, which causes the system to run out of memory.
@guios91 You are absolutely right! I changed it. Thanks
Now, It will be interesting to hear from people with 50,000+ domain users about the performance improvement of that change.
@Hechtov Happy to help.
After modification the scripts works well and takes no more than an hour or so for 30 000 users in one domain.
I've grabbed a fresh copy of the script, but still getting the issue where it never completes. Reasonably large domain, around 45K enabled accounts, 85K in total. It creates the first file (full output) at 275MB, and the cmd window shows that it's starting the Accounts Analysis, but never gets any further.
Thanks @dakers1980 for your feedback. In the upcoming future I'm going to publish an improved version of the ACLight scan that will be better in its results and overall performance. Meanwhile you can reduce the size of scan by removing few ACL queries - i.e. remove lines from 158 until 178.
Hi @mubix, @nwdean, @firefart , @guios91 and @dakers1980, I have uploaded the new and improved ACLight scan version (currently it's under the inner "ACLight2" folder).
Can you please check the new version in your networks and send me your feedback.
It should be more efficient and the architecture of the analysis is better.
Thanks
Hi Guys,
@mubix , @nwdean , @firefart , @guios91 , @dakers1980 ,
Could you please try ACLight2? The second version has much better performances.
I just want to close this opened issue ticket, and update the main version of the tool to its updated second version.
Share your thoughts,
Thanks