Setting up automatic testing for existing web application
emmaxtung opened this issue · 2 comments
Hi, I'm an intern that has been testing out memlab with some of the example scenarios and CLI commands. I'm now trying use it to detect memory leaks in an existing web application. I've looked at the continuous testing guide and some of the APIs but am still having a little trouble wrapping my head around how exactly to use everything together to set up and implement an efficient system for automating this.
Would you be able to provide me with an example of the process of how automated testing might be implemented? I know that for the example scenarios they depended on a URL and action coded in w/ Puppeteer. My question here is would we have to continue doing this for all the tests, or is it possible to test for memory leaks through other ways? I have had success writing scenarios that test very specific actions on pages but it feels unrealistic to keep doing this on a larger scale, and I believe this is where the automated testing would come in?
How does memlab operate at Meta? Is it constantly running tests, do the tests have specific scenarios coded, how are the results stored, how are the results analyzed, etc. Even just the basic skeletal structure of a fully implemented example continuous test might help me to understand this more.
I'm also not sure how I should be saving the results and analyzing/presenting them with this implementation. When I was going through the example scenarios, it was fine to save it to my disk and check the results through the CLI after each test since there were only a few. However, if I'll be implementing tests on a larger scale, I'm a little unsure on what the best way to handle the storage is. For each test that is ran in these large test suites, how should the results be saved, presented, and does the user need to go through the results each time to trace the leaks back like with those example scenarios in the documentation, or is there a more efficient way?
I apologize for the long message and sorry if any of this is confusing. Please let me know if I can rephrase any of the questions. I feel like I might be missing something about how memlab is used for running large test suites on existing apps. I have mostly been looking through the classes within the API module on the website.
MemLab essentially provides a mechanism for automating the detection of memory leaks once you have established the test scenarios. Although crafting a test scenario in MemLab is no more complex than in other testing frameworks such as Jest, end-to-end automation isn't currently a built-in feature of MemLab. MemLab generates the retainer trace to show why and how a memory leak exists in the heap. User still needs to examine the trace to understand and fix the leak (like we need to debug a stack trace when an exception occurs).
If your organization already has a suite of end-to-end tests running on Puppeteer, you have a few options. One option is to wrap the scenario logic into functions that can be reused within MemLab scenarios. Another option is to create a plugin compatible with your organization's existing testing framework. This plugin can capture heap snapshots during test runs and leverage the memlab find-leaks or memlab analyze command for detecting memory leaks.
When it comes to integrating MemLab with your CI/CD pipeline, there is a guide available which should help clarify the process: https://facebook.github.io/memlab/docs/guides/integration-and-file-structure
Consider MemLab as an API that takes certain inputs and generates specific outputs. How you choose to execute and collect the results at scale is entirely up to you. However, one suggestion would be to use the --work-dir flag to specify the location where you'd like to store the output of each test run. You could then create a script that gathers the output from this directory and stores it in your backend's existing datastore. You can find visual examples of potential result presentations in our engineering blog post: https://engineering.fb.com/2022/09/12/open-source/memlab/
I see, thank you for the clarification! Will definitely look into some of these options.