EMU plugin missing adversaries, planners, and abilities

[th3wyatt]

Describe the bug
Adversaries, abilities and planners from CTID are not imported when emu plugin is installed.

To Reproduce
Steps to reproduce the behavior:

1. build docker image using main branch and emu enabled in default.yml

Expected behavior
CTID emulation plans available when viewing adversaries.

Screenshots

2024-08-26 20:53:30 INFO     Invalid Github Gist personal API contact_gist.py:70
                             token provided. Gist C2 contact                    
                             will not be started.                               
                    INFO     Generating temporary SSH private   tunnel_ssh.py:26
                             key. Was unable to use provided                    
                             SSH private key                                    
2024-08-26 20:53:31 INFO     Enabled plugin: emu                  app_svc.py:116
                    INFO     Enabled plugin: magma                app_svc.py:116
                    INFO     Enabled plugin: manx                 app_svc.py:116
                    ERROR    Error importing plugin=builder, No   c_plugin.py:91
                             module named 'docker'                              
                    ERROR    Error loading plugin=builder,        c_plugin.py:59
                             'NoneType' object has no attribute                 
                             'description'                                      
                    INFO     Enabled plugin: fieldmanual          app_svc.py:116
                    INFO     Enabled plugin: debrief              app_svc.py:116
                    INFO     Enabled plugin: compass              app_svc.py:116
                    INFO     Enabled plugin: response             app_svc.py:116
                    INFO     Enabled plugin: stockpile            app_svc.py:116
2024-08-26 20:53:33 INFO     Enabled plugin: sandcat              app_svc.py:116
                    INFO     Enabled plugin: access               app_svc.py:116
                    INFO     Enabled plugin: training             app_svc.py:116
                    INFO     Creating SSH listener on 0.0.0.0,     logging.py:92
                             port 8022                                          
                    INFO     serving on 0.0.0.0:2222               server.py:741
                    WARNING  Unable to properly load .donut for  data_svc.py:436
                             payload                                            
                             plugins.stockpile.app.donut.donut_h                
                             andler due to failed import                        
                    WARNING  upx does not meet the minimum        app_svc.py:171
                             version of 0.0.0. Upx is an optional               
                             dependency which adds more                         
                             functionality.                                     
2024-08-26 20:53:55 WARNING  Ability referenced in adversary   c_adversary.py:90
                             ef4d997c-a0d1-4067-9efa-87c58682d                  
                             b71 but not found:                                 
                             df94858e92a23d274ac1d70133d9150f                   
                    WARNING  Ability referenced in adversary   c_adversary.py:90
                             ef4d997c-a0d1-4067-9efa-87c58682d                  
                             b71 but not found:                                 

A count of loaded CTID emulation plan adversaries and abilities should be here

I looked here for the Sandworm Group, which i used in an older version of Caldera

Desktop (please complete the following information):

• OS: Mac
• Browser Firefox
• Version 192.0.2

[github-actions]

Looks like your first issue -- we aim to respond to issues as quickly as possible. In the meantime, check out our documentation here: http://caldera.readthedocs.io/

[elegantmoose]

Did the payload install script run successfully?

[th3wyatt]

I think so.

LICENSE  __pycache__  app  conf  data  download_payloads.sh  gui  hook.py  payloads  requirements.txt  templates  tests
root@b3aa08ab9d14:/usr/src/app/plugins/emu# ls payloads
AdFind.zip  LICENSE.txt  NetSess.zip  PSTools.zip  README       nbtscan.exe  plink.exe  psexec.exe          putty.exe        tcping.exe  wce_v1_41beta_universal.zip
Changelog   NetSess.exe  PSTools      PsExec.exe   dnscat2.ps1  netsess.exe  pscp.exe   psexec_sandworm.py  secretsdump.exe  wce.exe     wmiexec.vbs
root@b3aa08ab9d14:/usr/src/app/plugins/emu# 

[th3wyatt]

It looks like it's not pulling down the emulation repo?

root@b3aa08ab9d14:/usr/src/app/plugins/emu# ls
LICENSE  __pycache__  app  conf  data  download_payloads.sh  gui  hook.py  payloads  requirements.txt  templates  tests
root@b3aa08ab9d14:/usr/src/app/plugins/emu# ls data
adversary-emulation-plans
root@b3aa08ab9d14:/usr/src/app/plugins/emu# ls data/adversary-emulation-plans/
turla

[th3wyatt]

I figured out the issue, i think.

In EMU, the hook.py looks for the repo folder and only clones if the folder doesn't exist

if not os.path.isdir(plugin_svc.repo_dir):
        await plugin_svc.clone_repo()

But the download_payloads.sh script creates that folder when populating one of the payloads. This is done when building the container from the dockerfile

if [ "$psexec_md5" = "84858ca42dc54947eea910e8fab5f668" ]
then
    target_dir="data/adversary-emulation-plans/turla/Resources/payloads/snake"
    mkdir -p "$target_dir" && cp payloads/PSTools/PsExec64.exe $target_dir/PsExec.exe
    echo "PsExec64.exe v2.4 copied to Turla payloads directory"
else
    echo "PsExec from PSTools.zip with MD5 '$psexec_md5' does not match v2.4 with MD5 of 84858ca42dc54947eea910e8fab5f668"
fi

So the folder already exists and then the repo is not downloaded. I'm not sure if this should be fixed in the Dockerfile or in the EMU module. My workaround is to just edit the hook to run even if the folder exists.

[elegantmoose]

Correct. So did the README instructions (https://github.com/mitre/emu/tree/master?tab=readme-ov-file#installation) on Emu not work? As you shouldnt get this error if you enable Emu plugin, start Caldera, stop Caldera, run payload script, restart Caldera.

[elegantmoose]

@th3wyatt Closing. Reopen if still have issues. 👍

[mllamazares]

@th3wyatt facing the same issue, but I couldn't replicate the fix. 😞

These are the steps I followed:

1. Download MITRE Caldera's repo: git clone https://github.com/mitre/caldera.git --recursive --branch master
2. Enable the Emu plugin in the conf/default.yml file by adding - emu to the plugin list
3. Copy conf/default.yml to conf/local.yml (this is necessary because the Dockerfile only checks if Emu is enabled in local.yml, not in default.yml)
4. Copying the magma env file: cp plugins/magma/.env.template plugins/magma/.env
5. Modified plugins/emu/hook.py to comment that if statement and running await plugin_svc.clone_repo() always
6. Build the image: docker build --build-arg WIN_BUILD=true . -t caldera:server
7. Run the server docker run -p 7010:7010 -p 7011:7011 -p 7012:7012 -p 8888:8888 caldera:server --insecure
8. Access http://localhost:8888 and login with admin:admin.
9. Click on the emu tab and they are still no adversaries or abilities loaded.

Any idea?

[th3wyatt]

Correct. So did the README instructions (https://github.com/mitre/emu/tree/master?tab=readme-ov-file#installation) on Emu not work? As you shouldnt get this error if you enable Emu plugin, start Caldera, stop Caldera, run payload script, restart Caldera.

Sorry for the super late reply, but I'm building the container with custom adversaries and abilities, then deploying that custom container. download_payloads.sh runs during container build. So, when i start caldera later, it doesn't download the emu repos because the folder already was created by download_payloads.sh during the container build.

[th3wyatt]

@th3wyatt facing the same issue, but I couldn't replicate the fix. 😞

These are the steps I followed:

1. Download MITRE Caldera's repo: `git clone https://github.com/mitre/caldera.git --recursive --branch master`

2. Enable the [Emu plugin](https://github.com/mitre/emu) in the `conf/default.yml` file by adding `- emu` to the plugin list

3. Copy `conf/default.yml` to `conf/local.yml` (this is necessary because [the Dockerfile only checks if Emu is enabled in local.yml](https://github.com/mitre/caldera/blob/master/Dockerfile#L72), not in default.yml)

4. Copying the magma env file: `cp plugins/magma/.env.template plugins/magma/.env`

5. **Modified `plugins/emu/hook.py` to comment that if statement and running `await plugin_svc.clone_repo()` always**

6. Build the image: `docker build --build-arg WIN_BUILD=true . -t caldera:server`

7. Run the server `docker run -p 7010:7010 -p 7011:7011 -p 7012:7012 -p 8888:8888 caldera:server --insecure`

8. Access `http://localhost:8888` and login with `admin:admin`.

9. Click on the `emu` tab and they are still no adversaries or abilities loaded.

Any idea?

The local.yml gets created if it doesn't exist from default.yml when building the container.

RUN python3 -c "import app; import app.utility.config_generator; app.utility.config_generator.ensure_local_config();"; \
    sed -i '/\- atomic/d' conf/local.yml;

However, if you're running with --insecure, it does not use local.yml and uses default.yml, from what I understand.

--insecure: Uses the conf/default.yml file for configuration, not recommended.

I modified my workaround to remove download_payloads.sh from the dockerfile and into emu_svc.py.

caldera/app/plugins/emu/app/emu_svc.py

At line ~74 I added a call for download_payloads.sh

if not os.path.exists(self.repo_dir) or not os.listdir(self.repo_dir):
            self.log.debug('cloning repo %s' % repo_url)
            check_call(['git', 'clone', '--depth', '1', repo_url, self.repo_dir], stdout=DEVNULL, stderr=STDOUT)
            self.log.debug('clone complete')
            check_call(['bash', '/usr/src/app/plugins/emu/download_payloads.sh'], stdout=DEVNULL, stderr=STDOUT)

So, when the container is built, download_payloads.sh is not run, the folder is not created. When starting the server, hook.py looks for the folder the first time, it's not there. It downloads the repos, then downloads payloads.