Mnemosyne

About

Mnemosyne has three main objectives:

Provide immutable persistence for hpfeeds.
Normalization of data to enable sensor agnostic analysis.
Expose the normalized data through a RESTful API.

Channels

Mnemosyne currently supports normalization of data from the following channels:

dionaea.capture
mwbinary.dionaea.sensorunique
kippo.sessions
glastopf.events
glastopf.files
thug.events
thug.files
beeswarm.hive

Preliminary REST API

Can be found at http://johnnykv.github.com/mnemosyne/WebAPI.html

Speciality services

Mnemosyne is currently serving a speciality service, which on one side collects live dorks from the Glastopf hpfeed, and on the other side correlate the collected data, which allows virgin Glastopf instances to bootstrap themselves by using a Mnemosyne service.

Example queries with curl

Login to the mnemosyne webservice with the provided credentials

curl -k -c cookies.txt -X POST -d "username=james&password=bond" https://mnemosyne.honeycloud.net:8282/login
[~]$ cat cookies.txt 

mnemosyne.honeycloud.net  FALSE	/	TRUE	2147487247	beaker.session.id	1f7x19deadbeef8f802fbabe18f1f01a

Malicious websites

Searching for malicious websites in the .ru TLD using a regex. The result of this query is list of urls and hashes of binaries extracted from those urls. At the moment most data in /urls is generated by Thug.

curl -k -b cookies.txt "<...>/api/v1/urls?url_regex=\.ru(\/|\:|$)" | python -mjson.tool

{
    "url": "http://xxxyyy.ru:8080/forum/links/django_version.php",
    "_id": "510c35bfc6b6082a30d50bba", 
    "extractions": [
        {
            "hashes": {
                "md5": "f3a80b8b26579a4bfc591f834c767d15", 
                "sha1": "22ff183597009e64ffc3d93f5bccfdb214d6a4bd", 
                "sha512": "63cc541a0743c95037da2fe86cfddc881ad0128665171c50958142f2d8b87a3e90c3085286f774aceb15e28d515ed7b24e399b19579f42da894da206945fe023"
            }, 
            "timestamp": "2013-01-11T15:57:24.107000"
        }, 
        {
            "hashes": {
                "md5": "b33310b69e12427d09e0123c385b3d47", 
                "sha1": "aed4552dfe85ed33f0b77a28abc48c9f831623c5", 
                "sha512": "ab2e1edb49a864c43864f432b0f18a4f097e90b262e4e2964814e022218ad2128a1cdb0402ee76eb75b382f53d145ae7ebd64d0bec7cd599d45ae9c799802b68"
            }, 
            "timestamp": "2013-01-11T15:57:24.107000"
        }, 
        {
            "hashes": {
                "md5": "1f17127f63c975e28710739092117676", 
                "sha1": "64da64d6d6aac36323d326282093e64c89dada40", 
                "sha512": "210055a3a8614da004d92ba6edda8222d483d5563789f9443bb9d5c06481b9674fdf0eef1929410e8c44f9d66c2c7c07e0109e98a5eb92f326a3e8130801f4e7"
            }, 
            "timestamp": "2013-01-11T15:57:24.107000"
        }
    ], 
    "hpfeeds_ids": [
        "50f028a009ce4533628b1af7"
    ]
}

Looking for files

curl -k -b cookies.txt "<...>/api/v1/files?hash=b420138b88eda83a51fea5298f72864a" | python -mjson.tool

{
    "files": [
        {
            "_id": "510c3ce5c6b6082a30d548f3",
            "content_guess": "PE32 executable (DLL) (GUI) Intel 80386, for MS Windows, UPX compressed",
            "data": "4d5a90000300000004000000ffff0000b8 <--- SNIP! --->",
            "encoding": "hex",
            "hashes": {
                "md5": "b420138b88eda83a51fea5298f72864a",
                "sha1": "0e644fc39a287e6f020ede6d6c9dd708b1a871ba",
                "sha512": "98a2110f389790b5fd66f50e26e85465b3d22662245969b1fd03025194ef7a00a928c3709b57e20d165876231cdab12d38b7ff17e5c173b6562e924dc4087d85"
            },
            "hpfeed_ids": [
                "50f3e41b09ce4533629cea00"
            ]
        }
    ]
}

Dorks

Dorks collected by Glastopf

curl -k -b cookies.txt "<...>/api/v1/aux/dorks?limit=10" | python -mjson.tool

{
    "dorks": [
        {
            "content": "/pivotx/includes/index.php", 
            "count": 716, 
            "firsttime": "2013-02-01T20:38:42+00:00", 
            "lasttime": "2013-01-14T16:20:51.504000", 
            "type": "inurl"
        }, 
        {
            "content": "/axis-cgi/mjpg/wp-content/themes/diner/timthumb.php", 
            "count": 545, 
            "firsttime": "2013-02-01T20:38:32+00:00", 
            "lasttime": "2013-01-14T16:26:03.036000", 
            "type": "inurl"
        }, 
        {
            "content": "/board/board/include/pivotx/includes/wp-content/pivotx/includes/timthumb.php", 
            "count": 493, 
            "firsttime": "2013-02-01T20:39:03+00:00", 
            "lasttime": "2013-01-14T10:55:50.197000", 
            "type": "inurl"
        },
        
        <--- SNIP --- >
        
         ]   
}

Sessions

Searching for all honeypot attacks comming from an specific source port.

curl -k -b cookies.txt "<...>/api/v1/sessions?source_port=37337" | python -mjson.tool

{
  "sessions": [
    {
          "_id": "510c2f1209ce45385d3ed584", 
          "honeypot": "dionaea", 
          "attachments": [
              {
                  "description": "Binary extraction", 
                  "hashes": {
                      "md5": "984cef500b81e7ad2f7a69d9208e64e6", 
                      "sha512": "e899155228a1d3b5ed9864a7fed944716b7b0a3061b76e0f720bf9f7f6c65c633d8fdd4799335b9d92238b4b18e8076718a87a5d7a6538fec4223f111224b5e5"
                  }
              }
          ], 
          "destination_ip": [
              "xxx.yyy.zzz.ppp"
          ], 
          "destination_port": 445, 
          "hpfeed_id": "50ec09b709ce451dac5c844e", 
          "protocol": "microsoft-ds", 
          "source_ip": "xxx.yy.xx.xxx", 
          "source_port": 37337, 
          "timestamp": "2013-01-08T11:57:43.390000"
      }, 
      {
          "_id": "510c2fbc09ce45385d3fcd16",
          "honeypot": "glastopf", 
          "destination_port": 80, 
          "hpfeed_id": "50ec74f509ce452427303b50", 
          "protocol": "http", 
          "session_http": {
              "request": {
                  "body": "", 
                  "header": "{<--- SNIP --->}", 
                  "host": "<--- SNIP --->", 
                  "url": "<--- SNIP --->", 
                  "verb": "GET"
              }
          }, 
          "source_ip": "xxx.zzz.yy.zzz", 
          "source_port": 37337, 
          "timestamp": "2013-01-08T13:28:15"
      },
      ]
}

ehackify/mnemosyne