csirtgadgets/massive-octo-spice

cif-smrt performance

Nibor62 opened this issue · 6 comments

Nibor62 commented

Hi,

I am currently testing CIF (master branch) and I have issues with cif-smrt. Its updates are quite slow and with source like Alienvault I get an error after ~15min while CPU usage stay really low (<10%). After Alienvault fails, all the other feeds goes into the same error (509).
I investigated a bit and it seem that starman is doing a lots of requests in background.
So,

  • Is it a known issue ?
  • I have seen that "enable_metadata" may disable somethings but I haven't found any documentation about it. And I still have a lot of requests (mostly AS number) getting sent. What is "enable_metadata" doing exactly ?
  • Is there any way to disable completely post working on data ?
wesyoung commented
  • it's a known feature, resolving things like ASN and Geo info for each IP. if you tcpdump the network traffic you should see lots of activity
  • enable_metadata isn't really that doc'd because not a lot of people use it, disabling it should get you the desired result (a default we're looking at changing on v3). it's very useful information, but we've seen with smaller deployments not everyone wants that type of data in their instance (or cares).
Nibor62 commented

Thank for the answer,

Actually I don't need this kind of information but putting enable_metadata in the conf doesn't seem to stop ASN request (still see them passing through with tcpdump). How am I supposed to disable it ?

wesyoung commented

iirc (it's been a while); try modifying /etc/cif/cif-starman.conf to look like:

{
#       "tlp_map" => {
#               "red" => "restricted",
#               "amber" => "privileged",
#               "green" => "limited",
#               "white" => "public"
#       }
         "enable_metadata" => '0',
}

and restart cif-starman.

this should be picked up here:

massive-octo-spice/src/lib/CIF/REST.pm

Line 68 in a6994e8

enable_meta_data => (defined($self->config('enable_metadata'))) ? $self->config('enable_metadata') : 1

https://github.com/csirtgadgets/massive-octo-spice/blob/develop/src/lib/CIF/Client.pm#L258

and keep the plugins from being called here:

https://github.com/csirtgadgets/massive-octo-spice/tree/develop/src/lib/CIF/Meta

ref:

https://github.com/csirtgadgets/massive-octo-spice/blob/f9e85d2c1af7e8d3cf77b0f0b86595dce0bcf0a6/hacking/platforms/ubuntu/cif-starman.conf

Nibor62 commented

aarrg, this language is giving me headache !
Well, actually, at this line :

massive-octo-spice/src/lib/CIF/REST.pm

Line 68 in a6994e8

enable_meta_data => (defined($self->config('enable_metadata'))) ? $self->config('enable_metadata') : 1

You are using the name 'enable_meta_data' which is named 'enable_metadata' here:
https://github.com/csirtgadgets/massive-octo-spice/blob/develop/src/lib/CIF/Client.pm#L35

If i correctly understand the darks incantations behind moose, it is a bug isn't it ?

I have tested to rename it and it effectively prevent "_process_metadata" to get called and bring a nice performance improvement (and actually allow me to parse alienvault without error).
I still have a lot of DNS request passing through, which came from cif-worker (after cif-smrt update finished). Any configuration options for this one ?

wesyoung commented

don't remember. might be a typo that "just stuck". like i said; most instances leverage the data so those sections haven't been all that refined (at-least in v2, v3 is a bit diff by default).

the dns req's are coming from cif-worker, which ... if you don't want that data, don't start that service.

Nibor62 commented

Well it's working thank you.
However there is still an issue with enable_meta_data naming. PL incoming

  • 👍1