GSoC Idea: Implement the Social Currency Metrics System in GrimoireLabs

Question

GSoC Idea: Implement the Social Currency Metrics System in GrimoireLabs

GeorgLink opened this issue 5 years ago · 145 comments

Idea: Implement the Social Currencyy Metrics System in Grimoire Labs

The Social Currency Metrics System (SCMS) is a qualitative data collection, processing, and measurement system that augments quantitative community management metrics already available in CHAOSS properties. Implementing the SCMS will ultimately help community leaders, power users, and other stakeholders leverage qualitative data for social listening so that they can rely less on the behaviors quantitative data tracks and more on community sentiment. The SCMS empowers community leaders to make decisions based on what community members freely share about their opinions, wants, and needs.

The SCMS shows why trends occur and identifies commonly missed pitfalls in conclusions taken from quantitative data. With an SCMS platform built natively into a CHAOSS Bitergia property, open-source communities can use it to facilitate members’ input in decisions essential to community health.

The purpose of this project is to:
Build the SCMS in one of CHAOSS project’s systems that collects qualitative data from several channels, displays that information for manipulation and tagging, and outputs it to a metric dashboard such as the SCMS information page found here.

The aims of this project are to:

Create an API that collects and returns data collected from a community of the GSoC participants’ choice.
Gain familiarity with and create a way to grade, tag, and process large amounts of data.
Develop creative ways to display complex datasets in the Bitergia Analytics System.
Investigate ways to process qualitative data at scale using AI or similar technology.
Encourage the larger development of social listening tools based on qualitative data.

Difficulty: Medium to Hard based on the level of implementation (machine learning).

Requirements: Python programming. Interest in machine learning. API Development some understanding of the social scientific process and qualitative data analysis

Recommended: Other API languages, interest in science of community management and anthropological studies of online worlds, willingness to learn CHAOSS Property Structures

Mentors: Dylan Marcy (SociallyConstructed.Online), Venia Logan (SociallyConstructed.Online), Valereo (Grimoire Labs), Georg (Advising only)

Micro-tasks

For becoming familiar with GrimoireLab, you can start by reading some documentation. You can find useful information at:

GrimoireLab Tutorial
Perceval: Software Project Data at Your Will
The GitHub repositories hosting the tools (https://github.com/chaoss/grimoirelab-perceval, https://github.com/chaoss/grimoirelab-sirmordred, https://github.com/chaoss/grimoirelab-...) and https://github.com/Bitergia/archimedes.

Once you're familiar with Grimoirelab, you can have a look at the following microtasks.

Microtask 0:
Download PyCharm and get familiar with it (for instance, you can follow this tutorial).
Microtask 1:
Set up Perceval to be executed from PyCharm.
Microtask 2:
Create a Python script to execute Perceval via its Python interface using the GitLab and GitHub backends. Feel free to select any target repository.
Microtask 3:
Based on the JSON documents produced by Perceval and its source code, try to answer the following questions:
- What is the meaning of the JSON attribute 'timestamp'?
- What is the meaning of the JSON attribute 'updated_on'?
- What is the meaning of the JSON attribute 'origin'?
- What is the meaning of the JSON attribute 'category'?
- How many categories do the GitHub and GitLab backends have?
- What is the meaning of the JSON attribute 'uuid'?
- What is the meaning of the JSON attribute search_fields?
- What is stored in the attribute data of each JSON document produced by Perceval?
Microtask 4:
Set up a dev environment to work on GrimoireLab. Have a look to https://github.com/chaoss/grimoirelab-sirmordred#setting-up-a-pycharm-dev-environment.
Microtask 5:
Execute micro-mordred to collect, enrich and visualize data from Git and GitHub repositories.
Microtask 6:
Using the dev tools in Kibiter, create a query that counts the number of unique authors on a Git repository from 2018-01-01 until 2019-01-01.
Microtask 7:
Install and use elasticdump to download the mapping and data of an ElasticSearch index (it can be anyone created in Microtask 5).
Microtask 8:
Execute micro-mordred to collect and enrich data from a groupsio repository. You need to register to a group (e.g., https://lists.onap.org/g/main) and follow the instructions at https://github.com/chaoss/grimoirelab-sirmordred#groupsio. Then, write a script to read the enriched index and import the attributes uuid, project, project_1, origin, grimoirelab_creation_date, body and subject_analyzed to a CSV file. Import the obtained file to an excel sheet (in a manual or automatic way).
Microtask 9:
Build a Data Table visualization in Kibiter (you can use the CHAOSS community dashboard) that shows for emails (mbox index) the text of emails (split row by Termbody_extract field).
Microtask 10:
Submit at least a PR to one of the GrimoireLab repositories to fix an issue, improve the documentation, etc.

abhiandthetruth commented 5 years ago

ok!

🚀1

abhiandthetruth commented 5 years ago

ok!

👍1

Answer 1 · 2020-02-22T11:00:40.000Z

For all students interested in this idea, please comment on this issue to get in touch with the mentors. This is the main communication channel.

Answer 2 · 2020-02-24T03:57:25.000Z

Hello Mentors,
I am Animesh Kumar. I am a student at NITK Surathkal. I would be interested in working on this idea. I have started working on the micro-tasks and will send a PR for the same soon.

Answer 3 · 2020-02-24T06:13:00.000Z

Thank you @animeshk08 for your interest in this idea! Don't hesitate to write here if you need feedback or help!

Answer 4 · 2020-02-24T17:26:20.000Z

Hello everyone,
I am currently working on the micro-tasks, will send the pr on completion. Hoping to have a good coding community ahead!

Answer 5 · 2020-02-25T07:53:41.000Z

HI @abhiandthetruth , great! Don't hesitate to write here if you need any help/clarification/feedback, thanks!

Answer 6 · 2020-02-25T09:53:22.000Z

Hey everyone,
I am Kunal Mehta currently in my Final Year of Computer Engineering and I really loved this project topic. I am currently working on micro-tasks and would love to contribute as much as I can to the open-source and would strive to be an active part of this organization in the future.

Answer 7 · 2020-02-25T10:17:12.000Z

Hi @kunakl07, thank you for your interest!
If you are looking for contributions, don't hesitate to have a look at the issues in the https://github.com/chaoss/grimoirelab* repos. Fixes and improvements on the grimoirelab tools documentation and tutorial are also welcomed! Finally, if you have some ideas in mind which aren't reflected in any existing issue, feel free to open a new issue. Thanks!

Answer 8 · 2020-02-25T10:20:46.000Z

Sure, Thank you for replying.

Answer 9 · 2020-02-27T14:12:40.000Z

@valeriocos I had an Idea in my mind. Should I make an Issue or are we free to discuss here?

Answer 10 · 2020-02-27T14:17:39.000Z

If the idea isn't related to this issue, please open a new one in this repo, thanks!

Answer 11 · 2020-02-27T14:18:08.000Z

Sure Thanks!

Answer 12 · 2020-02-27T22:03:50.000Z

@valeriocos sorry for mentioning it here but I just wanted your opinion before creating the issue since I am new to grimoire labs. Is adding a voice query feature relevant? It will be a great feature which will help the target analysts in their research in this age of voice assistants but may not make sense if grimoire lab is not that end user oriented. Do you think it is feasible? If it is I have a detailed proposal ready. Thanks in advance :).

Answer 13 · 2020-02-28T10:29:50.000Z

@abhiandthetruth thank you for sharing this interesting idea!

Having a voice assistant help to create queries in ElasticSearch could be useful from an end-user/analyst perspective. This could be interesting not just for GrimoireLab, but for the ElasticSearch community.

Maybe we can explore this idea (or voice assistant related ones) in one of your personal repositories. Once the implementation is mature, we can propose the inclusion of this repo in CHAOSS. WDYT?

Answer 14 · 2020-02-28T17:21:19.000Z

Glad that you liked the idea! Certainly voice interfaces are the future and the sooner we adopt it the better. I am excited to work on it but my priority is GSoC so I have to complete the microtasks. I will be working on the idea definitely. First we need to plan the structure of the software. I initially planned to use Sir Perceval as our first voice agent ;) and try to implement voice queries on him. Shall I proceed to create a fresh repo and add you as a contributor so that we can discuss there?

Answer 15 · 2020-02-28T17:51:20.000Z

Cool! Using Perceval is a good starting point!

Yes, please create the repo, thanks! (cc @jgbarah)

Answer 16 · 2020-02-28T20:55:06.000Z

repo created and invite sent to you! @valeriocos

Answer 17 · 2020-02-28T21:14:50.000Z

@abhiandthetruth, it's great to see your initiative in this. Would you be okay with having your repo public and sharing the link here? It is in the spirit of open source to share publicly and discuss in the open.

Answer 18 · 2020-02-28T21:24:24.000Z

Yes of course @GeorgLink. The repo link is https://github.com/abhiandthetruth/perceval-voice. I have opened an issue with a short description of the challenges I think we will face, the resources needed and to discuss the basic structure of the initial software. Everyone is welcome to contribute :).

Answer 19 · 2020-02-28T23:10:06.000Z

Comment moved to: abhiandthetruth/perceval-voice#2 to continue conversation about perceval-voice there.

Answer 20 · 2020-03-02T18:48:46.000Z

@valeriocos there seems to be a problem with search guard permission in my machine. I can't create indexes due to a 403. I have enriched it though, and the indices are showing up during creation of index. But when I click create index button nothing happens at the frontend. Following are the logs for the same.

elasticsearch_1 | [2020-03-02T18:41:15,716][INFO ][c.f.s.c.PrivilegesEvaluator] No index-level perm match for User [name=readall, roles=[readall], requestedTenant=null] [IndexType [index=.kibana, type=doc]] [Action [[indices:data/write/index]]] [RolesChecked [sg_own_index, sg_readall]]

elasticsearch_1 | [2020-03-02T18:41:15,716][INFO ][c.f.s.c.PrivilegesEvaluator] No permissions for {sg_own_index=[IndexType [index=.kibana, type=doc]], sg_readall=[IndexType [index=.kibana, type=doc]]}

kibiter_1 | {"type":"response","@timestamp":"2020-03-02T18:41:15Z","tags":[],"pid":1,"method":"post","statusCode":403,"req":{"url":"/api/saved_objects/index-pattern?overwrite=false","method":"post","headers":{"host":"localhost:5601","user-agent":"Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0","accept":"application/json, text/plain, /","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate","referer":"http://localhost:5601/app/kibana","content-type":"application/json;charset=utf-8","kbn-version":"6.1.4-1","content-length":"59","connection":"keep-alive"},"remoteAddress":"172.17.0.1","userAgent":"172.17.0.1","referer":"http://localhost:5601/app/kibana"},"res":{"statusCode":403,"responseTime":25,"contentLength":9},"message":"POST /api/saved_objects/index-pattern?overwrite=false 403 25ms - 9.0B"}

Also there appears to be an ssl error whenever I try to access the elastic search server. I am able to access it though after bypassing the warning.
Please help! :(

Answer 21 · 2020-03-02T19:04:02.000Z

@abhiandthetruth did you try https while accessing the elasticsearch server? This is required if you elasticsearch is of the secured version.

Answer 22 · 2020-03-02T19:06:29.000Z

Yup!

…

On Tue, Mar 3, 2020, 12:34 AM Abid Abdullah ***@***.***> wrote: @abhiandthetruth <https://github.com/abhiandthetruth> did you try https while accessing the elasticsearch server? This is required if you elasticsearch is of the secured version. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#288?email_source=notifications&email_token=ALNBECKX3O2TLK5V4KNTWCTRFP7KJA5CNFSM4KQKEER2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENQRD4A#issuecomment-593564144>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALNBECPIPPQ4YO4EIULAD4DRFP7KJANCNFSM4KQKEERQ> .

Answer 23 · 2020-03-02T19:12:16.000Z

@abhiandthetruth hmm
Try increasing the virtual memory of elasticsearch:
sudo sysctl -w vm.max_map_count=262144
This is valid only for the current session.

Answer 24 · 2020-03-02T19:16:06.000Z

@snack0verflow ya I have done that, that is why I was able to enrich it. The elastic search server can be accessed over https in my machine. I think it's an issue with search guard. But that was one of the problem I also encountered during my setup. Maybe we should mention it in the readme? Thanks for helping :)

Answer 25 · 2020-03-02T19:21:20.000Z

@snack0verflow are you using the same docker-compose.yml file or have you tweaked it? Is everything working fine in your machine? Can you try
curl -XGET https://admin:admin@localhost:9200
and tell me the response?

Answer 26 · 2020-03-02T19:29:35.000Z

@abhiandthetruth I got the same "dictionary" result that shows up on my browser. Just added -k to the curl command.
Yes I am using the same yml.

Answer 27 · 2020-03-02T19:31:13.000Z

does it give an ssl error without -k?

Answer 28 · 2020-03-02T19:32:38.000Z

Hi @abhiandthetruth
Not sure if this would help, but just try with just http everywhere instead of https.

Answer 29 · 2020-03-02T19:33:06.000Z

Not an error but certificate problem curl: (60)
Sorry I'm typing on my phone. My PC is offline.

Answer 30 · 2020-03-02T19:34:16.000Z

@vcrombie using http, there is a connection was reset error so can't do that. Thanks :)

Answer 31 · 2020-03-02T19:37:29.000Z

Not an error but certificate problem curl: (60)
Sorry I'm typing on my phone. My PC is offline.

Yes I am also having the same problem. I was just verifying whether this was the root cause, apparently it wasn't ! Thanks :)

Answer 32 · 2020-03-02T19:47:46.000Z

@abhiandthetruth hmm
I got a 403 as well, don't think it happened last time I tried creating index.
@valeriocos Here is the response to the UI payload if it helps:

{"message":"no permissions for [indices:data/write/index] and User [name=readall, roles=[readall], requestedTenant=null]: [security_exception] no permissions for [indices:data/write/index] and User [name=readall, roles=[readall], requestedTenant=null]","statusCode":403,"error":"Forbidden"}

Answer 33 · 2020-03-03T09:20:26.000Z

Sorry for the late reply!

I'm using the master branch of all components (note that a PR on mordred was just merged: chaoss/grimoirelab-sirmordred#412)

start from a fresh installation (docker prune -a --volumes)
use the docker-compose at https://github.com/chaoss/grimoirelab-sirmordred#source-code-and-docker
launch micro-mordred with the setup.cfg and projects.json at https://github.com/chaoss/grimoirelab-sirmordred/tree/master/utils (micro --raw --enrich --panels --cfg ./setup.cfg --backends git)
check that the data is visible through the dashboard

#288 (comment)
#288 (comment)

@snack0verflow @abhiandthetruth this message generally appears when you try to create an index pattern but you are not logged in Kibana (the login button is on the bottom left corner).

Can you write here the steps you did, so I can try to reproduce the behavior you report? Thanks

Answer 34 · 2020-03-03T14:10:31.000Z

@valeriocos logging in to kibana worked. Need to document this in micro-mordred?

Answer 35 · 2020-03-03T20:49:21.000Z

@snack0verflow are you able to execute the study 'enrich_areas_of_code:git'. The following error occurs while execution of mine,

2020-03-04 02:10:03,461 [git] Problem executing study enrich_areas_of_code:git, RequestError(400, 'search_phase_execution_exception', 'No mapping found for [metadata__timestamp] in order to sort on')
2020-03-04 02:10:03,462 RequestError(400, 'search_phase_execution_exception', 'No mapping found for [metadata__timestamp] in order to sort on')

while running.
The ui payload shows a RemortTransportationException, caused by a QueryShardException.

Answer 36 · 2020-03-03T20:51:47.000Z

And yes @valeriocos the 403 is solved by logging in. I think the user should be notified to login if that 403 is recieved by the frontend.

Answer 37 · 2020-03-04T09:16:23.000Z

#288 (comment)

@abhiandthetruth the error should pop up when the input index of the study is empty

@abhiandthetruth @snack0verflow it's a good idea to document the different problems you are having!

@snack0verflow can you add the section Troubleshooting to the mordred README before https://github.com/chaoss/grimoirelab-sirmordred#micro-mordred ? In this section you can add an entry to comment about #288 (comment)

Once this new section is created, @abhiandthetruth you could add 2 more entries to target:

WDYT?

Answer 38 · 2020-03-04T11:05:09.000Z

Definitely. We can name the section 'Debugging' @valeriocos @snack0verflow.

…

On Wed, Mar 4, 2020, 2:46 PM valerio ***@***.***> wrote: #288 (comment) <#288 (comment)> @abhiandthetruth <https://github.com/abhiandthetruth> the error should pop up when the input index of the study is empty @abhiandthetruth <https://github.com/abhiandthetruth> @snack0verflow <https://github.com/snack0verflow> it's a good idea to document the different problems you are having! @snack0verflow <https://github.com/snack0verflow> can you add the section Troubleshooting to the mordred README before https://github.com/chaoss/grimoirelab-sirmordred#micro-mordred ? In this section you can add an entry to comment about #288 (comment) <#288 (comment)> Once this new section is created, @abhiandthetruth <https://github.com/abhiandthetruth> you could add 2 more entries to target: - #288 (comment) <#288 (comment)> - #288 (comment) <#288 (comment)> WDYT? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#288?email_source=notifications&email_token=ALNBECN7A4736DZKSQSJUS3RFYL6RA5CNFSM4KQKEER2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENW7JRA#issuecomment-594408644>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALNBECKLXZHOI3RVL5HLHDLRFYL6RANCNFSM4KQKEERQ> .

Answer 39 · 2020-03-04T11:23:01.000Z

I'm fine with any name, @abhiandthetruth @snack0verflow feel free to decide one, thanks!

Answer 40 · 2020-03-06T03:56:48.000Z

Hi all, I am an Outreachy applicant, looking to contribute to this project.
To get started, I have to follow the micro tasks in order. Is that right?

Answer 41 · 2020-03-06T05:54:41.000Z

I'm fine with any name, @abhiandthetruth @snack0verflow feel free to decide one, thanks!

Sorry for late response, since @snack0verflow seems to be busy at the time, I will be proceeding with the addition. Also I missed that you had already named the section 😅. Sorry for the override @valeriocos :). Troubleshooting is better.

Answer 42 · 2020-03-06T06:25:48.000Z

ok thanks @abhiandthetruth

Answer 43 · 2020-03-06T06:46:20.000Z

#288 (comment)

Hi @vvijayalakshmi21, thank you for your interest in this idea. You don't have to follow an order to complete the microtasks.
Don't hesitate to write if you have more questions or need feedback.

Answer 44 · 2020-03-07T04:13:01.000Z

Hi everyone! I'm an Outreachy applicant from Argentina and, as my main academic background is in Social Sciences, I find this project very interesting. I would love to contribute to it. I will start with the micro-tasks.
See you!

Answer 45 · 2020-03-07T08:16:15.000Z

Hi @ferculell! if you need any help or clarification, please write in this issue.

Answer 46 · 2020-03-07T11:00:16.000Z

Hi! @valeriocos I have some technical doubts regarding this project. Is there a place where I can ping you (maybe an IRC or mail id). Sorry, have a bunch of doubts so I don't think this is the appropriate place to mention them :/

Answer 47 · 2020-03-07T11:52:21.000Z

Sure @animeshk08 , IRC is perfect (I can be there for the next hour), otherwise by email (valcos@bitergia.com)

Answer 48 · 2020-03-08T10:30:39.000Z

@valeriocos is the kidash argument --elastic_url-enrich changed to --elastic_url bcoz trying the former gave unrecognized error whereas the later works fine. The former is also not mentioned in the command kidash --help but the tutorials are using it freely. Is it a change in development?

Answer 49 · 2020-03-08T10:57:10.000Z

And also the tutorial sample mordred config still contains the bots_names param.

Answer 50 · 2020-03-08T11:27:52.000Z

@abhiandthetruth probably the doc hasn't been updated. Can you submit two PRs (one for #288 (comment) and one for #288 (comment))? thanks

Answer 51 · 2020-03-08T11:30:30.000Z

@valeriocos @abhiandthetruth I think I have removed the bots_names param in an earlier PR

Answer 52 · 2020-03-08T11:33:35.000Z

chaoss/grimoirelab-sirmordred#406 this one

Answer 53 · 2020-03-08T11:37:47.000Z

@snack0verflow it was for the setup.cfg in the grimoirelab-sirmordred repo this one is for the tutorials!

Answer 54 · 2020-03-08T12:20:48.000Z

@abhiandthetruth my bad, you can go ahead and fix that.

Answer 55 · 2020-03-08T22:06:29.000Z

@valeriocos finished all the micro-tasks!. I have made a private repo for that. It would be really helpful if you could review my work before submitting. So how should I proceed, shall I make it public and share the link? Thanks :). Also there is no Microtask-8 there in the list provided.

Answer 56 · 2020-03-09T07:13:45.000Z

@abhiandthetruth I'm going to add a task today (sorry for the inconvenience). WRT sharing your work, you can either invite me to your private repo or you can make it public and share the link. Choose what you prefer :)

Answer 57 · 2020-03-09T07:19:45.000Z

@valeriocos I have made it public for now :). Here is the link https://github.com/abhiandthetruth/Chaoss-Gsoc2020. Please review. Please tell if more detailed answers are needed.

Answer 58 · 2020-03-09T08:44:42.000Z

#288 (comment)

Task added! Please have a look, thanks

Answer 59 · 2020-03-09T10:01:48.000Z

Hi!! I am Tanvi Singhal an outreachy applicant interested in working on this project

Answer 60 · 2020-03-09T12:00:04.000Z

Hi @singhaltanvi , welcome on board! Don't hesitate to write here if you need help/feedback, thanks!

Answer 61 · 2020-03-12T16:32:55.000Z

Hello mentors, I have a few questions regarding the implementation of Social Currency Metric System :)

Is there going to be a feature to add these social currencies manually? In that case who can do so? Also how often they are to be added to generate a good reflective result?
How does one decide the weight score for these metrics?
The process of tagging the data and inserting these metrics is not going to be performed by GrimoireLab, to best of my understanding. What procedure will be followed to do so? Is it done manually? How much time and effort does it require to do such a task?

Thank you so much.

EDIT: Also for anyone interested in this project, some clarity can be gained by looking at this issue. :)

Answer 62 · 2020-03-12T18:33:18.000Z

Is the implementation architecture been decided yet or is it still in progress?

Answer 63 · 2020-03-12T22:04:18.000Z

Hi @animeshk08 ,

Hello mentors, I have a few questions regarding the implementation of Social Currency Metric System :)

Is there going to be a feature to add these social currencies manually? In that case who can do so? Also how often they are to be added to generate a good reflective result?

Social currency is an abstract idea. The concete things this project will be working with is data that GrimoireLab already collects: email text, tweet text, issue comment text, and other text. That text needs to be annotated or tagged (see metric definition). This may be a manual process where a person reads these messages and adds tags. What we need is that user interface that pulls in the messages from GrimoireLab and allows someone to add tags which are stored back to GrimoireLab. The other thing we need are visualizations to show how often a tag was used and report that back.

How does one decide the weight score for these metrics?

That is determined based on business goals someone has. In the visualization or reporting feature of this project, I imagine someone can set weights. However, weights are optional at this point and can be an advanced feature.

The process of tagging the data and inserting these metrics is not going to be performed by GrimoireLab, to best of my understanding. What procedure will be followed to do so? Is it done manually? How much time and effort does it require to do such a task?

I described the process above. GrimoireLab does not do tagging, a person will do the tagging manually in the first iteration and maybe an artificial intelligence in a stretch goal. How long it takes to do tagging depends on the size of the community and how many messages the members produce.

Answer 64 · 2020-03-13T02:47:26.000Z

Thank you @GeorgLink for the all the clarifications :) One more thing the user interface for adding the tags will also be provide be Grimoirelab or just the data fetching and visualisation part?? As I was having the notion that the data will be exported in excel or CSV format and then later imported again.

Answer 65 · 2020-03-13T05:39:19.000Z

Hello everyone!

I feel a little silly for not commenting on the main thread yet but I've been supporting some side messages so I'm going to put a master here.

But first, Introduction!
Skip this if you don't really care :)

My name is Samantha Venia Logan. I go by Venia. I am the originating creator of the Social Currency Metrics System and the CEO of SociallyConstructed.Online.

Both my co-founder, Dylan, and I have social-scientific backgrounds so we don't have any coding knowledge beyond marketing requirements. That said however I am a full-stack marketer and online community manager whereas Dylan is a copywriter and anthropologist by trade. We are mentors on this project to provide some insight into how the system works, what it's used for, and how the system should be built to support the community management and social capital process.

@GeorgLink and CHAOSS, however, are so wonderful and on the ball about this that it seems a lot of questions can be answered through the present resources provided.

I still want to extend my assistance to you all and offer myself up as a resource. I am primarily available by email at samantha@sociallyconstructed.online and I LOVE video calls so just let me know if you'd like to have an extended conversation. I'm also more than happy to schedule something as a group ^^ Let me know if you'd prefer that.

I am already very impressed with the passion, suggestions, and questions though. You've all got amazing ideas like the ability to add a voice-service to the system! A very innovative way to use it!

So to answer a few questions that have come around:

Is there going to be a feature to add these social currencies manually? In that case who can do so? Also how often they are to be added to generate a good reflective result?

The social currencies are to be defined in the codex which is the most important part. You must have a clear grounded definition for the term, an example of when it is applicable and an example of when it is not. Then you can either manually tag a term or teach a language processor to do it.

The SCMS is built to scale with the community. As it stands because it doesn't have a "home" system you can usually build it in the preferred tech stack of the community and as it scales new options for what you can do emerge. For CHAOSS, obviously, that will be a part of your build.

Off the cuff, it's more than possible to tack a language processing AI or algorithm to the back end of the system but by no means is this necessary. This means that as you tag content you're training the system to tag larger quantities of traffic and the system will automatically scale.

BUT,

Dylan and I STRONGLY recommend tagging your own traffic 1 hour per week no matter what you do, and then we suggest that you have 1 hour a week to meet with others to discuss the process as a whole. The MOST important part of this system is the tag codex because if you're tagging your data wrong (just like tagging UTM traffic in a marketing campaign) it can have huge consequences.

If you want to tack the language processer on this is a good opportunity to own and spearhead that part of the project, but it is VERY MUCH a stretch goal. :)

How does one decide the weight score for these metrics?

Score weights, as Georg stated are based on business goals. The MOST important part of the system is defining the terms in the codex. Using these tags you can then decide, "what counts as bad for this tag and what counts as a good response?" When you fill out when to use the term and when not to, how you should weight it typically falls in to place based on the tone of the sentiment and whether a comment counted against or for the tag. It should also be noted that research in marketing has shown that a 7-point scale is usually most accurate without being overwhelming. So I use a rating system of from -3 to +3 with 0 as a neutral default.

The process of tagging the data and inserting these metrics is not going to be performed by GrimoireLab, to best of my understanding. What procedure will be followed to do so? Is it done manually? How much time and effort does it require to do such a task?

As I stated above we will NEVER recommend that you not tag your traffic. The system scales and adding AI is possible but you should be reading the comments. If you give it to an automation task to read you're doing your community a massive disservice because you, the one with the power, aren't listening. So, 1 hour per week per person and then a 1 hour meeting to discuss:

Best and worst comments,
Trends found in the data,
What you'd like to track,
and how previous tracks have stacked up against the prior week.
Look at the dashboards to see if there's an initiative you can launch that improves the results
And poke holes at the validity of the data and see if there are any data sources you're missing.

This system works surprisingly like google analytics and tracking traffic quantitatively. You have to get all your traffic in to lanes, plan the way you're getting and analyzing it, and always trust, but verify your data

Reading through some materials online about Social Currency had a different set of dimensions.

Affiliation – Offering a sense of community among your customers.
Conversation – Engaging in the customer conversation around your brand.
Utility – Creating utility and value for customers. While helping them become more socially relevant.
Advocacy – Developing strong feelings of loyalty to a brand. To the point where customers recommend and even defend it.
Information – Giving valuable and shareable information.
Identity – Providing customers with their own identity in a group.
Can you provide me some references where the said dimensions in SCMS doc are mentioned? :) Or how are some of the former dimensions are dependent on the latter ones.

These are correct from an academic standpoint but a quick look will tell you, analyzing these from week to week in-situ with your community leaves a lot of processing and subjectivity. They're largely up in the air and hard to define. We wanted actionable definitions you could predict the next week on so we did 3 things to them:

It consolidated several different definitions and concepts from sociology, psychology, and anthropology to remove some ambiguity from the underlying concepts,
simplified the more abstract language into more commonly used terms in communities as best it could and
turned them in to lead metrics as opposed to lag metrics that can be used to predict a measurable response in the community from week to week.

These changes make it viable as a daily analytics system by making the collecting tagging and analytics portions easy to understand, applicable to any community, and capable of predicting community response to actions.

Conclusion / TL;DR
These are all amazing questions and I hope the long response doesn't disappoint. The TL;DR is that this system you're making is to support a cultural process so there are aspects of automation and user experience that should be left to the community like definitions. That said, however, building a system that facilitates more objective, actionable, and easier use of the qualitative data communities are producing is important. This system takes a complicated, lengthy, vague, and historical process and makes it a simple, predictive, concrete, and immersive process that scales to match the community it's reporting on.

If you have any questions, I'm more than happy to talk here, over email, over the phone, or on a hangout. It may, however, be best to keep the conversation here though so others benefit from your amazing questions).

Answer 66 · 2020-03-13T05:53:31.000Z

One more thing the user interface for adding the tags will also provide be Grimoirelab or just the data fetching and visualization part?? As I was having the notion that the data will be exported in Excel or CSV format and then later imported again.

Regarding this, it's best to make the tagging a part of the native platform because then you have to handle collaboration rules and permissions, but the CSV should export with the needed container columns to tag in whatever sheet software you prefer.

We have seen it the way you're envisioning though. In specific implementations like my use for Comcast Xfinity customer support, we had the system implemented in Microsoft SharePoint website. Support agents would export a CSV to desktop, fill in ticket notes with every call, and then use a micosoft sharepoiint addon to add their data back to the MySQL via CSV at the end of the day. It wasn't feasible to have all the agents on the floor input their data directly until we were confident it would work and then we eventually built the system in Medallion afterward.

Answer 67 · 2020-03-13T06:01:37.000Z

Thank you @samanthavenialogan for giving such detailed answers to our questions.:)

Regarding this, it's best to make the tagging a part of the native platform because then you have to handle collaboration rules and permissions, but the CSV should export with the needed container columns to tag in whatever sheet software you prefer.

Now I understand the direction in which you want to project to move forward and I will explore more to integrate this way of tagging the data on the SCMS platform.

Answer 68 · 2020-03-14T09:08:30.000Z

Hi, @valeriocos I have been going through the microtasks. However, while running micromordred I am getting the following error could you help me with it.

Thank you

Answer 69 · 2020-03-14T09:22:24.000Z

Hi @animeshk08 ,

Which docker-compose version are you running? The one with or without searchguard (https://github.com/chaoss/grimoirelab-sirmordred#docker-compose-with-searchguard)?
Can you post the output of curl -XGET <elasticsearch-url> -k ?
Can you post the logs of the elasticsearch container (i.e., docker logs <es-container>) ?
Please check the section https://github.com/chaoss/grimoirelab-sirmordred#troubleshooting, to see if what you are experiencing is already documented?

Thanks

Answer 70 · 2020-03-14T09:47:12.000Z

Hi @valeriocos Thank you for the quick response.

Which docker-compose version are you running? The one with or without searchguard (https://github.com/chaoss/grimoirelab-sirmordred#docker-compose-with-searchguard)?

Without search guard

Can you post the output of curl -XGET -k?

curl: (7) Failed to connect to localhost port 9200: Connection refused

Can you post the logs of the elasticsearch container (i.e., docker logs ) ?

While running this command docker-compose -f docker-compose.yml up I am getting the following logs.

Creating network "utils_default" with the default driver
Creating utils_mariadb_1       ... done
Creating utils_elasticsearch_1 ... done
Creating utils_kibiter_1       ... done
Attaching to utils_elasticsearch_1, utils_mariadb_1, utils_kibiter_1
elasticsearch_1  | [2020-03-14T09:43:44,198][INFO ][o.e.n.Node               ] [] initializing ...
elasticsearch_1  | [2020-03-14T09:43:45,711][INFO ][o.e.e.NodeEnvironment    ] [UZ6jRxG] using [1] data paths, mounts [[/ (overlay)]], net usable_space [36.5gb], net total_space [116.1gb], types [overlay]
elasticsearch_1  | [2020-03-14T09:43:45,711][INFO ][o.e.e.NodeEnvironment    ] [UZ6jRxG] heap size [1.9gb], compressed ordinary object pointers [true]
mariadb_1        | Initializing database
elasticsearch_1  | [2020-03-14T09:43:45,714][INFO ][o.e.n.Node               ] node name [UZ6jRxG] derived from node ID [UZ6jRxGpThm8DQCHjKjsLw]; set [node.name] to override
mariadb_1        | 200314  9:43:41 [Note] /usr/sbin/mysqld (mysqld 10.0.38-MariaDB-1~xenial) starting as process 68 ...
elasticsearch_1  | [2020-03-14T09:43:45,714][INFO ][o.e.n.Node               ] version[6.1.4], pid[1], build[d838f2d/2018-03-14T08:28:22.470Z], OS[Linux/5.3.0-40-generic/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_161/25.161-b14]
mariadb_1        | 200314  9:43:42 [Note] InnoDB: Using mutexes to ref count buffer pool pages
mariadb_1        | 200314  9:43:42 [Note] InnoDB: The InnoDB memory heap is disabled
mariadb_1        | 200314  9:43:42 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
mariadb_1        | 200314  9:43:42 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
mariadb_1        | 200314  9:43:42 [Note] InnoDB: Compressed tables use zlib 1.2.8
mariadb_1        | 200314  9:43:42 [Note] InnoDB: Using Linux native AIO
elasticsearch_1  | [2020-03-14T09:43:45,715][INFO ][o.e.n.Node               ] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -XX:+HeapDumpOnOutOfMemoryError, -Xms2g, -Xmx2g, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config]
mariadb_1        | 200314  9:43:42 [Note] InnoDB: Using CPU crc32 instructions
elasticsearch_1  | [2020-03-14T09:43:46,534][INFO ][o.e.p.PluginsService     ] [UZ6jRxG] loaded module [aggs-matrix-stats]
elasticsearch_1  | [2020-03-14T09:43:46,534][INFO ][o.e.p.PluginsService     ] [UZ6jRxG] loaded module [analysis-common]
elasticsearch_1  | [2020-03-14T09:43:46,534][INFO ][o.e.p.PluginsService     ] [UZ6jRxG] loaded module [ingest-common]
mariadb_1        | 200314  9:43:42 [Note] InnoDB: Initializing buffer pool, size = 256.0M
elasticsearch_1  | [2020-03-14T09:43:46,534][INFO ][o.e.p.PluginsService     ] [UZ6jRxG] loaded module [lang-expression]
elasticsearch_1  | [2020-03-14T09:43:46,534][INFO ][o.e.p.PluginsService     ] [UZ6jRxG] loaded module [lang-mustache]
mariadb_1        | 200314  9:43:42 [Note] InnoDB: Completed initialization of buffer pool
mariadb_1        | 200314  9:43:42 [Note] InnoDB: The first specified data file ./ibdata1 did not exist: a new database to be created!
mariadb_1        | 200314  9:43:42 [Note] InnoDB: Setting file ./ibdata1 size to 12 MB
elasticsearch_1  | [2020-03-14T09:43:46,534][INFO ][o.e.p.PluginsService     ] [UZ6jRxG] loaded module [lang-painless]
mariadb_1        | 200314  9:43:42 [Note] InnoDB: Database physically writes the file full: wait...
elasticsearch_1  | [2020-03-14T09:43:46,534][INFO ][o.e.p.PluginsService     ] [UZ6jRxG] loaded module [mapper-extras]
elasticsearch_1  | [2020-03-14T09:43:46,534][INFO ][o.e.p.PluginsService     ] [UZ6jRxG] loaded module [parent-join]
mariadb_1        | 200314  9:43:43 [Note] InnoDB: Setting log file ./ib_logfile101 size to 48 MB
elasticsearch_1  | [2020-03-14T09:43:46,535][INFO ][o.e.p.PluginsService     ] [UZ6jRxG] loaded module [percolator]
elasticsearch_1  | [2020-03-14T09:43:46,535][INFO ][o.e.p.PluginsService     ] [UZ6jRxG] loaded module [reindex]
mariadb_1        | 200314  9:43:44 [Note] InnoDB: Setting log file ./ib_logfile1 size to 48 MB
elasticsearch_1  | [2020-03-14T09:43:46,535][INFO ][o.e.p.PluginsService     ] [UZ6jRxG] loaded module [repository-url]
mariadb_1        | 200314  9:43:47 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0
mariadb_1        | 200314  9:43:47 [Warning] InnoDB: New log files created, LSN=45781
mariadb_1        | 200314  9:43:47 [Note] InnoDB: Doublewrite buffer not found: creating new
elasticsearch_1  | [2020-03-14T09:43:46,535][INFO ][o.e.p.PluginsService     ] [UZ6jRxG] loaded module [transport-netty4]
mariadb_1        | 200314  9:43:47 [Note] InnoDB: Doublewrite buffer created
elasticsearch_1  | [2020-03-14T09:43:46,535][INFO ][o.e.p.PluginsService     ] [UZ6jRxG] loaded module [tribe]
mariadb_1        | 200314  9:43:47 [Note] InnoDB: 128 rollback segment(s) are active.
mariadb_1        | 200314  9:43:47 [Warning] InnoDB: Creating foreign key constraint system tables.
elasticsearch_1  | [2020-03-14T09:43:46,535][INFO ][o.e.p.PluginsService     ] [UZ6jRxG] loaded plugin [ingest-geoip]
mariadb_1        | 200314  9:43:47 [Note] InnoDB: Foreign key constraint system tables created
mariadb_1        | 200314  9:43:47 [Note] InnoDB: Creating tablespace and datafile system tables.
elasticsearch_1  | [2020-03-14T09:43:46,535][INFO ][o.e.p.PluginsService     ] [UZ6jRxG] loaded plugin [ingest-user-agent]
mariadb_1        | 200314  9:43:47 [Note] InnoDB: Tablespace and datafile system tables created.
elasticsearch_1  | [2020-03-14T09:43:47,767][INFO ][o.e.d.DiscoveryModule    ] [UZ6jRxG] using discovery type [zen]
mariadb_1        | 200314  9:43:47 [Note] InnoDB: Waiting for purge to start
elasticsearch_1  | [2020-03-14T09:43:48,326][INFO ][o.e.n.Node               ] initialized
mariadb_1        | 200314  9:43:47 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.42-84.2 started; log sequence number 0
elasticsearch_1  | [2020-03-14T09:43:48,326][INFO ][o.e.n.Node               ] [UZ6jRxG] starting ...
elasticsearch_1  | [2020-03-14T09:43:48,487][INFO ][o.e.t.TransportService   ] [UZ6jRxG] publish_address {172.20.0.2:9300}, bound_addresses {0.0.0.0:9300}
elasticsearch_1  | [2020-03-14T09:43:48,495][INFO ][o.e.b.BootstrapChecks    ] [UZ6jRxG] bound or publishing to a non-loopback address, enforcing bootstrap checks
elasticsearch_1  | ERROR: [1] bootstrap checks failed
elasticsearch_1  | [1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
elasticsearch_1  | [2020-03-14T09:43:48,502][INFO ][o.e.n.Node               ] [UZ6jRxG] stopping ...
elasticsearch_1  | [2020-03-14T09:43:48,525][INFO ][o.e.n.Node               ] [UZ6jRxG] stopped
elasticsearch_1  | [2020-03-14T09:43:48,525][INFO ][o.e.n.Node               ] [UZ6jRxG] closing ...
elasticsearch_1  | [2020-03-14T09:43:48,531][INFO ][o.e.n.Node               ] [UZ6jRxG] closed
utils_elasticsearch_1 exited with code 78
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T09:43:55Z","tags":["status","plugin:kibana@6.1.4-1","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T09:43:55Z","tags":["status","plugin:elasticsearch@6.1.4-1","info"],"pid":1,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T09:43:55Z","tags":["status","plugin:console@6.1.4-1","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T09:43:55Z","tags":["status","plugin:metrics@6.1.4-1","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T09:43:55Z","tags":["error","elasticsearch","admin"],"pid":1,"message":"Request error, retrying\nHEAD http://elasticsearch:9200/ => connect ECONNREFUSED 127.0.0.1:9200"}
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T09:43:55Z","tags":["status","plugin:timelion@6.1.4-1","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T09:43:55Z","tags":["listening","info"],"pid":1,"message":"Server running at http://0.0.0.0:5601"}
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T09:43:55Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T09:43:55Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"}
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T09:43:55Z","tags":["status","plugin:elasticsearch@6.1.4-1","error"],"pid":1,"state":"red","message":"Status changed from yellow to red - Unable to connect to Elasticsearch at http://elasticsearch:9200.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
mariadb_1        | 200314  9:43:57 [Note] InnoDB: FTS optimize thread exiting.
mariadb_1        | 200314  9:43:57 [Note] InnoDB: Starting shutdown...
mariadb_1        | 200314  9:43:58 [Note] InnoDB: Waiting for page_cleaner to finish flushing of buffer pool
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T09:43:58Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T09:43:58Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"}
mariadb_1        | 200314  9:43:59 [Note] InnoDB: Shutdown completed; log sequence number 1616697
mariadb_1        | 200314  9:44:00 [Note] /usr/sbin/mysqld (mysqld 10.0.38-MariaDB-1~xenial) starting as process 94 ...
mariadb_1        | 200314  9:44:00 [Note] InnoDB: Using mutexes to ref count buffer pool pages
mariadb_1        | 200314  9:44:00 [Note] InnoDB: The InnoDB memory heap is disabled
mariadb_1        | 200314  9:44:00 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
mariadb_1        | 200314  9:44:00 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
mariadb_1        | 200314  9:44:00 [Note] InnoDB: Compressed tables use zlib 1.2.8
mariadb_1        | 200314  9:44:00 [Note] InnoDB: Using Linux native AIO
mariadb_1        | 200314  9:44:00 [Note] InnoDB: Using CPU crc32 instructions
mariadb_1        | 200314  9:44:00 [Note] InnoDB: Initializing buffer pool, size = 256.0M
mariadb_1        | 200314  9:44:00 [Note] InnoDB: Completed initialization of buffer pool
mariadb_1        | 200314  9:44:00 [Note] InnoDB: Highest supported file format is Barracuda.
mariadb_1        | 200314  9:44:00 [Note] InnoDB: 128 rollback segment(s) are active.
mariadb_1        | 200314  9:44:00 [Note] InnoDB: Waiting for purge to start
mariadb_1        | 200314  9:44:00 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.42-84.2 started; log sequence number 1616697
mariadb_1        | 200314  9:44:00 [Note] InnoDB: FTS optimize thread exiting.
mariadb_1        | 200314  9:44:00 [Note] InnoDB: Starting shutdown...
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T09:44:00Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T09:44:00Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"}
mariadb_1        | 200314  9:44:01 [Note] InnoDB: Waiting for page_cleaner to finish flushing of buffer pool
mariadb_1        | 200314  9:44:03 [Note] InnoDB: Shutdown completed; log sequence number 1616707
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T09:44:03Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T09:44:03Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"}
mariadb_1        | 
mariadb_1        | PLEASE REMEMBER TO SET A PASSWORD FOR THE MariaDB root USER !
mariadb_1        | To do so, start the server, then issue the following commands:
mariadb_1        | 
mariadb_1        | '/usr/bin/mysqladmin' -u root password 'new-password'
mariadb_1        | '/usr/bin/mysqladmin' -u root -h  password 'new-password'
mariadb_1        | 
mariadb_1        | Alternatively you can run:
mariadb_1        | '/usr/bin/mysql_secure_installation'
mariadb_1        | 
mariadb_1        | which will also give you the option of removing the test
mariadb_1        | databases and anonymous user created by default.  This is
mariadb_1        | strongly recommended for production servers.
mariadb_1        | 
mariadb_1        | See the MariaDB Knowledgebase at http://mariadb.com/kb or the
mariadb_1        | MySQL manual for more instructions.
mariadb_1        | 
mariadb_1        | Please report any problems at http://mariadb.org/jira
mariadb_1        | 
mariadb_1        | The latest information about MariaDB is available at http://mariadb.org/.
mariadb_1        | You can find additional information about the MySQL part at:
mariadb_1        | http://dev.mysql.com
mariadb_1        | Consider joining MariaDB's strong and vibrant community:
mariadb_1        | https://mariadb.org/get-involved/
mariadb_1        | 
mariadb_1        | Database initialized
mariadb_1        | 200314  9:44:03 [Note] mysqld (mysqld 10.0.38-MariaDB-1~xenial) starting as process 124 ...
mariadb_1        | MySQL init process in progress...
mariadb_1        | 200314  9:44:03 [Note] InnoDB: Using mutexes to ref count buffer pool pages
mariadb_1        | 200314  9:44:03 [Note] InnoDB: The InnoDB memory heap is disabled
mariadb_1        | 200314  9:44:03 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
mariadb_1        | 200314  9:44:03 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
mariadb_1        | 200314  9:44:03 [Note] InnoDB: Compressed tables use zlib 1.2.8
mariadb_1        | 200314  9:44:03 [Note] InnoDB: Using Linux native AIO
mariadb_1        | 200314  9:44:03 [Note] InnoDB: Using CPU crc32 instructions
mariadb_1        | 200314  9:44:03 [Note] InnoDB: Initializing buffer pool, size = 256.0M
mariadb_1        | 200314  9:44:03 [Note] InnoDB: Completed initialization of buffer pool
mariadb_1        | 200314  9:44:03 [Note] InnoDB: Highest supported file format is Barracuda.
mariadb_1        | 200314  9:44:03 [Note] InnoDB: 128 rollback segment(s) are active.
mariadb_1        | 200314  9:44:03 [Note] InnoDB: Waiting for purge to start
mariadb_1        | 200314  9:44:04 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.42-84.2 started; log sequence number 1616707
mariadb_1        | 200314  9:44:04 [Note] Plugin 'FEEDBACK' is disabled.
mariadb_1        | 200314  9:44:04 [Warning] 'user' entry 'root@2650998d4990' ignored in --skip-name-resolve mode.
mariadb_1        | 200314  9:44:04 [Warning] 'proxies_priv' entry '@% root@2650998d4990' ignored in --skip-name-resolve mode.
mariadb_1        | 200314  9:44:04 [Note] mysqld: ready for connections.
mariadb_1        | Version: '10.0.38-MariaDB-1~xenial'  socket: '/var/run/mysqld/mysqld.sock'  port: 0  mariadb.org binary distribution
mariadb_1        | Warning: Unable to load '/usr/share/zoneinfo/leap-seconds.list' as time zone. Skipping it.
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T09:44:06Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T09:44:06Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"}
mariadb_1        | 200314  9:44:06 [Warning] 'proxies_priv' entry '@% root@2650998d4990' ignored in --skip-name-resolve mode.
mariadb_1        | 
mariadb_1        | 200314  9:44:06 [Note] mysqld: Normal shutdown
mariadb_1        | 200314  9:44:06 [Note] Event Scheduler: Purging the queue. 0 events
mariadb_1        | 200314  9:44:06 [Note] InnoDB: FTS optimize thread exiting.
mariadb_1        | 200314  9:44:06 [Note] InnoDB: Starting shutdown...
mariadb_1        | 200314  9:44:06 [Note] InnoDB: Waiting for page_cleaner to finish flushing of buffer pool
mariadb_1        | 200314  9:44:07 [Note] InnoDB: Shutdown completed; log sequence number 1616717
mariadb_1        | 200314  9:44:08 [Note] mysqld: Shutdown complete
mariadb_1        | 
mariadb_1        | 
mariadb_1        | MySQL init process done. Ready for start up.
mariadb_1        | 
mariadb_1        | 200314  9:44:08 [Note] mysqld (mysqld 10.0.38-MariaDB-1~xenial) starting as process 1 ...
mariadb_1        | 200314  9:44:08 [Note] InnoDB: Using mutexes to ref count buffer pool pages
mariadb_1        | 200314  9:44:08 [Note] InnoDB: The InnoDB memory heap is disabled
mariadb_1        | 200314  9:44:08 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
mariadb_1        | 200314  9:44:08 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
mariadb_1        | 200314  9:44:08 [Note] InnoDB: Compressed tables use zlib 1.2.8
mariadb_1        | 200314  9:44:08 [Note] InnoDB: Using Linux native AIO
mariadb_1        | 200314  9:44:08 [Note] InnoDB: Using CPU crc32 instructions
mariadb_1        | 200314  9:44:08 [Note] InnoDB: Initializing buffer pool, size = 256.0M
mariadb_1        | 200314  9:44:08 [Note] InnoDB: Completed initialization of buffer pool
mariadb_1        | 200314  9:44:08 [Note] InnoDB: Highest supported file format is Barracuda.
mariadb_1        | 200314  9:44:08 [Note] InnoDB: 128 rollback segment(s) are active.
mariadb_1        | 200314  9:44:08 [Note] InnoDB: Waiting for purge to start
mariadb_1        | 200314  9:44:08 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.42-84.2 started; log sequence number 1616717
mariadb_1        | 200314  9:44:08 [Note] Plugin 'FEEDBACK' is disabled.
mariadb_1        | 200314  9:44:08 [Note] Server socket created on IP: '::'.
mariadb_1        | 200314  9:44:08 [Warning] 'proxies_priv' entry '@% root@2650998d4990' ignored in --skip-name-resolve mode.
mariadb_1        | 200314  9:44:08 [Note] mysqld: ready for connections.
mariadb_1        | Version: '10.0.38-MariaDB-1~xenial'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T09:44:08Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T09:44:08Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"}
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T09:44:11Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T09:44:11Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"}
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T09:44:13Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T09:44:13Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"}
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T09:44:16Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T09:44:16Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"}
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T09:44:18Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T09:44:18Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"}
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T09:44:21Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T09:44:21Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"}
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T09:44:23Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T09:44:23Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"}
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T09:44:26Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T09:44:26Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"}
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T09:44:28Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T09:44:28Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"}
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T09:44:31Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T09:44:31Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"}
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T09:44:33Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T09:44:33Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"}

Please check the section https://github.com/chaoss/grimoirelab-sirmordred#troubleshooting, to see if what you are experiencing is already documented?

Sorry I could not find something related to my problem here.

Answer 71 · 2020-03-14T09:53:05.000Z

Hi @animeshk08

Your error lies in max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]

You could try increasing it by : sysctl -w fs.file-max=65536

Hope it helps
Thanks

Answer 72 · 2020-03-14T10:02:55.000Z

HI @ria18405 I have tried running sysctl -w fs.file-max=65536 command already however the error still persists.
I have cross-checked that the value has changed as running sysctl fs.file-max
gives me the following output:
fs.file-max = 65536

EDIT: I also went ahead and tried to make this change permanent by add the following values to etc/security/limits.conf of my system.
#* soft nofile 65536
#* hard nofile 65536

Answer 73 · 2020-03-14T10:16:30.000Z

@animeshk08
Have you tried composing the docker again?
Composing the docker again might resolve the issue.

Answer 74 · 2020-03-14T10:17:03.000Z

@animeshk08 can you share you docker-compose.yml or the link to the tutorial you are following?

Answer 75 · 2020-03-14T10:25:11.000Z

Hi @abhiandthetruth. Thank you for the quick response

My docker-compose.yml

version: '2.2'

services:
    mariadb:
      image: mariadb:10.0
      expose:
        - "3306"
      environment:
        - MYSQL_ROOT_PASSWORD=
        - MYSQL_ALLOW_EMPTY_PASSWORD=yes

    elasticsearch:
      image: docker.elastic.co/elasticsearch/elasticsearch-oss:6.1.4
      command: /elasticsearch/bin/elasticsearch -E network.bind_host=0.0.0.0
      ports:
        - 9200:9200
      environment:
        - ES_JAVA_OPTS=-Xms2g -Xmx2g

    kibiter:
      restart: on-failure:5
      image: bitergia/kibiter:optimized-v6.1.4-4
      environment:
        - PROJECT_NAME=Demo
        - NODE_OPTIONS=--max-old-space-size=1000
        - ELASTICSEARCH_URL=http://elasticsearch:9200
      links:
        - elasticsearch
      ports:
        - 5601:5601

Answer 76 · 2020-03-14T10:38:39.000Z

I'm trying to debug the problem by running it on my system. BTW have you tried using the with searchguard version. It worked just fine with me just make sure you set the correct vm.max_map_count
sudo sysctl -w vm.max_map_count=262144

Answer 77 · 2020-03-14T10:55:17.000Z

Hi @abhiandthetruth @ria18405 @valeriocos Thank you everyone for looking into this issue. Its been resolved.

Modify the docker-compse.yml by adding the following lines inder elastic:

 ulimits:
          nofile:
              soft: 65536
              hard: 65536

The new docker-compose.yml looks as follows:

version: '2.2'

services:
    mariadb:
      image: mariadb:10.0
      expose:
        - "3306"
      environment:
        - MYSQL_ROOT_PASSWORD=
        - MYSQL_ALLOW_EMPTY_PASSWORD=yes

    elasticsearch:
      image: docker.elastic.co/elasticsearch/elasticsearch-oss:6.1.4
      command: /elasticsearch/bin/elasticsearch -E network.bind_host=0.0.0.0
      ports:
        - 9200:9200
      environment:
        - ES_JAVA_OPTS=-Xms2g -Xmx2g
      ulimits:
        nofile:
          soft: 65536
          hard: 65536

    kibiter:
      restart: on-failure:5
      image: bitergia/kibiter:optimized-v6.1.4-4
      environment:
        - PROJECT_NAME=Demo
        - NODE_OPTIONS=--max-old-space-size=1000
        - ELASTICSEARCH_URL=http://elasticsearch:9200
      links:
        - elasticsearch
      ports:
        - 5601:5601

@valeriocos Let me know if you would like to document this :)

Answer 78 · 2020-03-14T10:59:17.000Z

However, This error still persists: #288 (comment) :(

Answer 79 · 2020-03-14T11:01:50.000Z

whats the output of the curl command now?

Answer 80 · 2020-03-14T11:03:29.000Z

@animeshk08 have you changed the corresponding urls in setup.cfg?

Answer 81 · 2020-03-14T11:04:30.000Z

Output for curl:

{
  "name" : "Mmigh3Z",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "3QN9WZN4RzipqKaFeaGMtA",
  "version" : {
    "number" : "6.1.4",
    "build_hash" : "d838f2d",
    "build_date" : "2018-03-14T08:28:22.470Z",
    "build_snapshot" : false,
    "lucene_version" : "7.1.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

Logs for docker-compose -f docker-compose.yml up:


Recreating utils_mariadb_1       ... done
Recreating utils_elasticsearch_1 ... done
Recreating utils_kibiter_1       ... 
^CGracefully stopping... (press Ctrl+C again to force)
Stopping utils_mariadb_1         ... 
Stopping utils_elasticsearch_1   ... 
^CKilling utils_mariadb_1          ... 
Killing utils_elasticsearch_1    ... 
^CERROR: Aborting.
animesh@msi:~/glsource/grimoirelab-sirmordred/utils$ docker-compose -f docker-compose.yml down
Stopping utils_kibiter_1 ... done
Removing utils_kibiter_1              ... done
Removing utils_mariadb_1              ... done
Removing utils_elasticsearch_1        ... done
Removing 92bf5b79bc87_utils_kibiter_1 ... done
Removing network utils_default
animesh@msi:~/glsource/grimoirelab-sirmordred/utils$ docker-compose -f docker-compose.yml up
Creating network "utils_default" with the default driver
Creating utils_mariadb_1       ... done
Creating utils_elasticsearch_1 ... done
Creating utils_kibiter_1       ... done
Attaching to utils_elasticsearch_1, utils_mariadb_1, utils_kibiter_1
mariadb_1        | Initializing database
mariadb_1        | 200314 10:57:02 [Note] /usr/sbin/mysqld (mysqld 10.0.38-MariaDB-1~xenial) starting as process 68 ...
mariadb_1        | 200314 10:57:02 [Note] InnoDB: Using mutexes to ref count buffer pool pages
mariadb_1        | 200314 10:57:02 [Note] InnoDB: The InnoDB memory heap is disabled
mariadb_1        | 200314 10:57:02 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
mariadb_1        | 200314 10:57:02 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
mariadb_1        | 200314 10:57:02 [Note] InnoDB: Compressed tables use zlib 1.2.8
mariadb_1        | 200314 10:57:02 [Note] InnoDB: Using Linux native AIO
mariadb_1        | 200314 10:57:02 [Note] InnoDB: Using CPU crc32 instructions
mariadb_1        | 200314 10:57:02 [Note] InnoDB: Initializing buffer pool, size = 256.0M
mariadb_1        | 200314 10:57:02 [Note] InnoDB: Completed initialization of buffer pool
mariadb_1        | 200314 10:57:02 [Note] InnoDB: The first specified data file ./ibdata1 did not exist: a new database to be created!
mariadb_1        | 200314 10:57:02 [Note] InnoDB: Setting file ./ibdata1 size to 12 MB
mariadb_1        | 200314 10:57:02 [Note] InnoDB: Database physically writes the file full: wait...
mariadb_1        | 200314 10:57:03 [Note] InnoDB: Setting log file ./ib_logfile101 size to 48 MB
mariadb_1        | 200314 10:57:04 [Note] InnoDB: Setting log file ./ib_logfile1 size to 48 MB
mariadb_1        | 200314 10:57:06 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0
mariadb_1        | 200314 10:57:06 [Warning] InnoDB: New log files created, LSN=45781
mariadb_1        | 200314 10:57:06 [Note] InnoDB: Doublewrite buffer not found: creating new
mariadb_1        | 200314 10:57:07 [Note] InnoDB: Doublewrite buffer created
mariadb_1        | 200314 10:57:07 [Note] InnoDB: 128 rollback segment(s) are active.
mariadb_1        | 200314 10:57:07 [Warning] InnoDB: Creating foreign key constraint system tables.
mariadb_1        | 200314 10:57:07 [Note] InnoDB: Foreign key constraint system tables created
mariadb_1        | 200314 10:57:07 [Note] InnoDB: Creating tablespace and datafile system tables.
mariadb_1        | 200314 10:57:07 [Note] InnoDB: Tablespace and datafile system tables created.
mariadb_1        | 200314 10:57:07 [Note] InnoDB: Waiting for purge to start
mariadb_1        | 200314 10:57:07 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.42-84.2 started; log sequence number 0
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T10:57:10Z","tags":["status","plugin:kibana@6.1.4-1","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T10:57:10Z","tags":["status","plugin:elasticsearch@6.1.4-1","info"],"pid":1,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T10:57:10Z","tags":["error","elasticsearch","admin"],"pid":1,"message":"Request error, retrying\nHEAD http://elasticsearch:9200/ => connect ECONNREFUSED 172.29.0.2:9200"}
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T10:57:10Z","tags":["status","plugin:console@6.1.4-1","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T10:57:10Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T10:57:10Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"}
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T10:57:10Z","tags":["status","plugin:elasticsearch@6.1.4-1","error"],"pid":1,"state":"red","message":"Status changed from yellow to red - Unable to connect to Elasticsearch at http://elasticsearch:9200.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T10:57:10Z","tags":["status","plugin:metrics@6.1.4-1","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T10:57:10Z","tags":["status","plugin:timelion@6.1.4-1","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T10:57:10Z","tags":["listening","info"],"pid":1,"message":"Server running at http://0.0.0.0:5601"}
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T10:57:13Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T10:57:13Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"}
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T10:57:15Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T10:57:15Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"}
mariadb_1        | 200314 10:57:17 [Note] InnoDB: FTS optimize thread exiting.
mariadb_1        | 200314 10:57:17 [Note] InnoDB: Starting shutdown...
mariadb_1        | 200314 10:57:17 [Note] InnoDB: Waiting for page_cleaner to finish flushing of buffer pool
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T10:57:18Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T10:57:18Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"}
elasticsearch_1  | [2020-03-14T10:57:18,538][INFO ][o.e.n.Node               ] [] initializing ...
elasticsearch_1  | [2020-03-14T10:57:18,822][INFO ][o.e.e.NodeEnvironment    ] [p2Tnmy3] using [1] data paths, mounts [[/ (overlay)]], net usable_space [35.3gb], net total_space [116.1gb], types [overlay]
elasticsearch_1  | [2020-03-14T10:57:18,823][INFO ][o.e.e.NodeEnvironment    ] [p2Tnmy3] heap size [1.9gb], compressed ordinary object pointers [true]
elasticsearch_1  | [2020-03-14T10:57:18,825][INFO ][o.e.n.Node               ] node name [p2Tnmy3] derived from node ID [p2Tnmy3HRW6aZVQoOcVC8Q]; set [node.name] to override
elasticsearch_1  | [2020-03-14T10:57:18,826][INFO ][o.e.n.Node               ] version[6.1.4], pid[1], build[d838f2d/2018-03-14T08:28:22.470Z], OS[Linux/5.3.0-40-generic/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_161/25.161-b14]
elasticsearch_1  | [2020-03-14T10:57:18,826][INFO ][o.e.n.Node               ] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -XX:+HeapDumpOnOutOfMemoryError, -Xms2g, -Xmx2g, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config]
mariadb_1        | 200314 10:57:19 [Note] InnoDB: Shutdown completed; log sequence number 1616697
mariadb_1        | 200314 10:57:19 [Note] /usr/sbin/mysqld (mysqld 10.0.38-MariaDB-1~xenial) starting as process 94 ...
mariadb_1        | 200314 10:57:19 [Note] InnoDB: Using mutexes to ref count buffer pool pages
mariadb_1        | 200314 10:57:19 [Note] InnoDB: The InnoDB memory heap is disabled
mariadb_1        | 200314 10:57:19 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
mariadb_1        | 200314 10:57:19 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
mariadb_1        | 200314 10:57:19 [Note] InnoDB: Compressed tables use zlib 1.2.8
mariadb_1        | 200314 10:57:19 [Note] InnoDB: Using Linux native AIO
mariadb_1        | 200314 10:57:19 [Note] InnoDB: Using CPU crc32 instructions
mariadb_1        | 200314 10:57:19 [Note] InnoDB: Initializing buffer pool, size = 256.0M
mariadb_1        | 200314 10:57:19 [Note] InnoDB: Completed initialization of buffer pool
mariadb_1        | 200314 10:57:20 [Note] InnoDB: Highest supported file format is Barracuda.
elasticsearch_1  | [2020-03-14T10:57:20,226][INFO ][o.e.p.PluginsService     ] [p2Tnmy3] loaded module [aggs-matrix-stats]
elasticsearch_1  | [2020-03-14T10:57:20,226][INFO ][o.e.p.PluginsService     ] [p2Tnmy3] loaded module [analysis-common]
elasticsearch_1  | [2020-03-14T10:57:20,226][INFO ][o.e.p.PluginsService     ] [p2Tnmy3] loaded module [ingest-common]
elasticsearch_1  | [2020-03-14T10:57:20,226][INFO ][o.e.p.PluginsService     ] [p2Tnmy3] loaded module [lang-expression]
elasticsearch_1  | [2020-03-14T10:57:20,226][INFO ][o.e.p.PluginsService     ] [p2Tnmy3] loaded module [lang-mustache]
elasticsearch_1  | [2020-03-14T10:57:20,226][INFO ][o.e.p.PluginsService     ] [p2Tnmy3] loaded module [lang-painless]
elasticsearch_1  | [2020-03-14T10:57:20,227][INFO ][o.e.p.PluginsService     ] [p2Tnmy3] loaded module [mapper-extras]
elasticsearch_1  | [2020-03-14T10:57:20,227][INFO ][o.e.p.PluginsService     ] [p2Tnmy3] loaded module [parent-join]
elasticsearch_1  | [2020-03-14T10:57:20,227][INFO ][o.e.p.PluginsService     ] [p2Tnmy3] loaded module [percolator]
elasticsearch_1  | [2020-03-14T10:57:20,227][INFO ][o.e.p.PluginsService     ] [p2Tnmy3] loaded module [reindex]
elasticsearch_1  | [2020-03-14T10:57:20,227][INFO ][o.e.p.PluginsService     ] [p2Tnmy3] loaded module [repository-url]
elasticsearch_1  | [2020-03-14T10:57:20,227][INFO ][o.e.p.PluginsService     ] [p2Tnmy3] loaded module [transport-netty4]
elasticsearch_1  | [2020-03-14T10:57:20,227][INFO ][o.e.p.PluginsService     ] [p2Tnmy3] loaded module [tribe]
elasticsearch_1  | [2020-03-14T10:57:20,228][INFO ][o.e.p.PluginsService     ] [p2Tnmy3] loaded plugin [ingest-geoip]
elasticsearch_1  | [2020-03-14T10:57:20,228][INFO ][o.e.p.PluginsService     ] [p2Tnmy3] loaded plugin [ingest-user-agent]
mariadb_1        | 200314 10:57:20 [Note] InnoDB: 128 rollback segment(s) are active.
mariadb_1        | 200314 10:57:20 [Note] InnoDB: Waiting for purge to start
mariadb_1        | 200314 10:57:20 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.42-84.2 started; log sequence number 1616697
mariadb_1        | 200314 10:57:20 [Note] InnoDB: FTS optimize thread exiting.
mariadb_1        | 200314 10:57:20 [Note] InnoDB: Starting shutdown...
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T10:57:20Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T10:57:20Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"}
mariadb_1        | 200314 10:57:21 [Note] InnoDB: Waiting for page_cleaner to finish flushing of buffer pool
elasticsearch_1  | [2020-03-14T10:57:22,224][INFO ][o.e.d.DiscoveryModule    ] [p2Tnmy3] using discovery type [zen]
elasticsearch_1  | [2020-03-14T10:57:22,686][INFO ][o.e.n.Node               ] initialized
elasticsearch_1  | [2020-03-14T10:57:22,687][INFO ][o.e.n.Node               ] [p2Tnmy3] starting ...
elasticsearch_1  | [2020-03-14T10:57:22,859][INFO ][o.e.t.TransportService   ] [p2Tnmy3] publish_address {172.29.0.2:9300}, bound_addresses {0.0.0.0:9300}
elasticsearch_1  | [2020-03-14T10:57:22,865][INFO ][o.e.b.BootstrapChecks    ] [p2Tnmy3] bound or publishing to a non-loopback address, enforcing bootstrap checks
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T10:57:23Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T10:57:23Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"}
mariadb_1        | 200314 10:57:23 [Note] InnoDB: Shutdown completed; log sequence number 1616707
mariadb_1        | 
mariadb_1        | PLEASE REMEMBER TO SET A PASSWORD FOR THE MariaDB root USER !
mariadb_1        | To do so, start the server, then issue the following commands:
mariadb_1        | 
mariadb_1        | '/usr/bin/mysqladmin' -u root password 'new-password'
mariadb_1        | '/usr/bin/mysqladmin' -u root -h  password 'new-password'
mariadb_1        | 
mariadb_1        | Alternatively you can run:
mariadb_1        | '/usr/bin/mysql_secure_installation'
mariadb_1        | 
mariadb_1        | which will also give you the option of removing the test
mariadb_1        | databases and anonymous user created by default.  This is
mariadb_1        | strongly recommended for production servers.
mariadb_1        | 
mariadb_1        | See the MariaDB Knowledgebase at http://mariadb.com/kb or the
mariadb_1        | MySQL manual for more instructions.
mariadb_1        | 
mariadb_1        | Please report any problems at http://mariadb.org/jira
mariadb_1        | 
mariadb_1        | The latest information about MariaDB is available at http://mariadb.org/.
mariadb_1        | You can find additional information about the MySQL part at:
mariadb_1        | http://dev.mysql.com
mariadb_1        | Consider joining MariaDB's strong and vibrant community:
mariadb_1        | https://mariadb.org/get-involved/
mariadb_1        | 
mariadb_1        | Database initialized
mariadb_1        | MySQL init process in progress...
mariadb_1        | 200314 10:57:23 [Note] mysqld (mysqld 10.0.38-MariaDB-1~xenial) starting as process 124 ...
mariadb_1        | 200314 10:57:23 [Note] InnoDB: Using mutexes to ref count buffer pool pages
mariadb_1        | 200314 10:57:23 [Note] InnoDB: The InnoDB memory heap is disabled
mariadb_1        | 200314 10:57:23 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
mariadb_1        | 200314 10:57:23 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
mariadb_1        | 200314 10:57:23 [Note] InnoDB: Compressed tables use zlib 1.2.8
mariadb_1        | 200314 10:57:23 [Note] InnoDB: Using Linux native AIO
mariadb_1        | 200314 10:57:23 [Note] InnoDB: Using CPU crc32 instructions
mariadb_1        | 200314 10:57:23 [Note] InnoDB: Initializing buffer pool, size = 256.0M
mariadb_1        | 200314 10:57:23 [Note] InnoDB: Completed initialization of buffer pool
mariadb_1        | 200314 10:57:23 [Note] InnoDB: Highest supported file format is Barracuda.
mariadb_1        | 200314 10:57:23 [Note] InnoDB: 128 rollback segment(s) are active.
mariadb_1        | 200314 10:57:23 [Note] InnoDB: Waiting for purge to start
mariadb_1        | 200314 10:57:23 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.42-84.2 started; log sequence number 1616707
mariadb_1        | 200314 10:57:23 [Note] Plugin 'FEEDBACK' is disabled.
mariadb_1        | 200314 10:57:23 [Warning] 'user' entry 'root@8e7aa3d8a2d8' ignored in --skip-name-resolve mode.
mariadb_1        | 200314 10:57:23 [Warning] 'proxies_priv' entry '@% root@8e7aa3d8a2d8' ignored in --skip-name-resolve mode.
mariadb_1        | 200314 10:57:23 [Note] mysqld: ready for connections.
mariadb_1        | Version: '10.0.38-MariaDB-1~xenial'  socket: '/var/run/mysqld/mysqld.sock'  port: 0  mariadb.org binary distribution
mariadb_1        | Warning: Unable to load '/usr/share/zoneinfo/leap-seconds.list' as time zone. Skipping it.
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T10:57:25Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T10:57:25Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"}
mariadb_1        | 200314 10:57:25 [Warning] 'proxies_priv' entry '@% root@8e7aa3d8a2d8' ignored in --skip-name-resolve mode.
mariadb_1        | 
mariadb_1        | 200314 10:57:25 [Note] mysqld: Normal shutdown
mariadb_1        | 200314 10:57:25 [Note] Event Scheduler: Purging the queue. 0 events
mariadb_1        | 200314 10:57:25 [Note] InnoDB: FTS optimize thread exiting.
mariadb_1        | 200314 10:57:25 [Note] InnoDB: Starting shutdown...
elasticsearch_1  | [2020-03-14T10:57:25,917][INFO ][o.e.c.s.MasterService    ] [p2Tnmy3] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {p2Tnmy3}{p2Tnmy3HRW6aZVQoOcVC8Q}{GvYFKX8XT5OrVVx1oKA3TQ}{172.29.0.2}{172.29.0.2:9300}
elasticsearch_1  | [2020-03-14T10:57:25,923][INFO ][o.e.c.s.ClusterApplierService] [p2Tnmy3] new_master {p2Tnmy3}{p2Tnmy3HRW6aZVQoOcVC8Q}{GvYFKX8XT5OrVVx1oKA3TQ}{172.29.0.2}{172.29.0.2:9300}, reason: apply cluster state (from master [master {p2Tnmy3}{p2Tnmy3HRW6aZVQoOcVC8Q}{GvYFKX8XT5OrVVx1oKA3TQ}{172.29.0.2}{172.29.0.2:9300} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
elasticsearch_1  | [2020-03-14T10:57:25,934][INFO ][o.e.h.n.Netty4HttpServerTransport] [p2Tnmy3] publish_address {172.29.0.2:9200}, bound_addresses {0.0.0.0:9200}
elasticsearch_1  | [2020-03-14T10:57:25,935][INFO ][o.e.n.Node               ] [p2Tnmy3] started
mariadb_1        | 200314 10:57:25 [Note] InnoDB: Waiting for page_cleaner to finish flushing of buffer pool
elasticsearch_1  | [2020-03-14T10:57:26,088][INFO ][o.e.g.GatewayService     ] [p2Tnmy3] recovered [0] indices into cluster_state
mariadb_1        | 200314 10:57:27 [Note] InnoDB: Shutdown completed; log sequence number 1616717
mariadb_1        | 200314 10:57:27 [Note] mysqld: Shutdown complete
mariadb_1        | 
mariadb_1        | 
mariadb_1        | MySQL init process done. Ready for start up.
mariadb_1        | 
mariadb_1        | 200314 10:57:27 [Note] mysqld (mysqld 10.0.38-MariaDB-1~xenial) starting as process 1 ...
mariadb_1        | 200314 10:57:27 [Note] InnoDB: Using mutexes to ref count buffer pool pages
mariadb_1        | 200314 10:57:27 [Note] InnoDB: The InnoDB memory heap is disabled
mariadb_1        | 200314 10:57:27 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
mariadb_1        | 200314 10:57:27 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
mariadb_1        | 200314 10:57:27 [Note] InnoDB: Compressed tables use zlib 1.2.8
mariadb_1        | 200314 10:57:27 [Note] InnoDB: Using Linux native AIO
mariadb_1        | 200314 10:57:27 [Note] InnoDB: Using CPU crc32 instructions
mariadb_1        | 200314 10:57:27 [Note] InnoDB: Initializing buffer pool, size = 256.0M
mariadb_1        | 200314 10:57:27 [Note] InnoDB: Completed initialization of buffer pool
mariadb_1        | 200314 10:57:27 [Note] InnoDB: Highest supported file format is Barracuda.
mariadb_1        | 200314 10:57:27 [Note] InnoDB: 128 rollback segment(s) are active.
mariadb_1        | 200314 10:57:27 [Note] InnoDB: Waiting for purge to start
mariadb_1        | 200314 10:57:27 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.42-84.2 started; log sequence number 1616717
mariadb_1        | 200314 10:57:27 [Note] Plugin 'FEEDBACK' is disabled.
mariadb_1        | 200314 10:57:27 [Note] Server socket created on IP: '::'.
mariadb_1        | 200314 10:57:27 [Warning] 'proxies_priv' entry '@% root@8e7aa3d8a2d8' ignored in --skip-name-resolve mode.
mariadb_1        | 200314 10:57:27 [Note] mysqld: ready for connections.
mariadb_1        | Version: '10.0.38-MariaDB-1~xenial'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T10:57:28Z","tags":["warning"],"pid":1,"kibanaVersion":"6.1.4-1","nodes":[{"version":"6.1.4","http":{"publish_address":"172.29.0.2:9200"},"ip":"172.29.0.2"}],"message":"You're running Kibana 6.1.4-1 with some different versions of Elasticsearch. Update Kibana or Elasticsearch to the same version to prevent compatibility issues: v6.1.4 @ 172.29.0.2:9200 (172.29.0.2)"}
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T10:57:28Z","tags":["status","plugin:elasticsearch@6.1.4-1","info"],"pid":1,"state":"green","message":"Status changed from red to green - Ready","prevState":"red","prevMsg":"Unable to connect to Elasticsearch at http://elasticsearch:9200."}
kibiter_1        | {"type":"response","@timestamp":"2020-03-14T10:57:54Z","tags":[],"pid":1,"method":"get","statusCode":200,"req":{"url":"/","method":"get","headers":{"host":"localhost:5601","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:73.0) Gecko/20100101 Firefox/73.0","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate","connection":"keep-alive","upgrade-insecure-requests":"1","cache-control":"max-age=0"},"remoteAddress":"172.29.0.1","userAgent":"172.29.0.1"},"res":{"statusCode":200,"responseTime":64,"contentLength":9},"message":"GET / 200 64ms - 9.0B"}
kibiter_1        | {"type":"response","@timestamp":"2020-03-14T10:57:55Z","tags":[],"pid":1,"method":"get","statusCode":200,"req":{"url":"/app/kibana","method":"get","headers":{"host":"localhost:5601","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:73.0) Gecko/20100101 Firefox/73.0","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate","connection":"keep-alive","referer":"http://localhost:5601/","upgrade-insecure-requests":"1"},"remoteAddress":"172.29.0.1","userAgent":"172.29.0.1","referer":"http://localhost:5601/"},"res":{"statusCode":200,"responseTime":252,"contentLength":9},"message":"GET /app/kibana 200 252ms - 9.0B"}
kibiter_1        | {"type":"response","@timestamp":"2020-03-14T10:57:55Z","tags":[],"pid":1,"method":"get","statusCode":200,"req":{"url":"/bundles/kibana.style.css?v=1","method":"get","headers":{"host":"localhost:5601","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:73.0) Gecko/20100101 Firefox/73.0","accept":"text/css,*/*;q=0.1","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate","connection":"keep-alive","referer":"http://localhost:5601/app/kibana","if-modified-since":"Wed, 18 Sep 2019 17:15:31 GMT","if-none-match":"\"3db9d568504a51be3350c2ef4051f03178d8c55d-/bundles/-gzip\""},"remoteAddress":"172.29.0.1","userAgent":"172.29.0.1","referer":"http://localhost:5601/app/kibana"},"res":{"statusCode":200,"responseTime":36,"contentLength":9},"message":"GET /bundles/kibana.style.css?v=1 200 36ms - 9.0B"}
kibiter_1        | {"type":"response","@timestamp":"2020-03-14T10:57:55Z","tags":[],"pid":1,"method":"get","statusCode":200,"req":{"url":"/bundles/commons.style.css?v=1","method":"get","headers":{"host":"localhost:5601","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:73.0) Gecko/20100101 Firefox/73.0","accept":"text/css,*/*;q=0.1","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate","connection":"keep-alive","referer":"http://localhost:5601/app/kibana"},"remoteAddress":"172.29.0.1","userAgent":"172.29.0.1","referer":"http://localhost:5601/app/kibana"},"res":{"statusCode":200,"responseTime":43,"contentLength":9},"message":"GET /bundles/commons.style.css?v=1 200 43ms - 9.0B"}
kibiter_1        | {"type":"response","@timestamp":"2020-03-14T10:57:55Z","tags":[],"pid":1,"method":"get","statusCode":200,"req":{"url":"/bundles/commons.bundle.js?v=1","method":"get","headers":{"host":"localhost:5601","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:73.0) Gecko/20100101 Firefox/73.0","accept":"*/*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate","connection":"keep-alive","referer":"http://localhost:5601/app/kibana","if-modified-since":"Wed, 18 Sep 2019 17:15:31 GMT","if-none-match":"\"cc24657c5f57c8846b0ca2480431a2505a30d4ae-/bundles/-gzip\""},"remoteAddress":"172.29.0.1","userAgent":"172.29.0.1","referer":"http://localhost:5601/app/kibana"},"res":{"statusCode":200,"responseTime":106,"contentLength":9},"message":"GET /bundles/commons.bundle.js?v=1 200 106ms - 9.0B"}
kibiter_1        | {"type":"response","@timestamp":"2020-03-14T10:57:55Z","tags":[],"pid":1,"method":"get","statusCode":200,"req":{"url":"/bundles/kibana.bundle.js?v=1","method":"get","headers":{"host":"localhost:5601","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:73.0) Gecko/20100101 Firefox/73.0","accept":"*/*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate","connection":"keep-alive","referer":"http://localhost:5601/app/kibana","if-modified-since":"Wed, 18 Sep 2019 17:15:31 GMT","if-none-match":"\"6d1873a30f93e1e70e743bbaccfdfe0a666b914b-/bundles/-gzip\""},"remoteAddress":"172.29.0.1","userAgent":"172.29.0.1","referer":"http://localhost:5601/app/kibana"},"res":{"statusCode":200,"responseTime":404,"contentLength":9},"message":"GET /bundles/kibana.bundle.js?v=1 200 404ms - 9.0B"}
kibiter_1        | {"type":"response","@timestamp":"2020-03-14T10:57:57Z","tags":[],"pid":1,"method":"get","statusCode":200,"req":{"url":"/api/console/api_server?sense_version=%40%40SENSE_VERSION&apis=es_5_0","method":"get","headers":{"host":"localhost:5601","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:73.0) Gecko/20100101 Firefox/73.0","accept":"application/json, text/javascript, */*; q=0.01","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate","kbn-version":"6.1.4-1","x-requested-with":"XMLHttpRequest","connection":"keep-alive","referer":"http://localhost:5601/app/kibana"},"remoteAddress":"172.29.0.1","userAgent":"172.29.0.1","referer":"http://localhost:5601/app/kibana"},"res":{"statusCode":200,"responseTime":27,"contentLength":9},"message":"GET /api/console/api_server?sense_version=%40%40SENSE_VERSION&apis=es_5_0 200 27ms - 9.0B"}
kibiter_1        | {"type":"response","@timestamp":"2020-03-14T10:57:57Z","tags":[],"pid":1,"method":"get","statusCode":200,"req":{"url":"/plugins/kibana/assets/discover.svg","method":"get","headers":{"host":"localhost:5601","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:73.0) Gecko/20100101 Firefox/73.0","accept":"image/webp,*/*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate","connection":"keep-alive","referer":"http://localhost:5601/app/kibana"},"remoteAddress":"172.29.0.1","userAgent":"172.29.0.1","referer":"http://localhost:5601/app/kibana"},"res":{"statusCode":200,"responseTime":66,"contentLength":9},"message":"GET /plugins/kibana/assets/discover.svg 200 66ms - 9.0B"}
kibiter_1        | {"type":"response","@timestamp":"2020-03-14T10:57:57Z","tags":[],"pid":1,"method":"get","statusCode":304,"req":{"url":"/plugins/kibana/assets/visualize.svg","method":"get","headers":{"host":"localhost:5601","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:73.0) Gecko/20100101 Firefox/73.0","accept":"image/webp,*/*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate","connection":"keep-alive","referer":"http://localhost:5601/app/kibana","if-modified-since":"Wed, 18 Sep 2019 16:41:48 GMT","if-none-match":"\"4cc79a4d91bd0380d0c82a6b092f339d185670ef-gzip\""},"remoteAddress":"172.29.0.1","userAgent":"172.29.0.1","referer":"http://localhost:5601/app/kibana"},"res":{"statusCode":304,"responseTime":8,"contentLength":9},"message":"GET /plugins/kibana/assets/visualize.svg 304 8ms - 9.0B"}
kibiter_1        | {"type":"response","@timestamp":"2020-03-14T10:57:57Z","tags":[],"pid":1,"method":"get","statusCode":304,"req":{"url":"/plugins/kibana/assets/dashboard.svg","method":"get","headers":{"host":"localhost:5601","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:73.0) Gecko/20100101 Firefox/73.0","accept":"image/webp,*/*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate","connection":"keep-alive","referer":"http://localhost:5601/app/kibana","if-modified-since":"Wed, 18 Sep 2019 16:41:48 GMT","if-none-match":"\"42c2161fa64691414784868afdd722444460763a-gzip\""},"remoteAddress":"172.29.0.1","userAgent":"172.29.0.1","referer":"http://localhost:5601/app/kibana"},"res":{"statusCode":304,"responseTime":10,"contentLength":9},"message":"GET /plugins/kibana/assets/dashboard.svg 304 10ms - 9.0B"}
kibiter_1        | {"type":"response","@timestamp":"2020-03-14T10:57:57Z","tags":[],"pid":1,"method":"get","statusCode":304,"req":{"url":"/plugins/timelion/icon.svg","method":"get","headers":{"host":"localhost:5601","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:73.0) Gecko/20100101 Firefox/73.0","accept":"image/webp,*/*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate","connection":"keep-alive","referer":"http://localhost:5601/app/kibana","if-modified-since":"Wed, 18 Sep 2019 16:41:48 GMT","if-none-match":"\"cb793d5314d680b7d5ce130f0393a70b51989541-gzip\""},"remoteAddress":"172.29.0.1","userAgent":"172.29.0.1","referer":"http://localhost:5601/app/kibana"},"res":{"statusCode":304,"responseTime":16,"contentLength":9},"message":"GET /plugins/timelion/icon.svg 304 16ms - 9.0B"}
kibiter_1        | {"type":"response","@timestamp":"2020-03-14T10:57:57Z","tags":[],"pid":1,"method":"get","statusCode":304,"req":{"url":"/plugins/kibana/assets/settings.svg","method":"get","headers":{"host":"localhost:5601","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:73.0) Gecko/20100101 Firefox/73.0","accept":"image/webp,*/*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate","connection":"keep-alive","referer":"http://localhost:5601/app/kibana","if-modified-since":"Wed, 18 Sep 2019 16:41:48 GMT","if-none-match":"\"4f859e27d4917026ff1590805887902b14ce79d5-gzip\""},"remoteAddress":"172.29.0.1","userAgent":"172.29.0.1","referer":"http://localhost:5601/app/kibana"},"res":{"statusCode":304,"responseTime":18,"contentLength":9},"message":"GET /plugins/kibana/assets/settings.svg 304 18ms - 9.0B"}
kibiter_1        | {"type":"response","@timestamp":"2020-03-14T10:57:57Z","tags":[],"pid":1,"method":"get","statusCode":304,"req":{"url":"/plugins/kibana/assets/wrench.svg","method":"get","headers":{"host":"localhost:5601","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:73.0) Gecko/20100101 Firefox/73.0","accept":"image/webp,*/*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate","connection":"keep-alive","referer":"http://localhost:5601/app/kibana","if-modified-since":"Wed, 18 Sep 2019 16:41:48 GMT","if-none-match":"\"088a9a98c99e406dca2354af14f688ad84826b97-gzip\""},"remoteAddress":"172.29.0.1","userAgent":"172.29.0.1","referer":"http://localhost:5601/app/kibana"},"res":{"statusCode":304,"responseTime":20,"contentLength":9},"message":"GET /plugins/kibana/assets/wrench.svg 304 20ms - 9.0B"}
kibiter_1        | {"type":"response","@timestamp":"2020-03-14T10:57:57Z","tags":[],"pid":1,"method":"get","statusCode":304,"req":{"url":"/plugins/kibana/assets/play-circle.svg","method":"get","headers":{"host":"localhost:5601","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:73.0) Gecko/20100101 Firefox/73.0","accept":"image/webp,*/*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate","connection":"keep-alive","referer":"http://localhost:5601/app/kibana","if-modified-since":"Wed, 18 Sep 2019 16:41:48 GMT","if-none-match":"\"2433ecf38258f7121c835670b6993600e7657717-gzip\""},"remoteAddress":"172.29.0.1","userAgent":"172.29.0.1","referer":"http://localhost:5601/app/kibana"},"res":{"statusCode":304,"responseTime":3,"contentLength":9},"message":"GET /plugins/kibana/assets/play-circle.svg 304 3ms - 9.0B"}
kibiter_1        | {"type":"response","@timestamp":"2020-03-14T10:57:57Z","tags":[],"pid":1,"method":"get","statusCode":200,"req":{"url":"/bundles/4246f178bc05db44a8b68c7acf264597.svg","method":"get","headers":{"host":"localhost:5601","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:73.0) Gecko/20100101 Firefox/73.0","accept":"image/webp,*/*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate","connection":"keep-alive","referer":"http://localhost:5601/bundles/commons.style.css?v=1"},"remoteAddress":"172.29.0.1","userAgent":"172.29.0.1","referer":"http://localhost:5601/bundles/commons.style.css?v=1"},"res":{"statusCode":200,"responseTime":6,"contentLength":9},"message":"GET /bundles/4246f178bc05db44a8b68c7acf264597.svg 200 6ms - 9.0B"}
kibiter_1        | {"type":"response","@timestamp":"2020-03-14T10:57:57Z","tags":[],"pid":1,"method":"get","statusCode":200,"req":{"url":"/api/saved_objects/?type=index-pattern&per_page=10000&page=1","method":"get","headers":{"host":"localhost:5601","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:73.0) Gecko/20100101 Firefox/73.0","accept":"application/json, text/plain, */*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate","kbn-version":"6.1.4-1","connection":"keep-alive","referer":"http://localhost:5601/app/kibana"},"remoteAddress":"172.29.0.1","userAgent":"172.29.0.1","referer":"http://localhost:5601/app/kibana"},"res":{"statusCode":200,"responseTime":27,"contentLength":9},"message":"GET /api/saved_objects/?type=index-pattern&per_page=10000&page=1 200 27ms - 9.0B"}
kibiter_1        | {"type":"response","@timestamp":"2020-03-14T10:57:57Z","tags":[],"pid":1,"method":"get","statusCode":200,"req":{"url":"/api/saved_objects/?type=index-pattern&fields=title&per_page=10000&page=1","method":"get","headers":{"host":"localhost:5601","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:73.0) Gecko/20100101 Firefox/73.0","accept":"application/json, text/plain, */*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate","kbn-version":"6.1.4-1","connection":"keep-alive","referer":"http://localhost:5601/app/kibana"},"remoteAddress":"172.29.0.1","userAgent":"172.29.0.1","referer":"http://localhost:5601/app/kibana"},"res":{"statusCode":200,"responseTime":8,"contentLength":9},"message":"GET /api/saved_objects/?type=index-pattern&fields=title&per_page=10000&page=1 200 8ms - 9.0B"}
kibiter_1        | {"type":"response","@timestamp":"2020-03-14T10:57:57Z","tags":[],"pid":1,"method":"post","statusCode":404,"req":{"url":"/elasticsearch/.kibana/_search","method":"post","headers":{"host":"localhost:5601","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:73.0) Gecko/20100101 Firefox/73.0","accept":"application/json, text/plain, */*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate","content-type":"application/json","kbn-version":"6.1.4-1","content-length":"43","origin":"http://localhost:5601","connection":"keep-alive","referer":"http://localhost:5601/app/kibana"},"remoteAddress":"172.29.0.1","userAgent":"172.29.0.1","referer":"http://localhost:5601/app/kibana"},"res":{"statusCode":404,"responseTime":32,"contentLength":9},"message":"POST /elasticsearch/.kibana/_search 404 32ms - 9.0B"}
kibiter_1        | {"type":"response","@timestamp":"2020-03-14T10:57:57Z","tags":[],"pid":1,"method":"post","statusCode":404,"req":{"url":"/elasticsearch/.kibana/_search","method":"post","headers":{"host":"localhost:5601","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:73.0) Gecko/20100101 Firefox/73.0","accept":"application/json, text/plain, */*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate","content-type":"application/json","kbn-version":"6.1.4-1","content-length":"41","origin":"http://localhost:5601","connection":"keep-alive","referer":"http://localhost:5601/app/kibana"},"remoteAddress":"172.29.0.1","userAgent":"172.29.0.1","referer":"http://localhost:5601/app/kibana"},"res":{"statusCode":404,"responseTime":28,"contentLength":9},"message":"POST /elasticsearch/.kibana/_search 404 28ms - 9.0B"}
kibiter_1        | {"type":"response","@timestamp":"2020-03-14T10:57:57Z","tags":[],"pid":1,"method":"post","statusCode":200,"req":{"url":"/elasticsearch/*/_search","method":"post","headers":{"host":"localhost:5601","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:73.0) Gecko/20100101 Firefox/73.0","accept":"application/json, text/plain, */*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate","content-type":"application/json","kbn-version":"6.1.4-1","content-length":"68","origin":"http://localhost:5601","connection":"keep-alive","referer":"http://localhost:5601/app/kibana"},"remoteAddress":"172.29.0.1","userAgent":"172.29.0.1","referer":"http://localhost:5601/app/kibana"},"res":{"statusCode":200,"responseTime":34,"contentLength":9},"message":"POST /elasticsearch/*/_search 200 34ms - 9.0B"}
^CGracefully stopping... (press Ctrl+C again to force)
Stopping utils_kibiter_1       ... 
Stopping utils_mariadb_1       ... 
Stopping utils_elasticsearch_1 ... 
^CKilling utils_kibiter_1        ... 
Killing utils_mariadb_1        ... 
Killing utils_elasticsearch_1  ... 
^CERROR: Aborting.
animesh@msi:~/glsource/grimoirelab-sirmordred/utils$ docker-compose -f docker-compose.yml down
Removing utils_kibiter_1       ... done
Removing utils_mariadb_1       ... done
Removing utils_elasticsearch_1 ... done
Removing network utils_default
animesh@msi:~/glsource/grimoirelab-sirmordred/utils$ docker-compose -f docker-compose.yml up
Creating network "utils_default" with the default driver
Creating utils_mariadb_1       ... done
Creating utils_elasticsearch_1 ... done
Creating utils_kibiter_1       ... done
Attaching to utils_elasticsearch_1, utils_mariadb_1, utils_kibiter_1
mariadb_1        | Initializing database
mariadb_1        | 200314 11:00:53 [Note] /usr/sbin/mysqld (mysqld 10.0.38-MariaDB-1~xenial) starting as process 68 ...
elasticsearch_1  | [2020-03-14T11:00:55,154][INFO ][o.e.n.Node               ] [] initializing ...
mariadb_1        | 200314 11:00:54 [Note] InnoDB: Using mutexes to ref count buffer pool pages
elasticsearch_1  | [2020-03-14T11:00:55,809][INFO ][o.e.e.NodeEnvironment    ] [Mmigh3Z] using [1] data paths, mounts [[/ (overlay)]], net usable_space [35.2gb], net total_space [116.1gb], types [overlay]
mariadb_1        | 200314 11:00:54 [Note] InnoDB: The InnoDB memory heap is disabled
mariadb_1        | 200314 11:00:54 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
mariadb_1        | 200314 11:00:54 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
mariadb_1        | 200314 11:00:54 [Note] InnoDB: Compressed tables use zlib 1.2.8
elasticsearch_1  | [2020-03-14T11:00:55,810][INFO ][o.e.e.NodeEnvironment    ] [Mmigh3Z] heap size [1.9gb], compressed ordinary object pointers [true]
mariadb_1        | 200314 11:00:54 [Note] InnoDB: Using Linux native AIO
mariadb_1        | 200314 11:00:54 [Note] InnoDB: Using CPU crc32 instructions
elasticsearch_1  | [2020-03-14T11:00:55,920][INFO ][o.e.n.Node               ] node name [Mmigh3Z] derived from node ID [Mmigh3ZvRAKJ6jLpeeiHCA]; set [node.name] to override
elasticsearch_1  | [2020-03-14T11:00:55,920][INFO ][o.e.n.Node               ] version[6.1.4], pid[1], build[d838f2d/2018-03-14T08:28:22.470Z], OS[Linux/5.3.0-40-generic/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_161/25.161-b14]
mariadb_1        | 200314 11:00:54 [Note] InnoDB: Initializing buffer pool, size = 256.0M
mariadb_1        | 200314 11:00:54 [Note] InnoDB: Completed initialization of buffer pool
elasticsearch_1  | [2020-03-14T11:00:55,921][INFO ][o.e.n.Node               ] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -XX:+HeapDumpOnOutOfMemoryError, -Xms2g, -Xmx2g, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config]
mariadb_1        | 200314 11:00:54 [Note] InnoDB: The first specified data file ./ibdata1 did not exist: a new database to be created!
mariadb_1        | 200314 11:00:54 [Note] InnoDB: Setting file ./ibdata1 size to 12 MB
mariadb_1        | 200314 11:00:54 [Note] InnoDB: Database physically writes the file full: wait...
elasticsearch_1  | [2020-03-14T11:00:56,586][INFO ][o.e.p.PluginsService     ] [Mmigh3Z] loaded module [aggs-matrix-stats]
mariadb_1        | 200314 11:00:55 [Note] InnoDB: Setting log file ./ib_logfile101 size to 48 MB
elasticsearch_1  | [2020-03-14T11:00:56,587][INFO ][o.e.p.PluginsService     ] [Mmigh3Z] loaded module [analysis-common]
mariadb_1        | 200314 11:00:56 [Note] InnoDB: Setting log file ./ib_logfile1 size to 48 MB
elasticsearch_1  | [2020-03-14T11:00:56,587][INFO ][o.e.p.PluginsService     ] [Mmigh3Z] loaded module [ingest-common]
mariadb_1        | 200314 11:00:59 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0
mariadb_1        | 200314 11:00:59 [Warning] InnoDB: New log files created, LSN=45781
mariadb_1        | 200314 11:00:59 [Note] InnoDB: Doublewrite buffer not found: creating new
elasticsearch_1  | [2020-03-14T11:00:56,587][INFO ][o.e.p.PluginsService     ] [Mmigh3Z] loaded module [lang-expression]
mariadb_1        | 200314 11:01:00 [Note] InnoDB: Doublewrite buffer created
elasticsearch_1  | [2020-03-14T11:00:56,587][INFO ][o.e.p.PluginsService     ] [Mmigh3Z] loaded module [lang-mustache]
elasticsearch_1  | [2020-03-14T11:00:56,587][INFO ][o.e.p.PluginsService     ] [Mmigh3Z] loaded module [lang-painless]
elasticsearch_1  | [2020-03-14T11:00:56,587][INFO ][o.e.p.PluginsService     ] [Mmigh3Z] loaded module [mapper-extras]
mariadb_1        | 200314 11:01:00 [Note] InnoDB: 128 rollback segment(s) are active.
elasticsearch_1  | [2020-03-14T11:00:56,587][INFO ][o.e.p.PluginsService     ] [Mmigh3Z] loaded module [parent-join]
mariadb_1        | 200314 11:01:00 [Warning] InnoDB: Creating foreign key constraint system tables.
elasticsearch_1  | [2020-03-14T11:00:56,587][INFO ][o.e.p.PluginsService     ] [Mmigh3Z] loaded module [percolator]
mariadb_1        | 200314 11:01:00 [Note] InnoDB: Foreign key constraint system tables created
mariadb_1        | 200314 11:01:00 [Note] InnoDB: Creating tablespace and datafile system tables.
mariadb_1        | 200314 11:01:01 [Note] InnoDB: Tablespace and datafile system tables created.
elasticsearch_1  | [2020-03-14T11:00:56,587][INFO ][o.e.p.PluginsService     ] [Mmigh3Z] loaded module [reindex]
mariadb_1        | 200314 11:01:01 [Note] InnoDB: Waiting for purge to start
elasticsearch_1  | [2020-03-14T11:00:56,587][INFO ][o.e.p.PluginsService     ] [Mmigh3Z] loaded module [repository-url]
mariadb_1        | 200314 11:01:01 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.42-84.2 started; log sequence number 0
elasticsearch_1  | [2020-03-14T11:00:56,587][INFO ][o.e.p.PluginsService     ] [Mmigh3Z] loaded module [transport-netty4]
elasticsearch_1  | [2020-03-14T11:00:56,588][INFO ][o.e.p.PluginsService     ] [Mmigh3Z] loaded module [tribe]
elasticsearch_1  | [2020-03-14T11:00:56,588][INFO ][o.e.p.PluginsService     ] [Mmigh3Z] loaded plugin [ingest-geoip]
elasticsearch_1  | [2020-03-14T11:00:56,588][INFO ][o.e.p.PluginsService     ] [Mmigh3Z] loaded plugin [ingest-user-agent]
elasticsearch_1  | [2020-03-14T11:00:57,684][INFO ][o.e.d.DiscoveryModule    ] [Mmigh3Z] using discovery type [zen]
elasticsearch_1  | [2020-03-14T11:00:58,029][INFO ][o.e.n.Node               ] initialized
elasticsearch_1  | [2020-03-14T11:00:58,030][INFO ][o.e.n.Node               ] [Mmigh3Z] starting ...
elasticsearch_1  | [2020-03-14T11:00:58,116][INFO ][o.e.t.TransportService   ] [Mmigh3Z] publish_address {172.30.0.2:9300}, bound_addresses {0.0.0.0:9300}
elasticsearch_1  | [2020-03-14T11:00:58,122][INFO ][o.e.b.BootstrapChecks    ] [Mmigh3Z] bound or publishing to a non-loopback address, enforcing bootstrap checks
elasticsearch_1  | [2020-03-14T11:01:01,173][INFO ][o.e.c.s.MasterService    ] [Mmigh3Z] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {Mmigh3Z}{Mmigh3ZvRAKJ6jLpeeiHCA}{k2LJhg6BSOKQRuSy5O-OiQ}{172.30.0.2}{172.30.0.2:9300}
elasticsearch_1  | [2020-03-14T11:01:01,179][INFO ][o.e.c.s.ClusterApplierService] [Mmigh3Z] new_master {Mmigh3Z}{Mmigh3ZvRAKJ6jLpeeiHCA}{k2LJhg6BSOKQRuSy5O-OiQ}{172.30.0.2}{172.30.0.2:9300}, reason: apply cluster state (from master [master {Mmigh3Z}{Mmigh3ZvRAKJ6jLpeeiHCA}{k2LJhg6BSOKQRuSy5O-OiQ}{172.30.0.2}{172.30.0.2:9300} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
elasticsearch_1  | [2020-03-14T11:01:01,189][INFO ][o.e.h.n.Netty4HttpServerTransport] [Mmigh3Z] publish_address {172.30.0.2:9200}, bound_addresses {0.0.0.0:9200}
elasticsearch_1  | [2020-03-14T11:01:01,189][INFO ][o.e.n.Node               ] [Mmigh3Z] started
elasticsearch_1  | [2020-03-14T11:01:01,382][INFO ][o.e.g.GatewayService     ] [Mmigh3Z] recovered [0] indices into cluster_state
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T11:01:03Z","tags":["status","plugin:kibana@6.1.4-1","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T11:01:03Z","tags":["status","plugin:elasticsearch@6.1.4-1","info"],"pid":1,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T11:01:03Z","tags":["status","plugin:console@6.1.4-1","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T11:01:03Z","tags":["status","plugin:metrics@6.1.4-1","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T11:01:03Z","tags":["status","plugin:timelion@6.1.4-1","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T11:01:03Z","tags":["listening","info"],"pid":1,"message":"Server running at http://0.0.0.0:5601"}
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T11:01:03Z","tags":["warning"],"pid":1,"kibanaVersion":"6.1.4-1","nodes":[{"version":"6.1.4","http":{"publish_address":"172.30.0.2:9200"},"ip":"172.30.0.2"}],"message":"You're running Kibana 6.1.4-1 with some different versions of Elasticsearch. Update Kibana or Elasticsearch to the same version to prevent compatibility issues: v6.1.4 @ 172.30.0.2:9200 (172.30.0.2)"}
kibiter_1        | {"type":"log","@timestamp":"2020-03-14T11:01:03Z","tags":["status","plugin:elasticsearch@6.1.4-1","info"],"pid":1,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
mariadb_1        | 200314 11:01:11 [Note] InnoDB: FTS optimize thread exiting.
mariadb_1        | 200314 11:01:11 [Note] InnoDB: Starting shutdown...
mariadb_1        | 200314 11:01:12 [Note] InnoDB: Waiting for page_cleaner to finish flushing of buffer pool
mariadb_1        | 200314 11:01:14 [Note] InnoDB: Shutdown completed; log sequence number 1616697
mariadb_1        | 200314 11:01:14 [Note] /usr/sbin/mysqld (mysqld 10.0.38-MariaDB-1~xenial) starting as process 94 ...
mariadb_1        | 200314 11:01:14 [Note] InnoDB: Using mutexes to ref count buffer pool pages
mariadb_1        | 200314 11:01:14 [Note] InnoDB: The InnoDB memory heap is disabled
mariadb_1        | 200314 11:01:14 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
mariadb_1        | 200314 11:01:14 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
mariadb_1        | 200314 11:01:14 [Note] InnoDB: Compressed tables use zlib 1.2.8
mariadb_1        | 200314 11:01:14 [Note] InnoDB: Using Linux native AIO
mariadb_1        | 200314 11:01:14 [Note] InnoDB: Using CPU crc32 instructions
mariadb_1        | 200314 11:01:14 [Note] InnoDB: Initializing buffer pool, size = 256.0M
mariadb_1        | 200314 11:01:14 [Note] InnoDB: Completed initialization of buffer pool
mariadb_1        | 200314 11:01:14 [Note] InnoDB: Highest supported file format is Barracuda.
mariadb_1        | 200314 11:01:15 [Note] InnoDB: 128 rollback segment(s) are active.
mariadb_1        | 200314 11:01:15 [Note] InnoDB: Waiting for purge to start
mariadb_1        | 200314 11:01:15 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.42-84.2 started; log sequence number 1616697
mariadb_1        | 200314 11:01:15 [Note] InnoDB: FTS optimize thread exiting.
mariadb_1        | 200314 11:01:15 [Note] InnoDB: Starting shutdown...
mariadb_1        | 200314 11:01:16 [Note] InnoDB: Waiting for page_cleaner to finish flushing of buffer pool
mariadb_1        | 200314 11:01:18 [Note] InnoDB: Shutdown completed; log sequence number 1616707
mariadb_1        | 
mariadb_1        | PLEASE REMEMBER TO SET A PASSWORD FOR THE MariaDB root USER !
mariadb_1        | To do so, start the server, then issue the following commands:
mariadb_1        | 
mariadb_1        | '/usr/bin/mysqladmin' -u root password 'new-password'
mariadb_1        | '/usr/bin/mysqladmin' -u root -h  password 'new-password'
mariadb_1        | 
mariadb_1        | Alternatively you can run:
mariadb_1        | '/usr/bin/mysql_secure_installation'
mariadb_1        | 
mariadb_1        | which will also give you the option of removing the test
mariadb_1        | databases and anonymous user created by default.  This is
mariadb_1        | strongly recommended for production servers.
mariadb_1        | 
mariadb_1        | See the MariaDB Knowledgebase at http://mariadb.com/kb or the
mariadb_1        | MySQL manual for more instructions.
mariadb_1        | 
mariadb_1        | Please report any problems at http://mariadb.org/jira
mariadb_1        | 
mariadb_1        | The latest information about MariaDB is available at http://mariadb.org/.
mariadb_1        | You can find additional information about the MySQL part at:
mariadb_1        | http://dev.mysql.com
mariadb_1        | Consider joining MariaDB's strong and vibrant community:
mariadb_1        | https://mariadb.org/get-involved/
mariadb_1        | 
mariadb_1        | Database initialized
mariadb_1        | MySQL init process in progress...
mariadb_1        | 200314 11:01:18 [Note] mysqld (mysqld 10.0.38-MariaDB-1~xenial) starting as process 124 ...
mariadb_1        | 200314 11:01:18 [Note] InnoDB: Using mutexes to ref count buffer pool pages
mariadb_1        | 200314 11:01:18 [Note] InnoDB: The InnoDB memory heap is disabled
mariadb_1        | 200314 11:01:18 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
mariadb_1        | 200314 11:01:18 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
mariadb_1        | 200314 11:01:18 [Note] InnoDB: Compressed tables use zlib 1.2.8
mariadb_1        | 200314 11:01:18 [Note] InnoDB: Using Linux native AIO
mariadb_1        | 200314 11:01:18 [Note] InnoDB: Using CPU crc32 instructions
mariadb_1        | 200314 11:01:18 [Note] InnoDB: Initializing buffer pool, size = 256.0M
mariadb_1        | 200314 11:01:18 [Note] InnoDB: Completed initialization of buffer pool
mariadb_1        | 200314 11:01:18 [Note] InnoDB: Highest supported file format is Barracuda.
mariadb_1        | 200314 11:01:18 [Note] InnoDB: 128 rollback segment(s) are active.
mariadb_1        | 200314 11:01:18 [Note] InnoDB: Waiting for purge to start
mariadb_1        | 200314 11:01:18 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.42-84.2 started; log sequence number 1616707
mariadb_1        | 200314 11:01:18 [Note] Plugin 'FEEDBACK' is disabled.
mariadb_1        | 200314 11:01:18 [Warning] 'user' entry 'root@e2fd6f2028c6' ignored in --skip-name-resolve mode.
mariadb_1        | 200314 11:01:18 [Warning] 'proxies_priv' entry '@% root@e2fd6f2028c6' ignored in --skip-name-resolve mode.
mariadb_1        | 200314 11:01:19 [Note] mysqld: ready for connections.
mariadb_1        | Version: '10.0.38-MariaDB-1~xenial'  socket: '/var/run/mysqld/mysqld.sock'  port: 0  mariadb.org binary distribution
mariadb_1        | Warning: Unable to load '/usr/share/zoneinfo/leap-seconds.list' as time zone. Skipping it.
mariadb_1        | 200314 11:01:20 [Warning] 'proxies_priv' entry '@% root@e2fd6f2028c6' ignored in --skip-name-resolve mode.
mariadb_1        | 
mariadb_1        | 200314 11:01:20 [Note] mysqld: Normal shutdown
mariadb_1        | 200314 11:01:20 [Note] Event Scheduler: Purging the queue. 0 events
mariadb_1        | 200314 11:01:20 [Note] InnoDB: FTS optimize thread exiting.
mariadb_1        | 200314 11:01:20 [Note] InnoDB: Starting shutdown...
mariadb_1        | 200314 11:01:21 [Note] InnoDB: Waiting for page_cleaner to finish flushing of buffer pool
mariadb_1        | 200314 11:01:22 [Note] InnoDB: Shutdown completed; log sequence number 1616717
mariadb_1        | 200314 11:01:23 [Note] mysqld: Shutdown complete
mariadb_1        | 
mariadb_1        | 
mariadb_1        | MySQL init process done. Ready for start up.
mariadb_1        | 
mariadb_1        | 200314 11:01:23 [Note] mysqld (mysqld 10.0.38-MariaDB-1~xenial) starting as process 1 ...
mariadb_1        | 200314 11:01:23 [Note] InnoDB: Using mutexes to ref count buffer pool pages
mariadb_1        | 200314 11:01:23 [Note] InnoDB: The InnoDB memory heap is disabled
mariadb_1        | 200314 11:01:23 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
mariadb_1        | 200314 11:01:23 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
mariadb_1        | 200314 11:01:23 [Note] InnoDB: Compressed tables use zlib 1.2.8
mariadb_1        | 200314 11:01:23 [Note] InnoDB: Using Linux native AIO
mariadb_1        | 200314 11:01:23 [Note] InnoDB: Using CPU crc32 instructions
mariadb_1        | 200314 11:01:23 [Note] InnoDB: Initializing buffer pool, size = 256.0M
mariadb_1        | 200314 11:01:23 [Note] InnoDB: Completed initialization of buffer pool
mariadb_1        | 200314 11:01:23 [Note] InnoDB: Highest supported file format is Barracuda.
mariadb_1        | 200314 11:01:23 [Note] InnoDB: 128 rollback segment(s) are active.
mariadb_1        | 200314 11:01:23 [Note] InnoDB: Waiting for purge to start
mariadb_1        | 200314 11:01:23 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.42-84.2 started; log sequence number 1616717
mariadb_1        | 200314 11:01:23 [Note] Plugin 'FEEDBACK' is disabled.
mariadb_1        | 200314 11:01:23 [Note] Server socket created on IP: '::'.
mariadb_1        | 200314 11:01:23 [Warning] 'proxies_priv' entry '@% root@e2fd6f2028c6' ignored in --skip-name-resolve mode.
mariadb_1        | 200314 11:01:23 [Note] mysqld: ready for connections.

Answer 82 · 2020-03-14T11:04:59.000Z

Hi @abhiandthetruth

@animeshk08 have you changed the corresponding urls in setup.cfg?

Sorry I didn't get you.

Answer 83 · 2020-03-14T11:07:25.000Z

the setup.cfg file needs the url to connect to elasticsearch, in the provided file the urls are https://admin:admin@localhost:9200 which is not applicable in your case, you should change it too http://localhost:9200. Are you using the default setup.cfg? if yes then you need to change these params.

Answer 84 · 2020-03-14T11:43:35.000Z

Thank you @abhiandthetruth The issues have been resolved!!

Answer 85 · 2020-03-14T11:44:54.000Z

@animeshk08 I am curious as to what was the issue bcoz I couldn't reproduce the same on my machine. Was it a proxy issue? It'll help me understand the working of the system better :).

Answer 86 · 2020-03-14T11:55:37.000Z

@animeshk08 I am curious as to what was the issue bcoz I couldn't reproduce the same on my machine. Was it a proxy issue?

Hi @abhiandthetruth the initial issue was due to file descripter limit which was resolved as I mentioned. I think the docker was not able to read the limit configurations of my system. I still need to figure out why was that happening though. Will ping you once I have updates :)

Answer 87 · 2020-03-14T11:59:00.000Z

Thanks for the reply @animeshk08.

Answer 88 · 2020-03-14T14:33:37.000Z

Thank you @abhiandthetruth and @animeshk08 to sort this problem out. @animeshk08 could you add a section to the troubleshooting in sirmordred about the problem you had (#288 (comment))?

I found something on stackoverflow about your problem (
https://stackoverflow.com/questions/46771233/max-file-descriptors-for-elasticsearch-process-is-too-low), however I wasn't able to reproduce it on my machine.

Which are the specs of your machine?

Answer 89 · 2020-03-14T14:38:29.000Z

@valeriocos I think the readme size is becoming quite large. We can refract it. Like creating a troubleshooting.md. WDYT?

Answer 90 · 2020-03-14T14:44:52.000Z

I think the readme size is becoming quite large. We can refract it. Like creating a troubleshooting.md. WDYT?

I think the same. We can reference it in the current README and create a new one.
@valeriocos Are you doing this? I will add the trouble shooting for my issue once you have done in that case :)

Answer 91 · 2020-03-14T14:54:47.000Z

yes, it sounds good! @abhiandthetruth @animeshk08 do you think that the new md file should contain only troubleshooting info or maybe we can create a getting started md with a troubleshooting section into it? For instance the new fille could have this structure:

getting started (content from https://github.com/chaoss/grimoirelab-sirmordred#getting-started)
- setting up Pycharm dev environment (content from https://github.com/chaoss/grimoirelab-sirmordred#setting-up-a-pycharm-dev-environment)
- setting up docker-compose (content from https://github.com/chaoss/grimoirelab-sirmordred#source-code-and-docker)
- micro mordred (content from https://github.com/chaoss/grimoirelab-sirmordred#micro-mordred)
troubleshooting (content from https://github.com/chaoss/grimoirelab-sirmordred#troubleshooting)

please share your thoughts?

Answer 92 · 2020-03-14T14:59:05.000Z

@valeriocos I think we can have a complete "Getting Started" README file in which troubleshooting section is also included.

Answer 93 · 2020-03-14T15:14:20.000Z

ok! please @animeshk08 can you kickstart this new getting started readme? Maybe you can add the troubleshooting section plus the problem you found, @abhiandthetruth you could then focus on the content of the first section (getting started). WDYT?

Answer 94 · 2020-03-14T15:18:07.000Z

@valeriocos Sure I will start with it! :)

Answer 95 · 2020-03-14T15:18:19.000Z

thanks @animeshk08

Answer 96 · 2020-03-15T19:24:10.000Z

Hello Everyone,
Anyone who is working on Microtask 8, a few issues I came across might help you:

If you are trying to fetch the message from a group make sure it has "download_archives": true permission activated. This can be checked through this API.
For more details check this issue: chaoss/grimoirelab-perceval#586
Some of the index fields may differ. I found the following:
grimoirelab_creation_date ->grimoire_creation_date
body->body_extract
subject_analyzed -> Subject_analyzed

@valeriocos Correct me if I am wrong anywhere. Also, should I change the logs for groupio in perceval similar to pagure 404 exception?

Answer 97 · 2020-03-15T19:32:07.000Z

Thank you for reporting this info @animeshk08 . Consider to add it to the troubleshooting section (https://github.com/chaoss/grimoirelab-sirmordred/blob/master/Getting-Started.md#troubleshooting-).

@valeriocos Correct me if I am wrong anywhere. Also, should I change the logs for groupio in perceval similar to pagure 404 exception?

Yes, this will be really useful, thanks!

Answer 98 · 2020-03-15T19:36:54.000Z

Sure! I will get on with it! :)