oss-gate/workshop

OSS Gate Workshop: Online: 2024-07-27: jkyochen: Fluentd: Work log

jkyochen opened this issue · 19 comments

jkyochen commented

This is a work log of a "OSS Gate workshop".
"OSS Gate workshop" is an activity to increase OSS developers.
Here's been discussed in Japanese. Thanks.

作業ログ作成時の説明

以下のテンプレートを埋めてタイトルに設定します。埋め方例はスクロールすると見えてきます。

OSS Gate Workshop: ${LOCATION}: ${YEAR}-${MONTH}-${DAY}: ${ACCOUNT_NAME}: ${OSS_NAME}: Work log

タイトル例↓：

OSS Gate Workshop: Tokyo: 2017-01-16: kou: Rabbit: Work log

OSS Gateワークショップ関連情報

  • スライド：ワークショップの進行に使っているスライドがあります。
  • チャット：OSS開発に関することならなんでも相談できます。ワークショップが終わった後もオンラインで相談しながら継続的にOSSの開発に参加しましょう！
  • シナリオ：ワークショップの目的・内容・進め方の詳細が書いています。
  • 過去のビギナーの作業ログ：他の人の作業ログから学べることがいろいろあるはずです。
jkyochen commented

その前は二つソフト（RedisとMongodb）のライセンス、チェックしたいんです。

jkyochen commented

まずはRedisのライセンスチェックしよう。

Github https://github.com/redis/redis

ライセンスは Redis Source Available License 2.0 (RSALv2) Agreement

https://github.com/redis/redis?tab=License-1-ov-file#readme
https://redis.io/blog/redis-adopts-dual-source-available-licensing/

Redis Source Available License 2.0 (RSALv2) AgreementはOpensourceで探しできませんので、RedisはOSSじゃないです。

https://opensource.org/license/page/2?ls=Redis+Source

どころで

7.2Version前のライセンスは BSD-3-Clause と確認しました、まはOSSです。

https://github.com/redis/redis/tree/7.2.5?tab=License-1-ov-file
https://opensource.org/license/bsd-3-clause
https://redis.io/legal/licenses/

jkyochen commented

Mongodbのライセンスを確認しよう。

Github https://github.com/mongodb/mongo

ライセンスは Server Side Public License

https://github.com/mongodb/mongo?tab=License-1-ov-file#readme

Server Side Public LicenseはOpenSourceで探していませんので、MongodbはOSSじゃないです。

https://opensource.org/license?ls=Server+Side

jkyochen commented

それで今回作業のFluentdのライセンスを確認しよう。

Github https://github.com/fluent/fluentd

ライセンスはApache License Version 2.0, January 2004

https://github.com/fluent/fluentd?tab=Apache-2.0-1-ov-file#readme

Apache LicenseはOpenSourceで探しました、FluentdはOSSです。

https://opensource.org/license/apache-2-0

jkyochen commented

まずFluentdのREADMEを読みます。

https://github.com/fluent/fluentd?tab=readme-ov-file

あまり有用の情報がありませんでした、HomePageに行きます

https://docs.fluentd.org/

jkyochen commented

Documentからインストールしよう。

https://docs.fluentd.org/installation/before-install

これは本番インストールの準備です、今回開発環境するなので、この事前準備しないになります。

jkyochen commented

macOSでインストールのドキュメントからやります。

https://docs.fluentd.org/installation/install-by-dmg

jkyochen commented

macOSは二つバッジョンあって、td-agentとcalyptiaです、迷いました、ソースからインストールにチェンジしよう。

https://docs.fluentd.org/installation/install-from-source

jkyochen commented

ソースをCloneする。

$ git clone https://github.com/fluent/fluentd.git
$ cd fluentd
  • 👍1
jkyochen commented

Packageをインストールする。

➜  fluentd git:(master) bundle install
Fetching gem metadata from https://rubygems.org/......
Resolving dependencies...
Fetching fiber-storage 0.1.2
Fetching fiber-annotation 0.2.0
Fetching timers 4.3.5
Fetching protocol-http 0.26.8
Installing fiber-annotation 0.2.0
Installing fiber-storage 0.1.2
Installing timers 4.3.5
Installing protocol-http 0.26.8
Fetching protocol-hpack 1.4.3
Fetching traces 0.11.1
Fetching aws-eventstream 1.3.0
Installing protocol-hpack 1.4.3
Fetching aws-partitions 1.958.0
Fetching jmespath 1.6.2
Installing traces 0.11.1
Fetching concurrent-ruby 1.3.3
Installing aws-eventstream 1.3.0
Fetching cool.io 1.8.1
Installing aws-partitions 1.958.0
Fetching csv 3.3.0
Installing jmespath 1.6.2
Fetching docile 1.4.1
Installing concurrent-ruby 1.3.3
Installing cool.io 1.8.1 with native extensions
Installing csv 3.3.0
Installing docile 1.4.1
Fetching flexmock 2.3.8
Fetching http_parser.rb 0.8.0
Installing flexmock 2.3.8
Installing http_parser.rb 0.8.0 with native extensions
Fetching sigdump 0.2.5
Installing sigdump 0.2.5
Fetching strptime 0.2.5
Fetching yajl-ruby 1.4.3
Installing strptime 0.2.5 with native extensions
Installing yajl-ruby 1.4.3 with native extensions
Fetching parallel 1.25.1
Installing parallel 1.25.1
Fetching rr 3.1.0
Installing rr 3.1.0
Fetching simplecov-html 0.12.3
Installing simplecov-html 0.12.3
Fetching simplecov_json_formatter 0.1.4
Installing simplecov_json_formatter 0.1.4
Fetching timecop 0.9.8
Fetching fiber-local 1.1.0
Installing timecop 0.9.8
Installing fiber-local 1.1.0
Fetching protocol-http1 0.19.1
Installing protocol-http1 0.19.1
Fetching protocol-http2 0.16.0
Fetching oj 3.16.4
Installing protocol-http2 0.16.0
Installing oj 3.16.4 with native extensions
Fetching aws-sigv4 1.9.0
Installing aws-sigv4 1.9.0
Fetching serverengine 2.3.2
Fetching parallel_tests 0.15.4
Installing serverengine 2.3.2
Fetching test-unit 3.6.2
Installing parallel_tests 0.15.4
Installing test-unit 3.6.2
Fetching rexml 3.3.2
Installing rexml 3.3.2
Fetching simplecov 0.22.0
Installing simplecov 0.22.0
Fetching console 1.27.0
Fetching aws-sdk-core 3.201.3
Installing console 1.27.0
Installing aws-sdk-core 3.201.3
Fetching tzinfo-data 1.2024.1
Installing tzinfo-data 1.2024.1
Fetching test-unit-rr 1.0.5
Installing test-unit-rr 1.0.5
Fetching async 1.32.1
Installing async 1.32.1
Fetching async-io 1.43.2
Installing async-io 1.43.2
Fetching async-pool 0.7.0
Installing async-pool 0.7.0
Fetching async-http 0.64.2
Installing async-http 0.64.2
Bundle complete! 15 Gemfile dependencies, 54 gems now installed.
Gems in the group 'production' were not installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.
  • 👍1
jkyochen commented

Rake Buildする

➜  fluentd git:(master) ruby -v
ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin21]
➜  fluentd git:(master) bundle exec rake build
fluentd 1.17.0 built to pkg/fluentd-1.17.0.gem.
➜  fluentd git:(master) gem install pkg/fluentd-1.17.0.gem
Successfully installed fluentd-1.17.0
Parsing documentation for fluentd-1.17.0
Installing ri documentation for fluentd-1.17.0
Done installing documentation for fluentd after 5 seconds
1 gem installed

A new release of RubyGems is available: 3.5.3 → 3.5.16!
Run `gem update --system 3.5.16` to update your installation.
jkyochen commented

実行する。

➜  fluentd git:(master) fluentd --setup ./fluent
Installed ./fluent/fluent.conf.
➜  fluentd git:(master) ✗ fluentd -c ./fluent/fluent.conf -vv &
[1] 19649
➜  fluentd git:(master) ✗ 2024-07-27 14:27:46 +0900 [info]: fluent/log.rb:362:info: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2024-07-27 14:27:46 +0900 [info]: fluent/log.rb:362:info: parsing config file is succeeded path="./fluent/fluent.conf"
2024-07-27 14:27:46 +0900 [info]: fluent/log.rb:362:info: gem 'fluentd' version '1.17.0'
2024-07-27 14:27:46 +0900 [trace]: fluent/log.rb:319:trace: registered output plugin 'stdout'
2024-07-27 14:27:46 +0900 [trace]: fluent/log.rb:319:trace: registered metrics plugin 'local'
2024-07-27 14:27:46 +0900 [trace]: fluent/log.rb:319:trace: registered buffer plugin 'memory'
2024-07-27 14:27:46 +0900 [trace]: fluent/log.rb:319:trace: registered formatter plugin 'stdout'
2024-07-27 14:27:46 +0900 [trace]: fluent/log.rb:319:trace: registered formatter plugin 'json'
2024-07-27 14:27:46 +0900 [trace]: fluent/log.rb:319:trace: registered input plugin 'forward'
2024-07-27 14:27:47 +0900 [trace]: fluent/log.rb:319:trace: registered parser plugin 'in_http'
2024-07-27 14:27:47 +0900 [trace]: fluent/log.rb:319:trace: registered input plugin 'http'
2024-07-27 14:27:47 +0900 [trace]: fluent/log.rb:319:trace: registered parser plugin 'msgpack'
2024-07-27 14:27:47 +0900 [trace]: fluent/log.rb:319:trace: registered parser plugin 'json'
2024-07-27 14:27:47 +0900 [trace]: fluent/log.rb:319:trace: registered input plugin 'monitor_agent'
2024-07-27 14:27:47 +0900 [trace]: fluent/log.rb:319:trace: registered input plugin 'debug_agent'
2024-07-27 14:27:47 +0900 [debug]: fluent/log.rb:341:debug: No fluent logger for internal event
2024-07-27 14:27:47 +0900 [info]: fluent/log.rb:362:info: using configuration file: <ROOT>
  <source>
    @type forward
    @id forward_input
  </source>
  <source>
    @type http
    @id http_input
    port 8888
  </source>
  <source>
    @type monitor_agent
    @id monitor_agent_input
    port 24220
  </source>
  <source>
    @type debug_agent
    @id debug_agent_input
    bind "127.0.0.1"
    port 24230
  </source>
  <match debug.**>
    @type stdout
    @id stdout_output
  </match>
</ROOT>
2024-07-27 14:27:47 +0900 [info]: fluent/log.rb:362:info: starting fluentd-1.17.0 pid=19649 ruby="3.3.0"
2024-07-27 14:27:47 +0900 [info]: fluent/log.rb:362:info: spawn command to main:  cmdline=["/Users/lanlyhs/.asdf/installs/ruby/3.3.0/bin/ruby", "-Eascii-8bit:ascii-8bit", "/Users/lanlyhs/.asdf/installs/ruby/3.3.0/bin/fluentd", "-c", "./fluent/fluent.conf", "-vv", "--under-supervisor"]
2024-07-27 14:27:48 +0900 [info]: #0 fluent/log.rb:362:info: init worker0 logger path=nil rotate_age=nil rotate_size=nil
2024-07-27 14:27:48 +0900 [info]: fluent/log.rb:362:info: adding match pattern="debug.**" type="stdout"
2024-07-27 14:27:48 +0900 [trace]: #0 fluent/log.rb:319:trace: registered output plugin 'stdout'
2024-07-27 14:27:48 +0900 [trace]: #0 fluent/log.rb:319:trace: registered metrics plugin 'local'
2024-07-27 14:27:48 +0900 [trace]: #0 fluent/log.rb:319:trace: registered buffer plugin 'memory'
2024-07-27 14:27:48 +0900 [trace]: #0 fluent/log.rb:319:trace: registered formatter plugin 'stdout'
2024-07-27 14:27:48 +0900 [trace]: #0 fluent/log.rb:319:trace: registered formatter plugin 'json'
2024-07-27 14:27:48 +0900 [info]: fluent/log.rb:362:info: adding source type="forward"
2024-07-27 14:27:48 +0900 [trace]: #0 fluent/log.rb:319:trace: registered input plugin 'forward'
2024-07-27 14:27:48 +0900 [info]: fluent/log.rb:362:info: adding source type="http"
2024-07-27 14:27:48 +0900 [trace]: #0 fluent/log.rb:319:trace: registered parser plugin 'in_http'
2024-07-27 14:27:48 +0900 [trace]: #0 fluent/log.rb:319:trace: registered input plugin 'http'
2024-07-27 14:27:48 +0900 [trace]: #0 fluent/log.rb:319:trace: registered parser plugin 'msgpack'
2024-07-27 14:27:48 +0900 [trace]: #0 fluent/log.rb:319:trace: registered parser plugin 'json'
2024-07-27 14:27:48 +0900 [info]: fluent/log.rb:362:info: adding source type="monitor_agent"
2024-07-27 14:27:48 +0900 [trace]: #0 fluent/log.rb:319:trace: registered input plugin 'monitor_agent'
2024-07-27 14:27:48 +0900 [info]: fluent/log.rb:362:info: adding source type="debug_agent"
2024-07-27 14:27:48 +0900 [trace]: #0 fluent/log.rb:319:trace: registered input plugin 'debug_agent'
2024-07-27 14:27:48 +0900 [debug]: #0 fluent/log.rb:341:debug: No fluent logger for internal event
2024-07-27 14:27:48 +0900 [info]: #0 fluent/log.rb:362:info: starting fluentd worker pid=19824 ppid=19649 worker=0
2024-07-27 14:27:48 +0900 [info]: #0 [debug_agent_input] listening dRuby uri="druby://127.0.0.1:24230" object="Fluent::Engine" worker=0
2024-07-27 14:27:48 +0900 [debug]: #0 [monitor_agent_input] listening monitoring http server on http://0.0.0.0:24220/api/plugins for worker0
2024-07-27 14:27:48 +0900 [debug]: #0 [monitor_agent_input] Start async HTTP server listening http://0.0.0.0:24220
2024-07-27 14:27:48 +0900 [debug]: #0 [monitor_agent_input]   0.0s: Async::IO::Socket
      | Binding to #<Addrinfo: 0.0.0.0:24220 TCP>
2024-07-27 14:27:48 +0900 [debug]: #0 [http_input] listening http bind="0.0.0.0" port=8888
2024-07-27 14:27:48 +0900 [info]: #0 [forward_input] listening port port=24224 bind="0.0.0.0"
2024-07-27 14:27:48 +0900 [info]: #0 fluent/log.rb:362:info: fluentd worker is now running worker=0
2024-07-27 14:29:41 +0900 [trace]: #0 [forward_input] connected fluent socket addr="127.0.0.1" port=50822
2024-07-27 14:29:41 +0900 [trace]: #0 [forward_input] accepted fluent socket addr="127.0.0.1" port=50822
2024-07-27 14:29:41.179517000 +0900 debug.test: {"json":"message"}
  • 👍1
jkyochen commented

processを見る。

➜  fluentd git:(master) ✗ ps -ef | grep 19649
  501 19649  1776   0  2:27PM ttys000    0:01.52 /Users/〇〇/.asdf/installs/ruby/3.3.0/bin/ruby /Users/lanlyhs/.asdf/installs/ruby/3.3.0/bin/fluentd -c ./fluent/fluent.conf -vv
jkyochen commented

これから、HTTPからfluentdを通して、ファイルに写すすることをやります。

  • 👍1
jkyochen commented

fluentdにファイルをConfigする。

<match pattern>
  @type file
  path /Users/〇〇/librarys/program/github/dev/fluentd/log
  compress gzip
  <buffer>
    timekey 1s
    timekey_use_utc true
    timekey_wait 1m
  </buffer>
</match>

https://docs.fluentd.org/output/file

  • 👍1
daipom commented

ドキュメントの直し方がこのウェブサイトに書いてあるが、このリンクが古い。。。

https://www.fluentd.org/contributing

Documentation: Use Fluentd-Docs repository.

  • ❤️1
github-actions commented

おつかれさまでした！

ワークショップの終了にともないissueを閉じますが、このまま作業メモとして使っても構いません 👌

ワークショップの感想を集めています！

ブログなどに書かれた際は、このページへリンクの追加をお願いします 🙏

またの参加をお待ちしています！