🦸‍♀️ [OFFICIAL] Talk about Development 🦸‍♀️

Question

🦸‍♀️ [OFFICIAL] Talk about Development 🦸‍♀️

BaseMax opened this issue 4 years ago · 45 comments

BaseMax commented 4 years ago

Hi,

The DonyaOS is not yet ready and complete and we need the help of others.

Here's a chat room.

We can talk about the tasks we want to work on here.
And set a development path for ourselves.

Study and research can be one of the things that will always be needed in this project.

Please state if something goes wrong.
I appreciate you.

Answer 1 · 2020-08-13T22:54:02.000Z

At this point, we have a script that can generate a new Linux distribution and produce .ISO image file.
https://github.com/DonyaOS/Donya/blob/815216b91e7df81c9d3aebc1217cd9a2b3d698b3/donya.sh

But this is not enough.

We need to separate the scripts and have such a separate file. (Not a single .sh file)

For example, Scripts:

Downloading dependency files (Maybe someone has these file in their PC)
Compile kernel, base
Compile OS/BusyBox and other packages
Generate .ISO image file

Also Makefile can help to compile quickly.

Answer 2 · 2020-08-14T02:18:00.000Z

With Donya's package manager, I'd suggest a packaging system like KISS Linux (here's an overview: https://k1ss.org/package-system). What did you have in mind?

Answer 3 · 2020-08-14T05:01:56.000Z

OK, with package management, this is what I had in mind—just a suggestion, I'm not entirely sure what your plans are:

We make a repository in the DonyaOS GitHub organization, and we call it packages or something to that effect. This will be the default source of packages in the package manager. Then, the package manager (which I'll call dpak until we come up with a proper name) will look for packages within that GitHub repository, at the root, by filename; so if we were to do something like: dpak install ExamplePackage, then dpak would read the data in the ExamplePackage.yml file at the root of the packages repository. This is similar to how the Homebrew package manager on macOS works.

Each package file in the repository will have several standard data fields, which I suggest we represent in YAML. These fields will be description, license, homepage, dependencies, binary (optional), and install. The package manager will show the package metadata (including the list of dependencies) before prompting the user if they want to install (like with APT). The install variable is a series of commands which will be ran by the system to get the package installed, useful for if the package is compiled (like how Gentoo handles packages), not a binary. In the event a package is binary-only, then the binary variable will point to the URL of the binary release, and then dpak will simply install it into the system.
Here's an example of a package called dlib; stored in DonyaOS/packages/dlib.yml:

description: "C++ library for machine learning"
homepage: "http://dlib.net/"
binary: "http://dlib.net/files/dlib-19.21.tar.bz2"
license: "BSL-1.0"
dependencies:
- cmake
- jpeg
- libpng
- openblas
install: |
  git clone https://github.com/davisking/dlib.git
  cd dlib-master
  mkdir build
  cd build
  cmake ..
  cmake --build . --config Release

This package manager model has several key differences over currently existing package managers:

It's simple—package repositories are easy to create, because they're just GitHub repositories
- APT and Snapcraft struggle greatly in this department. Packaging software, and then publishing it into a repository, is a difficult process for developers who want to get their work installed via those package managers. Moving everything to a simple, GitHub-centric system will make publishing easier.
It allows for compiling packages rather than just getting binaries
- This is only really seen in Gentoo nowadays. Making a simpler alternative to Portage is a good idea.
Greater GitHub integration; by making each software source a GitHub repository, it will force software publishers to become familiar with the open-source ecosystem, particularly if they wish to publish in the primary DonyaOS package repository. This promotes open-source development.

Answer 4 · 2020-08-14T13:28:01.000Z

Thanks for your suggestion.
I did create packages repository in this organization.

https://github.com/DonyaOS/Packages

Answer 5 · 2020-08-14T13:31:16.000Z

We need to design the package manager software.
Something similar to apt in ubuntu, pacman in Arch, pkg in FreeBSD.

I think GoLang is a good choice for this purpose.
C is also good.

Easy command line is very important.

We can use an alias:

i for install.
u or r for uninstall and remove
s for search

For example:

d i php ; install php version 7.1
d i php7.4 ; install php version 7.2
d i gcc ; install gcc

d s php ; search all package with php perfix

d r php ; remove php package
d r php* ; remove all php prefix package

d i php* ; install all php prefix package

Display loading while downloading and package information before installing is a good feature.
Centos has this feature as well.

Ubuntu apt has a bad feature, this forces you to remove lock files if you stop this using ctrl+x, ctrl+c etc.

Answer 6 · 2020-08-14T13:50:28.000Z

Hi,

Does anyone want to work on the package manager or package service? @sudo-corvus @limeschool Or others
We welcome this help.
I think GoLang is a good choice for this purpose.

Please tell me your comment.
Thanks,

Answer 7 · 2020-08-14T14:05:00.000Z

PackageManager:
It's a client software, this should be installed at DonyaOS in default.
https://github.com/DonyaOS/PackageManager

Packages:
This is an API.
This is a web service, PackageManager will get his data from this repository.

https://github.com/DonyaOS/Packages

For example: If PackageManager wants to search in packages list, should get a list of packages from the Packages repository. and save this as a cache.

I think we can host packages files in GitHub.
Otherwise, we can create a server and host on that.
No problem with this.

Answer 8 · 2020-08-14T17:26:52.000Z

Hey @BaseMax i'm familiar with GoLang and i can help you guys with this!
Just let me know where to begin...

Answer 9 · 2020-08-14T20:42:13.000Z

Hey Amir, @amir-shiati
Thanks for letting me know you agree to work on it.
I gave you full access. Therefore, you can submit and send changes in the desired repository without any permission.
Thanks,

Answer 10 · 2020-08-15T02:14:59.000Z

I'm not familiar with Go or C. I'll try to make my own package manager in D, and then we can compare and pick which is best.

EDIT: D was not a good choice. Writing it in C# right now. It should perform rather well, with the time to complete an operation being almost entirely package-dependent and not noticeably impacted by any overhead.

Answer 11 · 2020-08-15T05:26:13.000Z

I will take this up in C I have been going through RPM to understand the process

Answer 12 · 2020-08-15T18:38:18.000Z

Hi,
@amir-shiati is working to create Package Manager.
Can you please start working on the Packages repository? @sudo-corvus, and @limeschool

Answer 13 · 2020-08-15T19:15:53.000Z

all right on it @BaseMax

Answer 14 · 2020-08-16T06:01:15.000Z

Just as a side project:

I've made a package manager in C# that works rather fast and does not require installing any dependencies to use it.

It's called gany and does not use the KISS structure currently in the DonyaOS/Packages repository, but rather a simpler one where each package is a single file. For an example of this, see my own package repository which has a package definition for gany itself, allowing for self-updating.
gany can build packages, and it can download binary versions as well. Pretty neat, though a little incomplete right now.

Since you seem to be going with the KISS package structure, however, what packages should @sudo-corvus and I put in DonyaOS/Packages? Should we just copy over the default KISS software repository?

Answer 15 · 2020-08-16T06:57:38.000Z

Hi Emil,

I was excited to see your activity.
Where is package definition of the gany? (Your link is broken)

Anyway, you has full-access to this organization and can push your code/commits directly.

I've made a package manager in C#...

I not sure about this, Usually, C# is not a good choice for Linux commands And this is not a common thing.

This is a good language though.
But C# is compiled into IL, not binary.

To install gany, you need 'libkrb5-dev' & '.NET Core 3.1' to build...

However, if this is necessary, We can think of rewriting it in another language. @amir-shiati, Have you done anything?
This can give him a good view of how to develop a package manager.

KISS package structure...

I just use this as the first version for a starter package.

We can think about and develop a new structure.
Anyway, I like your idea about the KISS structure.
And this is great for most purposes.

For example ffmpeg:
https://github.com/kisslinux/repo/tree/master/extra/ffmpeg

build
checksums
depends
sources
version

Please show me your current packages repository.

Update:
https://github.com/limeschool/packages

You can add some sample packages just for testing:
https://github.com/kisslinux/repo/tree/master/core/curl

Thanks,

Answer 16 · 2020-08-16T07:13:01.000Z

Please push your repositories to the Donya organization. @limeschool

Please explain where the name gany comes from?
donya and d was not a short and better name?

Some feature needed for package manager:

Checksum
Ability to display the size of all files (before start to downloading)

In your structure file size is not calculable. (This is missing side by side with bash commands)

Display loading x% while downloading files...

Answer 17 · 2020-08-16T08:18:31.000Z

A sample: https://github.com/DonyaOS/Packages/blob/master/core/curl/package.donya

Direct HTTP/S request: https://donyaos.github.io/Packages/core/curl/package.donya

In this case, But we need to know the name of the group. (core at this sample)
Otherwise, we have to not group the packages.

Answer 18 · 2020-08-16T08:39:27.000Z

If a package need multi-file to download.
How we can control run make/configure in different downloaded files?

I no mean dependency. I mean multi required file to download.

DonyaOS/Packages@9947725

We can do unrar x, unzip, and tar xvf automatically in the package manager.

But I think we have to write cd manually...
What do you think?

Answer 19 · 2020-08-16T08:51:13.000Z

About the package manager:

Consider that before the installation step (install in the package file), All required packages must be installed.

This must be done in a recursive manner.
Because the required packages may also depend on other packages.

Answer 20 · 2020-08-16T09:01:34.000Z

Hey everyone!
this is a simple diagram showing how the package manager is going to work, please take a look at it and let me know if any other blocks needs to be added to the diagram...

Answer 21 · 2020-08-16T09:09:30.000Z

And also this is an example of how a package would be installed:
lets say the user wants to install curl:
$ donya install curl
or
$ donya i curl

the package manager would send a request to the package manager api, the package manager api would search for curl and then return (if exists) the url of curls "package.donya" file which is for example this, the package manager would download the file and open it, then after that:
1- it would install all the required packages in a recursive manner.
2- it would download the package file.
3- it would check the checksum to make sure that the downloaded file is not corrupted.
4- it would run the given instructions to install package.

Let me know what you guys think...

Answer 22 · 2020-08-16T10:14:36.000Z

Thanks, Amir, That's fine. @amir-shiati

Answer 23 · 2020-08-16T10:16:32.000Z

Who is currently responsible for designing the API? @sudo-corvus, and @limeschool
We have already assigned two people.

One of our problems is to host files at GitHub.
Do we will can to develop API and web server based on GitHub?

We can set up files on own server, and connect the GitHub repository to server using an SSH Action.

But I think we can follow a better way.

Update:

Also, Using the GitHub API might be another way to do this.
https://developer.github.com/v3/
https://developer.github.com/v3/search/#search-code

Answer 24 · 2020-08-16T10:29:40.000Z

Hey everyone,
For hosting the API oh github i came across this,
However the api is static.

Answer 25 · 2020-08-16T10:38:51.000Z

Thanks for sharing this. It's awesome. @MatMoore

https://github.com/MatMoore/jargon

By using github pages you are accepting their usage limits which means that your API could still go down if it gets too much traffic, and they don’t recommend using it for commercial purposes:

GitHub Pages is not intended for or allowed to be used as a free web hosting service to run your online business, e-commerce site, or any other website that is primarily directed at either facilitating commercial transactions or providing commercial software as a service (SaaS).

https://docs.github.com/en/github/working-with-github-pages/about-github-pages

Published GitHub Pages sites may be no larger than 1 GB.
GitHub Pages sites have a soft bandwidth limit of 100GB per month.
GitHub Pages sites have a soft limit of 10 builds per hour.

Answer 26 · 2020-08-16T15:23:27.000Z

We cannot implement a powerful search API using GitHub pages.
GitHub pages execute a script file(such as ruby) and create folder/files.

You can watch the gh-pages branch of the mentioned example:
https://github.com/MatMoore/jargon/tree/gh-pages

We can cache a list of packages and store this in local system, finally search between packages using package manager client software.

Answer 27 · 2020-08-17T17:04:24.000Z

Hi @limeschool,
Thanks for your commits and message.

I do not think we can find an algorithm that finds the exact file-size of compiled files from the source code.
However, we can specify the approximate size of the data in the package information.

Can you combine the test branch with the main branch in Packages repository?
We need to go to the next steps.

Answer 28 · 2020-08-18T14:07:19.000Z

The figure of pkg in FreeBSD:

root@base:~ # pkg install php74

Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
The following 4 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
	libargon2: 20190702
	libxml2: 2.9.10
	pcre2: 10.35
	php74: 7.4.8

Number of packages to be installed: 4

The process will require 40 MiB more space.
6 MiB to be downloaded.

Proceed with this action? [y/N]: wrong_key
Please type 'Y[es]' or 'N[o]' to make a selection

Proceed with this action? [y/N]: y
[1/4] Fetching php74-7.4.8.txz: 100%    4 MiB   4.2MB/s    00:01    
[2/4] Fetching libxml2-2.9.10.txz: 100%  828 KiB 848.0kB/s    00:01    
[3/4] Fetching libargon2-20190702.txz: 100%   64 KiB  65.5kB/s    00:01    
[4/4] Fetching pcre2-10.35.txz: 100%    1 MiB   1.3MB/s    00:01    
Checking integrity... done (0 conflicting)
[1/4] Installing libxml2-2.9.10...
[1/4] Extracting libxml2-2.9.10: 100%
[2/4] Installing libargon2-20190702...
[2/4] Extracting libargon2-20190702: 100%
[3/4] Installing pcre2-10.35...
[3/4] Extracting pcre2-10.35: 100%
[4/4] Installing php74-7.4.8...
[4/4] Extracting php74-7.4.8: 100%

Answer 29 · 2020-08-18T14:11:31.000Z

The figure of pkg search in FreeBSD:

root@base:~ # pkg search mariadb | grep 10.3

mariadb103-client-10.3.23      Multithreaded SQL database (client)
mariadb103-server-10.3.23      Multithreaded SQL database (server)

root@base:~ # pkg search mariadb

mariadb-connector-c-3.1.9      MariaDB database connector for C
mariadb-connector-odbc-3.1.7_1 MariaDB database connector for odbc
mariadb103-client-10.3.23      Multithreaded SQL database (client)
mariadb103-server-10.3.23      Multithreaded SQL database (server)
mariadb104-client-10.4.13_4    Multithreaded SQL database (client)
mariadb104-server-10.4.13_4    Multithreaded SQL database (server)
rubygem-azure_mgmt_mariadb-0.17.3 Microsoft Azure Microsoft Azure MariaDB Library for Ruby Client Library for Ruby

Answer 30 · 2020-08-27T10:23:02.000Z

Hi. Where did the package manager get to and how did it happen? @amir-shiati
https://github.com/DonyaOS/PackageManager

Answer 31 · 2020-08-27T10:36:34.000Z

Welcome @jbampton,

Because you are good at Go Language. Can you help with the package manager?
Currently, one person works on it, And development is very slow and needs to help.

This is the last thing I saw or heard about the Package Manager and I have no news. @amir-shiati

As Amir said: I know that this package is going to help:
https://github.com/spf13/cobra (CLI interactions)

Regards,
Max

Answer 32 · 2020-08-30T20:53:38.000Z

Hi, Good news: Since esmaeel works, we will add Grub to the project this week. It seems that Amir does not have the opportunity and there is no news about the package manager yet. I have not received an answer yet, You are expected to at least respond even if you do not do anything. He did the initial part, but we do not have access to the code. Sorry, Maybe I call to Amir today. @amir-shiati We invite another member to continue, we need a package manager sooner. Can you do this? @jbampton -- Regards, Max

Answer 33 · 2020-08-30T21:04:26.000Z

Hey, Can you adding at least most useful packages to repository? The more, the better... @sudo-corvus No problem if there is something wrong with the commands. We can improve step by step. -- Regards, Max

Answer 34 · 2020-08-31T00:21:19.000Z

I would appreciate if anyone can continue the package manager.
I created the file and load the cobra library.

If cobra is not suitable to needs, we can use another library. There is no limitations. Or even pure code :)
cc @amir-shiati

https://github.com/DonyaOS/PackageManager/blob/master/src/donya.go

Are you ready to continue? @jbampton

Answer 35 · 2020-08-31T00:42:37.000Z

Hi @BaseMax !! Yes I would like to help on this go code. I am really tired here now and it's only lunch time.

I need to read this whole issue fully once again.

I have not used cobra before. I did use -> https://github.com/urfave/cli

Answer 36 · 2020-08-31T00:59:24.000Z

Hey, Thanks,

No limitation for CLI library. Enjoy by way :)

P.S: Just a small draft for Package Manager in PHP 7.4, to show how we can control arguments. (Pure code in PHP)
https://github.com/BaseMax/php-cli-apps

Answer 37 · 2020-08-31T07:59:51.000Z

Hey everyone.
@BaseMax i do apologize for late response.
I have been really busy this whole week.
I will get back to you guys as soon as i'm done with this client.
wish you all the best.

Answer 38 · 2020-08-31T12:06:04.000Z

Hey Amir,

Thank you for sending files to me.
John, Look at Amir source code, it's great. modular programming... @jbampton
Separate file for different functions of Donya package manager.
This was a good idea, and can help us to make source code more clear.

With the permission of Amir. file:
donya.zip

We can follow this structure while developing package manager.

What do you think?

Update:

Thanks to Amir: DonyaOS/PackageManager#5

Answer 39 · 2020-08-31T13:54:19.000Z

Hey friends,

How you will installing/downloading a package in the package manager? @jbampton
You need to clone Package repositories as cache? Or will need a API?

If need a API:
We will need a API or somethings like this for Searching or finding packages.
@sudo-corvus, and @limeschool

Otherwise: We will not need a API.
Only download zip file: https://github.com/DonyaOS/Packages/archive/master.zip, then unzip and save as cache.
e.g: $ donya update, will update local repository.

Related messages:

Answer 40 · 2020-09-02T19:28:31.000Z

Hi guys,

Thank you for inviting me to this project!

Here is what I think:

Statically compiled binaries are a relic from the past. Every modern OS supports dynamic loading of shared libraries. That said, I believe the first thing you need to do is to integrate the GNU C Library in the OS or at least provide it as a package / package dependency. If you stick to statically compiled Busybox linked against GLIBC, you'll end up with broken DNS resolving and therefore (almost) unusable network tools. That's the main reason I switched Minimal Linux Live from statically compiled Busybox to dynamically compiled Busybox. If you use Musl instead of GLIBC then you won't have DNS issues but you won't be able to use other Linux binaries that are taken directly, because they are most certainly linked against GLIBC.
An already existing package manager for static binaries is static-get. In fact, I'm providing static-get as an optional overlay bundle in Minimal Linux Live.

I'll keep an eye on the project development.

Regards,
Ivan Davidov

Answer 41 · 2020-09-11T22:10:59.000Z

Hello everyone,

Thanks for the effort of @esmaeelE
We managed to get to the next stage of work.

The previous script is now obsolete.
And new scripts from step1.sh, and step2.sh, step3.sh to step5.sh are.
We need help to clean and improve the scripts.

This will require several files whose links are as follows:

Unfortunately I could not upload the files directly to the repository. (Limitation is 100MB)

For this reason, I will soon write a script that he can download with the help of wget if he does not have the desired file.

Regards,
Max

Answer 42 · 2020-09-11T22:38:05.000Z

Regarding kernel dependencies, I wrote a script to automatically download all the required files.
If possible, please check and test: dependencies.sh

If you have a Linux (VM), So you can test and run scripts.
Ask a question, if you face a problem or error while running scripts.

We need feedback and want to improve scripts. Please send PR.

Answer 43 · 2020-09-14T08:24:00.000Z

donyaOS todo list

First things first

About 5 step scripts

Rewrite, add comment, add a wrapper script to run them
Express original version that I grab from Linux journal, something like:
This work based on an excellent tutorial by Petros Koutoupis on Linux journal DIY: Build a Custom Minimal Linux Distribution from Source
Combine with dependencies.sh by @BaseMax
Release them as newer and current version of donyaOS
Speed up compile time
Eliminate need for cross compile, is it possible?

Then

Make an .iso from latest output, i.e. LJOS-build.tar.xz
Test on virtualbox, qemu, vmware and physical system
Write a step by step tutorial to make and install donyaOS

Other things

Always use donyaOS as official name of project
Write a motto for donyaOS
donyaOS, Yet another Linux distro without any bullshit
When using donyaOS not clusterfuck will happen
Collect things relates to donya, for example an Iranian comedy movie Donya, In Persian دنیا
donyaOS must remain update, use latest version of packages if possible before 2020 finished
Full comment each line of code in donyaOS scripts
donyaOS will work out of the box without any pain
Add terminal multiplexer and window manager support as a simple UI before use desktop environment
Write a step by step tutorial, How to build a real and useful operating system in 2020 as we can
Create a sub project to collect learning material for OS
donyaOS must remain simple and fun forever
Turn donyaOS a platform for learning OS and how to make a new one but keep is simple
Select a design methodology for example KISS Principle as stated by esr in his book The Art of Unix Programming
A temp architecture for donyaOS

kernel
base
layer_0: bootloader, busybox, more essential tools for compliting busybox
layer_1: drivers, nettools, package manager
layer_2: window manager, tmux, base packages for other apps (compiler collection)
application: all application that user desired

All of them is my opinion, feel free to express anything wrong and must be change beside that as English is not my mother tongue please correct me.

I thing this can be used as a todo for donyaOS

Answer 44 · 2020-09-20T22:18:13.000Z

Hi,
Some new changes were made to the repository, sorry for the separate commits.

Also a guide references for users who want to learn more:

Some related links

Thanks to @esmaeelE

Answer 45 · 2020-09-20T23:15:04.000Z

The Next proposed version of donyaOS release totally based on the Petros Koutoupis article on Linux journal.

DIY: Build a Custom Minimal Linux Distribution from Source
Build a Custom Minimal Linux Distribution from Source, Part II

We want to extent it as a base for future developments

In the next step I will

Solve donyaOS script issues, some of them depicted in the picture.
Make an iso image that targets X86/64 architecture to install donyaOS