Todo

Question

Todo

Closed this issue 3 years ago · 20 comments

WenheLI commented 4 years ago

This will be a thread for discussion about what we should cover in the future.

IMO, we should have four major aspects:

Tabular
Text
Image
Sound

And we should implement some common features like dataset splitting, shuffle, data augmentation, visualizing?

Besides, data-type specific operations are also needed:

Generic
- Train test split
- Random sampler
- Scalers (Standard, Robust, MinMax)
Image
- Basic I/O
- Filter
- Crop
- Resize
- Rotating
- Augmentation
- color_mode
- Normalize
- (Some more ideas)
Text
- Embedding
- Serialize
- stats analysis
- Tokenization
- Stemming
- Stopword removal
- Padding
Soung
- FFT?
Tabular Data
- Encodings
- I/O

Discussions are welcome here!

steveoni commented 4 years ago

yes

👍1

Answer 1 · 2020-10-26T13:32:55.000Z

Typescript or vanilla Javascript?

Answer 2 · 2020-10-26T15:15:11.000Z

Typescript or vanilla Javascript?

Implement in TypeScript?

Answer 3 · 2020-10-27T02:09:46.000Z

Since danfo has already covered some basic interactions with table data, shall we firstly support more types of data first? I.E. Image data first?

/cc @steveoni @risenW

Answer 4 · 2020-10-27T07:17:59.000Z

Since danfo has already covered some basic interactions with table data, shall we firstly support more types of data first? I.E. Image data first?

/cc @steveoni @risenW

Or we could create a base app that integrates with danfo just for the sake of having an API to test.

Answer 5 · 2020-10-27T07:30:32.000Z

@WenheLI Do we really need visualization?

Answer 6 · 2020-10-27T07:38:38.000Z

IMO, the visualization could be achieved by other JavaScript libs in the community.

Answer 7 · 2020-10-27T07:59:41.000Z

@WenheLI Do we really need visualization?

Aye, I think visualization is kinda out of this repo's scope!

Answer 8 · 2020-10-27T09:02:33.000Z

This will be a thread for discussion about what we should cover in the future.

IMO, we should have four major aspects:

Tabular

Text

Image

Sound

And we should implement some common features like dataset splitting, shuffle, data augmentation, visualizing?

Besides, data-type specific operations are also needed:

Generic

Train test split

Random sampler

Scalers (Standard, Robust, MinMax)

Image

Basic I/O

Filter

Crop

Resize

Rotating

Augmentation

color_mode

Normalize

(Some more ideas)

Text

Embedding

Serialize

stats analysis

Tokenization

Stemming

Stopword removal

Padding

Soung

FFT?

Tabular Data

Encodings

I/O

Discussions are welcome here!

@WenheLI @steveoni I added some updates to this

Answer 9 · 2020-10-27T09:53:56.000Z

@risenW Looks solid! I think we have covered the most common methods for data processing. I think for now we can just follow this todo list and implement them step by step.
IMO, for the first step, we could wrap danfo and export some tabular related data-processing methods. And then move to image processing and some generic processing methods.

/cc @yorkie @steveoni @risenW

Answer 10 · 2020-10-27T10:53:13.000Z

@risenW Looks solid! I think we have covered the most common methods for data processing. I think for now we can just follow this todo list and implement them step by step.
IMO, for the first step, we could wrap danfo and export some tabular related data-processing methods. And then move to image processing and some generic processing methods.

/cc @yorkie @steveoni @risenW

Yes, let's work with this. So I think the next step will be to define some code structure/styling and guide. Then we can create a basic skaffold application. @WenheLI , do you have style guide, or should I quickly draft one?

Answer 11 · 2020-10-27T12:20:25.000Z

Yes, let's work with this. So I think the next step will be to define some code structure/styling and guide. Then we can create a basic skaffold application. @WenheLI , do you have style guide, or should I quickly draft one?

@risenW Sure! Let's get things on track! It's kinda late at my time now. If it is possible, could you write a draft one. And we can review it later!

Answer 12 · 2020-10-27T13:26:13.000Z

Yes, let's work with this. So I think the next step will be to define some code structure/styling and guide. Then we can create a basic skaffold application. @WenheLI , do you have style guide, or should I quickly draft one?

@risenW Sure! Let's get things on track! It's kinda late at my time now. If it is possible, could you write a draft one. And we can review it later!

Okay, I'll draft one later today.

Answer 13 · 2020-10-27T22:00:17.000Z

@yorkie @steveoni @WenheLI I created the base project skaffold, and also added a contributing guide. We can discuss what should be kept/discarded in the guide, as it is not final.

Answer 14 · 2020-10-28T03:03:51.000Z

@risenW - This looks terrific!! The Contributing Guide looks perfect! I will add more details in the scaffold and then we can turn this repo to public! This looks promising!

Answer 15 · 2020-10-28T08:08:39.000Z

@risenW Just update the codebase with ts. If this is okay with you, shall we turn this to public?

Answer 16 · 2020-10-28T08:31:22.000Z

@risenW Just update the codebase with ts. If this is okay with you, shall we turn this to public?

Looks great! Yea we can make it public. Quick question, are we using Typescript or Vanilla JS?

Answer 17 · 2020-10-28T09:33:27.000Z

@risenW Just update the codebase with ts. If this is okay with you, shall we turn this to public?

Looks great! Yea we can make it public. Quick question, are we using Typescript or Vanilla JS?

Definitely TypeScript. It makes life easier!

Answer 18 · 2020-10-28T10:27:12.000Z

@risenW Just update the codebase with ts. If this is okay with you, shall we turn this to public?

Looks great! Yea we can make it public. Quick question, are we using Typescript or Vanilla JS?

Definitely TypeScript. It makes life easier!

Okay, noted 😁

Answer 19 · 2021-04-16T15:13:37.000Z

Just removed because this is stale and almost done.