This is a mlsql-lang example project.
- Download Visual Studio Code
- Download MLSQL Lang VSCode plugin
After installation of Visual Studio Code
, switch to Extensions
tab, click ...
on right side of search bar, find the item Install from VSIX...
, choose the VSCode plugin we had already downloaded in preview step and intall it.
Notice that for now we only support Ligt theme in vscode. Select Code > Preferences > Color Theme :
In command palette popup,select the light color:
Click Clone or download
and select Download ZIP
, then you get the package of this project. Unzip it in you desktop.
Select File > Open... and choose the location where we unzip this project.
./src/try_mlsql
is a good start point for you to learn MLSQL../src/a_tour_of_mlsql
you can learn full picture of MLSQL Lang../src/examples/examples
there are many mlsql code snippets in this notebook../analysis/example/cifar10/ResizeImage
teach you how to processing image distributly../analysis/example/cifar10/DistributeTFTrainning
teach you how to train DL by tensorflow distributly and deploy model as UDF .
Ray
is a build-in plugin in MLSQL which can execute Python script.
The power part is that you can access the data in target table in Python and pass the result processed back as a new table.
Some limitation for now:
- The schema of python output should be specified mannually.
!python conf "schema=st(field(a,long))";
The basic python dependencies:
pyarrow==4.0.1
ray[default]
aiohttp==3.7.4
pandas>=1.0.5; python_version < '3.7'
pandas>=1.2.0; python_version >= '3.7'
requests
matplotlib~=3.3.4
uuid~=1.30
pyjava
Suppose you can have created virtual python enviroment called ray1.8.0
(this will used by example in this project by default).
conda create --name ray1.8.0 python=3.6
and make sure you have the aforementioned dependencies are also installed.