binghui89/SF2

Process CAISO data
1. Collect_CAISO_xxxx.ipynb -> Download raw CAISO OASIS files (directory CAISO_OASIS)
2. Process_CAISO_xxxx.ipynb -> Process raw CAISO data and generate for_flexiramp_summary_RTD/RTPD.csv, only format processing and no calculation
3. Peep_CAISO_xxxx.ipynb -> Read for_flexiramp_summary_RTD or RTPD.csv and generate ./tmp_excels/df_rtd.csv or df_rfpd.csv, calculate advisory renewable forecasts and net load errors using persistence forecast and show snapshots

Process of the revision in August 2021.
1. Use lab_ghi2power.m to convert raw IBM GHI into power.
2. load_ibm_5sites_carlo.m (previously: load_ibm_5sites.m)
3. load_caiso_data.m (previously the same, but it is updated. Need ./tmp_excels/df_rtd.csv and df_rtpd.csv)

Process of all kNN runs in August 2021 revision
1. 1-dim classifiers (example in lab_revision_1.m)
1.1. Run single site with all 1-dim classifiers.

    % Sample MATLAB code
    for this_month = 3:4
        this_year = 2020; 
        knn_rtpd_1site_1dim;
        close all;
        save(strcat('knn_rtpd_puresolar_', int2str(this_month), '.mat'));
        clearvars -except 'this_month';
    end

1.2. Run dynamic selection processing

    % Sample MATLAB code
    clear;
    for this_month = 4:12 % Target month for dynamic selection
        tic;
        knn_1 = load(strcat('knn_rtpd_puresolar_', int2str(this_month-1), '.mat')); 
        knn_2 = load(strcat('knn_rtpd_puresolar_', int2str(this_month), '.mat'));
        % Sometimes the elements of cell_baseline_rtpd are empty except the
        % first column, so this is a preprocessing process.
        knn_1.cell_baseline_rtpd = knn_1.cell_baseline_rtpd(:, 1);
        knn_2.cell_baseline_rtpd = knn_2.cell_baseline_rtpd(:, 1);
        knn_rtpd_post;
        save(strcat('knn_post_rtpd_puresolar_', int2str(this_month), '_complete.mat'));
        fprintf('%s written to file.\n', strcat('knn_post_rtpd_puresolar_', int2str(this_month), '_complete.mat'));
        toc;
        clearvars -except 'this_month';
        close all;
    end

2. PCA classifiers (example in lab_revision.m)
2.1. Run all PCA classifiers (1 principal component to 3 principal components)

    % Sample MATLAB code
    for this_month = 6:12
        tic;
        this_year = 2020; 
        write_flag = 0;
        knn_rtpd_5site_pca;
        close all;
        save(strcat('knn_rtpd_puresolar_', int2str(this_month), '.5site.pca.mat'));
        toc;
        clearvars -except 'this_month';
    end

2.2. Run dynamic selection

    clear;
    for this_month = 4:12 % Target month for dynamic selection
        tic;
        knn_1 = load(strcat('knn_rtpd_puresolar_', int2str(this_month-1), '.5site.pca.mat')); 
        knn_2 = load(strcat('knn_rtpd_puresolar_', int2str(this_month), '.5site.pca.mat'));
        knn_rtpd_post;
        save(strcat('knn_post_rtpd_puresolar_', int2str(this_month), '_complete.pca.mat'));
        fprintf('%s written to file.\n', strcat('knn_post_rtpd_puresolar_', int2str(this_month), '_complete.pca.mat'));
        toc;
        clearvars -except 'this_month';
        close all;
    end

Visualization
For PCA results, which include almost everything in Fig.9 of the revision:
>> this_month = 8; visualization_rtpd_pca_other_month;