nguyenngoclongdev/vs-data-sync

Unable to generate any output, documentation?

Closed this issue · 6 comments

andyg2 commented

Please describe feature/problem details and solution you'd like.
Some documentation for configuring the database.json file

I've tried a lot of variations but I can't seem to get anything working.

Also not sure how to run the diff/parsing.

Config

{
  "$schema": "https://cdn.statically.io/gh/nguyenngoclongdev/cdn/main/schema/v1/data-sync.json",
  "verbose": true,
  "patterns": {
    "patternName": {
      "source": {
        "database": "crm_dev",
        "host": "localhost",
        "port": 3306,
        "user": "root",
        "password": ""
      },
      "target": {
        "database": "crm_dist",
        "host": "localhost",
        "port": 3306,
        "user": "root",
        "password": ""
      },
      "diff": {
        "format": true,
        "tables": [
          {
            "name": "customers",
            "where": "1",
            "orderBy": "customerID",
            "schema": "$schema"
          }
        ]
      },
      "migrate": {
        "noInsert": false,
        "noUpdate": false,
        "noDelete": false,
        "noRowAffected": "warn",
        "multipleRowAffected": "throw"
      }
    }
  }
}

Terminal output

2023-08-30 02:45:30.680 [info] Starting Data Sync extensions.
2023-08-30 02:45:45.239 [info] Click on refresh the info activity
2023-08-30 02:48:01.347 [info] Click on refresh the compare activity {"skipReanalyze":false,"currentPattern":"","migrateName":""}
2023-08-30 02:48:29.633 [info] Click on refresh the info activity
2023-08-30 02:48:29.634 [info] Click on refresh the compare activity {"skipReanalyze":true,"currentPattern":"","migrateName":""}
2023-08-30 02:48:29.634 [info] Click on refresh the diff activity {"skipReanalyze":true,"currentPattern":"","migrateName":""}
2023-08-30 02:48:39.744 [info] Click on refresh the info activity
2023-08-30 02:48:51.081 [info] Click on refresh the compare activity {"skipReanalyze":false,"currentPattern":"","migrateName":""}
2023-08-30 02:48:58.816 [info] Click on refresh the diff activity {"skipReanalyze":false,"currentPattern":"","migrateName":""}
2023-08-30 02:49:07.258 [info] Click on info tree item 'patternName'
2023-08-30 02:49:12.869 [info] Generating all snapshot files...
2023-08-30 02:49:12.869 [info] Generating target snapshot with db connection 'postgres://root:******@localhost:3306/liquidhub_target'....
2023-08-30 02:49:26.443 [error] Failed to analyze data! A system error occurred (ENOENT: no such file or directory, open 'xxxxxxxxxxxxxxxxxxxxxxxxxxxx\web\migrations\2023-08-30_patternName_test1\session.json')
2023-08-30 02:50:34.225 [info] Click on refresh the info activity
2023-08-30 02:51:10.424 [info] Click on refresh the info activity
2023-08-30 02:51:19.110 [info] Click on refresh the info activity
2023-08-30 02:51:19.110 [info] Click on refresh the compare activity {"skipReanalyze":true,"currentPattern":"","migrateName":""}
2023-08-30 02:51:19.110 [info] Click on refresh the diff activity {"skipReanalyze":true,"currentPattern":"","migrateName":""}
2023-08-30 02:51:37.744 [info] Click on refresh the info activity
2023-08-30 02:51:43.461 [info] Click on refresh the info activity
2023-08-30 02:51:43.462 [info] Click on refresh the compare activity {"skipReanalyze":true,"currentPattern":"","migrateName":""}
2023-08-30 02:51:43.462 [info] Click on refresh the diff activity {"skipReanalyze":true,"currentPattern":"","migrateName":""}
2023-08-30 02:52:27.892 [error] Failed to analyze data! A system error occurred (ENOENT: no such file or directory, open 'xxxxxxxxxxxxxxxxxxxxxxxxxxxx\web\migrations\2023-08-30_patternName_test1\database.json')
2023-08-30 02:52:40.517 [info] Click on refresh the info activity
2023-08-30 02:53:02.936 [info] Click on refresh the info activity
2023-08-30 02:53:02.936 [info] Click on refresh the compare activity {"skipReanalyze":true,"currentPattern":"","migrateName":""}
2023-08-30 02:53:02.936 [info] Click on refresh the diff activity {"skipReanalyze":true,"currentPattern":"","migrateName":""}
2023-08-30 03:03:08.727 [info] Click on refresh the info activity
2023-08-30 03:03:21.622 [info] Click on refresh the compare activity {"skipReanalyze":false,"currentPattern":"","migrateName":""}
2023-08-30 03:03:41.790 [info] Click on refresh the info activity
2023-08-30 03:03:59.758 [info] Click on refresh the compare activity {"skipReanalyze":false,"currentPattern":"","migrateName":""}
2023-08-30 03:04:01.022 [info] Click on refresh the compare activity {"skipReanalyze":false,"currentPattern":"","migrateName":""}
2023-08-30 03:04:01.206 [info] Click on refresh the compare activity {"skipReanalyze":false,"currentPattern":"","migrateName":""}
2023-08-30 03:04:01.406 [info] Click on refresh the compare activity {"skipReanalyze":false,"currentPattern":"","migrateName":""}
2023-08-30 03:04:31.494 [info] Click on refresh the info activity
2023-08-30 03:05:16.062 [info] Click on refresh the info activity
2023-08-30 03:05:38.878 [info] Click on refresh the info activity
2023-08-30 03:06:21.541 [info] Click on refresh the info activity
2023-08-30 03:07:24.134 [info] Click on refresh the info activity
2023-08-30 03:07:33.981 [info] Click on refresh the compare activity {"skipReanalyze":false,"currentPattern":"","migrateName":""}
2023-08-30 03:07:35.182 [info] Click on refresh the diff activity {"skipReanalyze":false,"currentPattern":"","migrateName":""}
2023-08-30 03:07:45.253 [info] Click on refresh the info activity
2023-08-30 03:07:59.592 [error] Failed to analyze data! A system error occurred (ENOENT: no such file or directory, open
nguyenngoclongdev commented

Hi @andyg2

Thanks for your feedback.
I have checked your config information. It appears that there are some incorrect configurations, which prevented the creation of snapshot files before data comparison.

{
  "$schema": "https://cdn.statically.io/gh/nguyenngoclongdev/cdn/main/schema/v1/data-sync.json",
  "verbose": true,
  "patterns": {
    "patternName": {
      "source": {
        "database": "crm_dev",
        "host": "localhost",
        "port": 3306,
        "user": "root",
        "password": "" 👈 Is it correct if the password information is left empty?
      },
      "target": {
        "database": "crm_dist",
        "host": "localhost",
        "port": 3306,
        "user": "root",
        "password": "" 👈 Is it correct if the password information is left empty?
      },
      "diff": {
        "format": true,
        "tables": [
          {
            "name": "customers",
            "where": "1", 👈 Since it is a WHERE statement, I need to know the column you intend to filter. Therefore, please specify the columnName in this section, for example, id = 1.
            "orderBy": "customerID",
            "schema": "$schema" 👈 For PostgreSQL, should this be set to 'public' or something similar?
          }
        ]
      },
      "migrate": {
        "noInsert": false,
        "noUpdate": false,
        "noDelete": false,
        "noRowAffected": "warn",
        "multipleRowAffected": "throw"
      }
    }
  }
}

Please let me know if you have any further questions or concerns.

andyg2 commented

Thanks for the info, it's mysql, not postgres, there's no password as local machine. I tried leaving the schema empty but that also didn't work.

For the WHERE statement, I would like to query all records.

SELECT * FROM customers;

or

SELECT * FROM customers WHERE 1;

I've tried all of these:

          {
            "name": "customers",
            "where": "1",
            "orderBy": "customerID",
            "schema": "" 
          }       
          {
            "name": "customers",
            "orderBy": "customerID",
          }       
          {
            "name": "customers",
            "where": "1",
            "orderBy": "customerID",
          }       
          {
            "name": "customers",
            "where": "1",
            "orderBy": "customerID",
            "schema": "public" 
          }       
          {
            "name": "customers",
            "where": "customerID != 0",
            "orderBy": "customerID",
          }

I appreciate your help :)

nguyenngoclongdev commented

Oh, I Understand!

Currently, the Data Sync extension only works with PostgreSQL because I originally developed it exclusively for my company's use.
However, if you are interested in supporting it, I can open-source the extension, and you can create a MySQL adapter to connect with MySQL.

andyg2 commented

Currently, the Data Sync extension only works with PostgreSQL because I originally developed it exclusively for my company's use. However, if you are interested in supporting it, I can open-source the extension, and you can create a MySQL adapter to connect with MySQL.

Yeah I can certainly try! Thanks

nguyenngoclongdev commented

Hi @andyg2

I have opened the source code. You can explore it and create an adapter for MySQL.
Firstly, I suggest focusing on these two files:

  • src/commands/actions/generateSnapshotsAsync.ts
  • src/commands/actions/generateMigrateAsync.ts

If you have any questions, please let me know.

andyg2 commented

Hi, sorry, my situation has changed here I can't work on the MySQL adapter. I appreciate you took the time to open source the code base and I feel really bad about just vanishing. I'll let you know if I can get on to it in the future.
Feel free to close this issue again.

  • ❤️1