toluaina/pgsync

can I create schema with where condition like this?

Closed this issue · 6 comments

PGSync version: 2.1.9

Postgres version: 14

Elasticsearch version: 7.14.2

Redis version: 7.0.0

Python version: 3.7

Problem Description:
can I create schema with where condition like this?

SELECT * FROM users WHERE user_status = 'active';

[
    {
        "database": "db_name",
        "index": "user-index",
        "nodes": {
            "table": "product",
            "columns": [
                "id","name","desc","status"
            ],
            "children": [
                {
                    "table": "users",
                    "columns": [
                    ],
                    "relationship": {
                        "variant": "scalar",
                        "type": "one_to_one",
                        "foreign_key": {
                            "child": ["product_id"],
                            "parent": ["id"]
                        }
                    }
                }
            ]
        }
    }
]

I mean, how can I add WHERE user_status = 'active' in schema.

thank you in advnace

Error Message (if any):



You would need to use the plugin feature for this.
Here is an example and documentation

example code:

class MyFilterPlugin(plugin.Plugin):
    """Example Filter plugin."""

    name: str = "Filter"

    def transform(self, doc: dict, **kwargs) -> dict:
        """Demonstrates how to perform a filter."""
        doc_id = kwargs["_id"]
        doc_index = kwargs["_index"]
        if doc["id"] == 1:
            return None

        return doc

@toluaina how can I add PYTHONPATH? sorry I am newbie in python.

export PYTHONPATH=$PYTHONPATH:/this/is/the/path/to/plugin

This is further documented here
You can add a ltemporary log/print statement to the plugin to ensure it is getting called when you run pgsync

I already made like this:

# echo $PYTHONPATH
/plugins
# ls
documents  plugins  runserver.sh  wait-for-it.sh
# cd plugins
# ls
__init__.py  activestatusplugin.py

but still got error:

pgsync-1         | Traceback (most recent call last):
pgsync-1         |   File "/usr/local/bin/bootstrap", line 70, in <module>
pgsync-1         |     main()
pgsync-1         |   File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1128, in __call__
pgsync-1         |     return self.main(*args, **kwargs)
pgsync-1         |   File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1053, in main
pgsync-1         |     rv = self.invoke(ctx)
pgsync-1         |   File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1395, in invoke
pgsync-1         |     return ctx.invoke(self.callback, **ctx.params)
pgsync-1         |   File "/usr/local/lib/python3.7/site-packages/click/core.py", line 754, in invoke
pgsync-1         |     return __callback(*args, **kwargs)
pgsync-1         |   File "/usr/local/bin/bootstrap", line 60, in main
pgsync-1         |     document, verbose=verbose, repl_slots=False, **kwargs
pgsync-1         |   File "/usr/local/lib/python3.7/site-packages/pgsync/sync.py", line 94, in __init__
pgsync-1         |     self.validate(repl_slots=repl_slots)
pgsync-1         |   File "/usr/local/lib/python3.7/site-packages/pgsync/sync.py", line 175, in validate
pgsync-1         |     root: Node = self.tree.build(self.nodes)
pgsync-1         |   File "/usr/local/lib/python3.7/site-packages/pgsync/node.py", line 251, in build
pgsync-1         |     raise NodeAttributeError(f"Unknown node attribute(s): {attrs}")
pgsync-1         | pgsync.exc.NodeAttributeError: "Unknown node attribute(s): {'plugins'}"
pgsync-1         |  0:00:00.038423 (0.04 sec)
pgsync-1         | Traceback (most recent call last):
pgsync-1         |   File "/usr/local/bin/pgsync", line 7, in <module>
pgsync-1         |     sync.main()
pgsync-1         |   File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1128, in __call__
pgsync-1         |     return self.main(*args, **kwargs)
pgsync-1         |   File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1053, in main
pgsync-1         |     rv = self.invoke(ctx)
pgsync-1         |   File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1395, in invoke
pgsync-1         |     return ctx.invoke(self.callback, **ctx.params)
pgsync-1         |   File "/usr/local/lib/python3.7/site-packages/click/core.py", line 754, in invoke
pgsync-1         |     return __callback(*args, **kwargs)
pgsync-1         |   File "/usr/local/lib/python3.7/site-packages/pgsync/sync.py", line 1248, in main
pgsync-1         |     sync: Sync = Sync(document, verbose=verbose, **kwargs)
pgsync-1         |   File "/usr/local/lib/python3.7/site-packages/pgsync/sync.py", line 94, in __init__
pgsync-1         |     self.validate(repl_slots=repl_slots)
pgsync-1         |   File "/usr/local/lib/python3.7/site-packages/pgsync/sync.py", line 175, in validate
pgsync-1         |     root: Node = self.tree.build(self.nodes)
pgsync-1         |   File "/usr/local/lib/python3.7/site-packages/pgsync/node.py", line 251, in build
pgsync-1         |     raise NodeAttributeError(f"Unknown node attribute(s): {attrs}")
pgsync-1         | pgsync.exc.NodeAttributeError: "Unknown node attribute(s): {'plugins'}"

please help me @toluaina

@phsophea101

I think you would be better served dropping me a message on discord taina#4505.
It appears you have an error in your schema.json with the way you have declared the plugins

@toluaina Its working now,