Missing files
Closed this issue · 20 comments
Hello Francois,
I didn't upgrade past version 1.0.5.6 but now my folder on the LabVIEW Open Source Project Folder is completely different and missing key packages like the Client. This causing my LV application to be broken. See my screenshot of an older version and my current file location. I see from another issue you are reorganizing the packages, but I am not sure how to fix this issue. Do I need to run everyone of the VIP that you created?
There are some dependencies that have been namespaced, which would break the link to your 1.0 package.
I don't think you will be able to use broker 1.0 with the other packages which I've updated to 3.1.x
I think I'll need to specify exclusions in the vip file so that this is not possible to install.
Can you tell me the list of package versions you have installed? OpenSerializer, Data manipulation, MQTT control packets, etc
I'm also having a lot of trouble using or following this project. The dependencies seem to be scattered all over the place and there's no clear documentation of how to use the project.
I actually decided to use the https://github.com/cowen71/mqtt-LabVIEW project mostly because it compiled and worked when I brought it down from github and I only need a client implementation. I've since forked that project (https://github.com/jkoplo/mqtt-LabVIEW), re-organized a lot of it and begun to realize some of the core MQTT specs were never implemented.
Going through this project again, I realize that my implementation has converged quite a bit with this project. I'd love to join forces and help develop, but the current structure of the code/repos is not friendly for community involvement...
Here are the packages I have installed that relate to the word open:
Thanks for that feedback. @jkoplo
I've tried to find time this weekend to create a comprehensive video to show the installation steps from A to Z. I bumped into an issue in the Unit testing of the client which I decided to resolve before moving forward.
For installing from Git project directly, I believe this is a matter going through the steps consistently, which I agree it's not documented properly. I heavily rely on VI packages, so this needs to be the first step before even opening the project. If dependencies are not installed first, you will hit a wall and that's definitely a recurring theme in LabVIEW world. I know there are a few GLA Summit presentations that directly speak about that.
I'll create a video for this specifically. I think it's important and I thank you for the genuine feedback.
@russellgsystems I can't be certain because I'm not home with my dev environment, not I think it might be OpenSerializer that should be downgraded to 1.0.0.
I'll take a look closer this evening if that's not it.
Thanks for the quick response. The timing is frustrating - I looked at your library a month ago and deemed the broker overkill for my needs and the code hard to bring into my project, so I forked the cowen71 repo and have spent a couple weeks hacking it into shape. Then I decided to check back on how you handled something and realized you're actively developing the client now...
My company doesn't license the Enterprise version of VIPM - I think some of this would be easier with those .vipc files? That being said, it's probably best if open-source projects can depend on as few proprietary tools as possible...
You don't need a Pro license to install the packages, just to build a vipc file. But I assumed you know this already.
Sorry it's difficult to answer from a cell phone... I think you need to install the vipc file located at the root of this project: https://github.com/LabVIEW-Open-Source/MQTT-Client
Since you don't mind the broker part, if you install the MQTT-Client.vipc file and then open the client project 3.1.x source code, that should work.
One thing to know is that there is sometimes a small delay (up to couple days) for package updates to end up on VIPM community... So this VIPC file might not contain the latest packages of they haven't already been published by VIPM.io
I could add a VIPC file that contains the actual packages. I think it would help, wouldn't it?
And I just realized you published to VIPM, making all this much easier. I'm playing with the examples now and they seem to be working against my local docker mosquitto broker. I probably owe you several beers.
To be completely honest, I've been out of the LV game for a bit and the evolution of VIPM is one of the things I need to get caught up on. Things seem to be working for my purposes right now.
I'll need to dig through your toolkit and line it up with how I was using the other adapted code. We're sharing an MQTT connection amongst several cloned state machines so I've got some interesting routing/subscribing but it's built at a level above the MQTT messages and should be easily adapted to a different client library.
My serialization is also different - using JSON with a loose schema around header/payload, but I think I can handle that by overloading your serialization classes.
@jkoplo that's why the project is broken down into bits.
You'll want to extend the OpenSerializer class with a serializer.JSON class and inject it in the client. Look at the base64 extension. The exact same can be done in JSON. I've done it before for a customer of mine. I just didn't have time to create another version from scratch which does not reuse their IP... But you can do it in a few minutes by wrapping jsontext or JKI JSON.
You'll also want to extend the connection class with your special connection needs.
Both of them should work orthogonally with the new 3.1.x version of the client.
That will be in my video this weekend too...
Yup. I already have my own set of serialization/payload classes that implement several encodings. I'll see if my implementation is generic enough that I can PR it to your project.
I think the connection is fine - my routing implementation has more to do with duplicating received events to multiple queues when they come in - it's generic enough that the underlying MQTT implementation doesn't matter.
I did look at your TLS implementation and noticed it's not built up for mutual auth. I'll look into adding that as either a new class or as part of the a VI in the TLS class depending on complexity. I've already got that implemented and tested in some local code.
Cool. Open a new issue so we can track that discussion on a separate thread. You'll want to override the handshake method, most likely.
For template, I created a local Queue connection which does a very crude hmac type of handshake.
@russellgsystems
I think the DataManipulation would also need an downgrade. (Post Edit: actually, no... the DataManipulation is fine with 1.3.0)
Alternatively, you can upgrade the MQTT Client and Broker to 3.1.x.which should resolve as before, except if you were injecting your own Serializer class or perhaps any reference to control packets. In the migration, MQTT Client and MQTT Control packets were extracted out of the MQTT Broker library, so it would definitely need to be relinked.
I didn't upgrade past version 1.0.5.6 but now my folder on the LabVIEW Open Source Project Folder is completely different and missing key packages like the Client. This causing my LV application to be broken. See my screenshot of an older version and my current file location. I see from another issue you are reorganizing the packages, but I am not sure how to fix this issue. Do I need to run everyone of the VIP that you created?
Going back to your original post, with a clearer view from actually sitting at a computer (no cell phone)...
I believe that if you upgrade to Control Packets 3.1.1, you will need to also upgrade your Broker and install the Client code.
Most likely, you are using the client code only, so the Broker is actually not important to be installed anymore. It is a bonus for testing locally, for sure, but not required.
My suggestion for course of action is this:
1- Install MQTT Client 3.1.2.9. It will update all the dependencies to the latest.
2- You can keep MQTT Broker 3.1.1 installed
3- You'll want to install at least one connection class: MQTT TCP Connection
(You'll see that there is a missing dependency in the LabVIEW open Source palette... this is temporary until the dependency is updated on VIPM Community repo.)
4- You could have relinking issues with any Client public methods that are called in separate VIs (not inside a library). If they are in a library, it will definitely scan vi.lib to find the new location. Because it is not under Broker folder, it might require you to point to its new location once.
5- The only part I don,t know, is if the Serializer stuff will relink. If you have not extended the OpenSerializer class, it will depend on whether or not you make use of the marshalling and unmarshalling methods outside classes or not. It could be that you will find "Not an Object" indicators on your block diagrams. Perhaps not.
I started a wiki page here, to document the migration process:
https://github.com/LabVIEW-Open-Source/LV-MQTT-Broker/wiki/Migration-from-Broker-1.0-to-3.1.x
@russellgsystems
I've come up with a partial migration VI which will bring you close to up and running in a single click.
- Open your project, after the migration to MQTT Broker and Client 3.1.1
- Let LabVIEW try to relink anything it can, or ignore all to go straight to the project view
You're gonna see loads of these:
I've created a Test Client vi which looks a bit like this after the migration:
- Launch the tool attached to this page (migration project.vi)... with its subVI (migration_store)
It will list all VIs in your project and go through their subVIs, find the ones that are missing and replace with the new path.
You'll need to open them and save them. I did not go through the process of implementing the Undo support, so you should probably save them yourself after inspection.
Some artefacts are not yet replaced automatically, but the hook is there in case that becomes necessary. Hopefully it gets you close to where you can do the rest manually.
Written in LV2019:
MQTT Broker Migration Helper Tool.zip
Thanks @francois-normandin that got me started and then I went through and manually replace the rest of the VIs. Your documentation was helpful so I knew to add the TCP connection. I used your client class in another class and I had to remove your class and add it back in my class data so my class was not broken. But I think I got it now.
I don't know if this was in this thread that I mentioned I'd be recording youtube videos about this project, to get some kind of tutorials going, but I'll say this here anyway.
I've just recorded about 90 minutes of MQTT content today. I still need to edit the videos, so it might take a few days to find the time to do so, but it's really a matter of time.
Thanks for pushing me to do it quickly...
I'll be closing this issue now.
I've sent a bunch of new packages on VIPM Community, but they haven't been published yet.
Look at the release 3.1.2 page for details...
Great thanks @francois-normandin!