UAP / UWP support
neaorin opened this issue · 46 comments
When trying to install the Couchbase.Lite package into a Universal Windows Platform (UWP) application project, I get the message below:
Couchbase.Lite 1.1.2 is not compatible with UAP,Version=v10.0.
This is one of our long term goals but it has no timeline yet. Making Couchbase Lite compatible with the Windows Runtime is an enormous undertaking.
If you could list the main TODOs, I would try to contribute to this effort. Thanks
Any update on this? Since UWP is picking up (more and more applications are building for win10), have you reconsidered support for UWP?
It's not that we are considering whether or not to do it, it's just that in our current setup we don't have the manpower to take this on. I'll bring it up today in our meeting though.
Hey, thanks for bringing it up. Please let us know if you decide a timeline. Finger's crossed.
From what I hear the pressure for this is increasing, so after our release of 1.2 I think we are going to start planning on how to integrate.
Great to hear! The missing UWP support is the only thing that keeps us from using couchbase lite in our project.
Latest round of build errors using latest Universal SDK, and after getting rid of some noise errors (like missing SerializableAttribute, logging issues, etc).
@borrrden: I've also pushed my spike work so that you can see what exactly I did to get the build error list. Some of the stuff was hacky, but done simply to get rid of compile errors. Some was legit (like refactoring calls to Stream.Close over to Stream.Dispose instead), and the project/sln files themselves are also keepable.
Any Idea of a roadmap for UWP in the meantime? I'm just in the process on deciding which storage solution I will use for my App. IOs and Android will be the first, but UWP hast to follow.
None yet. There are some irresolvable blocking issues that we have to think hard about or wait for them to get implemented (missing API from WinRT). However, that being said, we do have some users who are setting up a kind of phone gap scenario using the REST API between a regular Windows service (or console app) and UWP applications.
You mean WinRT is missing some APIs that Android and IOs support? (is WinRT and UWP Windows 10 the same API set?)
UWP Windows 10 WinRT Whatever it is called these days, it's missing some regular .NET APIs that we use. That's what I meant. I definitely want to start looking for alternatives once we get .NET into continuous integration though.
That's good. If I know, it will come I will now start with it under Android and IOs
It is coming, the unfortunate open ended question (for now) is when. I'll be parading all over everywhere once we have more solid information but I wouldn't wait around for it.
One bit of good news: according to the latest ApiPort scan, we're down to 80 missing type/member references with respect to NetCore 5.0, and only 35 issues remain if you filter out those that have a "Recommended Change".
Great. Rooting for you guys :D
Since this has been moved from in progress to the backlog do we think uwp support will come in 2016? Thanks
I tried again to see what would happen if I compile against uap-10.0, and here are some results:
System.Security.Cryptographyis unavailable, and no streaming encryption API seems to exist in UWPSystem.Diagnostics.Tracedoes not exist, need to rewrite logging for UWPSerializableAttributedoes not exist (possibly unneeded)- WebSocket-Sharp is not compatible with UWP (Need to rewrite the listener setup logic and web socket change tracker)
System.Threadis unavailable (one of our task schedulers uses a private thread implementation)Environment.GetFolderPathdoes not exist (need an alternative way to find the local application data folder, but I haven't checked how to do it yet)Assembly.GetExecutingAssemblyandAssembly.GetCustomAttributeare unavailable (used for reading the git version info out of the assembly, could be cut if needed...)StringComparison.InvariantCultureIgnoreCasedoes not exist (Not sure what the alternative is)NetworkInterfaceis now completely different and doesn't provide any info or callbacks about network changes (this will completely ruin online / offline reaction without a workaround)
This is just for the core library, I haven't checked the plugins yet but I am pretty sure that SortKey class is missing.
As a general consideration, missing bits starting with "System" often end up in "Windows"
(1) Windows.Security.Cryptography
This sample avoids DataProtection (in case the Manifest issue is breaking)
But it requires the entire stream load in memory which is not workable for large streams.
Bouncy Castle has a stream interface and a PCL (ok for UWP and Android) version is here:
https://github.com/onovotny/BouncyCastle-PCL
(2) Trace is in Windows.Foundation.Diagnostics
(3) no comment ATM.
(4) This functionality needs to be Async enabled anyways so a library replacement is probably a good idea across all targets.
https://msdn.microsoft.com/en-us/windows/uwp/networking/websockets
https://msdn.microsoft.com/en-us/windows/uwp/networking/network-communications-in-the-background
Scanning the issues in WebSocket-Sharp, the author does not seem interested in supporting UWP.
Here are some cross-platform projects that might work:
https://github.com/NVentimiglia/Websockets.PCL
https://github.com/aspnet/WebSockets (targets Core 1.0)
Supports every imaginable target:
https://github.com/kerryjiang/WebSocket4Net
https://github.com/kerryjiang/SuperSocket.ClientEngine
(5) is an interesting issue since Microsoft is attempting to support Stephan's brilliant design efforts with Tasks, TPL and DataFlow to bring people into a modern era of parallelism, but plain-old-threads refuse to die. It appears that the legacy System.Thread is scheduled for inclusion as soon as possible:
https://github.com/dotnet/corefx/issues/2576
The current work-around (or better way to do it depending on point of view) is:
"by passing TaskCreationOptions.LongRunning and TaskScheduler.Default to StartNew, a dedicated thread will be created for that one Task instance. To be fair, that's the current behavior and it's not documented as such, but I doubt it would change in the future due to a fair amount of code that has taken a dependency on it."
(6) ApplicationData.LocalFolder
Overview:
https://blogs.windows.com/buildingapps/2016/05/10/getting-started-storing-app-data-locally/
(7) Possibly see:
https://github.com/SimpleGitVersion/SGV-Net
(8) no comment ATM
(9) This one is interesting and at first glance almost nothing about network status is located where you think it should be. But it is such a fundamental item for the mobility of UWP, it becomes a core system function.
Also due to mobile and power management issues, any App can miss out on state changes because it has simply been kicked out of memory. So there is a background task trigger system to permit Apps to be responsive to system state changes even when they are not running.
https://msdn.microsoft.com/windows/uwp/launch-resume/support-your-app-with-background-tasks
"Your app can respond to system-generated events by registering a background task with the SystemTrigger class. An app can use any of the following system event triggers (defined in SystemTriggerType)"
"InternetAvailable" etc in SystemTriggerType
https://msdn.microsoft.com/library/windows/apps/br224839
Here is a walkthrough:
http://social.technet.microsoft.com/wiki/contents/articles/26925.background-task-network-change-trigger.aspx
And also this useful API has a rather obtuse name:
Windows.Networking.Connectivity.NetworkInformation.GetInternetConnectionProfile();
Note that you can't plan to solve the issue by simply polling with that API call unless connection only matters when the App is restored but here is an example of a user created callback based on that:
Also this github project might be useful:
https://github.com/deepak-rathi/InternetSpeedUWP
I just wandered by in the process of making a list of UWP compatible databases. Of course I can't list Couchbase in my UWP Tools list right now, so please drop me a note when you have it working. I probably won't get around to providing pointers for the items I skipped above but I'll be happy to help with any questions you throw my way.
https://github.com/10UWP/The-UWP-Tools-List
As a general note, I have not looked at your code but it might be helpful to consider that when porting "old" code, converting to Async can trigger a cascade of architectural changes (for the better IMO which you can backport to Android etc)
It's hard to say, but my goal is to still have it in 2016. After the 1.3 release we are going to assess platform support going forward (including the possibility of dropping some, which would free up time for UWP). If there are any public API changes that need to be made (aside from additions), however, then we are in trouble. Hopefully it's limited to internal changes only.
Thanks for the detailed analysis. The reason for using DataProtection is so that we can leave the key generation up to the OS, and avoid bundling keys along with applications.
As for websocket-sharp, the reason I use this one in particular is because it is a full HTTP server with the possibility for websocket connections on endpoints (i.e. this endpoint is websocket, and everything else is regular HTTP). That needs to be considered with the implementation. Actually this is not being used yet, but is there for when the P2P engine adds support for web sockets. This will need be needed for Couchbase.Lite.dll, but for Couchbase.Lite.Listener.dll so perhaps I can start rolling out with the core only.
I was aware of the behavior of 5, but didn't want to depend on an implementation detail. Also the way that the database write thread works is by checking to see if new commands are queued re-entrantly (i.e. called from an operation that began on that thread) and if so, they skip the queue and execute immediately whereas others will join the queue and wait for the current operation to finish. It's easier if the thread remains the same.
As for 9, "InternetAvailable" is not enough, I need to know if the device has an IP address (they could be replicating P2P over a wireless AP that isn't connected to the Internet, which is a case I have personally seen). I will review that solution and see if it is possible. All of this will happen after the 1.3 release, however. Currently that means "near the end of this month at the soonest"
(9) InternetAvailable was one example of many triggers in the Enum.
Once a background task is fired up from the trigger invoking, then you can do more detailed investigation via the ConnectionProfile query. I haven't examined the GitHub code yet, but the last project I listed seems to also yield a network quality status which is groovy but probably not needed.
(4) The kerryjiang projects I referenced seem to include well developed clients and also servers - They are just all separate projects. I didn't check the license and other details that can make it otherwise non-viable.
Also, UWP has a lot of various socket-triggers which can wake up a powered down App which is something that would be difficult to include in a cross platform library. There are a few UWP specific libraries but it's hard to see how your architecture could support the many unique advantages of UWP while still having something that isn't a spaghetti library mess for other (less capable) platforms.
I sure hate LCD solutions such as the horrid PCL but creating a superset cross-platform solution that leverages the best of every platform is a general challenge that I am not aware of anyone solving. Some sort of proxy interface layer with a Capabilities object directing code paths out of a PlatformMatrix object :)
(5) You can always communicate with the LongRunning Task as if it is a thread and be sure the underlying thread will never change. If you make a proxy layer, then you can just discard the code when System.Thread gets exposed. If it was up to me, some whiteboard time might yield a cross platform solution based on the TPL since that was first standardized in .NET 4.0 and if Stephan says something is so then it is not really an implementation detail, it is just something not yet documented!
I would imagine that a NoSQL DB should be able to take advantage of all that parallelism support but I have come across a few articles where parallelism in a DB is considered to be a waste of time.
I think you should consider filing a bug on the DataProtection API requiring a Corporate account in the App manifest.
Microsoft responds a lot stronger to bugs based on an actual use-case by a real "customer" even if the customer is just one of the billion people running Windows and you have a great example of a practical reason to make that change.
For some reason, well written abstract cases of the type "your design would be better if" just seem to die quickly somewhere in their response chain.
I have had a bit if a discussion going here:
on the dismal state of data storage with UWP.
To the extent that all software projects and products compete in a vast ecosystem, you have an opportunity of unknown value to really stand out as an embedded DB solution for UWP. I think your partially connected sync solution is vital to mobile data use and something so obvious is also rather scarce. I filed a request with EF for that but since they can't even get a single embedded SQLite to work right, I'm not anticipating much.
They are however, planning to implement a EF Database Provider for their Azure NoSQL and you could use that OSS perhaps as a starting point to implement your own EF provider which would get you a lot of attention in the .NET world.
This certainly gives me more motivation, but last time I rushed into a platform I totally screwed myself. Rest assured that this is already something which has had its importance raised several times and I WILL be doing it. It's just a matter of fitting it in while not falling behind the other implementations (I'm only one person, after all). I've seen the hoops that others are jumping through to get Couchbase running on UWP (running a service and communicating between app and service via REST API) and it is not pretty.
This is a brief overview of Async for DB from the EF 6 release:
https://msdn.microsoft.com/en-us/data/jj819165.aspx
Although this thinking predates UWP by quite a bit, it becomes one of the first questions a UWP developer will ask you when you finally do the port so that is worth some whiteboard time perhaps.
Running any DB on the Windows desktop needs the UWP App to be sideloaded because it will fail store certification. I suppose you could make a VS Template for a "runtime brokered component" which could package up a sideload sample for customers until the port is done depending on how often people complain about it. My UWP tools list has an entry for a Github project that provide a starting point for that.
So when porting,you need to make sure the embedded Couchbase will pass cert. There is Util to test that. The one thing that is not immediately obvious is that you need to compile release builds fairly frequently during development which is a new "thing" with UWP. Thats because the Release compile path uses the .NET Native runtime via the VC++ backend for efficient machine code generation. It has the potential to break things that work fine in the Debug build. Particularly with a large codebase. This issue can hit any libraries you use such as that networking one where the author is not supporting a UWP target. Nothing insightful here, just perhaps saving you some head scratching moments - avoiding Reflection covers a lot of it. All the AOC differences are documented in detail and reviewing it ahead of time saves recoding stuff if it hits you. Many people never run into this due to lots of small UWP apps out there and hence the awareness is rather low.
I agree with you about doing it right. Things like .NET Native and missing API etc seem like an irritating hurdle but they are also there to make very efficient responsive code that can make your product stand out if you take advantage of it.
@10Dev I have some exciting news. I have a branch where I have powered through the changes to at least get Couchbase.Lite.dll and Couchbase.Lite.Storage.SystemSQLite.dll compiling and a simple spike test program working. You can see probably hundreds of lines of willy nilly changes so it's all just to get things running but they ARE running in Debug builds so far (will test with Release as per your note though). I will have to go back and make sure things can still work for regular .NET as well but I think this is a hurdle that is overcome.
I still can't find the Trace class so I've replaced it with the Debug class. Other than that some of the missing API has become available through Nuget.
The two places I am concerned about still are the server side of web sockets (Though I haven't actually implemented that yet) and the missing SortKey class from System.Globalization. However I was able to at least get some fundamental functionality for the client side web sockets using the .NET web socket namespace. I seem to remember Mono not having implemented this functionality yet which is the reason I sought out an external library in the first place but a cursory glance at the current Mono source indicates that at least the client side seems to be implemented now. That's something I need to confirm though. If so I can drop the web socket lib dependency.
One thing to note is that in order to run on UWP, the app needs to reference the SQLite for UWP extension (which can be installed and referenced through visual studio). I don't think I can take a nuget dep on that so I need to see if it is possible to automate or not.
My WIP is on the feature/uwp branch of the repo.
Unless I am mistaken, using the UWP SQLite from .raw should be an option. In fact, I think this works in 0.9.3, but my next release will have relevant improvements.
CoreFX is using ICU4C for SortKey (et. al.) on non-Win32 platforms, and according to a discussion on the Lucene.NET dev list, a Microsoft engineer @conniey recommended https://github.com/sillsdev/icu-dotnet in a recent Lucene.NET contrib as the best C# binding.
Nothing final yet, but it looks like UWP support will be targeted for the 2.0 release (still a long way off) and I'm inclined to simply move the whole library to .NET Standard so I don't need to have separate projects for each platform.
Great news! I really hope for a single NuGet package which can be used on both UWP & Xamarin, and other .NET platforms.
If I understand the move to .NET Standard correctly, it means branch https://github.com/couchbase/couchbase-lite-net/tree/feature/uwp will be experimental, and that the structure of Couchbase Lite for .NET will significantly change in release 2.0. Is that correct?
Do you expect anything ready in 2016, or do we need to wait longer? By “anything” I also include beta releases, which would be nice.
Keep up the good work!
@runarj I expect I'll have something by the end of 2016, if only the core components, but the official bits won't come until 2.0.
As far as the structure changing in 2.0, this is our first major version number bump and so we finally get a chance to erase all the things we did that we wish we didn't and apply all the feedback and lessons learned during 1.x in a way that we couldn't while tied to a frozen public API. I can't be more specific than that at the moment but when I CAN say something specific I will probably make a forum post jumping for joy.
Oh, but as for the uwp branch it's there to let me play around and see what is needed for the switch. UWP and .NET standard 1.4 are pretty similar as far as I can tell so if I can get things stable against UWP then I can probably build for .NET standard 1.4, and possibly lower.
Cool new github feature -> https://github.com/couchbase/couchbase-lite-net/projects/1
Ok, updated info. It turned out that although UWP is compatible with .NET standard 1.4, it doesn't mean the API coverage is similar. There are a lot of missing API calls in 1.4 that makes 1.4 a bit impractical. I will fall back to getting a UWP build with the goal to target .NET Standard 2.0 (A huge jump in API coverage and certainly on par with UWP so we would basically almost get it for free).
Relevant post: https://blogs.msdn.microsoft.com/dotnet/2016/09/26/introducing-net-standard/
Ok I have another update. We recently wrapped up our conference in which we shared initial plans for our 2.0 release. The second half indicates that we will be moving a lot of the storage logic down to the native level and out of C# / Java. This puts me in an awkward place because even if I spent lots of time getting things working in 1.x, it will just be redone again later. Furthermore, I'm still working with the managers to get our official support matrix finalized. I am pushing a lot on this because it is important to me, and hopefully I will have the final answer next week! Depending on the answer I get, I can say for sure whether or not I can get UWP support into 1.x.
One thing I noticed is that ForestDB makes a lot of violations that prevent it from passing WACK. I can't figure out how to use WACK properly, and so I just tried compiling it with WINAPI_FAMILY=WINAPI_FAMILY_PC_APP and noting the missing methods. SQLite, as expected, compiles fine provided the correct macros. ForestDB is managed by a different team who works on our server product so I'll have to see if they have the bandwidth to do this (not terribly likely) otherwise it will probably be dropped for a long period for UWP.
I apologize that it seems like I am doing nothing but trust me I am constantly bringing this up.
Thanks for continuing to push for UWP support!
I've been getting a head start on the 2.0 API work and here is the story:
It has been decided that we will push out a 2.0 DP for Objective-C first, then Swift, and then .NET, etc. Once the DPs have started we aim to continue to release them every two weeks. These will not be complete, but piece by piece the 2.0 API will come together. I've pretty much started over for a lot of things in 2.0 and so I can focus on making sure everything is UWP compatible right from the beginning. In fact so far everything is building on .NET Standard 1.4, and I have a unit test suite that runs on .NET Core (which can be extended to .NET Framework 4.6.1), UWP, Xamarin iOS, and Xamarin Android (all based on the same DLL!).
At the moment I've worked through some basic CRUD, direct object model serialization / deserialization, initial LINQ querying via model, etc. The work I am doing is on a branch but the branch is not QUITE ready to be cloned to other machines yet. However, you can check it out if you feel like it on the feature/api-v2 branch. Actually, I'm lending my time to help get the Objective-C DP out faster and so I'm doing the .NET stuff in my free time until then but I'm trying to keep things more or less lined up. Here are a few things I have done to make the process go faster:
-
Every variant of our C++ core is building in CI via Jenkins (private for the time being) and desktop platforms are also being unit tested along the way (still trying to work out how to unit test mobile targets via Jenkins on a machine in a data center)
-
I made a script that will almost entirely auto generate type safe C# bindings to the C++ core via its C interface so whenever a change is made I just need to run the script and have a quick look over it. These bindings are also built and all unit tests are ported to C# and repeated to ensure that bindings are functioning correctly. This is also in CI for desktop platforms.
This will allow me to rapidly both develop the .NET API for Couchbase Lite 2.0, as well as react to any changes in the underlying core. This should give me the velocity I need for biweekly DP releasing and I hope everyone will get on board with trying it out.
On a more general note, the DP period for 2.0 will be an important time to submit your feedback for the API in general. This few month window will be a relaxed time where the API can change and evolve as needed. Once the GA is out, it is locked in and any further changes will need to be discussed and approved by the entire Lite team (they will still need this in the DP period, but the changes are much more likely to be done quickly).
Also be aware that 2.0 will not have .NET 3.5 support. The entire .NET world is converging onto .NET Standard and it doesn't make sense to spend so much effort (the entire TPL and System.Net.Http assemblies are backported and included) and give up so much (DLR and async) to hang onto an aging platform. This also means that Unity3D is on hold until they perform their runtime and profile upgrade (as of this writing the latter is in the research phase).
I guarantee you will see even more movement on this in February! It's getting close to DP quality.
The first developer build is available. UWP support is included, but because of an issue I am facing with the UWP test runner from Microsoft.
Note that .NET Native toolset will not work yet! You will need to disable it until future notice (of course I plan to have it compatible, but which method I fix the issue in will be dependent on whether or not we include the API that is causing the compilation failure).
I think this is enough to finally close this ticket. See here for more information about 2.0.
Hi, couchbase is available in uwp project? if yes, how can i add the sdk to uwp project.
I tried to add couchbase lite 1.4 to uwp project but I can't add it.
It's not available for 1.4. As mentioned above, UWP will be supported from version 2.0 (biweekly prerelease developer builds are available to track its progress)
@borrrden Hi,
I've looked at "Release Notes" page and it says that replication is available in the build 5, however the latest version that I can get via Nuget is 2.0.0-db004. Is the build 5 just unreleased yet?
It's ready technically but we're having problems with deployment so my hands are a bit tied with the schedule. Actually db6 is already done so hopefully we can play a bit of catch up. Another piece of bad news is that replication got delayed to db6 but as I mentioned it is done and I'm hoping that we get the deployment issues sorted out so that db5 and db6 can be released back to back pretty much.