dotnet/runtime

Developers can benefit from enhanced Date and Time types and Timezone support

tarekgh opened this issue · 24 comments

tarekgh commented

Date and Time features proposal

This issue is capturing the Date and Time candidate tasks for .NET 6.0 release. It lists every proposed feature and the reason for having it in the candidate list.

Features List

Features Details

Time zone ids

Currently when running on Windows, .NET read the time zone data from Windows registry which include the time zone Windows Ids e.g. Standard Pacific Time. Trying to create a TimeZoneInfo using IANA time zone Ids on Windows will fail and will get exception. The reason of that is .NET don't have any source of getting such Ids and map it to Windows Ids.

The same problem occur on Linux/OSX too. On Linux/OSX we read the time zone data for the installed time zone package which include only IANA time zone Ids e.g. America/Los_Angeles. Trying to create a TimeZoneInfo using Windows Ids will fail and throw exception.

It has been requested multiple times to support IANA Ids on Windows and support Windows Ids on Linux/OSX. #14929 show the demand on this feature and capture the discussions and the demand on it. We delayed the support of this feature because we didn't have a good source of the time zone data which can help converting the IANA Ids to Windows Ids and vice versa. Also, .NET avoided carrying the data because servicing such data will be big deal especially time zone data is very dynamic and constantly change.

We have been recommending using the TimeZoneConverter library as a workaround which helped a lot to unblock user's scenarios. The issue is some users cannot take dependency on none core packages and want to have this support natively in the .NET.

Fortunately, in .NET 5.0 we have started to use ICU library for the globalization support. ICU has time zone APIs which can help in converting Windows Ids to IANA Ids and vice versa. .NET can take advantage of ICU to implement such feature.

ICU provide the TimeZone class which provide the methods TimeZone::getWindowsID and TimeZone::getIDForWindowsID can be used to implement the feature. There are some catches though which we need to consider as part of the feature design:

  • The ICU time zone APIs is introduced in ICU version 52 while .NET supports ICU from versions 50. We are looking at all Linux platforms that will be supported by .NET 6.0 and find out if we can require ICU 52 as the minimum supported version. If we cannot raise the minimum ICU version to 52, then will not support the feature when using ICU version 50 and 51.
  • On Windows we support running using NLS and not using ICU. We either will not support the feature when switching to NLS mode. Or we consider loading ICU in NLS mode too for calling time zone APIs. In recent Windows 10 versions, ICU get loaded in many processes anyway so there will not be much overhead.
  • ICU doesn’t provide plain C APIs for time zone functionality. To use ICU for Id conversion will require to call C++ ICU APIs especially if we load ICU dynamically. There can be some challenges with that for binding to the C++ dynamically. This will need more investigation to handle that. We can investigate getting the needed information using ures_* APIs
  • .NET can run on Windows down-level versions which will not have ICU. That mean we’ll not support the Ids conversion feature on such Windows versions except if the apps decide to opt-in using the local ICU feature which the app will carry ICU as part of the app.
  • Although we'll allow creating TimeZoneInfo objects using either IANA or Windows Ids whereever we run on any platform, it may worth thinking if need to expose any needed APIs (e.g. TimeZoneInfo.WindowsId, TimeZoneInfo.IANAId).

Adjustment Rules Enhancement

When we ported the TimeZoneInfo code to Linux and exposed the APIs that handle the time zone adjustment rules, there was some limitations. The original design of the AdjustmentRule was based on how the time zone data format we read from Windows Registry while the data we read from Linux is very different. Although we tried to make it work with this limitation, it still not returning a prefect results in all cases. This issue is reported many times in issues like #44475, #20626, and #20624. The proposal here is do the following:

  • Currently we maintain the internal UTC office for each rule. Exposing this offset is going to help in general as seeing some users using reflection to get this value. Exposing this offset is going to help in Linux scenarios too.
  • Linux time zone data cannot be satisfied with the current AdjustmentRule APIs. We’ll need to expose more APIs to allow Linux rules scenarios.
  • Need to investigate the current reported issues and fix any wrong calculations. This is related to the previous bullet which we may need to expose a new API for that as needed.

Date & Time types

.NET expose DateTime and DateTimeOffset for handling date and time. It has been requested multiple time to expose Date only and Time only structs. There are many scenarios users need to handle Dates without caring about time at all. And scenarios need to handle time without caring about Date. Users today try to work around that by using DateTime or DateTimeOffset but users need to truncate the parts they don’t need and want to ensure the remaining values is not affected. This is error prone as DateTime and DateTimeOffset also carry some time zone related things and it is not meant handle absolute date and time.

There is detailed discussions about such requested in the reported issues #14744, #1740 and #14089. Initially we have suggested to expose these features outside the core as a NuGet package. This is implemented in corefxlab https://github.com/dotnet/corefxlab/tree/master/src/System.Time. Users still requesting this functionality to be in core and they wanted to avoid dependencies on third party libraries and make sense such functionality be in the core. The corefxlab implementation is a good start as it has the full exposure and working functionality. Need just some polishing to be moved to core. Some points we need to consider when moving it to core:

  • The type names are important. Time ok to use but it could be many users define their own Time type. When exposing Time that can conflict with user types. This is a generic issue that usually occur when using names that can conflict with user defined types (Span is a good example of that). It is important to have the right name in core though for discoverability purpose. Date is more problematic name as Visual Basic exposed aliased type with that name. This can be issue there and we need to evaluate if we can easily do something to help with that conflicts or just try to find another name.
  • Interop between these types and existing types DateTime and DateTimeOffset need to be carefully designed to avoid any confusion for users to be clear which type can be used in which scenario.
  • Because the current corefxlab implementation is done outside the core, it had to do some hacks to enable functionality like formatting and parsing. When moving the types to core we need to have clear design for such functionality and re-implement these parts.
  • Need to decide if we’ll support any serialization support.

Miscellaneous Fixes

We have some other issues and features related to dates and time. Investigating and resolving these issues would help the users in general.

danmoseley commented

@tarekgh this is great, I retitled it in user-focused terms as it's a user story

  • 👍2
shtratos commented

A shout from the Java side: please have a look at the work done in JSR-310 to introduce modern date/time abstractions in Java 8: https://www.threeten.org/

Java has had a long and painful history with time handling (e.g. java.util.Date is mutable).
IIRC these modern types have originated as Joda time library which became widely popular and eventually assimilated into the JDK.

Also, having a standard Clock time provider is genuinely nice for injecting it as a dependency in your classes to make them testable and make dependency on the time explicit. Sadly, I frequently see time-bombs going off in our .NET tests because someone decided that using System.DateTime.UtcNow directly is good enough.

tarekgh commented

@shtratos we are aware of Joda time and there is already the Noda time library implementing that. The Clock concept is a good idea we can consider. Do you have specific concern with the proposal we have here?

  • 👍1
Clockwork-Muse commented

@tarekgh - I think the call there is (once again) to implement a full date/time library in core, not to limit things to just the basic Date/Time types.

Outside of restating my own desires in that direction, getting new types into a new System.Time namespace (and having them not reference System.DateTime) would be my only concerns given the limitations that have been previously discussed.

tarekgh commented

@Clockwork-Muse When we reach the design phase, we'll discuss the options having the new types in its own library or be inside System.Private.Corlib. Also, the design should discuss the options having the new types in a new namespace. I think this all valid points to explore. I agree the new types shouldn't have any dependencies on DateTime but may consider have implicit/explicit conversion for interop usages. that would be part of the design discussion too. Thanks for the feedback.

mattjohnsonpint commented

Just to add some thoughts. (All of the below is IMHO, of course):

  • Java needed to deprecate their old types (java.util.Date, et. al.). It was necessary. However, doing so required cascading changes that are still rippling throughout the Java ecosystem. Replacing foundational types in APIs is no trivial matter.
  • .NET isn't in the same position as Java. The foundational time types (DateTime, DateTimeOffset, TimeSpan, and TimeZoneInfo) aren't broken, there are just a few gaps to close. Deprecating any of them would create unnecessary havoc and toil.
  • Keeping the existing types and also adding a whole java-time/noda-time like API would just add confusion. I would envision a ton of questions like "Why should I use System.Time.LocalDateTime instead of System.DateTime?", which would have no great answer.
  • A namespace like System.Time would be really confusing for types like Time. (System.Time.Time?) It's would also be confusing when the other foundational time types are sitting in System, but a few stragglers were not.
  • I feel strongly that the two new types (currently named Date and Time) should be in the exact same namespace and package as the existing date and time types. This will promote usage and allow them to get the same treatment throughout the framework and ecosystem. Putting them elsewhere demotes them to being unimportant. Don't underestimate the power of Intellisense discovery.
  • There will always be Noda Time and other libraries. The ecosystem has gotten much better about custom type mapping (ex, EF Core Value Converters), and many adaptation libraries have been written for Noda Time (ex, Postgres NodaTime Type Plugin). Adding a NodaTime-like API in a System.Time package won't change that. It would likely work into the ecosystem in the same way, offering little additional value over Noda Time. Developers that want comprehensive, robust, domain-correct date/time handling can and should use Noda Time. Developers that want ease of use and compatibility with decades of existing code can and should stick with the built-in type system.
  • At the end of they day, closing the gaps in the existing system will offer the most value while causing the least disruption. I believe it is a solid plan.
  • 👍3
mattjohnsonpint commented

I'll also add two more examples:

  • JavaScript is currently going through the same pain-removal process that Java did. The standard Date object will be eventually replaced by the types being proposed in Temporal. JavaScript has long felt the pain of problems with the Date object, which is very much like java.util.Date.

  • Python is not going through that process. Their DateTime type has been mostly fine. They recently added full IANA time support with zoneinfo in Python 3.9 via PEP-615. But otherwise, no major changes. The Python datetime module's available types include date, time, datetime, timedelta for representing data, along with tzinfo and timezone (and now zoneinfo) for working with time zones. Sound familiar? It's much like .NET's existing system, except they have date and time and we are missing them.

mattjohnsonpint commented

I do agree we could add a ISystemClock and SystemClock to this proposal. They have already demonstrated usefulness via Microsoft.AspNetCore.Authentication.SystemClock and Microsoft.Owin.Infrastructure.SystemClock. Having a common one would let these be consolidated in a future version and would prevent yet more from being created.

mattjohnsonpint commented

FYI, here's an example where having a Date type would be useful.
https://stackoverflow.com/q/65296637/634824

mattjohnsonpint commented

I would also like to add that a very common use case is to collect an end-user's time zone from their browser in a web application using the JavaScript: Intl.DateTimeFormat().resolvedOptions().timeZone defined in the ECMA-402 standard. That time zone ID is often sent to a server, such as with ASP.Net, where it needs to be interpreted.

Today, one has two choices to handle this server-side in ASP.Net running on Windows:

  • Using my TimeZoneConverter library to map to a similar Windows ID and loading the corresponding TimeZoneInfo object.
  • Using the Noda Time library which includes its own IANA derived time zone data.

Going forward, I would like to see a way to do this directly in ASP.Net without requiring libraries. In my opinion, an end-to-end example in ASP.Net of picking a time zone by starting with the browser's default time zone and using it in C# code would be very useful to end users.

  • 👍1
justingrant commented

I led the design work for the IANA-timezone-aware type called ZonedDateTime in JavaScript Temporal. One learning from this design is that types like .Net's DateTime and DateTimeOffset are inherently dangerous when used with (non-UTC) timezones because they don't adjust for DST changes or other political changes in time zone offsets. For example, it's easy to obtain a DateTimeOffset in a particular time zone, but once you manipulate it (e.g add one day, or change the month to January) you'll have no idea if the resulting offset is correct or not.

So one suggestion I'd have is to add a new type (or expand functionality of existing types) to include a type that will automatically adjust the offset according to the rules of the time zone as the underlying date/time changes. The data model of such a type would be [UTC value, IANA time zone].

Another learning is that naming types "Date" and "Time" can potentially be problematic because so many platforms use those names to mean types that include more than just a plain date or plain time.

In JavaScript we're going to call those types PlainDate and PlainTime. In a previous company, our (.NET) codebase called these types DateOnly and TimeOnly. The general idea is that it's good to label subset types so users know they're getting a constrained subset.

Feel free to reach out if you have more questions about Temporal. I worked on the .NET 1.0 API long ago when I worked at MSFT so I have some .NET context too, although not as much recent knowledge as @mj1856.

  • 👍2
mattjohnsonpint commented

Thanks for your feedback @justingrant!

I tend to agree with most of your points. Though keep in mind that the challenge we face with .NET is that unlike Java and JavaScript, we aren't (IMHO) in a great position to overhaul the entire date/time type system. The existing structures DateTime, DateTimeOffset, and TimeSpan are so engrained into the framework and surrounding ecosystem that they will most definitely continued to be used regardless of what future decisions are made. Thus, we are in a position of filling the deficiencies, not overhauling.

That's been the driving principle behind the need to add Date and Time types. I agree we could have a healthy debate about what to name those in .NET. Some thoughts on naming:

  • I think Date is a great name for a date-only type, but we need to consider that VB.Net has forever had a Date keyword carried over from VB6 that directly maps to DateTime. They are already synonymous and interchangeable in VB.Net code. Thus, if we call it Date, VB users would have some interesting bugs crop up for sure.
  • .NET users often discover features by typeahead/intellisense. So if choosing a different name, I would prefer DateOnly rather than PlainDate - because as one starts typing D, a, t, they will see DateOnly, DateTime, and DateTimeOffset in that order.
  • I prefer TimeOfDay over Time or TimeOnly, as explains its purpose better in contrast to the existing TimeSpan duration type. Again, I recommend against PlainTime for the same typeahead/intellisense reasons.

There is also a concern about how to handle existing behaviors. Are any breaking changes to be tolerated? I could see the following breaking changes being reasonable and useful in light of those two new types. (I'll use DateOnly and TimeOfDay to illustrate here):

  • DateTime.Date and DateTimeOffset.Date properties changed from DateTime to DateOnly
  • DateTime.TimeOfDay and DateTimeOffset.TimeOfDay properties changed from TimeSpan to TimeOfDay
  • Various mappings in ADO.Net, SQL Client, etc. (too numerous to iterate here). One example: DbType.Date mapping changed from DateTime to DateOnly
mattjohnsonpint commented

With regard to a ZonedDateTime type - there's no reason we couldn't add that, but I don't feel it's as warranted or urgent as the other times. I'd be interested in hearing others in the .NET community's perspective on this.

mattjohnsonpint commented

Also, In regard to the first two items on the features list at the top of this issue - I'd actually be in favor of a whole new time zone type, or overhauling the TimeZoneInfo type with many breaking changes. We got the design quite wrong with System.TimeZone and thus created System.TimeZoneInfo in attempt to fix it - but we still got it wrong by not supporting the full complexity of time zone data that is evident in the IANA TZ database. If we're going to try again to get it right, then we need to not be constrained by the design decisions made in TimeZoneInfo.

  • 👍1
tarekgh commented

Also, In regard to the first two items on the features list at the top of this issue - I'd actually be in favor of a whole new time zone type, or overhauling the TimeZoneInfo type with many breaking changes. We got the design quite wrong with System.TimeZone and thus created System.TimeZoneInfo in attempt to fix it - but we still got it wrong by not supporting the full complexity of time zone data that is evident in the IANA TZ database. If we're going to try again to get it right, then we need to not be constrained by the design decisions made in TimeZoneInfo.

What exactly the breaking changes you are suggesting in TimeZoneInfo?

Clockwork-Muse commented

.... another breaking change to consider (that has almost no chance of happening) - remove DateTimeKind.

Personally I feel some programmers may be lead astray by DateTimeOffset - not necessarily that they believe it will change the offset, but that they may forget that offsets do, in fact, change. DateTime has its own issues in this area. DateTimeZoned is where I prefer the direction that JSR-310 took over NodaTime, in that you can add relative offsets (days, months, etc), instead of just absolute offsets (minutes, hours), which is going to be the most "natural" thing to do in the apps that actually require such a type (calendar/scheduling).

That aside, all this is why I've been wanting to just go the Java route and add a whole new date/time API.

mattjohnsonpint commented

@Clockwork-Muse - I don't believe removing DateTimeKind would be possible at this point. At least, not without comprehensive overhaul. (But I agree with you on the pain it causes.)

mattjohnsonpint commented

@tarekgh - Let me get back to you on the TimeZoneInfo changes. I would like to put a rough prototype together and see what edge cases are teased out.

  • 👍1
justingrant commented

Note that the current TimeSpan type isn't good for date arithmetic because it doesn't distinguish between day deltas (which must be adjusted for DST) and time deltas (which generally don't care about DST), and because it lacks months and years. So if there's a DateOnly type added, then would there also be a DateSpan type needed too, presumably with Years, Months, Days properties?

With regard to a ZonedDateTime type - there's no reason we couldn't add that, but I don't feel it's as warranted or urgent as the other times. I'd be interested in hearing others in the .NET community's perspective on this.

The main reason to add a ZonedDateTime type (or DateTimeZoned or DateTimeWithZone or something else that's friendlier for IntelliSense) is to support timezone-safe arithmetic of days and larger units (e.g. reschedule this appt 7 days later at the same local time) and other manipulations (e.g. set the day to 1 of an existing date/time value while keeping local time constant). These operations are inherently DST-unsafe in today's .NET types. Adding and removing days is a fairly common activity--although not as common as dealing with plain dates or plain times, I agree! I'd be happy to help advise the design work if there's interest.

  • if choosing a different name, I would prefer DateOnly rather than PlainDate - because as one starts typing D, a, t, they will see DateOnly, DateTime, and DateTimeOffset in that order.

Agree! I actually tried to recommend those names for that reason in Temporal, but that didn't fly there. ;-)

Although does Visual Studio IntelliSense still limit itself to "starts with" matching? Or does it now act like VS Code where "match inside" also works? If the latter then the placement of the "Date" / "Time" part is less critical than the past. That said, it's helpful for other places where alphabetic sorting is used, e.g. documentation, so I'd still recommend XxxxOnly vs. PrefixXxxx.

mattjohnsonpint commented

Thinking a bit more about the suggestion of including an IClock implementation. Since we are talking about changing .NET itself, I think it would be more valuable if we provided a way to intercept the clock, affecting the existing Now, UtcNow and Today properties. That would allow time-specific testing of existing code.

mattjohnsonpint commented

FYI, I'm working on a PR for the clock virtualization. This has also been discussed in #36617 (and previously in #29139). I'll add to that thread, but IMHO it should also be part of this top-level user story.

tarekgh commented

FYI, I'm working on a PR for the clock virtualization. This has also been discussed in #36617 (and previously in #29139). I'll add to that thread, but IMHO it should also be part of this top-level user story.

Please keep discussing the issue on #36617 and not here. the reason is #36617 is talking in general about mockability and also protentional breaking changes for removing the ISystemClock types from all other extension libraries. This is not about just adding a simple interface. The design review video on the issue is discussing all of that.

  • 👍1
tarekgh commented

All items in this issue already taken care of.

  • 🎉1
  • 🚀1