Exiv2/exiv2

Future of Exiv2

Opened this issue · 40 comments

alexvanderberkel commented

The development of v0.27.4 is on track. @clanmills has volunteered to help one last time by shipping v0.27.4 before he retires. I've set up a Zoom meeting to discuss the following two questions:

  1. Should Exiv2 join OIN (see #1447) .
  2. The future of Exiv2 after v0.27.4

Zoom meeting details:
Topic: Future of Exiv2
Time: Feb 27, 2021 06:00 PM Amsterdam, Berlin, Rome, Stockholm, Vienna
Join Zoom Meeting
https://us05web.zoom.us/j/87388802811?pwd=RStyUUdRak1aNmRENnRKZ3k1Z3U3Zz09
Meeting ID: 873 8880 2811
Passcode: S0qPG1

This is the list of contributors / people who forked and/or worked on Exiv2 over the years on GitHub.
@1480c1 @1div0 @354380o726502 @5l1v3r1 @a17r @adammartak @ahuggel @alexvanderberkel @AmaniBHH
@AndreasMartin72 @andreilabato @Anmol92verma @anmolvermamm @arkiruthis @asp @astippich @Aurora-Community
@awilfox @badola @bellaperez @bentglasstube @BioDataAnalysis @bkuhls @blockspacer @boardman @BohuaCao
@Cadcorp @caomw @carlosbasecodeit @Cethric @cgilles @chan0415 @clanmills @clayne @clthomas
@codingdave @connectthefuture @criadoperez @cryptomilk @csparker247 @CTF-s @cygwin-lem
@czgnp @D4N @danielkaneider @danileo @dbeichl @Deneb99 @derselbst @devil1437

clanmills commented

@boardhead: I hope you will be able to join us.

clanmills commented

Slides for the meeting.
Future of Exiv2.pdf

  • 👍2
alexvanderberkel commented

Meeting notes of 2021-02-27

Participants:

  • Robin Mills
  • Volker Griels-Grabsch
  • Luis Diaz
  • Christoph Hasse
  • Peter Kovar
  • Alexander Esseling
  • Thomas Beutlich

Join OIN (open invention network)?

  • Joining OIN is being suspended in the short-term
  • Decision to evaluate joining the KDE project
  • Benefits: KDE is already part of the open invention network
  • KDE project might bring new contributor and/or maintainer to the project

Exiv2 after v0.27.4?

  • No new maintainer has volunteered.
  • We agreed to simplify the branch structure and "modernise" the code (C++20 support)
  • Current master branch v0.28 will be labeled as old-master and frozen.
  • 0.27-maintenance will be frozen.
  • We'll branch 0.27-maintenance to 'main' and make it the default branch.
  • Luis will port Unique pointer code from old-master to main.
  • Robin will release v0.28 from 'main' in the Fall.

There is no active plant for development of Exiv2 beyond 2021.
We hope to have a week-end retreat in Germany in the Fall to celebrate our achievements.

  • 👍1
  • ❤️3
vog commented

It was nice to see you again after such a long time!

Just a small nitpick: I'm still in favor of naming the new main branch 'master' rather than 'main', because to any outsider, 'master' is the most obvious name for "the git branch to use" (for final testing, pull requests, etc.).

  • 👍1
phako commented

That's the new default in git, github and many other installations...

  • 👍1
clanmills commented

Thanks, Volker @vog. Maybe I'll do that. Yesterday I changed the name of 'master' to 'old-master' to discourage anybody from submitting further PRs. I also set '0.27-maintenance' as the default branch.

I should close all the open PRs on 'master' and explain what's going to happen.

By the time we ship 0.27.4 on 2021-05-22, it'll probably be safe and reasonable to branch an new 'master' from 0.27-maintenance. I want to retain the 0.27-maintenance branch in case we need a 0.27.5 security update in future. We haven't had a security issue raised on the code for almost 18 months.

Your comments are appreciated. Thank you for contributing.

@phako. Are you saying 'main' is the new default? I thought somebody said that during the meeting on Saturday.

phako commented

yes, at least for github, see https://github.com/github/renaming. I thought git itself had switched as well but I cannot find any reference for it in quick search, only heated discussions why this is "PC gone mad"

  • 👍1
imsodin commented

It's not (yet?) a default in git, but the default branch has recently(ish) been made configurable: https://github.blog/2020-07-27-highlights-from-git-2-28/#introducing-init-defaultbranch
If you are already restructuring branches, I do suggest using main. I do expect it to be the de-facto norm soon-ish - at least my muscle memory already does m-a-i-TAB (weirdly I didn't drop the TAB and go with m-a-i-n). And the only downside I see is having to switch existing setups from master to main, that's why I think an unrelated branch-restructuration is a good moment to make the switch.

vog commented

@phako @imsodin Thanks for clarifying the issue with master! I wasn't aware of that latest developments and forming consensus about switching from master to main.

I fully agree that, given those circumstances, main is the better choice.

By the way, I never liked the wording master that much. Even if it didn't have any connotation to slavery, it was simply a bad naming. You would never guess this default branch name unless you are already familiar with Git. Nevertheless, I followed this naming, because following a suboptimal convention is better than following no conventions at all. It is still a pity that the community bends towards "main" rather than "default" - the latter being the default branch name in Mercurial, and an equally good choice, and then would be automatically compatible with the second-most popular DVCS. But then, perhaps Mercurial will be moving to main in the future as well.

  • 👍1
clanmills commented

Thanks you everybody for your comments. I've had a message from @alexvanderberkel saying 'main' is the new 'master'!

Let's stick to the plan. On 2021-05-22, I'll branch 0.27-maintenance -> main and make 'main' the default branch. I've already renamed 'master' -> 'old-master' to discourage its use.

  • 👍4
boardhead commented

@boardhead: I hope you will be able to join us.

Sorry, I missed this. I haven't been keeping up with all of the Exiv2 posts recently.

1div0 commented

Version 1.0 with JPEG XL and WebP 2, right?

imsodin commented

Just a thought about maybe switching to 1.0: I agree that considering the maturity it is an appropriate step, however it might rise the (mental) bar for anyone considering to step up to help maintaining exiv2. On the flip side you don't win much if anything from doing it.

clanmills commented

@1div0. WebP has been supported since 2016. You're right. I should add JXL. We only need to add a few lines of code to support JXL/bmff. JXL/codestream is still being negotiated by the standards committee and will add ICC support. Exiv2 doesn't provide ICC support for every image format.

@imsodin
When Alison and I stayed with Andreas and his family in 2016, Andreas and I had our one-and-only in depth discussion about Exiv2. About v1.00, he said "until we have the Unified Metadata Container, we can't declare 1.0. BTW there's an implementation in the unstable branch in SVN.". Last week, I finally had the time to pull that down and investigate it. #1505

It was written about 2012, so the merge will not be easy. However, it's a huge win to add this. It enables the code base to support other metadata standards such as ICC/tags, PNG/text and so on. Now is the time to get this done. Additionally, we now have BMFF support. BigTiff is supported in tvisitor.cpp and can be ported into Exiv2. So, I really believe that we are very close to having Exiv2 v1.00 with a stable API which will endure.

The other reason is to claim it's done instead of leaving Exiv2 with an undefined future. What happens to Exiv2 or OSS/metadata processing after v1.00? We don't know. However it's nice to claim that I shepherded the code from 0.22 (about 2012) to successful completion.

At the weekend, @hassec asked me: "Who will take over fixing everything in Exiv2 after you, Robin?".
By declaring v1.00, I can answer Nobody. Exiv2 is complete.

clanmills commented

Ah WebP2. I haven't thought about that at all.

I have no idea why Google, Apple and all those people keep re-inventing the wheel. In the process of writing the book, I discovered that the current versions of BMP (the Native Windows bitmap) now supports ICC profiles. I believe BMP files from Windows/1.0 will be respected by current windows. I believe Windows/1.0 will open BMP files created on the current windows. I believe Adobe Acrobat which will have its 30th birthday in a couple of years is just as durable.

Canon have used 3 totally incompatible raw image formats in about 15 years. Why?

clanmills commented

Exiv2 v1.00
Updated: 2021-04-1

We are now within sight of Exiv2 v1.00 and could achieve that by the end of 2021. In many ways this is a more desirable goal than KDE. Exiv2 declaring v1.00 says Exiv2 is complete. If KDE wish to take over the project in 2022, that will be fine.

I think it's better to use our limited resources to reach Exiv2 v1.00 on GitHub and not put effort into working with KDE.

Topic Status Comment Owner
To reach v1.0
API Good Unified Metadata Container Milos
C++ Code Good C++ modernisation Luis
Documentation Good Improve README-TESTS.md Robin
Image Handlers Good Add BigTiff
Join OIN and/or KDE Good Alex
Localisation Fair Slovak and Czech
Brazilian Portuguese
Crowdin
German		 Peter
Leonardo
Leonardo/Robin
Alex
Project management Good Alex
Release Engineering Good Robin
Samples Good exiv2 -ps enhancements Milos
Security Good Create Security Advisories Kev
Source Code Good Reformat with clang-format Luis
Team Weekend Good September 2021 in Bavaria Alex
Test Framework Good Improve bash_tests Peter
Web Site Good Move to GitHub Robin
XMPsdk Good Consider removing from code base Robin
Deserves Attention
Bugs Good Monitor and respond Robin
Users Good Customer Satisfaction Follow Up Alex
Post v1.0
JXL/codestream Waiting for Standard
WebP2 Waiting for Standard
Deep Test Coverage raw.pixls.us and testimages.pixls.us
Platforms Good Support for Mobile
Preview/Thumbnails Fair The code is weak
Servers Good KDE/GitLab Transition
In Good Order
Back-porting Good
Build Good
Build Server Good
Contributors Good
Enhancements Good
Licensing Good
Scheduling Good
Tools Good

We hope to have a Zoom meeting on the weekend of 24 April or 1 May to discuss the plan and assignments.
It looks the moment as though there will be four groups of 3 contributors in each group:

Group Members Priority Tasks
code Luis, Rosen, Pydera C++11, jp2image.cpp, security
metadata Milos, Christoph, Tom Unified Metadata Container, Canon AFInfo, JPEG>64k
t_l Peter, Leonardo and AlexS. Test and localisation
pm AlexE, Kev, Yuahm Robin Project Management, Security, User support, Release

The schedule looks like:

Topic Date Purpose
Kickoff 2021-05-01 Zoom meeting to discuss the task list
Priority Complete 2021-06-30 Priority features done
Bavaria Week-end 2021-09-18 Fun meet-up. Hiking, talking, boozing
Code Complete 2021-09-30 Finish all shipping features
RC1 2021-10-21 First cut
Code Freeze/RC2 2021-11-15 No more changes
GM 2021-12-15 Shipped

The detailed task/priority list is still below. I hope each group will complete their Priority tasks by 2021-06-30. If we have to drop lower priority tasks to make the schedule, we will. If we hit Priority Complete in late June, success is certain.

The commitment of time per contributor should be of the order of 100 hours. I suspect the metadata group tasks will require more than 100 hours per contributor. I will work on/with any group as requested in addition to dealing with users, release engineering and project management.

Please be aware that the task list is based on open issues on 2021-04-13. New issues will be reported during the life of the project.

  Meaning Priority Description
Required 1 Deliver by Priority Complete
Required 2 Deliver by Code Complete
Optional 3 Will drop if necessary
Objective 4 Not a task
Cannot 5 Not scheduled/no resource
Issue Category Priority Title
#1557 code 1 C++ Modernisation Luis
#1538 code 1 32/64 bit support in Exiv2 Pydera
#1235 code 1 Add ARM job to Travis-CI
#1228 code 1 UBSan violations
#898 code 1 Add Safe::cast
#1355 code 1 Simplify cmake/compilerFlags.cmake
#1062 code 1 Investigate JPG2000 check_boxes
#950 code 1 CVE infinite loop in Exiv2::Jp2Image::encodeJp2Header
#1537 code 2 Fix special case of box size == 1 in jp2image.cpp
#1525 code 2 Replace jp2image with bmffimage
#1501 code 2 make uninstall leaves artefacts in standard locations
#1542 code 2 Function name free() in class EXIV2API DataBuf
#1406 code 2 fcf-protection only exists on x86
#1405 code 2 readMetadata - improve warning & error handling
#1404 code 2 kerAliasesNotSupported: Please sendXMP packet to ...
#1348 code 2 Move utilities out of the Image class
#1217 code 2 CMake does not check for duplicates
#917 code 2 ABI compatibility checker
#1320 code 2 clang-tidy: sort includes LLVM style
#1022 code 2 Add RAII Exiv2::Library to init/free resources
#956 code 2 Add compiler flags to test software security
#1514 code 3 Replace every EXV_ symbol with EXIV2_
#1417 code 3 Improve console output under Windows for UTF-8
#1060 code 3 Replace deprecated SHGetFolderPathA
#565 code 3 Test for alpha bit detection in VP8L chunks
#1552 code 4 Refactoring Tasks for v1.00
#1279 code 5 Unicode Basic Multilingual Plane
#1505 metadata 1 Unified Metadata Container Milos
#1515 metadata 1 Investigate removing xmpsdk from code base
#1343 metadata 1 Comment and EXIF data cannot be read from PNG files
#1549 metadata 1 Implement Adobe Exif >64k in JPEG
#1026 metadata 1 Rewrite sanity checks for CiffComponent size & offset
#459 metadata 1 add namespace to XMP toolkit - exiv2 and libexempi
#1543 metadata 1 Canon AFInfo Rotation makerNote
#1541 metadata 2 Add pretty printer for GPSImgDirection & GPSTrack
#1524 metadata 2 Error: Directory Samsung2 with 21313 entries considered invalid; not read.
#1419 metadata 2 Support for tag PrintImageMatching 0xc4a5
#1416 metadata 2 Iptc.Application2.DateCreated/TimeCreated formats are different for read/write
#1397 metadata 2 Support for DNG 1.5 Milos
#1334 metadata 2 report Roll Angle Tom
#279 metadata 2 Whitebalance values
#1085 metadata 2 Fix Irix lens names in pentaxmn_int.cpp
#280 metadata 2 Reading temperature values form RAW files
#1081 metadata 3 Changing Tag shortens MakerNote on Olympus
#1401 metadata 3 XmpParser::encode() swallowing exceptions
#1547 metadata 3 fix_1416_iptc_DateCreated
#1556 metadata 5 BigTiff Support
#1506 metadata 5 JXL/codestream support
#710 metadata 5 Extract PreviewImage from JPEG
#904 metadata 5 Canon 1Ds thumbnail colour corrupted
metadata 5 WebP2 Support
#1516 t_l 1 Build is using incorrect include path Fixed
#1510 t_l 1 Crowdin localisation Leonardo
#549 t_l 2 Localisation for v1.00
#1421 t_l 2 Fix regression in Canon lens detection Alex S
#1428 t_l 2 Test all Canon lenses Alex S
#930 t_l 2 Add test for issue 755
#896 t_l 2 Enable git LFS testing
#1550 t_l 4 Development of the python test framework Peter
pm 1 Security Kev
#1447 pm 1 Joining OIN Alex
pm 2 Week-end in Bavaria Alex
#1494 pm 2 Joining KDE community Alex
pm 4 User support Robin
#1521 pm 2 Deprecate exiv2.org and binary builds for v1.00 Robin
#1462 pm 3 Exiv2 Project Status Robin
#1018 pm 4 Exiv2 RoadMap Robin
  • 👍2
1div0 commented

I can and will finish localisation to Slovak & Czech, my primary and secondary languages.

1div0 commented

Also, please correct JXP/codestream to JXL as an abbreviation of JPEG XL. Thanks.

clanmills commented

Thanks for your input, Peter (@1div0). This is forming a nice project. Nobody has to do a lot of work and nobody is stuck with this forever and ever.

Apparently KDE have localisation folks to whom we can send the files. I've never discovered how to access that channel.

I'm thinking about bullding RC2 next week to include the JXL/bmff code. The code will then be frozen and I will open branch 'main'. I'm meeting with Luis this afternoon to discuss auto_ptr/C++20.

1div0 commented

I do have a very good experience with https://crowdin.com for ML based localisation. Probably worth to explore.

lbschenkel commented

If there's interest, I could volunteer to localize to Brazilian Portuguese (my native language). I also have previous experience with Crowdin.

clanmills commented

@lbschenkel Thanks. I've updated the task table. You've got that task.

Localisation is documented in README.md. It's driven by gettext technology. I believe this is a commonly used approach.

Some companies provide free services and products to open-source projects. Perhaps you could explore that with Crowdin. As Exiv2 has no revenue, so we are unable to pay for a license.

lbschenkel commented

I'm also familiar with gettext, so that is not a problem to me. However, Crowdin can be easier for non-developers to contribute translations since they can do it directly via the website compared to editing PO files, which is a bit awkward.

Crowdin is free-of-charge for open source projects, and I could apply on behalf of the project, but only the project lead can do so: https://crowdin.com/page/open-source-project-setup-request

clanmills commented

I've filled in the application form. I think they expect you to have already created something on their platform. I don't want to start learning about Crowdin. Is it a big job to tool up to use this platform? Can you set up the necessary resources?

lbschenkel commented

It's not hard. I can volunteer to set the project up as an experiment if you want, and then you can judge if it's something that you think that could be useful to the project.

But I think if I do that, it'll be tied to my account at first. I should be able to transfer it to somebody else's account.

  • 👍1
clanmills commented

I have opened a new issue "Investigate Localisation using Crowdin" for further discussion. #1510

Please be aware this topic "The Future of Exiv2" is circulated to about 200 people who have contributed to Exiv2 since 2004.

  • 👍1
alexvanderberkel commented

Hi guys,

I warmly welcome you to the kick-off meeting for our work on Exiv2. Please find attached the prepared meeting slides and meeting details.

The meeting is scheduled for:
2021-04-24 10:00 UTC / 11:00 BST / 12:00 CEST / 23:00 NZDT

Zoom details
https://us02web.zoom.us/j/9129667335?pwd=cjFvWHNnMlpwOExYSE44dHFrdWtyZz09
Meeting ID: 912 966 7335
Passcode: 2ngkdc

The full backlog for the project can be found on GitHub:
#1466

Exiv2 v1.0.pdf

alexvanderberkel commented

I just realized that we cannot use the zoom details of Robin and he is not attending. Please use this link

Video call link: https://meet.google.com/ceo-uckx-uof

postscript-dev commented

For Exiv2 1.0, the 'code complete' is scheduled for 2021-09-30. Is this still going ahead?

clanmills commented

When I put that in the schedule, I was intending to drive the project and "make it happen". Code-complete is an important milestone to achieve the goal of releasing v1.00 by the end of 2021.

I've now retired and leaving Alex to manage the project. He's likely to call a zoom meeting to ask everybody about goals to which they can commit and deliver.

My impression is that maintenance, security fixes and user support for Exiv2 is going well. Development of v1.00 is sluggish. However Luis has done the C++11 modernisation and that's important enough to release "main" even although other priority one features (such as Unified Metadata Container and removing xmpsdk from the code base) have not been developed.

kevinbackhouse commented

Do all major API changes need to be finished before 1.0 is released? I would quite like to try switching the standard size type from long to size_t, but that's going to a non-trivial project. I also have some smaller changes that I would like to make, like some ongoing changes to DataBuf and adding readOrThrow to BasicIo.

clanmills commented

It is very desirable to make big API changes before declaring v1.00. I know Alex has been intending to call a team meeting and this kind of thing should be discussed. Developing every priority one features probably cannot be achieved for several years.

Perhaps a good way forward is to aim to release main as v0.9 with C++11 support next year (say June 2022?). The intention with the v0.9 family is to avoid API changes on a series of half-yearly releases (v0.9.1, v0.9.2 and so on). Eventually reaching v1.00 in a few years time.

alexvanderberkel commented

I think we discuss when we think we are able to release 1.0 and who's committed to work on certain topics. Putting new things into the backlog I would say doesn't put us in a bad situation as long as we have people to work on it.

alexvanderberkel commented

I have set up a call in the past for 1.0 but there was only one participant. That is why I have not scheduled another call so far.

clanmills commented

I think it's kind of difficult to commit to a schedule without full time engineers. I was able to deliver releases to a schedule because I was working full-time on Exiv2. Everybody has full-time and demanding jobs.

With the Paris Climate Treaty, Ms. Figueres asked the countries "what can you do?" and drew the canvas from the bids. And I think that's how you'll have to approach Exiv2. Ask "What can you do by March 2021?" and that's the spec for v0.9.

Always allow 3 months from Code-complete to GM to allow for several RCs. If code is to ship in June, it has to be written, documented and tested by the end of March.

clanmills commented

Release notes for v0.27.5: #1018 (comment)

  • 👍3