README

Members of the OASIS Universal Business Language (UBL) TC create and manage technical content in this TC GitHub repository ( https://github.com/oasis-tcs/ubl ) as part of the TC's chartered work (i.e., the program of work and deliverables described in its charter).

OASIS TC GitHub repositories, as described in GitHub Repositories for OASIS TC Members' Chartered Work, are governed by the OASIS TC Process, IPR Policy, and other policies, similar to TC Wikis, TC JIRA issues tracking instances, TC SVN/Subversion repositories, etc. While they make use of public GitHub repositories, these TC GitHub repositories are distinct from OASIS TC Open Repositories, which are used for development of open source licensed content.

Description

The purpose of this repository is to support committee-member collaborative activity in developing releases of the UBL committee's flagship work product, the UBL OASIS Standard. The initial Maintainer is G. Ken Holman (GitHub: gkholman) of Crane Softwrights Ltd.. Until further notice, questions regarding this repository may be directed to Ken.

The Universal Business Language (UBL) is an open library of standard electronic XML business documents for procurement and transportation such as purchase orders, invoices, transport logistics and waybills.

UBL defines "a generic XML interchange format for business documents that can be restricted or extended to meet the requirements of particular industries. Specifically, UBL provides the following:

A suite of structured business objects and their associated semantics expressed as reusable data components and common business documents
A library of XML schemas for reusable data components such as "Address", "Item", and "Payment" — the common data elements of everyday business documents
A set of XML schemas for common business documents such as "Order", "Despatch Advice", and "Invoice" that are constructed from the UBL library components and can be used in generic procurement and transportation contexts..."

Published results

The published results are not stored in the GitHub repository, per se.

See the GitHub Actions tab for the results of publishing processes and distribution packaging, being careful to note the branch indicated for the desired commit message. As branches are merged the commit message is inherited in the merged branch.

You must be signed in to GitHub in order to download the results from the "Artifacts" section on the action's build result page. If not, you will get a "404 Not Found" error for the build result page.

Each downloaded result is doubly-zipped: the outer zip for GitHub extraction purposes and the inner zips for posting to Kavi and distribution. Moreover, the actual content is copied in the distribution in its own third ZIP file for posting. The archive-only/ subdirectory is not meant for inclusion in the final home in the OASIS document server, only for archive purposes in Kavi.

At particular milestones (typically when the main branch is updated) these files are archived in the committee Kavi documentation (members only) (Public access).

The results in the Actions tab eventually are deleted automatically by GitHub after 90 days, but if you have no need for a particular build result, you can delete it to save space.

Branches, roles, and protocol for contributions

Three branches are restricted (by policy, not all by software so please be careful):

main - this is content that has been reviewed by committee members and considered acceptable to be backed up on Kavi and, if necessary, distributed for its intended purpose (which may be for testing or for production use, not necessarily for final use)
review - this is content from the editor that has not been reviewed by committee members yet, and so is not considered agreed-upon for its intended purpose, but the editor has incorporated input from other sources into a package for review; when there is consensus about the content of the review branch, it is snapshot in the main branch
ubl-version-stage - (e.g. ubl-2.3-csd05) this is content the editor is working on while the review branch is being reviewed by committee members; this is not to be considered final and may be in a state of disrepair while it is being worked on; when the editor is ready for the committee to review it, it will be checked into the review branch

A main copy is not necessarily the final copy, but simply a copy of a review copy whose review has been completed.

Two roles are identified.

Editors are responsible for incorporating into review copies (for committee consideration) and main copies (already accepted by the committee) the suggestions made by the maintainers.
Maintainers create and maintain their own branches and are asked not to check in any changes to the main, review, or ubl-version-stage branches reserved for editors.

Maintainers can create and delete any number of their own branches as they see fit. Maintainers are reminded to pull the active review changes frequently so as not to diverge far from the work already progressed by the editor. Contributions are requested to be submitted by pull requests against the current review branch to be incorporated by the editor.

Maintainers can use any XML editing tool to make their changes to the specification document. See "Preview results" below regarding how they can preview their XML edits locally.

Other files and directories can change however needed by the maintainer.

Become a maintainer by sending your GitHub account name and associated email address to the committee chair(s) requesting that OASIS TC Admin grant you the required privileges in the repository.

Detailed steps

The act of checking a committed branch to GitHub automatically triggers the generation of all artefacts and the publishing of the authored XML into OASIS layout PDF, OASIS layout HTML, and ISO Directives Part 2 layout PDF. See "Published results" below. The results of the build are found in the Actions tab on GitHub. Note that the results are automatically deleted from GitHub after 90 days.

This is the relationship between the main, review, and all other branches:

The maintainer creates their own personal other branch of their own naming, not overlapping with the name used by any other maintainer or editor, and always pulls from the review branch into it. Pulling from the review branch must be done after every time the editor updates the review branch.
The maintainer makes the changes they wish to their local copy of the other branch. A local preview facility allows the maintainer to preview in a web browser their edits to the specification fully formatted as an OASIS specification. When completed they commit their changes and push their changes to GitHub in their private branch.
Every push to GitHub triggers a GitHub Action that forwards a copy of the XML to the https://www.RealtaOnline.com API entry point specific to the desired outputs.
Réalta prepares the HTML and PDF outputs for the OASIS layout and the PDF output for the ISO Directives Part 2 layout.
Réalta returns to GitHub the published results in a ZIP file. The Action’s script unzips the results and packages them in a doubly-zipped ZIP file. The outer ZIP is used to wrap the Action’s artifacts results. The inner two ZIP files wrap for posting to the OASIS Kavi server, respectively, the work product inputs and intermediate files for archive purposes, and the set of work product files to be posted by TC Administration to the OASIS Docs server. These Action results are transient and not placed into the GitHub code repository. Eventually GitHub deletes old Action results. The maintainer downloads the GitHub Action’s artifacts ZIP file for their review and, if desired, local backup. If they wish to make changes, they return to step 2 and repeat the process.
When the maintainer is satisfied with their work to be reviewed by the editor and other team members, they send a pull request to the editor describing their changes that they have committed to their other branch.
The editor reviews the pull request and, if satisfied with the contribution, they pull the server’s copy of the other branch into their local environment and merge it into their local UBL-{version}-{stage} branch or directly into their local review branch. Typically the editor will continue to make any changes they wish in their local UBL-{version}-{stage} branch, pushing their intermediate work to review their progress.
When the editor has incorporated changes from all of the contributing maintainers and is prepared to make the review document available, they merge their local UBL-{version}-{stage} branch into their local review branch and push their review branch to GitHub. This push automatically triggers the GitHub Action running the complete publishing process that returns the work product ready for the editor to download from GitHub to review. The GitHub Action results are posted to Kavi for the committee to download the published results and review. If maintainers wish to make changes they return to step 1 (not step 2) and repeat the process.
When the editor has accommodated all of the feedback from committee members regarding the review and wishes to archive a snapshot for public use, they merge their local review branch into their local main branch.
The editor pushes their local main branch to GitHub. This push automatically triggers the GitHub Action running the complete publishing process that returns the work product ready for the editor to download from GitHub. The artifacts ZIP is manually unpacked revealing the two inner ZIPs that are then uploaded to the OASIS Kavi server for posterity and to fulfill the obligation to the public that intermediate work products be easily available. Following the OASIS TC Process the committee can qualify a snapshot on the OASIS Kavi server to be uploaded by OASIS TC Administration to the OASIS Docs server.

If you are changing the Google spreadsheet but not any of your repository files, you cannot do a git push until you create a temporary file in your directory and then push the temporary file to the repository. The next time around, you can delete the temporary file in your directory and then push the deletion to the repository. Either of these steps will trigger GitHub actions that will reach out to the Google spreadsheet and run the process.

Two examples of the use of temporary files in a shell script are:

touch trigger.txt - creates the temporary file
rm trigger.txt - deletes the temporary file

Two examples of the use of temporary files in a DOS script are:

echo >trigger.txt - creates the temporary file
del trigger.txt - deletes the temporary file

To determine if the files are ready for sending to the project editors, look in the results in the archive-only directory for these files summarizing any problems with the submission:

build.exitcode.{timestamp}.txt - exit code from the execution of the Ant build script
build.console.{timestamp}.txt - console log of the execution of the Ant build script
check-ubl-version-stage-ubl-version-pstage.html - e.g. check-ubl-2.3-csd05-ubl-2.3-cs01.html - report of differences between the current version/stage and the previous version/stage
check-ubl-version-stage-ubl-peversion.html - e.g. check-ubl-2.3-csd05-ubl-2.2 - report of differences between the current version/stage and the previous version.

There should be no ".txt" files in the returned base directory. Various ".txt" files are possible to report problems with the build. The problems should be explained in the file itself or correlated in the build.console.{timestamp}.txt file:

ATTENTION-new-entities.txt indicates that in the archive-only/ subdirectory has a new-entities/ directory with new entity files that were created by your changes to the document, schema summary, and party XML files; if there are any new entity files, you must replace your local copies with those new entities in order for the validation in your XML editor to match the validation results on the server; if the new-entities directory does not exist, then there is no need to update your local entity files.
INTEGRITY-PROBLEMS.txt - there are files referenced in the hub document that do not exist, or there are files that exist and are not referenced in the hub document
LIST-OF-PROBLEM-CODE-LISTS.txt - genericode files in the cl/ directory that are not genericode-schema valid
MISMATCHED-TEST-SAMPLES-SH-WARNING.txt - the number of invocations in the val/testsamples.sh script does not match the number of examples
MISMATCHED-TEST-SAMPLES-BAT-WARNING.txt - the number of invocations in the val/testsamples.bat script does not match the number of examples
NDR-SPELL-CHECK-WARNING.txt - see the unexpetedWords.txt for a list of words that are not found in English or the UBL dictionary
UNEXPECTED-SAMPLES-NS-PI-DETAILS.txt - open the file to see a list of unexpected namespace declarations found in sample files
UNEXPECTED-TEST-RESULT-WARNING.txt - the val/test.sh script did not end cleanly
UNEXPECTED-TEST-SAMPLES-RESULT-WARNING.txt - at least one invocation in the val/testsamples.sh did not validate

When editors create their final edits, the results from the git push should be ready to be published without any modifications. If there are errant or unexpected files in the directory, the build process needs to address the discrepancies.

Creating a new release

Configuring the artefacts

The build.sh invocation points to the particular set of parameters to use to create the artefacts. The editors maintain this file for the technical committee version of the UBL package. Alternative configurations for subcommittees can create branches of this repository, but once created, they are not to be merged into the main, results, or ubl-version-stage branches. As is true for other maintainers, changes are submitted as a pull request to review. The subcommittee build.sh is not used by the technical committee.

Am example for the Transportation Subcommittee might be along the lines of UBL 2.3 CSD03 that follows stage UBL 2.3 csd02 and version UBL 2.2:

export title="UBL 2.3 TSC"
export package=UBL-2.3-TSC
export UBLversion=2.3
export UBLstage=csd03
export UBLprevStageVersion=2.3
export UBLprevStage=csd02
export UBLprevVersion=2.2
export rawdir=raw
export includeISO=false

export libGoogle=https://docs.google.com/spreadsheets/d/OKIu..library.spreadsheet..Cpp5O
export docGoogle=https://docs.google.com/spreadsheets/d/iJ5D..documents.spreadsheet..5h1tA
export sigGoogle=https://docs.google.com/spreadsheets/d/4mc6..signature.spreadsheet..OXE5T

The Technical Committee main publication parameters are along the lines of UBL 2.3 CSD05 that follows stage UBL 2.3 CS01 and version UBL 2.2:

export title="UBL 2.3"
export package=UBL-2.3
export UBLversion=2.3
export UBLstage=csd05
export UBLprevStageVersion=2.3
export UBLprevStage=cs01
export UBLprevVersion=2.2
export rawdir=raw
export includeISO=false

export libGoogle=https://docs.google.com/spreadsheets/d/1bWAhvsb8..library.spreadsheet..iCMLhgUrHFzY
export docGoogle=https://docs.google.com/spreadsheets/d/1P-3yEI0..documents.spreadsheet..0NTmINxksPQ
export sigGoogle=https://docs.google.com/spreadsheets/d/1bWAhvsb8..signature.spreadsheet..hgUrHoiCML

The document models are downloaded directly from Google during the GitHub Action. The titles of the Google documents are irrelevant. Each document is referenced in the invocation script as follows:

libGoogle=https://docs.google.com/spreadsheets/d/1bWAhvsb8..library.spreadsheet..iCMLhgUrHFzY

the URL of your Google spreadsheet for the common library (be sure not to include the "/edit..." at the end of the browser bar)

docGoogle=https://docs.google.com/spreadsheets/d/1P-3yEI0..documents.spreadsheet..0NTmINxksPQ

the URL of your Google spreadsheet for the documents (be sure not to include the "/edit..." at the end of the browser bar)

sigGoogle=https://docs.google.com/spreadsheets/d/1bWAhvsb8..signature.spreadsheet..hgUrHoiCML

the URL of your Google spreadsheet for the common library (be sure not to include the "/edit..." at the end of the browser bar)

Before the process runs, the spreadsheets on Google should reflect the new stage string in the spreadsheet name. Examples:

UBL 2.3 Library Elements Spreadsheet - CS01 master
UBL 2.3 Document Elements Spreadsheet - CS01 master

The genericode file of the UBL vocabulary of the previously-published stage must replace that of the previous previously-published stage.

e.g. if the stage previous to csd05 was cs01, then for cs02 replace UBL-Entities-2.3-cs01.gc with UBL-Entities-2.3-csd05.gc (the renamed UBL-Entities-2.3.gc file from the csd05 release)
do the same for the UBL-Signature-Entities-*.gc files

Each revision (e.g. CSDXX) influences the following configuration files for the artefacts:

target identification when converting ODS to genericode
- ident-UBL.xml
- ident-UBL-Signature.xml
XSD and JSON schema configuration
- config-UBL.xml
  - IMPORTANT NOTE: there is version information in a comment at the top of this configuration file that, when changed, must be manually added to comments in the eight UBL-*.xsd XSD schema fragments found in the directory raw/xsd/common
  - Library: OASIS Universal Business Language (UBL) 2.x *STAGE-UPPER-CASE*
  - http://docs.oasis-open.org/ubl/*stage-lower-case*-UBL-2.x/
  - Release Date: *DATE*
- config-UBL-Signature.xml
  - IMPORTANT NOTE: this file has the same version information as found in config-UBL.xml that needs to be updated as required

Each version (e.g. 2.X) influences the following configuration files for the artefacts:

CVA master file for code list second-pass validation
- UBL-CVA-Skeleton.cva
shell wrapper for generated CVA Schematron pattern
- UBL-DefaultDTQ-shell.sch
spell-check word list (each line is a word not in the dictionary that is allowed to be in UBL, including misspellings from previous versions of UBL that cannot be repaired due to backward compatibility)
- spellcheck-UBL.txt
Google bug-avoidance model name massage directives
- massageModelName.xml
documentary ODS template skeleton for generating spreadsheet results
- skeletonDisplayEditSubset.ods

Configuring the hub document

All references in the hub document UBL.xml to version, revision, and date information must be maintained in a set of general entities maintained in the internal document type subset along the lines of the following for CSD05 that followed CS01:

	<!ENTITY name "UBL">
	<!ENTITY pversion "2.2">
	<!ENTITY version  "2.3">
	<!ENTITY pstage   "cs01">
	<!ENTITY PSTAGE   "CS01">
	<!ENTITY stage    "csd05">
	<!ENTITY STAGE    "CSD05">
	<!ENTITY isostage "WD">
	<!ENTITY standard "Committee Specification Draft 05">
	<!ENTITY stagetext "Committee Specification Draft 05">
	<!ENTITY standard "Committee Specification 01xxxxxx">
	<!ENTITY stagetext "Committee Specification 01xxxxxx">
	<!ENTITY standard "Committee Specification Draft 04 / Working Draft 03xxxxx">
	<!ENTITY stagetext "Committee Specification Draft 04 Working Draft 03xxxxx">
	<!ENTITY this-loc "https://docs.oasis-open.org/ubl/&stage;-UBL-2.3">
	<!ENTITY previous-loc "https://docs.oasis-open.org/ubl/&pstage;-UBL-2.3">
	<!ENTITY latest-loc "https://docs.oasis-open.org/ubl">
	<!ENTITY pubdate "12 May 2021">
	<!--remove time from pub date-->
	<!ENTITY pubyear "2021">
	<!ENTITY pubdate-iso "ccyy-mm-dd">
	<!ENTITY reldate-iso "ccyy-mm">

Recall that only the first declaration of a given general entity is respected. This allows prototypical declarations to be maintained after the active declaration is specified in the file.

There are a number of SYSTEM general entities whose content is not edited by hand. Rather, they are generated from the inputs of other configuration files UBL-{version}-Party-summary-information.xml and UBL-{version}-Schema-summary-information.xml that are the responsibility of the maintainer to make consistent with their expectations of the hub document.

Source files, processing stylesheets, and resulting entities generated or used in the building of the hub document:

UBL.xml is the raw main hub document that gets massaged, assembled, and processed in the final UBL-{version}.xml
UBL-{prevVersion}-modified.xml is the final UBL-{prevVersion}.xml hub document from the previous version of UBL possibly modified from the original; this is processed by hub2processSummary.xsl:
- summary-processes-ent.xml
UBL-{version}-Party-summary-information.xml processed by partydoc2db.xsl:
- summary-parties-ent.xml
UBL-{version}-Schema-summary-information.xml processed by schemadoc2db.xsl:
- summary-namespaces-ent.xml
- summary-schemas-ent.xml
- summary-examples-ent.xml
the artefacts building process creates the following four entity files for comparing current versions to old versions:
- old2newDoc-from-previous-version-documents-ent.xml
- old2newDoc-from-previous-version-library-ent.xml
- old2newDoc-from-previous-stage-documents-ent.xml
- old2newDoc-from-previous-stage-library-ent.xml

The old2newDoc version entities are always in play in the hub document, through to the completion of OASIS Standard. As the work proceeds in stages during working drafts and committee specification drafts, the old2newDoc stage entities are incorporated into the appendix comparing the previous stage to the current stage. Once the model has been completed and the work is progressing through Committee Specification to OASIS Standard, that stage-perspective section of the documentation is commented out and not included.

Please note the documentation above regarding ATTENTION-new-entities.txt and how to remove that file from the build process by replacing the authoring environment "old" entities with the build environment "new" entities.

Subdirectories:

images - original revisable source vector artwork in .svg or .drawio (incomplete set of files because many originals have been lost; please add originals here using same base name as published)
art - high-res PNG artwork for print publishing (max width: 5.7in/14.5cm)
- at a resolution of 600dpi: max width: 3425 pixels
- at a resolution of 400dpi: max width: 2283 pixels
- at a resolution of 300dpi (minimum): max width: 1713 pixels
htmlart - low-res PNG artwork for web publishing (max width: 750 pixels)
os-UBL-2.0 - code list subdirectory copied from released version of UBL 2.0
os-UBL-2.1 - code list subdirectory copied from released version of UBL 2.1
os-UBL-2.2 - code list subdirectory copied from released version of UBL 2.2
raw - pre-populated content of the results directory with skeleton contents
- raw/cl - code list content from having run the code list tooling and results here:
  - https://www.oasis-open.org/committees/document.php?document_id=67039 - tooling
  - https://www.oasis-open.org/committees/document.php?document_id=67038 - results
  - the master-code-list-UBL-*.xml file records the input information to the code list generation
- raw/json - sample JSON instances converted from the sample XML instances
- raw/json-schema - hand-authored JSON Schema fragments
- raw/mod - hand-authored model documentation fragments
- raw/val - demonstration validation environment and validation of the XML samples
- raw/xml - sample XML instances
- raw/xsd - hand-authored XML Schema fragments
utilities - tools used to generate outputs

Image creation using the http://draw.io tool:

copy UBL-2.3-Pre-awardProcess.drawio as a starting point full-width image with your own file name
open the copy in http://draw.io and modify it as required
save the vector file in the images directory
using the menu item File / Export as / Advanced...
- select PNG format
- set the DPI to 600 using "Custom"
- set the width of the image to a maximum 3425 pixels
  - the zoom should be less than or equal to 100%
  - of course if the image is not the full width of the page the width count can be less to proportionally less than 3425
- turn off the transparent background and any border width
- save the resulting high-res PNG file into the art directory
- copy the high-res PNG file into the htmlart directory
- using GIMP or some other pixel image manipulation tool, scale the htmlart copy to be a maximum of 750 pixels (or proportionally smaller) and 96 DPI

Preview results

Intermediate edits saved to the local UBL.xml file can be previewed instantly in a browser on your computer. It is recommended that one do this to establish their edits are satisfactory before checking in to GitHub to trigger the published results. The preview does not regenerate the SYSTEM general entities, so it may be necessary to push your intermediate work to your branch in order to obtain replacement entity files for your repository from the published results.

Note that the preview HTML presents the high-resolution PDF images, not the low-resolution HTML images and so the images are dramatically oversized in the browser window. This is not the case for the published HTML returned from GitHub.

Once the XML has been opened in the browser, it is necessary only to refresh the browser window after each save of the XML edits. It is not necessary to go through again the drag-and-drop or open requests.

Opening the XML in Windows:

drag and drop the XML source onto Internet Explorer, or "right-click, Open with..., Internet Explorer"
use Ctrl-R or F5 to refresh the browser after editing the file
this does not work with Firefox, Chrome, or Edge browsers

Opening the XML in Mac OSX:

drag and drop the XML source onto Safari, or "right-click, Open With, Safari"
use Cmd-R to refresh the browser after editing the file
this does not work with Firefox or Chrome browsers

Producing results

Every git push to the repository triggers the GitHub Action execution of build-github.sh to invoke whichever set of results is needed. The action takes about 25 minutes of processing on GitHub to create all of the artefacts. The resulting ZIP is about 160Mb and when unzipped provides the two archive and distribution ZIP files with the results. Editors post these results as-is to Kavi. OASIS TC Administration posts the distribution package to the https://docs.oasis-open.org/ubl OASIS web site.

Results

The build result (after about 30 minutes on GitHub) in the target directory:

UBL-2.x-{stage}-{timestamp}/ - distribution artefacts
UBL-2.x-{stage}-{timestamp}/archive-only/ - archive artefacts

Note that in the archive directory are the files:

build.console.{timestamp}.txt - console log of the execution of the Ant build script
build.exitcode.{timestamp}.txt - exit code from the execution of the Ant build script

Housekeeping

The return ZIP file is doubly-zipped, once on the web site to reduce storage costs, and once by GitHub to return the two ZIP artefacts from the server. When finished and the file is not for public access, please delete the download artefact off of GitHub to reduce storage costs.

To do:

when 2.4 is started, remove the "-modified" suffix of the previous version hub document

Contributions

As stated in this repository's CONTRIBUTING file, contributors to this repository are expected to be Members of the OASIS UBL TC, for any substantive change requests. Anyone wishing to contribute to this GitHub project and participate in the TC's technical activity is invited to join as an OASIS TC Member. Public feedback is also accepted, subject to the terms of the OASIS Feedback License.

Licensing

Please see the LICENSE file for description of the license terms and OASIS policies applicable to the TC's work in this GitHub project. Content in this repository is intended to be part of the UBL TC's permanent record of activity, visible and freely available for all to use, subject to applicable OASIS policies, as presented in the repository LICENSE file.

Contact

Please send questions or comments about OASIS TC GitHub repositories to Chet Ensign. For questions about content in this repository, please contact the TC Chair or Co-Chairs as listed on the the UBL TC's home page.