Add option to optimize VCDIFF decoder when VCD_TARGET will not be used as source segment

[GoogleCodeExporter]

The open-vcdiff decoder currently supports the full VCDIFF format as
specified in RFC 3284.  The open-vcdiff encoder uses only a subset of the
features available in that format.

One example is using target data as the source segment.  RFC 3284 allows
each delta window to specify one source segment to be referenced by COPY
instructions.  This source segment can come either from the dictionary
(also known as source file) or from anywhere in the previously decoded
target file.  The latter possibility means that the decoder must preserve
the entire contents of the previously decoded target file in memory, or at
least be prepared to load any given portion of the target file into memory
on demand.

The open-vcdiff encoder will never use the previously decoded target file
as the source window.  It always uses a source window that starts at offset
0 of the dictionary and includes the entire dictionary contents.

If the decoder could be guaranteed that the encoder would never send a
delta window that used previously decoded target data as the source window,
then it would no longer have to save the previously decoded target file in
memory, resulting in significant memory savings and eliminating one memcpy
operation.

Proposal: add a decoder option that prohibits the use of VCD_TARGET as the
source window type.  If this option were enabled and a delta window
containing the VCD_TARGET flag were passed to the decoder, it would fail
with a decoding error.

The Xdelta encoder (another open-source VCDIFF package) does not use
VCD_TARGET either, according to its author.

A restriction has been added to the Appendix of the SDCH protocol
(http://sdch.googlegroups.com/web/Shared_Dictionary_Compression_over_HTTP.pdf)
to prohibit the use of VCD_TARGET for the VCDIFF source segment in the
context of SDCH.

Original issue reported on code.google.com by openvcd...@gmail.com on 11 Sep 2008 at 11:24

[GoogleCodeExporter]

> Proposal: add a decoder option that prohibits the use of VCD_TARGET as the
> source window type.  If this option were enabled and a delta window
> containing the VCD_TARGET flag were passed to the decoder, it would fail
> with a decoding error.

How is this any different/better than simply telling the decoder not to support
VCD_TARGET and failing if it sees a VCD_TARGET-coded window?

Original comment by josh.mac...@gmail.com on 12 Sep 2008 at 1:14

[GoogleCodeExporter]

> > Proposal: add a decoder option that prohibits the use of VCD_TARGET as the
> > source window type.  If this option were enabled and a delta window
> > containing the VCD_TARGET flag were passed to the decoder, it would fail
> > with a decoding error.

> How is this any different/better than simply telling the decoder not to 
support
> VCD_TARGET and failing if it sees a VCD_TARGET-coded window?

I believe we are talking about the same thing: adding a flag to the
VCDiffStreamingDecoder object that causes it to fail if it sees VCD_TARGET in 
the
source specification.  I am not proposing changing the VCDIFF format.

Original comment by openvcd...@gmail.com on 12 Sep 2008 at 3:12

[GoogleCodeExporter]

Implemented in open-vcdiff version 0.6.  Added a new interface
VCStreamingDecoder::SetAllowVcdTarget to control whether the VCD_TARGET flag 
may be
used.  The default value is true.  Setting the option to false allows the 
decoder to
reclaim some of the memory used to store decoded target windows prior to the 
current
one.  If this option is set to false and the decoder finds a VCD_TARGET flag, 
it will
issue an error and refuse to continue decoding.
The executable "vcdiff" has a corresponding command-line flag 
--allow_vcd_target=false.

Original comment by openvcd...@gmail.com on 9 Apr 2009 at 7:29

• Changed state: Fixed