Re: Huffman compression comes to GPC!

"Dr A A Olowofoyeku" wrote:

I have just completed work on an Huffman compression unit which now supports GPC. The unit exports two main (alternative) compression functions and two decompression functions. The routines are "dead easy" to use, and the compression ratio compares well with ZIP's "deflate".

I have tried to make the code portable, and I believe that I have achieved this (it works fine with TP, BP, Delphi, FreePascal, and GPC). AFAIK there are no "endian" issues, but I stand corrected.

Unfortunately, there seem to be some. Files compressed on a little-endian machine don't uncompress on a big-endian one (or vice versa). Comparing the same file compressed on both kinds of machines shows differences in byte ordering.

I don't have the time to study the code throughly enough to say exactly what the problem it, but at first sight, WDBuf might be the problem, since it contains 16-bit words which are written to a file.

To solve the problem, of course, you first have to decide if you want the data files to be little or big endian. If they're following some standard, this will be given, otherwise, if you invented the format, it's up to you. Let's assume you want them to be little endian.

Convert each element of the buffer to little endian using the ConvertToLittleEndian procedure, e.g.: ConvertToLittleEndian (WdBuf [i], SizeOf (WdBuf [i]))

You can do this either immediately after writing the values into the buffer, or before the BlockWrite. You don't need any ifdef for this -- the routine will simply not change the values on a little endian system.

Similarly, before reading values from the buffer (e.g. after the BlockRead statements), convert them back using ConvertFromLittleEndian.

BTW, if you want to test the changes and don't have access to a big endian system, you might use big endian data files (perhaps only for testing, if you want little endian data files otherwise), i.e. use ConvertToBigEndian and ConvertFromBigEndian instead of ConvertToLittleEndian and ConvertFromLittleEndian, and verify that you can still compress and decompress files correctly.

If you need to still compile the code with BP, you can use the following routines which are part of a GPC compatibility unit for BP which will be included in the next GPC release. (Just install the unit below where BP will find it, but GPC won't, then do a `uses GPC' in the code, so BP will use the unit below, and GPC will use its own GPC unit.)

Frank

-- Frank Heckenbach, frank@fjf.gnu.de, http://fjf.gnu.de/ GPC To-Do list, latest features, fixed bugs: http://agnes.dida.physik.uni-essen.de/~gnu-pascal/todo.html

unit GPC;

interface

const Bits_Big_Endian = False; Bytes_Big_Endian = False; Words_Big_Endian = False; Need_Alignment = False; MaxVarSize = $fff0;

type SizeType = Word;

procedure ReverseBytes (var Buf; Count : SizeType); procedure ConvertFromLittleEndian (var Buf; Count : SizeType); procedure ConvertFromBigEndian (var Buf; Count : SizeType); procedure ConvertToLittleEndian (var Buf; Count : SizeType); procedure ConvertToBigEndian (var Buf; Count : SizeType);

function BlockReadLittleEndian (var aFile : File; var Buf; Count : SizeType) : SizeType; function BlockReadBigEndian (var aFile : File; var Buf; Count : SizeType) : SizeType; function BlockWriteLittleEndian (var aFile : File; const Buf; Count : SizeType) : SizeType; function BlockWriteBigEndian (var aFile : File; const Buf; Count : SizeType) : SizeType;

implementation

procedure ReverseBytes (var Buf; Count : SizeType); var i : SizeType; b : Byte; ByteBuf : array [1 .. MaxVarSize] of Byte absolute Buf; begin for i := 1 to Count div 2 do begin b := ByteBuf [i]; ByteBuf [i] := ByteBuf [Count + 1 - i]; ByteBuf [Count + 1 - i] := b end end;

procedure ConvertFromLittleEndian (var Buf; Count : SizeType); begin if Bytes_Big_Endian then ReverseBytes (Buf, Count) end;

procedure ConvertFromBigEndian (var Buf; Count : SizeType); begin if not Bytes_Big_Endian then ReverseBytes (Buf, Count) end;

procedure ConvertToLittleEndian (var Buf; Count : SizeType); begin if Bytes_Big_Endian then ReverseBytes (Buf, Count) end;

procedure ConvertToBigEndian (var Buf; Count : SizeType); begin if not Bytes_Big_Endian then ReverseBytes (Buf, Count) end;

function BlockReadLittleEndian (var aFile : File; var Buf; Count : SizeType) : SizeType; var Result : SizeType; begin BlockRead (aFile, Buf, Count, Result); if InOutRes = 0 then ConvertFromLittleEndian (Buf, Count); BlockReadLittleEndian := Result end;

function BlockReadBigEndian (var aFile : File; var Buf; Count : SizeType) : SizeType; var Result : SizeType; begin BlockRead (aFile, Buf, Count, Result); if InOutRes = 0 then ConvertFromBigEndian (Buf, Count); BlockReadBigEndian := Result end;

function BlockWriteLittleEndian (var aFile : File; const Buf; Count : SizeType) : SizeType; var Result : SizeType; p : Pointer; begin GetMem (p, Count); Move (Buf, p^, Count); ConvertToLittleEndian (p^, Count); BlockWrite (aFile, p^, Count, Result); FreeMem (p, Count); BlockWriteLittleEndian := Result end;

function BlockWriteBigEndian (var aFile : File; const Buf; Count : SizeType) : SizeType; var Result : SizeType; p : Pointer; begin GetMem (p, Count); Move (Buf, p^, Count); ConvertToBigEndian (p^, Count); BlockWrite (aFile, p^, Count, Result); FreeMem (p, Count); BlockWriteBigEndian := Result end;

end.