New data packer

[Nyh]

Oops, right. I always forget that addx etc. use the X-flag, not the C-flag.

It is in the name!
I think it is a pity there is no bxc and bxs branch instruction, I could do some interesting jumps with that instruction...

[troed]

The new n0 depacker has an excellent depacking speed, outperforming m7 and m4 by a big margin: m7=40.855 s m4=48.498 s and n0=31.310s. The compression ratio is somewhere between the m4 packer and de m7 packer, for some files even worse as the m4 packer. The size of the n0 depacker is very small: only 6462 bytes.

Absolutely awesome. Thank you.

[Anima]

The Ni n0 packer is small and fast.

great work, thanks!

Have you considered taking MAME (program) ROM files for testing? I.e. from CPS games like Final Fight, Ghouls'N'Ghosts, etc.? The program data is mostly 512 kB up to 4 MB in size so this could be a good measure.

Please note that CPS2 ROMs are encrypted so you should stick with CPS1 games but I think there are enough to choose from.

[Nyh]

I wrote a new n1 packer but the current status of the packer is too much bit fiddeling for an Atari ST. Then I wrote a faster version of the n0 depacker and changed the meaning of a bit in the n0 packer to improve speed of the n0 fast depacker. I wish to thank all who send me test files. The idea to use Mame ROMS was great. I now have a total of 2.6 GB of Mame ROM files. Those files were very useful for finding bugs in my compression algorithms.

Current status of the project is:

Code: Select all

3255838 Calgary corpus test files
1129921 n0 fast  18.70 s  96 bytes
1129921 n0       31.58 s  62 bytes
1006902 m7       41.03 s 738 bytes
1181817 m4       48.61 s 132 bytes
1100971 n1       84.73 s 140 bytes

I think I will leave the n1 packer for now and do a rewrite of the sliding dictionary engine. This should improve the packing ratio of all packing modi and allow for bigger dictionaries.

I might do a first 'official' release of the packer this weekend including the dump mode, this all depends on how far I_A is with the dump mode.

Hans -Mr Ni!- Wessels

[metalages]

Huge improvement in performance

[Nyh]

Huge improvement in performance

50% more code 40% more speed. But I think a depacker size of 96 bytes is still acceptable.

Hans -Mr Ni!- Wessels

[metalages]

Huge improvement in performance

50% more code 40% more speed. But I think a depacker size of 96 bytes is still acceptable.

Hans -Mr Ni!- Wessels

Indeed

[metalages]

About preparing data before packing I have recently made a tool for .MOD file.
Not tested a lot yet :$
https://github.com/jhubrt/demOS/tree/ma ... S/MOD2MODX
The goal is to encode sample in simple DPCM (no prediction) instead of PCM.
It makes significant saving when packing processed MOD instead of original MOD file.

[Nyh]

Good news from the data compression front. The new sliding dictionary engine is working and results are great. I converted the arj m4 and the Mr Ni! n0 packer to the new sliding dictionary.

On the Calgary corpus:
n0: 1109042 bytes, was 1129921
m4: 1157988 bytes, was 1181817

Atari data:
n0: 38260991, was 38695256
m4: 38208660, was 38489297

and Mame data:
n0: 983897073, was 1007548997
m4: 1009605460, was 1018901400

I even managed to make compression slow on fast machines, but GUPS is not about fast compression, it is about superb compression and fast depack speeds. Next project is to convert arj m7 to the new sliding dictionary engine. I hope I can crack the 1000000 bytes barrier on the Calgary corpus.

As always the lastest version is available at: https://github.com/HansWessels/gup

Hans -Mr Ni!- Wessels

[Nyh]

I have finished testing the new n1 packer. While testing I found an error in de costs function that had a negative effect on the packing ratio of both arj m4 and the n1 packer. (The n1 packer is basically the arj m4 extended to a 128 k dictionary and a maximum match of 65 k.)

The current results are:
Calagary corpus:
m4: 1157578
n0:1109042
n1: 1075954
m7: 1006902

Atari:
m4: 38202849
n0: 38260991
n1: 36733409
m7: 34816258

Mame:
m4: 1009473046
n0: 983897073
n1: 941690196
m7: 886030557

The speed and size of the n1 depacker is roughly the same as m4. It is a bit faster because the are more and larger matches possible. It is a few bytes bigger because the original size is encoded in the data stream.

I started research on the n2 packer. The n2 packer should be better as the upx packer.

Hans -Mr Ni!- Wessels

[metalages]

Tx for the update.
I am curious about the ratio n2 will achieve

[Nyh]

I am curious about the ratio n2 will achieve

Results are mixed, some data packs good, some less, depends on the file size. Now I am working to an packer that is almost compatible with the 'nrv2s decompression in pure 68k asm, by ross' asm source I found. I removed some obvious clutter and I am not sure about the a3 constant: -$d00. I think this constant is way too large.

But by using that source as depacker I am 100% sure I will get an better compression ratio as nrv2s at the same depack speed. The n3 depacker will go beyond that with unknown speed consequences.

Hans -Mr Ni!- Wessels

[Nyh]

First results for the N2 packer are in:
Calagary corpus:
m4: 1157561
n0: 1109027
n1: 1075939
n2: 1054681 depack time 27.3 s
m7: 1006902


Atari:
m4: 38201619
n0: 38260991
n1: 36733409
n2: 35846214
m7: 34816258

There is one small problem: de C functions generate no errors while depacking but the (reference) Atari ST assembly function has 4 CRC errors. I will have to look into that. The compression got even slower. I will fix that later too (I hope).

I still have an idea for squeezing even more bits from the packed data but that will have to wait until after the weekend.

Hans -Mr Ni!- Wessels

[Nyh]

I fixed the issue with the CRC error. All seems to work OK now.

Can someone test how the n2 packer stacks up against the upx nrv2s?

Hans -Mr Ni!- Wessels

[metalages]

Can someone test how the n2 packer stacks up against the upx nrv2s?

Needs linux ?

[Nyh]

I can compile it for Windows 64.

compress:
gup a -n2 -r test.arj testfiles\*

show result:
gup l test.arj

Hans -Mr Ni!- Wessels

[Nyh]

Firs release of GUP packer.

I managed to squeeze out some extra bytes in serveral packing modes. I think now is the time to do a first release of the GUP packer. The 'official' release is at https://github.com/HansWessels/gup/releases. But the zip file with the Win32 packer is also attached to this post. I wish to thank Ger Hobbelt for helping me with the non compression parts of this release.

Have fun with this release!

Hans -Mr Ni! Wessels

[Anima]

Nice tool! Thanks for sharing.

[metalages]

Need to find time to test this.
(i was looking for a replacement of arj m4 for better ratio)

[Nyh]

Today I released a new version of the packer. Included no is the gupdump utility to dump the compressed files contained in the .ARJ file is separate files. I also improved the depacking speed of the m7 decoder and added an even faster m7 decoder:

Code: Select all

depacker bytes time [s] packed size
ni_n0       62    31.40     1109027
ni_n0_f     96    18.77     1109027
ni_n1      148    48.13     1075939
ni_n1_e    122    47.87     1075939
ni_n1_f    210    37.33     1075939
ni_n2      100    50.55     1053663
ni_n2_f    152    26.98     1053663

arj_m4     132    49.24     1157561
arj_m4_e   118    49.24     1157561
arj_m7     720    38.17     1006944
arj_m7_e   706    38.17     1006944
arj_m7_f   784    32.15     1006944

Next on the list is reorganizing the compression routines so I can use the new sliding dictionary routines for m7 packing too.

Hans

[Nyh]

I have changed the n2 packer format a bit. Most files packed with the old format can be depacked with the new format decoder. There might be special cases where it won't work. I haven't seen them yet. The new format enables better compression. It is still work in progress. I expect further improvements on the n2 packer to be made. Currently the n2 packer packs the Calgary corpus into 104720 bytes, the old packer needed 1053663 bytes.

Mode 4 went from 1157561 bytes to 1157561 bytes by fixing an error in the sliding dictionary code.

Hans

[Nyh]

I just released a new version of the gup packer. Most work has been done on the n2 packer. Currently the n2 packer packs the Calgary corpus into 1044629 bytes. With larger files the n2 packer might get very slow. But is the objective is best packing ratio and not good packing speed I will leave it as it is.

I will now try to get the m7 packer to use the new routines hoping to improve on the m7 packing ratio.

Hans

[metalages]

Is there a way to build gup from sources of your repository ?

[Nyh]

Yes, I think so. I never tried it but it should work.
For Linux it is an easy: ./configure and make

For windows and visual C: gup\win32\msvc2022\gup.sln

It works for me... do you have somewhere a problem?

Hans

[metalages]

I'll try the windows way and keep you aware