New data packer

GFA, ASM, STOS, ...

Moderators: simonsunnyboy, Mug UK, Zorro 2, Moderator Team

User avatar
Nyh
Atari God
Atari God
Posts: 1524
Joined: Tue Oct 12, 2004 2:25 pm
Location: Netherlands

Re: New data packer

Post by Nyh »

ThorstenOtto wrote: Sun Oct 16, 2022 9:18 am Oops, right. I always forget that addx etc. use the X-flag, not the C-flag.
It is in the name!
I think it is a pity there is no bxc and bxs branch instruction, I could do some interesting jumps with that instruction... ;-)
User avatar
troed
Atari God
Atari God
Posts: 1635
Joined: Mon Apr 30, 2012 6:20 pm
Location: Sweden

Re: New data packer

Post by troed »

Nyh wrote: Sat Oct 15, 2022 11:49 am The new n0 depacker has an excellent depacking speed, outperforming m7 and m4 by a big margin: m7=40.855 s m4=48.498 s and n0=31.310s. The compression ratio is somewhere between the m4 packer and de m7 packer, for some files even worse as the m4 packer. The size of the n0 depacker is very small: only 6462 bytes. :)
Absolutely awesome. Thank you.
User avatar
Anima
Atari Super Hero
Atari Super Hero
Posts: 843
Joined: Fri Mar 06, 2009 9:43 am
Contact:

Re: New data packer

Post by Anima »

Nyh wrote: Sat Oct 15, 2022 11:49 am The Ni n0 packer is small and fast.
great work, thanks! :cheers:
Anima wrote: Sat Oct 15, 2022 12:58 pm Have you considered taking MAME (program) ROM files for testing? I.e. from CPS games like Final Fight, Ghouls'N'Ghosts, etc.? The program data is mostly 512 kB up to 4 MB in size so this could be a good measure.
Please note that CPS2 ROMs are encrypted so you should stick with CPS1 games but I think there are enough to choose from. ;)
User avatar
Nyh
Atari God
Atari God
Posts: 1524
Joined: Tue Oct 12, 2004 2:25 pm
Location: Netherlands

Re: New data packer

Post by Nyh »

I wrote a new n1 packer but the current status of the packer is too much bit fiddeling for an Atari ST. Then I wrote a faster version of the n0 depacker and changed the meaning of a bit in the n0 packer to improve speed of the n0 fast depacker. I wish to thank all who send me test files. The idea to use Mame ROMS was great. I now have a total of 2.6 GB of Mame ROM files. Those files were very useful for finding bugs in my compression algorithms.

Current status of the project is:

Code: Select all

3255838 Calgary corpus test files
1129921 n0 fast  18.70 s  96 bytes
1129921 n0       31.58 s  62 bytes
1006902 m7       41.03 s 738 bytes
1181817 m4       48.61 s 132 bytes
1100971 n1       84.73 s 140 bytes
I think I will leave the n1 packer for now and do a rewrite of the sliding dictionary engine. This should improve the packing ratio of all packing modi and allow for bigger dictionaries.

I might do a first 'official' release of the packer this weekend including the dump mode, this all depends on how far I_A is with the dump mode.

Hans -Mr Ni!- Wessels
User avatar
metalages
Captain Atari
Captain Atari
Posts: 306
Joined: Thu Jun 06, 2013 5:14 pm
Location: France
Contact:

Re: New data packer

Post by metalages »

Huge improvement in performance ;)
User avatar
Nyh
Atari God
Atari God
Posts: 1524
Joined: Tue Oct 12, 2004 2:25 pm
Location: Netherlands

Re: New data packer

Post by Nyh »

metalages wrote: Sun Oct 30, 2022 9:50 am Huge improvement in performance ;)
50% more code 40% more speed. But I think a depacker size of 96 bytes is still acceptable.

Hans -Mr Ni!- Wessels
User avatar
metalages
Captain Atari
Captain Atari
Posts: 306
Joined: Thu Jun 06, 2013 5:14 pm
Location: France
Contact:

Re: New data packer

Post by metalages »

Nyh wrote: Sun Oct 30, 2022 12:23 pm
metalages wrote: Sun Oct 30, 2022 9:50 am Huge improvement in performance ;)
50% more code 40% more speed. But I think a depacker size of 96 bytes is still acceptable.

Hans -Mr Ni!- Wessels
Indeed :)
User avatar
metalages
Captain Atari
Captain Atari
Posts: 306
Joined: Thu Jun 06, 2013 5:14 pm
Location: France
Contact:

Re: New data packer

Post by metalages »

About preparing data before packing I have recently made a tool for .MOD file.
Not tested a lot yet :$
https://github.com/jhubrt/demOS/tree/ma ... S/MOD2MODX
The goal is to encode sample in simple DPCM (no prediction) instead of PCM.
It makes significant saving when packing processed MOD instead of original MOD file.
User avatar
Nyh
Atari God
Atari God
Posts: 1524
Joined: Tue Oct 12, 2004 2:25 pm
Location: Netherlands

Re: New data packer

Post by Nyh »

Good news from the data compression front. The new sliding dictionary engine is working and results are great. I converted the arj m4 and the Mr Ni! n0 packer to the new sliding dictionary.

On the Calgary corpus:
n0: 1109042 bytes, was 1129921
m4: 1157988 bytes, was 1181817

Atari data:
n0: 38260991, was 38695256
m4: 38208660, was 38489297

and Mame data:
n0: 983897073, was 1007548997
m4: 1009605460, was 1018901400

I even managed to make compression slow on fast machines, but GUPS is not about fast compression, it is about superb compression and fast depack speeds. Next project is to convert arj m7 to the new sliding dictionary engine. I hope I can crack the 1000000 bytes barrier on the Calgary corpus.

As always the lastest version is available at: https://github.com/HansWessels/gup

Hans -Mr Ni!- Wessels
User avatar
Nyh
Atari God
Atari God
Posts: 1524
Joined: Tue Oct 12, 2004 2:25 pm
Location: Netherlands

Re: New data packer

Post by Nyh »

I have finished testing the new n1 packer. While testing I found an error in de costs function that had a negative effect on the packing ratio of both arj m4 and the n1 packer. (The n1 packer is basically the arj m4 extended to a 128 k dictionary and a maximum match of 65 k.)

The current results are:
Calagary corpus:
m4: 1157578
n0:1109042
n1: 1075954
m7: 1006902

Atari:
m4: 38202849
n0: 38260991
n1: 36733409
m7: 34816258

Mame:
m4: 1009473046
n0: 983897073
n1: 941690196
m7: 886030557

The speed and size of the n1 depacker is roughly the same as m4. It is a bit faster because the are more and larger matches possible. It is a few bytes bigger because the original size is encoded in the data stream.

I started research on the n2 packer. The n2 packer should be better as the upx packer.

Hans -Mr Ni!- Wessels
User avatar
metalages
Captain Atari
Captain Atari
Posts: 306
Joined: Thu Jun 06, 2013 5:14 pm
Location: France
Contact:

Re: New data packer

Post by metalages »

Tx for the update.
I am curious about the ratio n2 will achieve :)
User avatar
Nyh
Atari God
Atari God
Posts: 1524
Joined: Tue Oct 12, 2004 2:25 pm
Location: Netherlands

Re: New data packer

Post by Nyh »

metalages wrote: Tue Dec 06, 2022 7:41 am I am curious about the ratio n2 will achieve :)
Results are mixed, some data packs good, some less, depends on the file size. Now I am working to an packer that is almost compatible with the 'nrv2s decompression in pure 68k asm, by ross' asm source I found. I removed some obvious clutter and I am not sure about the a3 constant: -$d00. I think this constant is way too large.

But by using that source as depacker I am 100% sure I will get an better compression ratio as nrv2s at the same depack speed. The n3 depacker will go beyond that with unknown speed consequences.

Hans -Mr Ni!- Wessels
User avatar
Nyh
Atari God
Atari God
Posts: 1524
Joined: Tue Oct 12, 2004 2:25 pm
Location: Netherlands

Re: New data packer

Post by Nyh »

First results for the N2 packer are in:
Calagary corpus:
m4: 1157561
n0: 1109027
n1: 1075939
n2: 1054681 depack time 27.3 s
m7: 1006902


Atari:
m4: 38201619
n0: 38260991
n1: 36733409
n2: 35846214
m7: 34816258

There is one small problem: de C functions generate no errors while depacking but the (reference) Atari ST assembly function has 4 CRC errors. I will have to look into that. The compression got even slower. I will fix that later too (I hope).

I still have an idea for squeezing even more bits from the packed data but that will have to wait until after the weekend.

Hans -Mr Ni!- Wessels
User avatar
Nyh
Atari God
Atari God
Posts: 1524
Joined: Tue Oct 12, 2004 2:25 pm
Location: Netherlands

Re: New data packer

Post by Nyh »

I fixed the issue with the CRC error. All seems to work OK now.

Can someone test how the n2 packer stacks up against the upx nrv2s?

Hans -Mr Ni!- Wessels
User avatar
metalages
Captain Atari
Captain Atari
Posts: 306
Joined: Thu Jun 06, 2013 5:14 pm
Location: France
Contact:

Re: New data packer

Post by metalages »

Nyh wrote: Mon Dec 12, 2022 3:07 pm Can someone test how the n2 packer stacks up against the upx nrv2s?
Needs linux ?
User avatar
Nyh
Atari God
Atari God
Posts: 1524
Joined: Tue Oct 12, 2004 2:25 pm
Location: Netherlands

Re: New data packer

Post by Nyh »

I can compile it for Windows 64.

compress:
gup a -n2 -r test.arj testfiles\*

show result:
gup l test.arj

Hans -Mr Ni!- Wessels
You do not have the required permissions to view the files attached to this post.
User avatar
Nyh
Atari God
Atari God
Posts: 1524
Joined: Tue Oct 12, 2004 2:25 pm
Location: Netherlands

Re: New data packer

Post by Nyh »

Firs release of GUP packer.

I managed to squeeze out some extra bytes in serveral packing modes. I think now is the time to do a first release of the GUP packer. The 'official' release is at https://github.com/HansWessels/gup/releases. But the zip file with the Win32 packer is also attached to this post. I wish to thank Ger Hobbelt for helping me with the non compression parts of this release.

Have fun with this release!

Hans -Mr Ni! Wessels
You do not have the required permissions to view the files attached to this post.
User avatar
Anima
Atari Super Hero
Atari Super Hero
Posts: 843
Joined: Fri Mar 06, 2009 9:43 am
Contact:

Re: New data packer

Post by Anima »

Nice tool! Thanks for sharing.
User avatar
metalages
Captain Atari
Captain Atari
Posts: 306
Joined: Thu Jun 06, 2013 5:14 pm
Location: France
Contact:

Re: New data packer

Post by metalages »

Need to find time to test this.
(i was looking for a replacement of arj m4 for better ratio)
Post Reply

Return to “Coding”