[A]tari [G]ame [T]ools - 2D prototyping engine for STE

GFA, ASM, STOS, ...

Moderators: simonsunnyboy, Mug UK, Zorro 2, Moderator Team

Post Reply
User avatar
dml
Fuji Shaped Bastard
Fuji Shaped Bastard
Posts: 3951
Joined: Sat Jun 30, 2012 9:33 am

Re: [A]tari [G]ame [T]ools - 2D prototyping engine for STE

Post by dml »

I have added ZX0 compression support, recommended earlier by Anima.

Apparently compression ratio is quite impressive, almost matching arj -m7 but much faster at unpacking. However it is sloooow to compress on the host side so it is maybe best avoided until the end of a project for a quick clean-build turnaround.

I updated the table:

Code: Select all

#==============================================================================
#       test(abreed):   ticks(200hz)    tiles   sprite  map 
#------------------------------------------------------------------------------
#       wrap            42              100.0%  100.0%  100.0%
#       lz77            216              75.4%   51.3%   36.5%  <- quickest
#       arj4            902              62.8%   36.6%   19.8%
#       lzs1            273              58.8%   36.3%   21.6%  <- balance for speed
#       pk2e            492              56.7%   33.6%   20.2%
#       lzs2            363              54.4%   31.9%   19.1%  <- best overall balance
#       zx0             517              53.8%   31.9%   19.2%  <- slow to compress
#       arj7            826              53.6%   31.1%   18.0%  <- best compression
#==============================================================================
User avatar
prog99
Captain Atari
Captain Atari
Posts: 159
Joined: Thu Jun 19, 2003 8:08 pm
Location: Ross & Cromarty
Contact:

Re: [A]tari [G]ame [T]ools - 2D prototyping engine for STE

Post by prog99 »

Hi Doug, somethings not right with fdirs_32angles.h (just a snippet of the error)

Code: Select all

In file included from abreed.cpp:100:0:
/agt-tools/agtools/agtsys/tables/fdirs_32angles.h: At global scope:
/agt-tools/agtools/agtsys/tables/fdirs_32angles.h:39:1: error: narrowing conversion of '4294954511' from 'unsigned int' to 's32 {aka long int}' inside { } [-Wnarrowing]
 };
 ^
/agt-tools/agtools/agtsys/tables/fdirs_32angles.h:39:1: error: narrowing conversion of '4294942217' from 'unsigned int' to 's32 {aka long int}' inside { } [-Wnarrowing]
/agt-tools/agtools/agtsys/tables/fdirs_32angles.h:39:1: error: narrowing conversion of '4294930887' from 'unsigned int' to 's32 {aka long int}' inside { } [-Wnarrowing]
/agt-tools/agtools/agtsys/tables/fdirs_32angles.h:39:1: error: narrowing conversion of '4294920956' from 'unsigned int' to 's32 {aka long int}' inside { } [-Wnarrowing]
All my real skills are undervalued
User avatar
dml
Fuji Shaped Bastard
Fuji Shaped Bastard
Posts: 3951
Joined: Sat Jun 30, 2012 9:33 am

Re: [A]tari [G]ame [T]ools - 2D prototyping engine for STE

Post by dml »

Ok I'll take a look. I'll build all the samples today with gcc 7.5 in case there are more of these.
User avatar
prog99
Captain Atari
Captain Atari
Posts: 159
Joined: Thu Jun 19, 2003 8:08 pm
Location: Ross & Cromarty
Contact:

Re: [A]tari [G]ame [T]ools - 2D prototyping engine for STE

Post by prog99 »

I’ll also do a build with Mikros latest gcc too when I find some time to compile it up.

edit:well this is a tad embarrassing. I'd been using the latest build all along but telling agt it was 4.6.4. Given I am cloning git repos every day at work you'd think I would know better but clearly not!
All my real skills are undervalued
User avatar
dml
Fuji Shaped Bastard
Fuji Shaped Bastard
Posts: 3951
Joined: Sat Jun 30, 2012 9:33 am

Re: [A]tari [G]ame [T]ools - 2D prototyping engine for STE

Post by dml »

I pushed a fix for that issue (reproduced with gcc 7.5) and checked that the sample builds. I haven't had time to check the other samples with this GCC but I'll do it soon.
User avatar
prog99
Captain Atari
Captain Atari
Posts: 159
Joined: Thu Jun 19, 2003 8:08 pm
Location: Ross & Cromarty
Contact:

Re: [A]tari [G]ame [T]ools - 2D prototyping engine for STE

Post by prog99 »

Tried a build and it worked. Thanks

My docker image is on docker hub but private at the moment but can make it public if anyone wants to give it a whirl?
All my real skills are undervalued
User avatar
dml
Fuji Shaped Bastard
Fuji Shaped Bastard
Posts: 3951
Joined: Sat Jun 30, 2012 9:33 am

Re: [A]tari [G]ame [T]ools - 2D prototyping engine for STE

Post by dml »

Have pushed more changes tonight which add support for PCS multipalette image display, for title screens etc.

I did this before but it was a temporary hack and not really practical to use. Now it is integrated properly and supports asset compression.

Adding a PCS title screen is now quite easy. Just create a modified version of assets_com.sh called assets_pcs.sh with...

Code: Select all

# convert .png to .pcs
${AGTBIN}/pcs -dg -s 2 -pct ${SRC}/type1_272.csv -cd 4 -f 2 -m 5 -nnd 10 -dt 0 -wp ${SRC}/myimage.png

# convert .pcs to .agi for use in engine
${AGTBIN}/pcs2agi ${SRC}/myimage.PCS ${OUT}/title.agi

# optionally compress it
${PACK} ${OUT}/title.agi zx0
Run this script when you need to re-generate the titlescreen from the art (these are not ideal settings just quick settings!).

...next make sure the final asset is listed in the Makefile (so it gets transferred to the right disk/remote_share when you use 'make disk1' etc.)

Code: Select all

ASSETS_COMMON = \
	$(ASSETS)/title.agi \
...in the game code, call the display routine with the .agi filename and a timeout value (it times out or you can exit with spacebar).

Code: Select all

	STE_ShowAGI416("title.agi", 100);
The .agi format really just prepares the PCS in a raw-memory form ready to display without reorganisation. Its possible to load PCS directly but it is slower and involves multiple memory allocations and copying/skipping stuff etc. The .agi format is better for the runtime side.

I added an optimization which converts bitplanes to nibbles to help with the compression ratio (the loader reverses this of course). It doesn't make a huge difference but 5-10% saving seems typical on these images.

Last, I did some work on all the displayroutines (normal game display code, Nickel rasters, border removal, PCS images) to reduce their size. Code that was scanline-unrolled is now looped and NOP streams are now rendered into the code using loops and selective instructions to burn equivalent time in a lot less space.
User avatar
TheNameOfTheGame
Fuji Shaped Bastard
Fuji Shaped Bastard
Posts: 2203
Joined: Mon Jul 23, 2012 8:57 pm
Location: Almost Heaven, West Virginia

Re: [A]tari [G]ame [T]ools - 2D prototyping engine for STE

Post by TheNameOfTheGame »

Nice set of improvements!
sigmate
Atari freak
Atari freak
Posts: 52
Joined: Tue Mar 16, 2021 11:58 am

Re: [A]tari [G]ame [T]ools - 2D prototyping engine for STE

Post by sigmate »

Thank you so much for the hard work!

@prog99 I’d be happy to use your Docker image as well!
User avatar
prog99
Captain Atari
Captain Atari
Posts: 159
Joined: Thu Jun 19, 2003 8:08 pm
Location: Ross & Cromarty
Contact:

Re: [A]tari [G]ame [T]ools - 2D prototyping engine for STE

Post by prog99 »

sigmate wrote: Fri Aug 26, 2022 6:17 am Thank you so much for the hard work!

@prog99 I’d be happy to use your Docker image as well!
Sure, it’s a holiday weekend in the uk and I’m off climbing but will do a new build and make it public next week.
Edit: more likely the week after, forgot work would like to see my presence in an office!
All my real skills are undervalued
User avatar
prog99
Captain Atari
Captain Atari
Posts: 159
Joined: Thu Jun 19, 2003 8:08 pm
Location: Ross & Cromarty
Contact:

Re: [A]tari [G]ame [T]ools - 2D prototyping engine for STE

Post by prog99 »

Whilst I remember. Still no pcs build for x86 Linux? I saw there were binaries provided for arm64, macOS & windows.

Adding wine to the image bumps the size up quite considerably!
All my real skills are undervalued
User avatar
metalages
Captain Atari
Captain Atari
Posts: 306
Joined: Thu Jun 06, 2013 5:14 pm
Location: France
Contact:

Re: [A]tari [G]ame [T]ools - 2D prototyping engine for STE

Post by metalages »

Wine is for arjbeta ?
User avatar
prog99
Captain Atari
Captain Atari
Posts: 159
Joined: Thu Jun 19, 2003 8:08 pm
Location: Ross & Cromarty
Contact:

Re: [A]tari [G]ame [T]ools - 2D prototyping engine for STE

Post by prog99 »

metalages wrote: Fri Aug 26, 2022 1:10 pm Wine is for arjbeta ?
According to the docs pcs as well.
All my real skills are undervalued
User avatar
dml
Fuji Shaped Bastard
Fuji Shaped Bastard
Posts: 3951
Joined: Sat Jun 30, 2012 9:33 am

Re: [A]tari [G]ame [T]ools - 2D prototyping engine for STE

Post by dml »

The issue with PCS is really just that I haven't made a public repo for it so it can be built from source (but it can be built from source - and should be fine on Linux).

I'll start looking at that soon. It's a much bigger project than the other tools so it's is lagging a bit.

(I mean - I can and do build it from source here for different targets including RPI, but nobody else can until I share the source :) )
Last edited by dml on Fri Aug 26, 2022 6:42 pm, edited 1 time in total.
User avatar
dml
Fuji Shaped Bastard
Fuji Shaped Bastard
Posts: 3951
Joined: Sat Jun 30, 2012 9:33 am

Re: [A]tari [G]ame [T]ools - 2D prototyping engine for STE

Post by dml »

I have added nibble-formatted compression for 8x8 and 16x16 base layer tilesets. It does make a measurable difference to the compressed sizes.

It might be possible to do this for transparent-overlay tilesets but it will take a bit longer (and I don't think anyone is using that feature yet anyway).
User avatar
dml
Fuji Shaped Bastard
Fuji Shaped Bastard
Posts: 3951
Joined: Sat Jun 30, 2012 9:33 am

Re: [A]tari [G]ame [T]ools - 2D prototyping engine for STE

Post by dml »

I will be adding a new sprite format soon. It is a hybrid of the existing EMS and EMX methods, faster than the former, uses less memory than the latter.

EMS uses preshifted endmask data, but with a common code implementation. Low memory overhead but misses out on code-generation optimizations specific to the sprite data. This has been the main choice for sprites with many animation frames.

EMX uses code-generated preshifts, so each preshift of each frame uses unique code with endmasks burned into that code. It always wins for speed, but it takes a lot of space when using all 16 preshifts (positional x-resolution of 1 pixel). As a compromise, the preshift resolution can optionally be limited at sprite cutting time to save memory, at the cost of grid-locked drawing (2, 4, 8 or 16 pixel grid).

The new EMH method combines these techniques by using preshifted endmask data (like EMS) combined with code-generation (like EMX), but with a common generator across all preshifts for a given frame. It is also able to render multiple sprite chunks in one piece of generated code (unlike EMS/EMX at the moment which divide them into separately drawn components). There is an additional memory overhead per animation frame - for the generated code, based mostly on the sprite height - but preshift cost is much lower than EMX because the code is reused.


I'm still testing this and measuring results but I'll post a table when the code is ready which can be used as a guide for selecting the best format in each case. I began adding this feature a long time ago but the generator was tricky and I shelved it - when I returned to it recently I could not remember how it worked (!) which didn't help.



Generally speaking...

EMX is always the fastest option but has a high memory cost and so becomes limiting for animation esp. for large animated sprites.

EMS is has so far been the better choice for animation, and works well with tall/slim sprites with low memory cost per animation frame.

EMH now fills the middle ground, being good for animation on larger/wider sprites. Compared with EMS it has a low cost per frame preshift but an extra cost per animation frame, based on the frame height. It has a fixed drawing cost for any x-coord, unlike the other two, which can have spiking costs as they move (especially EMS). It is ideal for large, moving animated objects.


The SLAB format always beats the other 3 on memory saving because it does not use preshifting and stores only active colour pixels. It is good for very wide sprites with a polygonal contour and no interior holes since it is a polygonal scanconverter for blitter spans. But it is a special case compared with the others.
User avatar
dml
Fuji Shaped Bastard
Fuji Shaped Bastard
Posts: 3951
Joined: Sat Jun 30, 2012 9:33 am

Re: [A]tari [G]ame [T]ools - 2D prototyping engine for STE

Post by dml »

Have pushed out all the collected changes, mostly related to sprites & optimization.

The table below is a guide to various sprite costs - drawing time in scanlines, RAM use and compressed size on disk.

It is just a guide - not exact, because the timing measurements include the engine code wrapped around the drawing of a single sprite, some of that being in C. It is not just the timing for the sprite pixels (!). This makes the measurements for the smaller sprites a bit off. The extra overhead disappears when drawing multiple sprites since the engine deals with them in batches in 68k. But the table is close enough to assess which formats are good for different things. If i get time I'll redo the table with batches for more accuracy.

The EMX format is fastest in all cases. EMH saves memory for large sprites with animation. EMS is more general and supports exact x- & y-clipping. SLABS take minimal RAM and can be fast for large areas with no holes and a simple outline.

agtcut has been updated to use multiple cores when annealing code-generated sprites. some other changes of this kind have been rolled in.

Code: Select all

--------------------------------------------------------------------------------------------------------
settings:
--------------------------------------------------------------------------------------------------------
EMS:                    -cm emspr -emc 
EMH:                    -cm emhspr -emc     -emxopt 17,12
EMX:                    -cm emxspr -emc -nc -emxopt 17,12
SLAB:                   -cm slabs / -cm slabrestore

========================================================================================================
Single-frame, small size:
========================================================================================================
                        drawtime(hi:lo) cleartime       totaltime       memory(+slr)    LZSA file(+slr)
--------------------------------------------------------------------------------------------------------
bob: [32x32] * 1 frame
--------------------------------------------------------------------------------------------------------
 EMS                     19:17          10:7             29              3854            1031
 EMH                     20:20          "                30              3222            1071
 EMX                     16:15          "                26              7606            1795
 SLAB                    29:28          10:7             39              1118(+314)      776(+76)
--------------------------------------------------------------------------------------------------------

========================================================================================================
Single-frame, large size:
========================================================================================================
                        drawtime(hi:lo) cleartime       totaltime       memory(+slr)    LZSA file(+slr)
--------------------------------------------------------------------------------------------------------
xenboss: [144x96] * 1 frame
--------------------------------------------------------------------------------------------------------
 EMS                    152:125         71:64           223             38226            5502
 EMH                     93:93          "               164             15824            4387
 EMX                     65:63          "               136             29726            7135
 SLAB                   112:110         65:58           177              6922(+1060)     3270(+286)
--------------------------------------------------------------------------------------------------------
ms_heli: [93x59] * 1 frame 
--------------------------------------------------------------------------------------------------------
 EMS                     72:57          32:28           104             16302            4864
 EMH                     52:52          "                84             12590            4963
 EMX                     42:41          "                74             29656           10144
 SLAB                    94:92          31:27           125              4612(+674)      3061(+222)
--------------------------------------------------------------------------------------------------------
ms_slug: [150x106] * 1 frame
--------------------------------------------------------------------------------------------------------
 EMS                    172:163         83:76           255             42214            7947
 EMH                    102:102         "               185             21678            8231
 EMX                     73:71          "               156             40226           12632
 SLAB                   127:126         67:64           194              8388(+1018)     6022(+225)
--------------------------------------------------------------------------------------------------------
bug80w: [80x105] * 1 frame
--------------------------------------------------------------------------------------------------------
 EMS                     90:82          48:40           138             24708            4810
 EMH                     75:75          "               123             14466            4664
 EMX                     54:53          "               102             27644            7351
 SLAB                    92:90          46:44           138              5294(+952)      3335(+139)

========================================================================================================
Animated, medium size (IMPORTANT: size = cutting rectangle, real sprite area varies per frame)
========================================================================================================
                        drawtime(hi:lo) cleartime       totaltime       memory(+slr)    LZSA file(+slr)
--------------------------------------------------------------------------------------------------------
villager: [31x32] * 20 frames
--------------------------------------------------------------------------------------------------------
 EMS                     18:16          9:7              27             57770           13645
 EMH                     18:18          "                27             62662           16899
 EMX                     15:14          "                24            158022           35367
 SLAB                    28:27          9:7              37             21848(+5590)     7505(+202)
--------------------------------------------------------------------------------------------------------
ms_player: [72x72] * 80 frames
--------------------------------------------------------------------------------------------------------
 EMS                     21:19         11:8              32            439314           
 EMH                     24:23         "                 35            439452          120306
 EMX                     18:18         "                 29           1104442            
 SLAB                    46:45         13:10             59            180914(+36174)     

========================================================================================================
Animated, large size (IMPORTANT: size = cutting rectangle, real sprite area varies per frame)
========================================================================================================
                        drawtime(hi:lo) cleartime       totaltime       memory(+slr)    LZSA file(+slr)
--------------------------------------------------------------------------------------------------------
ms_tank2: [72x64] * 15 frames
--------------------------------------------------------------------------------------------------------
 EMS                     53:48         27:23             80            186268           44923
 EMH                     45:45         "                 72            138046           47047     
 EMX                     32:31         "                 59            259308           78211
 SLAB                    60:59         28:26             88             43794(+8806)    29595(+1587)
--------------------------------------------------------------------------------------------------------
cad_dino: [220x88] * 4 frames
--------------------------------------------------------------------------------------------------------
 EMS                    171:165        83:78            254            180534           15298
 EMH                     91:91         "                174             91718           17172
 EMX                     69:67         "                152            172664           40078
 SLAB                   120:119        63:62            183             32436(+3578)    14798(+939)
--------------------------------------------------------------------------------------------------------
cad_colt: [80x103] * 6 frames
--------------------------------------------------------------------------------------------------------
 EMS                     93:85         47:40            140            147442           27385
 EMH                     78:78         "                125            113392           34563
 EMX                     64:57         "                111            261742           73139
 SLAB                   136:135        45:37            181             45168(+6344)    15002(+1067)
Last edited by dml on Tue Aug 30, 2022 7:52 pm, edited 1 time in total.
User avatar
dml
Fuji Shaped Bastard
Fuji Shaped Bastard
Posts: 3951
Joined: Sat Jun 30, 2012 9:33 am

Re: [A]tari [G]ame [T]ools - 2D prototyping engine for STE

Post by dml »

I re-measured the 32x32 'bob' case with a batch of 12 and it takes about 13 scanlines to draw, compared with 16:15 in the table.

This still includes the batch handling time, clip testing and so on - it's probably about 11-12 scans to just draw the pixels of one bob.
User avatar
TheNameOfTheGame
Fuji Shaped Bastard
Fuji Shaped Bastard
Posts: 2203
Joined: Mon Jul 23, 2012 8:57 pm
Location: Almost Heaven, West Virginia

Re: [A]tari [G]ame [T]ools - 2D prototyping engine for STE

Post by TheNameOfTheGame »

Thanks for the update. Great to have some more options for sprite drawing. Thanks for the multi-core also. That will speed things up hopefully with that process.

Hmm, do I see Cadillacs and Dinosaurs sprites at the end of your table :).
Playmobil
Captain Atari
Captain Atari
Posts: 283
Joined: Fri Nov 13, 2015 7:40 pm

Re: [A]tari [G]ame [T]ools - 2D prototyping engine for STE

Post by Playmobil »

I don't know how to say that in english, but :

Is a Windows version possible, something like a package or an .msi installer can be provided...

Downloading 1 file, just execute the instal file, and All components will installed under Windows, in good directories, etc, etc...

Sorry for my poor english...

I have some bases on C, C#, C++, and I very want trying AGT, but the installation looks very very not handy/easy...
User avatar
prog99
Captain Atari
Captain Atari
Posts: 159
Joined: Thu Jun 19, 2003 8:08 pm
Location: Ross & Cromarty
Contact:

Re: [A]tari [G]ame [T]ools - 2D prototyping engine for STE

Post by prog99 »

My docker image would achieve this if your capable of installing and using docker desktop or podman in windows.
Better wait and see what the others say though.
All my real skills are undervalued
User avatar
viking272
Atari Super Hero
Atari Super Hero
Posts: 706
Joined: Mon Oct 13, 2008 12:50 pm
Location: west of London, UK

Re: [A]tari [G]ame [T]ools - 2D prototyping engine for STE

Post by viking272 »

prog99 wrote: Wed Aug 31, 2022 7:49 am My docker image would achieve this if your capable of installing and using docker desktop or podman in windows.
Better wait and see what the others say though.
Docker would be amazing I run this already for something else on a small Linux box
User avatar
dml
Fuji Shaped Bastard
Fuji Shaped Bastard
Posts: 3951
Joined: Sat Jun 30, 2012 9:33 am

Re: [A]tari [G]ame [T]ools - 2D prototyping engine for STE

Post by dml »

Playmobil wrote: Tue Aug 30, 2022 10:09 pm I have some bases on C, C#, C++, and I very want trying AGT, but the installation looks very very not handy/easy...
In some ways, the Windows version is easier to set up than the others, because it is being developed/tested mainly on Windows and all the needed executables are already pre-built and in the repo.

The hard part is (A) installing/setting up Cygwin or WSL and (B) installing the compiler. Once you have those two things solved, grabbing AGT and making it work is very easy.


On the other platforms such as Mac, Linux, the (A) part is a non-issue because the shell is already available and (B) the compiler is usually ok to just build from source without too many issues. But it may be necessary to build some of the other AGT tools from source as they are not in the repo.

If someone was able to set up a docker or similar environment for Windows with (A) and (B) already solved, with a clone of the AGT repo, it would just be a case of refreshing AGT from time to time to get updates.
Playmobil
Captain Atari
Captain Atari
Posts: 283
Joined: Fri Nov 13, 2015 7:40 pm

Re: [A]tari [G]ame [T]ools - 2D prototyping engine for STE

Post by Playmobil »

Thanks @prog99 & @dml !
User avatar
dml
Fuji Shaped Bastard
Fuji Shaped Bastard
Posts: 3951
Joined: Sat Jun 30, 2012 9:33 am

Re: [A]tari [G]ame [T]ools - 2D prototyping engine for STE

Post by dml »

I have pushed out a bunch of improvements, mainly to the sprite system. Some of it optimizations and some are consistency fixes.

- EMX & EMH sprite generator improvements, drawing is slightly quicker
- optimization to cliptesting on all paths
- hitflash/drawflags is now observed on all (non-fastpath) sprite paths (e.g. EMXSPR uses it, EMXSPRQ ignores it for slight speed gain on abundant objects like bullets, particles etc.).
- drawflags encoding now supports blinking, strobe-fill (both with decay counter) and invisible mode on these paths. the state for this is per-entity-instance so drawing the same sprite for different entities with different drawflags is ok.
- SLAB (and all other paths) drawing now works with x-guardband clipping. previously SLABs would turn invisible if they touched the left viewport edge.
- fixed bug in SLR custom clear shapes not always working correctly with guardband clipping
- EMX & EMH sprite generator annealing is now multithreaded (it uses 3/4 of available cores while annealing)
- more detailed stats on generated EMX/EMH sprite frames, to help optimize settings for less memory or fewer cycles

Some improvements to PCS also (mainly bugfixes and threadpool now used when generating dualfield & superpalettes).

There are some more optimizations on the way, mainly for EMH but they will probably not surface until next weekend or later.
Post Reply

Return to “Coding”