[A]tari [G]ame [T]ools - 2D prototyping engine for STE

[dml]

I have made decent progress with this since I last wrote - but it has been a bit slower than expected. I underestimated the number of issues getting everything correct across various host systems (Cygwin, MacOS, Linux/RPI). There were several bugs in tools and some nasty scripting related compatibility problems that I hadn't noticed before. Some of my tools were VC++ based and needed some changes to build cleanly under each host.

The main issue has been the amount of effort involved in converting all of the samples to use the new build system. This is not difficult, just a lot of effort and careful testing (over 3 computers with git in the middle, not always helping esp. on Cygwin). Each sample caused some new issue - like transferring assets with parent directories intact for e.g. some examples with music - in a host compatible way - and so on.

Anyway the issues seem to be drying up and I am mostly just converting the remaining samples now. I'll post when I think it's worth trying out the new version. Probably at the weekend when I return to it.


It's worth noting that Windows is still the only host system that supports all the *optional* tools, mainly because some of those tools are not particularly portable or open-source, or x86 only - or whatever. I have set things up so AGT (or the samples) do not depend on such tools and in some cases will fall back if used on a non-supported host. e.g. trying to use certain asset packing modes will fallback on non-Windows to plain CRC-wrapping. Some of this is resolvable by finding/building MacOS/Linux binaries if available, or wrapping with wine (for x86 hosts anyway). But in some cases, best avoided. With more time I'll try to prepare some better default options e.g. UCL library for NRV2e and probably LZSA. For now we have LZ77 (or just CRC) as the compatible asset wrapping options - the others (NRV2e, ARJ) depend on Windows/x86 binaries meantime.

I have removed most of the rules related to preparing disk images or setting up host transfers because it was a huge mess and there are many different options. Instead I set up a post_image.sh script as a placeholder to turn disk image directories into images, or transmit them somewhere (like ultraDev, PARCP, whatever needed). For now it remains blank until I get time to catch up with those.

The gcc-toolchain mess has also been moved out to separate rules files, making it easier to deal with support for those. However I have only been testing with the more common GCC 4.6.4 so far (Vincent's distro on Cygwin & MacOS, Mikro's distro on RPI). I did test with the >=6.2 ELF based compilers at some earlier point and it was also working but will need to check it again after recent changes.

[troed]

Hi dml,

Before anyone else makes a comment about it, let me be the first to say that while SYNC did pre-display a pre-release of a "game engine" for plain ST at SV this weekend, it is by nowhere near nor intended to ever be anything like AGT In fact, I'd rather hope if anything that someone else would maybe port over the scrolling technique if there was ever a need.

As to why we didn't use AGT, it's because we aren't developing an actual game Just the fastest possible implementation of the new scrolling technology.

/Troed

[TheNameOfTheGame]

Interesting. Anywhere to see the new scrolling technique in action?

AGT scrolling is pretty fast now . But options are always good if maybe it could be used in the AGT system in some way.

[swapd0]

Interesting. Anywhere to see the new scrolling technique in action?

https://youtu.be/K0MoAhGEkn8?t=16813

[dml]

let me be the first to say that while SYNC did pre-display a pre-release of a "game engine" for plain ST at SV this weekend,

Hi! I just saw it a few minutes ago - very impressive to see that on an STFM. Congratulations!


I did make an attempt at converting some AGT samples to use ljbk's 4-pixel hardscroll on STFM and it worked well enough but IIRC that scroll technique was not 100% on all machines in all waitstates. It seemed quite close though. Of course I had to mask off the left/right border every refresh to hide the shift, reducing the area a bit.

I didn't convert many samples STFM mainly because I was finding it a lot of work to keep on top of the existing ones, and then specific versions for STF which could never use quite the same code with the extra preshift buffers etc... and the STE versions had faster sprites for the most part so not all the samples were 1:1 conversion - anyway the STF side didn't keep up with the STE side after the first view tests. But that general approach is almost interchangable with the STE version, give or take some extra bits.

Anyway your game/demo preview looks really fast and nice big sprites! Look forward to seeing more.

[dml]

I have merged the latest version back to the master/default branch and updated the top level Wiki docs for setup & prerequisites.

This version is much more self-contained and should be easier to get working in more environments. Binaries are provided for Windows, MacOS/x64 and RPI-OS 64. Changes for other environments should be minimal though.

I have not had time to test on Linux desktop systems but it will be necessary to build AGT tools locally since I can't provide working binaries for unknown environments. There could be some teething issues here but given that I have it working on RPI, it shouldn't be a big deal....


A guide has been added to the Wiki for some compiler toolchain options, which is likely needed for non-Windows systems.

Some of the samples (and all of the tutorials!) are missing from this version - I haven't had time to convert them all. But I will add them incrementally over the next week or two. The existing demos/samples should all work though.

The most important changes you'll need to know about:

* Remove any previous AGT tool bin/ directories from your $PATH, as this will interfere
* Undefine AGTROOT, AGTBIN globally in your environment. these are no longer required and could also interfere
* Check that you have a working compiler toolchain before trying to use with AGT - this is the main external prerequisite and first thing to go wrong

Building the samples is now easier than before:

> cd demos/bosscore
> make

Everything should built automatically. If it doesn't, I'll need to investigate.

More detail on building samples (since this area has seen the most significant changes):

> make

Makes everything not already made, remakes on certain changes (code, or source assets listed explicitly and optionally in your Makefile)

> make clean-all

Cleans the sample back to a nearly-clean-checkout (some diagnostic output may remain but all code & assets will be removed)

> make disk1 (disk2)

Makes just one of potentially multiple disks in a project. Each project can define which sets of assets map to which disks.

> make images

(in development) - converts the defined disk1,disk2 (etc) directories into disk images for distribution (or transmit to debug environment, or whatever else is required).


Assets:

> make assets

Generates just the runtime game assets from the source assets (.png files etc.)

> make clean-assets

Cleans the generated asset files and associated cache tags

> make clean-cache

Cleans the asset cache tags. Will trigger a rebuild of runtime assets on the next make, but does not physically remove the files.

> make assets_common

Makes only those assets common to all game 'stages' or levels

> make assets_s1

Makes assets only for stage 1 (if relevant)

> make colourmaps

For projects which use PCS-generated colourmaps/superpalettes (which is a slow process) this target will update the colourmaps. This is not part of a normal make or make-assets. It must be invoked manually.

Note that it is still possible to manually generate assets by invoking scripts, outwith the Make dependency rules e.g.

> ./assets_com.sh

This is more or less equivalent to 'makedata.sh' in previous versions of AGT. Note that some of the more complex demos may split assets over several scripts to illustrate how a larger project might be managed (with common files, multiple stages, a hud etc. etc.)


Other important notes:

- All built materials (code, assets, cache tags) are placed under directories local to the project (build/, assets/, cache/ disk1/ etc.) so things are easier to clean and there is much less mess. (The clean rules generally remove these directories).
- The AGT library is now built locally per-project, so there should be no more conflicts over AGT_CONFIG_??? defines which differ between projects.
- Dependencies should generally work better than before. I'm still making some changes here though.

There are many other changes and new features but I don't have time to list them here. I'll post separately on those. I just wanted to get a new version up ASAP so more users can hopefully get it working with fewer headaches...

[dml]

The latest and most important 'feature' update to AGT is transparent support for packed assets.

It is very easy to pack individual files in the assets_??.sh script for each project. Just add the following line:

Code: Select all

${PACK} ${OUT}/myasset.ext lz77

The last argument is the packing mode. There are 7 of these covering popular packers/decompressors on 68k.

Code: Select all

wrap: just wrap with a CRC, no compression
lz77: LZ77
lzs1: LZSA-1
lzs2: LZSA-2 (alias: lzsa)
pk2e: NRV2e/pack2e
arj4: ARJBETA -m4
arj7: ARJBETA -m7

(Note: appropriate credits for those packers/decompressors are on the Wiki, in the repo source etc.)

Only Windows supports all packer tools (pack2e, arjbeta being the tricky ones currently). The others have binaries or can be built locally from source on any host environment.

All packed assets have a file- and memory- CRC (first is high fidelity, second is quick) to catch disk and any potential packer errors. This is always-on for the moment but I will make them configurable soon.

It is only necessary to add a new define to your project makefile to support all compression types:

-DAGT_CONFIG_PACKER_ANY

Nothing else required. The engine knows how to deal with all the packing codes transparently when it sees them.

I plan to do more work on asset management but it probably won't surface for a while. The missing samples will appear first.

[EDIT]

There are individual AGT_CONFIG_PACKER_??? defines to enable specific decompressors e.g. AGT_CONFIG_PACKER_LZ77. This saves a bit of ram, but probably would be a last resort since the code for these decompressors is so small.

[Eero Tamminen]

> make disk1 (disk2)

Makes just one of potentially multiple disks in a project. Each project can define which sets of assets map to which disks.

Maybe add also zip target as there are external tools for converting / transferring content to disks / real machines?

(E.g. Hatari includes "zip2st" script for converting ZIP or directory into Atari disk image using "mtools".)

[dml]

Thanks for that tip, seems like a decent idea.

I haven't done anything yet about the disk imaging step but I'll take this into account when I get there - there is sometimes overlap between disk imaging and preparation for remote HW testing.

[Eero Tamminen]

Only Windows supports all packer tools (pack2e, arjbeta being the tricky ones currently). The others have binaries or can be built locally from source on any host environment.

There's ARJ available for Linux too, at least Debian based ones:
* https://packages.debian.org/bullseye/arj
* https://packages.ubuntu.com/jammy/arj

(Homepage link points to Sourceforge.)

[dml]

There's ARJ available for Linux too, at least Debian based ones:

It's not clear if these will even work because the m68k decompressor is tied to two very specific packing modes and probably a specific version of ARJ (that being arjbeta.exe). Of course, YMMV - I don't see any harm in trying it

[dml]

I forgot to add - arjbeta.exe is one of two tools needed to pack arj assets. The second tool parses the arj index and extracts the raw blocks for the decompressor. So it's not enough to just build the correct ARJ on Linux unfortunately. Still, it seems like a solvable problem if there is a need.

Hans Wessels (Mr Ni!) did the real hard work on ARJ - not sure if he is still around here reading/posting but if anyone would know the requirements it is Hans.

In case it is interesting I collected some stats (found in scripts/pack.sh) from one of the samples. But I expect this is old news for anyone familiar with most of these packers already. One test project isn't enough for a clear picture but it does tie with other results posted on this board previously.

Code: Select all

#==============================================================================
#       Wrap asset using a packer or a simple CRC
#==============================================================================
#       supported wrapping modes:
#==============================================================================
#                       win  mac  rpi  linux  info/credit
#==============================================================================
#       - wrap          y    y    y    y       wrap with CRC only, no compression
#       - lz77          y    y    y    y       LZ77 decoder (ray/tscc)
#       - lzs1          y    y    y    y       LZSA-1 decoder (tAt/Avena)
#       - pk2e          y    n    n    n       NRV2e decoder (Markus Franz Xaver Johannes Oberhumer)
#       - lzs2(lzsa)    y    y    y    y       LZSA-2 decoder (tAt/Avena)
#       - arj4          y    x    x    x       ARJ -m4 decoder (Mr Ni!/Tos-crew)
#       - arj7          y    x    x    x       ARJ -m7 decoder (Mr Ni!/Tos-crew)
#==============================================================================
# y = supported
# n = not supported currently
# x = not likely, any time soon
#==============================================================================
#       test(abreed):   ticks(200hz)    tiles   sprite  map 
#------------------------------------------------------------------------------
#       wrap            42              100.0%  100.0%  100.0%
#       lz77            216              75.4%   51.3%   36.5%
#       arj4            902              62.8%   36.6%   19.8%
#       lzs1            273              58.8%   36.3%   21.6%  <- favour speed
#       pk2e            492              56.7%   33.6%   20.2%
#       lzs2            363              54.4%   31.9%   19.1%  <- best balance
#       arj7            826              53.6%   31.1%   18.0%  <- favour compression
#==============================================================================

[TheNameOfTheGame]

Thanks for the updated version. Just wanted to give some feedback on testing the linux version.

Since I have the toolchain already installed, the first thing I tried was to get a demo to compile.

Of course, it gave an error that the "rmac" assembler was missing. config.sh returns "x86_64" for my arch.

So heading to ggn's rmac site http://rmac.is-slick.com/ I downloaded the linux binary, made a directory "x86_64" under agtools/bin/Linux and copied the rmac binary there and made it executable.

Next, was an error for "packwrap" not found. This was easy to correct by going to agtools/tools/packwrap and running "make" there.

This fixed the tools error and the next issue was assets. The "make" reported assets not found. There was a script in the abreed demo folder "assets_com.sh". Running this script reported "agtcut" not found. To compile and install this tool was simply a matter of going to "agtools/tools/agtcut" and running "make". After that, the "assets_com.sh" script completed successfully.

These changes allowed the "abreed" demo to build and it is able to be run in hatari:

abreed.png

So overall, not too difficult to get the first demo built and running. Looking good so far!

[dml]

Hi, that's great feedback - thanks.

I just tested on Ubuntu for WSL (Windows) and had basically the same experience. I just had to rename the Win64 'rmac.exe' to 'rmac' and the rest was the same.

I do need to create a build skeleton which makes all the natively-buildable tools in one command - but yes, currently you need to build those 3 individually. Although they do install themselves locally for you, when built.

Sorry to be more specific, the ones needing locally built are: (and this documented on the Host Build Environments bit on the wiki)

tools/agtcut
tools/packwrap
tools/3rdparty/lz77

(And obtain RMAC builds from ggn's RMAC homepage, or built it yourself).

For Windows native, Mac/64 and RPI/64 there are binaries for these in the repo for convenience but these can be rebuilt locally too.

[TheNameOfTheGame]

Further linux testing:

"bosscore" demo - noticed a minor issue during the assets build:

assets_com.sh

Code: Select all

warning: host [Linux] does not support packing code [pk2e], fallback to [wrap]
wrapping assets/bullet.emx with [wrap]

"h-shmup" demo: running "assets_com.sh" doesn't seem to make the "assets" directory it needs. I get messages like this:

Code: Select all

outputting data...
emitted sprite sequence [assets/sparks.emx]
...done
warning: could not create review TGA file: assets/sparks_vis.tga
emitted spritesheet visualisation [assets/sparks_vis.tga]

but no actual "assets" folder exists. Manually creating an "assets" folder clears all this up. "make" builds the demo ok.

"isoscroll" demo - same "assets" folder issue as above, but other that that builds fine.

So, should the assets, assets/music, assets/sounds folder be created by the scripts automatically or should it exist already i.e. manually create them in the project directory?

After building, the demos are all running good in hatari. Pretty smooth workflow as it stands. Great work.

[dml]

Hi, yes assets/ should be created automatically by the Makefile.

However it normally runs as a side effect of running 'make' or 'make disk1' or 'make assets'. It will probably be an issue if you run the asset scripts manually (since asset_tree is a prerequisite and handled by Make, not by the script).

I'll consider adding the directory step to the script itself. It does seem like an inconsistency.


BTW I have updated the Makefile page on the Wiki so it now makes sense for the new layout.

https://bitbucket.org/d_m_l/agtools/wiki/The%20Makefile

[EDIT]

> warning: host [Linux] does not support packing code [pk2e], fallback to [wrap]
> wrapping assets/bullet.emx with [wrap]

This issue is actually a documented thing now (as of today!) - Linux doesn't have an executable for pack2e (or arjbeta). Trying to use a packer that doesn't exist will cause it to fallback to a plain crc-wrap. So this is 'normal' behaviour at the moment.

However I do plan to change the default packing mode for the existing samples very soon to something that's available on all hosts (probably lz77 which is in the repo - or lzsa which can be built from github). This will remove the warning from the samples.

[TheNameOfTheGame]

Ok, great, sounds good!

[dml]

I have pushed fixes for those issues now.

[dml]

I have added the missing samples (the original ones, not the pending ones) and started adding the tutorial samples.

[dml]

In case it causes any unexpected issues - I did some branch maintenance and deletion on BitBucket last night mainly due to interactions with the existing Mercurial repo I had been working in locally - so pulling a new version from BitBucket might prompt you to 'merge changes' which don't exist. A rebase should be fine but if in doubt, delete your working dir and refetch the repo. This was a one-time thing so normal pulls should be fine from now on.

[TheNameOfTheGame]

Thanks for the improvements. Very smooth build process on linux now. All demos, examples and the tutorial build with just a "make" now.

Did notice one issue with the "layeranim" example. Running "make" on a clean directory resulted in a couple of files not found:

Code: Select all

wrapping assets/map.cct with [lz77]

 LZ77 packer  v1.3     � by ray//.tSCc. 2003-2005 

Failed to open assets/map.cct
PackWrap: compressed asset wrapper for AGT : dml/2022
Could not stat original file: assets/map.cct

wrapping assets/mapo.cct with [lz77]

 LZ77 packer  v1.3     � by ray//.tSCc. 2003-2005 

Failed to open assets/mapo.cct
PackWrap: compressed asset wrapper for AGT : dml/2022
Could not stat original file: assets/mapo.cct

Doing a "make clean" and re-running "make" cleared up the issues. Not sure what caused it in the first place though so thought I would mention it.

[dml]

Ok thanks, I'll look into that.

[dml]

Ok thanks, I'll look into that.

There were errors in the assets_com.sh script for that example which I have now fixed. I have also made some other scripting changes which will not let these packing errors pass unnoticed. However it is a minor issue (since the example runs anyway) and the update needs testing on other systems so I won't push anything right now. It will get pushed with other changes later.

[dml]

Have pushed out a collection of fixes and changes today. These impact agtcut and packwrap, affecting the asset storage format slightly. (A 'make clean-all' will be required for any samples built with a previous version to avoid loading errors).

Today's changes at the top. Earier changes (the recent repo rework) summarised below.

- add AGT_CONFIG_NO_WRAPCRC/AGT_CONFIG_NO_UNWRAPCRC optimizations to speed up asset loading for finished demos
- fix a slightly lossy 'lite-crc' on post-unwrapping step
- optimize NickelPort screen splitting impl. to be more robust against blits
- remove deprecated display modes (now handled by Nickel)
- fix Nickel stop/end events to silence raster properly on correct scanline (low border tearing bug)
- fix pack.sh script to stop with error if packing fails0
- add initial support for ultraDev HW remote testing (via a remote share), allows built projects to be booted on real STE over USB
- remote testing builds are now a build target 'make remote'
- fix script issue with layeranim asset packing
- misc. improvements to packwrap tool incl. usage feedback


------

[repo rework]

* improved Makefile rules:
- all build intermediates placed under ./build/ within project tree
- all intermediate assets placed under ./assets/ within project tree (which then get remapped to disk targets)
- agtsys library rebuilt per-project, not shared across projects, fixing engine AGT_CONFIG_?? conflicts
- distinct disks are now build targets e.g. 'make disk1', producing a final directory ready for testing or distro
- 'images' build target can optionally image built disk directories via script for distribution
- asset groups are build prerequisites, optionally refreshing targets if asset sources change
- asset groups are build targets e.g. 'make assets' 'make assets_common' 'make assets_s1' for specific game stages only
- asset groups are tagged in a build cache to prevent re-building unnecessary
- PCS colourmaps are now a build target for projects using them, but *optionally* auto-generated when missing
- removed a lot of redundant stuff, simplified rules
- cleaning is now more specific:
- clean: clean just the project + engine code
- clean-all: clean all build outputs including assets, tag-cache, disks & images
- clean-assets: clean the generated data assets & associated tag-cache
- clean-cache: clean just the tag-cache
- clean-images: clean any prepared disk images
- clean-disk1: clean built disk1 directory (..disk2 etc.)

* improved toolchain:
- all primary tools can now be built on different hosts using the same Makefiles (no more VC++ requirement)
- works natively on Mac
- tested on x64 only, so far but should work on M1
- now works on RPI/ARM host environment (although I must rebuild the tools manually at the moment, may sometimes lag...)
- AGTROOT no longer needs configured before use, it is configured relative to each project by the project Makefile
- AGTBIN not required at all
- tools built for different hosts placed under ./bin/<os>/<arch>/ subtree
- agtsys/bin/ directory should no longer be in the default search $PATH (or it may now cause issues!). only the gcc toochain should be on $PATH
- RMAC assembler is found by agtcut without requiring -rmac path on some hosts (or other previous CWD hacks)
- simplified support for different GCC toochain packages including Mikro's local-configured versions
- 'wine' no longer required on Mac/Linux, except for some optional packers (many are native windows or x86 only, not open-source)
- tools no longer wrapped in bash scripts for host env redirection
- all windows tools are statically-linked for compatibility (works on x86/x64 various Windows versions)
- multiple asset packing/wrapping modes now supported transparently, with very simple rules to apply:
- raw, no wrapping (some assets still have their own internal CRC)
- CRC wrapped (with no packing - more consistent application of CRC, quicker loading than raw mode)
- pack2e/nrv2e
- ray's lz77
- tat's lzsa 1/2
- arj -m4
- arj -m7
- all packing methods apply pre- and post- CRC (some assets still support their own CRC when used raw)
- fixed/caught numerous bugs and pitfalls in agtcut, pcs usage
- packer wrapping tool improved to apply pre- and post- wrapping CRC for disk-read and unpacker error detection

* improved asset building scripts
- all precious source assets now located in ./source_assets/ within project tree
- common assets and level/stage-specific assets organised and generated as separate tasks/scripts
- common and stage-specific asset groups can be mapped to specific disks
- demos/samples use reworked scripts to make asset cutting easier to read/understand

* improved AGT engine:
- all assets now loaded using common asset handler which understands wrapped assets using various packers
- (except .csv and .xml files, which remain raw for now)
- better error checking, reporting during asset loading etc.

[Anima]

Nice progress.

Wouldn’t it be a good idea to drop Cygwin support at all due to WSL being available on Windows machines? I suppose that you‘ll keep it for older Windows platforms being able to use AGT?!

In respect of compression ZX0 might be a good alternative as well.