Support for automated tests

[Heiko Behrens]

Hey, I am new to this forum

Currently, I am evaluating libraries to decode standard image formats such as (animated) GIF, (animated) PNG and JPEG on limited device I am working on. I was previously involved in firmware development at Pebble (the Smartwatch) where we modified uPNG (https://github.com/elanthis/upng) to support APNG – unfortunately, this work was never open-sourced.

uGFX looks like a promising candidate, it's structure is well designed and there was no problem integrating its features of gdispImage into the larger system by writing adapter functions around GFILE and GDisplay.

During this process, I was stumbling upon several smaller and larger limitations. Most-noticably, the JPEG implementation didn't even compile without a change. As it's current practice for me and my team to drive any work with tests, I started to write automated tests to measure memory leaks, memory footprint, among actual functional correctness. Going forward, I will integrate this into a larger setup of ours that runs the decoders against a set of reference images (to test in bulk which features are insufficiently supported by comparing the decoded output against the output of a trusted decoder). If uGFX lives up to my expectations, I want to add support for alpha blending and APNG. But before I start, I wanted to see if I could avoid the risk of regressions and wanted to double-check:

I could not find any automated tests in the repository. Are there any unit-tests or otherwise automated (integration) tests I can run to verify that changes don't negatively affect the current behavior? Looking at the most-recent pull-requests, e.g. https://git.ugfx.io/uGFX/uGFX/pulls/22/files or https://git.ugfx.io/uGFX/uGFX/pulls/19 I cannot see how people manage their tests. In fact, I don't see any CI system being configured. If not, are there plans to do so?

Cheers,
Heiko

[inmarket]

Thank you for your kind comments!

With regard to the image decoders, yes the JPG decoder is the odd one out. It was recently added based on contributed code. It is the only decoder that currently requires full in memory decoding (rather than decoding as it is being displayed). It is also the least tested of the decoders. It is basically there for completeness until it can be rewritten to be more consistent with the other decoders. Whilst my primary effort is currently on the upcoming uGFX V3 which is concentrating on the GDisplay/GDriver interface I hope over Christmas to be able to clean up the JPG decoder.

With regard to test suites for the image decoders, all the decoders except JPG (and native) have been run against image suites during development. These image suites were not added to the uGFX library itself for a  few reasons 1. They are of little use to most uGFX users, 2. There were questions regarding licensing that were created by including them, and 3. They are freely available elsewhere on the internet.

Automated unit testing is something that is sorely missing currently in uGFX. Whilst we have a good range of "demo" apps that does not constitute proper unit tests. The reason for this is currently just resourcing. uGFX has grown quickly from a hobby project so there are still many gaps.

With regard to your ideas for contribution - we would love it! We are happy to provide assistance wherever we can to help and are happy to change aspects in order to achieve improved development rigor.

Regarding alpha blending and APNG, one of the difficulties currently is the lack of support for alpha at the GDISP/GDisplay interface. Whilst doing manual alpha blending is possible by reading back from the display surface, many displays do not support this. To date we have tended to cheat in these situations by only alpha blending when blending against a constant background colour and using an alpha cliff in other situations. In uGFX V3 we are adding alpha channel support into the GDISP/GDisplay/GDriver interfaces. This should make alpha blending and APNG much more easily obtainable. It was the intention that some time after uGFX v3.0 that the image decoders would get some rework to use new GDISP capabilities such as streaming support in order to again reduce resource requirements and improve speed.

Any help you can provide would absolutely be appreciated!

[Heiko Behrens]

Thanks for the quick response.

I see that you're addressing quite a few topics in your response going from limitations in the JPEG decoder over reasoning behind integration tests, lack of unit-tests, wish for contributions, ideas around alpha blending, and uGFX 3.0. While this helps me to understand the overall situation, I'd like to focus on automated testing in this thread. My goal is to make sure I don't break known behavior with my work so that I can contribute with confidence. Likewise, I don't want to invest time in test coverage that may already exist.

Quote

With regard to test suites for the image decoders, all the decoders except JPG (and native) have been run against image suites during development.

Does this mean that there's another repository with images and scripts to run the uGFX against expectations of those on a development environment? I am thinking of a CLI that decodes an image and re-encodes it using a trustworthy encoder (we use lodepng for these cases) orchestrated by scripts that traverse over input files to compare them against expected output.

Quote

Whilst we have a good range of "demo" apps that does not constitute proper unit tests.

That sounds like a good starting point. How do I use those to verify correctness and more so, how do I compile them in the first place (ideally on my development machine to run them locally)? From what I see, their readme states "Only the main files are contained. No compile-able projects".

[inmarket]

There is no other repository with images etc. We simply used the demo image programs in the repository and just changed the image to display and then visually compared that with the sample output provided by the image suite. From memory there are good suites I found for BMP, GIF and PNG. A simple Google search should find them.

The building of demo programs is development platform and target platform specific which is why no build makefile is included with the demos even though the source code itself for most demos is platform independent. Instead we have a generic gmake based makefile that will work for most platforms with minor tweaks for each platform. For example look in /boards/base/stm32f429/examples. You will see our generic makefile. This can be customised for any platform (Dev and target device and embedded os) very simply. In particular there is one variable to set to build any particular demo for the specified platform. All the complex supporting files are in /tools/gmake_scripts. There is a good wiki article on the website on doing a first time build under windows.

Of course using a makefile is only one way to build uGFX. Many people use our "single-file-make" system particularly with IDE's. This method however does have some functionality limitations (to be fixed in v3).

So, for a automated building and testing something based around our generic makefile would be a good start. The test Dev platform and target platform would need to be specified - my suggestion is it should support Linux/Linux and win32/win32 as programs built for a different target device would be hard verify correctness.

I hope that helps

[Joel Bodenmann]

Hello Heiko and welcome to the µGFX-Community!

First of all I'd also like to thank you for your feedback - we appreciate it a lot!  

@inmarket basically addressed everything. I'd just like to put more emphasis on the fact that we'd really appreciate it if somebody would start building up an automated testing environment. It's definitely something that's currently absolutely missing but required.