I have not looked in detail into the approval process for apps so apologies if i say something totally off. From what I understand and app needs a certain amount of positive votes in order to get out of testing.

What I wonder is:

o are there defined test cases for a project/applications
- unit tests for especially scripted applications (python
=> I would expect the answer to be yes, as most projects get ported and hopefully have the tests executed that many bring along.
- functional tests for well - functionality.
=> I saw these in the wiki, however not very well organized at the time I checked
- integration tests - to check that components that can interact actually do that as expected
=> also saw these in the wiki, just not too much of them (i was loking for n800 testing, it might have improved in the meantime)


o are results from running these test tracked

This is the more interesting item from my point of view. If I understand it right then only the 'summery' from the 'testers' gets collected and once the 'PASSED' level reaches a certain point, the application is considerd tested successfully.
The problem i see there - and that is actually also the main reason for me answering on that topic at all - is that this process can be easily cheated, which in fact already happens. Another maemo developer whom i know private asked me to please give him the thumbs up so his package finally makes it into extras ...
I guess this is not the only occasion of this happening.

I can't say what the easiest technical approach there would be but know that there exists test tracking software for manual test cases that is open source. There is even a test extension for bugzilla itself called 'testopia' - it's pretty bloated and can be quite slow - so I'd look for something simpler but I really don;t care how the testing gets tracked in the end as long as it's tracked.

On the topic of rerunning all tests on a version/release upgrade - when the test cases are sorted in several dimensions, one can start selecting tests specifically for the area changes in the upgrade happened and run this subset instead of all tests.


o is it easy to add test cases?

In my opinion there have not to be that many testcases, just something more than the big "thumbs up / thumbs down".

More important is that anyone involved is able to easy add tests he thinks might cover functionality he needs and that this gets propagated. These most likely will be of varying quality but if you have some test case staging area and some regular reviews then I would expect that coverage especially in the area of functional tests will rise fast.

Also, having such a system, Nokia could add their own tests for new features as well as upcoming bugfixes and give dedicted beta testers a better guide on what to check.

Maybe all these things got covered in the meantime, when i was interested in n800 testing a year ago, the major problem in my pov
was the missing tool to track test results.



Once you have some defined tests, it becomes possible to better determine the actual quality of the application and also interested users that would like to use it, but are not sure if something from extras-testing might screw their device, can see a more detailed overview on what they have to expect when package is installed

Examples of basic tests I personally would expect having to at least pass to get stuff even into testing (i would expect this is already written down somewhere

o package integraty
- goes to /opt
- no errors from install scripts in the package
- ...

o basic function
- appears in the menu
- can be started (very importan for python apps
- does not exceed defined cpu usage per time if not used
- does not exceed defined disk space usage per time
- runs as user
- can be terminated from gui without remains like processes, temporal files and alike


o normal function (these now of course depend on the application, so I list the some I'd like to see for an image viewer)
- opens an image
- can rotate
- can zoom
- can handle X amount of images in defined time
- ...


When these succeed, put the package in testing and let the real features and advanced functionality of an app be tested, like changing options, attaching tags, and so on.

I myself install randomly software from testing and devel as I generally know what I'm doing and have no problem with reflashing the device. So I would love to give feedback on what actually works, and not only the bugs i see and that feedback could help users as well as developers.

Tell me where to report results - and please don't send me to a wiki page

Let's leave this one the master and merge the other into it.

What is the actual fact -- isn't it impossible to make bug-free software? Is't that a byproduct of the Godel principle that you can't construct a complete mathematics, or something?

I agree with the Texrat idea "a single trusted gatekeeper with full control' till things are straightened out. Of course, the identity of the gatekeeper is important. I nominate Texrat.

Endless delay at this critical juncture is not a good solution -- it's what we seem to have now. The N900 is not going to be a new device forever.

Ever read the book Blink, which suggests that a flash impression is often better than a carefully considered one?

this is actually an old Unix Sys Admin "joke", as quite often you might type "rm -rf ./*" (i.e. remove everything below the current directory), but if you miss the "." it wipes everything off the filesystem.

Hence, experienced admins developed methods of avoiding situations of having to type that type of thing due to having at least once accidentally done it themselves.
The most common reason for having to type this, was to remove some software.

I actually wouldn't mind the work but I don't think I'm qualified yet.

I'm sure you would do a much better job than the current system.

Heh. First I need my replacement N900...

There we have it! You already have tested vigorously the USB port, so surely you can test apps too...

Oh now you're scaring me. No more usb port testing!

Please stay on topic.