[Gta04-owner] [Community] About cost and business models for community hardware

[H. Nikolaus Schaller]

Hi,
I have cross-posted to the gta04-owners list because there may also be high interest in this discussion.

Am 25.09.2015 um 11:58 schrieb Neil Jerram <neil at ossau.homelinux.net>:

> 
> 
> On 20/09/15 16:16, H. Nikolaus Schaller wrote:
>> Hi,
>> 
>> Am 15.09.2015 um 22:32 schrieb Neil Jerram <neil at ossau.homelinux.net>:
>> 
>>> Hi Nikolaus!
>>> 
>>> I've been wondering about writing something along these lines, and your enquiry has provided the required activation energy!
>>> 
>>> On 15/09/15 17:10, H. Nikolaus Schaller wrote:
>>>> So what is your opinion?
>>> 
>>> As a potential purchaser, who has already been around for two cycles of this journey (i.e. GTA02 and GTA04), my current feeling is that for real (eventual) success there needs to be a solid plan that goes beyond just hardware.  In addition to all the hardware work, which I assume Goldelico would propose to do, someone needs to undertake to provide and guarantee a minimal level of working software,
>> 
>> well we also try to do that in addition to hardware (see our projects list [1]):
>> * we support boot loader
>> * we work and maintain the Letux kernel
>> * we adapt and maintain the Replicant for GTA04
>> 
>> [1]: http://projects.goldelico.com/projects/
> 
> Yes, and that is great, but my main point here is that - IIRC - apart from the boot loader work these projects were not part of your original GTA04 expectations, and you have taken them on progressively as it has become clear that they are needed.

Indeed. They all are needed for years. And since they did not happen without orchestration, we did take care of it.

Another point is that to sell hardware and development services based on the GTA04 design, Goldelico needs to run its own “Board Support Package” which is kernel + UI.

>  On the kernel front, I believe your original expectation was just that _some_ working kernel was needed, and not necessarily to upstream everything.

Initially we just wanted to get a working system. Based on the 2.6.32 kernel for the BeagleBoard XM. The main purpose was to test hardware and have something in NAND flash that boots and demonstrates basic operation.

And as it turned out that directly upstreaming is simpler than backporting upstream interesting features to a fork, we changed that strategy.

So what initially started as basic GTA04 kernel support became the “Letux distribution” (with kernel, Replicant and hopefully other UI stacks).
 
> 
> So now, at the start of a new cycle,

I don’t see a new cycle here. It is just that after focussing on software improvements we now are lucky to produce a handful of new devices that are also compatible

> we have a better idea of what minimal SW work should be planned for a successful project.

Indeed.

> 
> As a secondary point, I'm not sure that _you_ should need to be adapting and maintaining Replicant, or any other particular distro - because people like me could do that if there was a fully stable kernel base.

Well, 3.12 *is* stable and feature complete in the sense that all bells and whistles can be turned on and off. Some are not loud enough or draw too much energy, but in an ideal world it should be possible to fix it in the background.

And Lukas therefore did do what nobody else was doing: adapt Replicant to this kernel and improve the kernel where needed (e.g. sensor bug fixes).

What would have happened if we hadn’t done this approach? I am sure: nothing. There wouldn’t be a working Replicant now and in the next 10 years and no useable kernel beyond 3.7.

> There are other points in my list that I can’t address myself, and don't believe that anyone is addressing, such as 3G stability,

Here we depend too much on Imagination and TI to do anything :( And unless we buy 10 Mio chips per year they won’t even talk with us.

> and perhaps your SW resource would be better applied there.
> 
>> We haven’t been talking about that for a while (except Replicant) because the kernel was unusable due to an X11 crash bug (which has been fixed in the kerner just 2 weeks ago).
>> 
>>> which I think should mean:
>>> 
>>> - complete and upstreamed Linux kernel support, for all hardware
>>> - demonstration of an acceptably low battery current in suspend
>>> - demonstration of stable modem usage, for calls and data (at least 3G)
>>> - demonstration of reliably acceptable audio quality in voice calls
>>> - demonstration of stable wifi that reaches spec’d bandwidth and still uses acceptable current.
>> 
>> Yes, this is definitively what we all want to see.
>> 
>> A problem is how to demonstrate thais with the limited number of people really working on it. And demonstrate it before building hardware. Basically someone must pay for building these demonstrators (which might fail). I.e. take the risk of such a failure.
>> 
>> But most of your questions can already be answered:
>> 
>> * Upstreaming is tough (you have to convince maintainers and accept that they try to enforce different - and sometimes conflicting - goals).
> 
> With respect, I think that a little more humility and patience is needed here.  I have recently been upstreaming various changes into OpenStack, which I think is a broadly similar project and culture to the kernel.
> 
> My experience has been that one gets a lot of feedback, and initially some of it seems to be unhelpful, some seems to be pointlessly nitpicking, some seems to be contradicting other feedback, and so on. But I have found that if I just address things step by step - and have a lot of patience! - that everything eventually resolves, and the things that appeared to be pointless actually do have a point.  Try to be neutral - i.e. open as to the 'how' - on everything other than the fact that you do want a particular piece of function to work.  And if you do get two upstream developers who appear to be contradicting each other, just describe that, while addressing both of them, and they’ll probably sort it out.

Patience makes things slow down… And we risk to become obsolete by being too patient :)

In my experience it takes >1 year for a feature to get fully accepted. Here we must all work together…

Another aspect why it takes so long and needs patience is that none of us completely understands the kernel. Even after years you are confronted with concepts and terms you never have been aware. So fixing one line of code may need hours of research what it is about and how components work together. And what the currently recommended API is etc.

So the tough part is that you must learn and understand 10 times as much as you want to solve. Or we need people who have this knowledge but they are rarely volunteers…

> 
> 
>> * low battery current had been shown quite a while ago with the 3.7 kernel - but we did not yet get it down to the same low level with any more recent kernel (so it is a kernel issue and not hardware)
> 
> This begs an interesting question: once everything the GTA04 needs is upstream and working well, how do you ensure that it stays working well?  I would hope that the answer to this includes automated test systems somewhere, and not just ‘test manually each time a new kernel is released'.

That is a topic that also came to my mind recently. Until a while ago I thought if everything is upstream we have won: we can lean back and others will develop everything for us :)

But I have seen a lot of regressions in -rc1 kernels… So that someone who owns a GTA04 must actively test each kernel release and report bugs. But reporting bugs triggers the expectation to submit fixes… So upstreaming work is a never ending story.

And that needs some coordination of volunteers. Hence we just offer all the projects I have mentioned/listed. But we can’t “speed up” volunteers. Or we find a budget to pay freelancers - which is the original questions of this discussion.

> 
> Surely the Linaro / ARM / TI people must have an answer to this - could it be that the GTA04 needs to be ‘registered' in some way with them, so they can include it in their regression testing?

I don’t know if and how they are doing. TI runs their own kernel fork and recommends this to their customers. Some TI employees upstream and are even maintainers of subsystems. But even they introduce regressions.

There are also compile farms which report compiler errors and I have recently read about someone who has a box made of different ARM boards and compiles + auto-boots each linux-next release to detect if a board fails to boot. But our problems rarely are compile or boot problems. They are during operation. Or missing features. This is difficult to test and very difficult (or expensive) to automate.

So I think there is no such quality management/control as you expect. It works by *community*. I.e. the more owners a device type has, the more people test the new things and find and report more bugs.

On the other hand we have a script called ‘hw-test’ for a long time (every GTA04 has been tested with it) and that test report also shows regressions if e.g. the SDIO interface would be broken, it won’t report working WiFi any more. So this is sort of a regression test everybody can run on his/her own GTA04 device after installing a new kernel.

> 
>> * stable modem is IMHO already shown with Replicant (there are some tricks to stabilize the modem)
> 
> That's interesting.  Can you give or point to more details?
> 
> Specifically, is there now an effective recipe for:
> 
> - detecting when the modem has gone into a bad state
> 
> - restarting it?

Lukas can comment better on this.

> 
> (By the way, since seeing the modem reset problem on the GTA04, I've been noticing times when I think something similar is happening on my Blackberries: the modem inexplicably goes 'off' for a while, then comes back.  So it may be that it is actually common for the modem firmware to be unstable, and that the wider OS is expected just to handle it and restart the modem.)

Yes, such firmware is a black box - and especially 3G as well for the module hardware manufacturer. They purchase parts of the 3G firmware stack from Qualcomm and just add some AT commands to handle special hardware features (power on/off etc.). If there is a bug in the firmware, they can’t fix it - and have to wait for Qualcomm to fix it. Probably Qualcomm will even deny that a bug exists in their firmware :) This introduces delays beyond lifetime of handhelds. So they are usually never fixed… I.e.. nobody tries 100% bug free - there will be an equilibrium of “good enough for most”. We only have a problem of we do not belong to “most”.

> 
>> * audio quality as well (hardware voice routing in GTA04A4 and A5)
> 
> Excellent.
> 
> I still have an A3, myself, so still need reliable SW routing.  I spent ages on this, some time ago, and it appeared to be insoluble.  But there was definitely a kernel audio problem (*) that is now fixed; probably that was preventing the SW routing from working reliably, or at least obscuring the investigation.
> 
> (*) I reported on the list that if I ran a loop to play a WAV file 5 times, it would only actually play about 2 times out of the 5. Unfortunately I can't find the link for that now.
> 
>> * Neil has reported some fixes foe the SDIO kernel drivers to get high WiFi bandwidth
> 
> Right.  But they need to be upstream.

Indeed. He and the maintainers are working on it for several months. That is what I mean with “tough” work…

> 
>>> 
>>> Whoever takes that on, the plan also needs to include an understanding that they and Goldelico will work together as needed to achieve those things, through a combination of whatever SW and HW work is needed.
>> 
>> I am convinced that it does not work if two separate entities are working on hardware and software.
>> 
>> We had that for the QtMoko project. QtMoko did a wonderful job to support more and more features - up to a point where Radek lost interest.
> 
> I think you could draw the wrong conclusion from that.  I.e., that you should prepare and maintain a full distribution yourselves.

Well, if nobody maintains there will be no progress. And maintainer should be the entity that is most interested in. Obviously we are more interested in now than Radek. So the conclusion is simple: if nobody else volunteers, we have to do.

> 
> I think this was lesson #1 from the GTA02 project: producing both the HW and a full software distribution was just too much

Not really. It was too much at that time. Because code was much more fragmented and everything was new to everybody. And nothing to build on.

We have a different situation now: smart phones are everywhere. Good Android features did appear in the kernel. Libraries exist etc.

And IMHO Openmoko did make the mistake to throw away existing code and start over from scratch several times.

> , even though they had the benefit of much greater scale than GTA04.  Things are easier now, thanks to previous attempts, Android, Replicant, some of the Linux ecosystem being touch-conscious etc.  But are they easier enough?

Depends on how qualified the contributors are. I think they should have collected more experience.

> 
> I think Radek ‘lost interest' because of lack of some of the lower level points in my list above, principally the battery consumption.

Well, with 3.7 it was almost good.

A problem for me is that QtMoko has some very nice tools to support battery consumption measurements. But because it does neither run on 3.12 or later kernels, we can’t even test battery consumption easily. How to improve the kernel if a good GUI is missing.

So neither one can succeed with each other. And therefore I do not believe that such projects can be separated. They must somehow be bundled and orchestrated.

> 
> I have not been in Radek's league, but my own interest has taken regular hits from those points.  I'm very interested in distro development and maintenance, but it seems like every time I become interested again and put in some work, I hit a brick wall somewhere.  E.g. the SW audio routing work.  E.g. reliability of sending and receiving SMS.  E.g. that it's difficult to keep the GTA04 around with me and use it experimentally, because the battery doesn't last long.  E.g. the 3G data stability.  E.g. touchscreen jitter.

I know. All these have had temporoary solutions by individuals. Unfortunately it was never integrated into a whole common system.

> 
> So I really think that if all those low level problems are solved, and the kernel is upstream, other people can and will make the distros.

I think we can only solve these low level problems, if distros are actively developed. Otherwise kernel developers don’t even know what they want to have and how well the kernel behaves.

BTW: power consumption is not only a kernel problem. Distros can do very good or bad things. Android for example has introduced “wakelocks” to be able to power down subsystems if they are not needed.

Talking about Android: there must be a reason why Google did develop their own kernel and didn’t use/wait for upstream to become stable… They didn’t do that for fun (well, to some extent).

> 
>> I tried to revitalize it under the Goldelico/OpenPhoenux umbrella but failed to get it compiled.
>> 
>> Therefore I think it must be a single entity being responsible for hard- and software.
>> 
>> This does of course not exclude subcontractors working on parts of the whole picture.
>> 
>> We already have that in both HW and SW. Production is done by a subcontractor. Lukas is working as a subcontractor on Replicant. Marek for the kernel. But speed is limited by the budget that is available.
>> 
>> The software work is something which is financed from the gta04 project donations we collect. And HW subcontractors will be paid as soon as they build the GTA04A5 boards.
>> 
>>> 
>>> That would be a plan that I could support again.  I don’t think I would support a new hardware-only plan - even if I do still have fun playing with my GTA04 from time to time.
>> 
>> There is no hardware only plan. The GTA04A5 is just happening now, because we can build it now. It is intended to support all the software that has been built up since the last production run. But software work will continue in parallel.
> 
> Right.  On this primary overall point, it seems we agree.
> 
>> A problem we really have (don’t laugh) is that if someone wants to help to improve the software needs a piece of hardware to do testing. We even don’t have such a spare a device to give to potential developers. Therefore waiting for software to be finished first and then provide new hardware with rock-solid and well tested software doesn’t even work. For us. Non-free ecosystems have collected a lot of money at the stock exchanges to be able to finance alpha and beta tests and to publish new releases only if the most critical bugs have been fixed.
> 
> Yes, that is really hard.
> 
>>>> Should our projects be “profitable” or more be a “charity” based on volunteer work?
>>> 
>>> If the plan is right - i.e. as above - I am absolutely happy for you to plan for it to be profitable for you.
>> 
>> Well, what would we need to charge that really hiring more software developer is profitable?
>> 
>> A rule of thumb estimate is approx. 100EUR for each GTA04A5.
>> 
>> With that budget it could be possible to hire 2-3 more full-time software developers to iron out all known issues (maybe except upstreaming because it depends on the speed maintainers do their work). But only after the GTA04A5 come out of production.
>> 
>> What do you think - would adding such a “software development” price-increase be acceptable?
> 
> Well, I have one other worry.  The GTA02 was eventually 'left behind' because of only having 2G when the rest of the world had 3G.  So, even if the software eventually approached a working state, it was not usable for many things that people expect from up-to-date smartphones.
> 
> How soon might the same thing happen to GTA04, because of not having 4G?
> 
> If that question has a clear answer, it's the timescale that you and we have for making GTA04 a success.  (Because, actually, I don’t believe there’s very much _other_ innovation happening in smartphones now.)

> So, if your plan delivers success in that timescale, I think an extra 100EUR would be acceptable.

Oh, that is in interesting question I have never thought about. Yes, you are right. As soon as the networks or interfaces of a GTA04 are no longer available we can just put it into a museum :)

Let’s go through the list:
* WLAN, BT - for a long time
* USB2 - for a long time
* Mini-USB cables - for long time
* 2.5mm headset jack - well it is more difficult to buy a stereo headset with microphone but it still exists
* 2G - will be turned off in the US but in Europe it still is in operation (I think even CSCD = 9.6kBit/s acoustic modem mode still works)
* 3G - is mature
* 4G - network coverage is coming but still not everywhere
* 4G voice - hm. I think this is not even really in operation

So I would expect this to be not earlier than in 5 years. More likely 10 or 15 until 5G comes and 3G networks are turned off.

That is plenty of time to do something! For kernel, for distros and hardware.

Phew - quite a long answer :)

BR,
Nikolaus