[Gta04-owner] Modem reenummerating (was: Re: Crowdfunding an Ubuntu smartphone)

[Dr. H. Nikolaus Schaller]

Hi Radek, Neil, others,

Am 03.09.2013 um 14:26 schrieb Dr. H. Nikolaus Schaller:

> Hi Radek,
> 
> Am 31.08.2013 um 12:32 schrieb Dr. H. Nikolaus Schaller:
> 
>> 
>> Am 30.08.2013 um 08:07 schrieb Radek Polak:
>> 
>>> On Thursday, August 29, 2013 01:00:54 PM Dr. H. Nikolaus Schaller wrote:
>>>> Am 29.08.2013 um 12:56 schrieb Radek Polak:
>>>>> On Thursday, August 29, 2013 11:50:26 AM Dr. H. Nikolaus Schaller wrote:
>>>>>> Am 29.08.2013 um 10:50 schrieb Radek Polak:
>>>>>> 
>>>>>> Looks like becoming a Heisenberg-Bug. The closer you look the less you
>>>>>> can grab it...
>>>>> 
>>>>> unfortunately yes...
>>>>> 
>>>>>>> I think that it has lost it after playing with AT_OPSYS. My GTA04 is
>>>>>>> brand new and i havent used AT_OPSYS command until yesterday.
>>>>>> 
>>>>>> Unfortunately I don't know what the factory setting is.
>>>>> 
>>>>> You can use AT_OPSYS? to find out on brand new GTA04 board if such board
>>>>> exists.
>>>> 
>>>> Hm. I don't think so :( All of them have been tested including some modem
>>>> commands... So unless we produce more, there is no really virgin GTA04
>>>> board :(
>>> 
>>> I think "some AT commands" dont matter unless you did AT_OPSYS.
>>> 
>>> Btw mine phone is now very stable. 315 calls and 0 reenumerates.
>> 
>> If I get it right, this is with OPSYS=0,2.
>> 
>> Hm. What I still don't get into a clear hypothesis is that it depends on
>> 2G vs. 3G AND suspend vs. no suspend.
>> 
>> If it would show up only when using suspend I would conjecture that
>> the EHCI/USB interface does not handle CPU suspends correctly. Maybe
>> irritating the modem which might not be prepared that the host is suspending
>> or getting into a bad state [1]
>> 
>> But the dependence on 2G vs. 3G does not fit into this picture.
>> 
>> BTW: in my setup I still have to try to find out why my GTA04 used for testing
>> only sees 2G (AT_ONCI?) - even if I issue OPSYS=3,2. Maybe I have played
>> with some other (persistent) settings of the modem. So at the moment I can't
>> try to reproduce 3G reenumerations.
>> 
>> -- hns
> 
> I have now a better setup to test this issue.
> 
> The major problem was that the antenna connection in the GTA04 was broken
> and therefore the signal strength wasn't good enough for 3G and the modem
> simply refused to switch to 3G mode even with OPSYS=3,2 because it did not
> even see a 3G base station.
> 
> But after replacing the antenna, I could now see reenumeration in ~10% of the
> incoming (and aborted) 3G calls.
> 
> To me it looks as if the first RING signal is not even being printed, but once I
> had a RING + some other unsolicted message. The caller is redirected to
> the mailbox quite immediately.
> 
> And, it is not only the modem reenumerating, but it also appears to be
> sort of resetting.
> 
> I conjecture that from the fact that parameters like AT+CRC=1 and AT+CLIP=1
> are reset when I reopen /dev/ttyHS_Application.
> 
> Hm. We may have some resonance of the 2.1 GHz UMTS signals with
> the ON_KEY transistor...
> 
> But I don't have the impression that the modem is switched off and on, because I
> don't have to wake it up again. I.e. the USB connection is reenumerated
> automatically. If we would have RF disturbances there would be some chance
> that the modem stays turned off after the transmission stops.
> 
> I have not enabled PIN checks on this SIM card for easier testing, so I can't see
> if the modem was really powered off, because then it would ask for the PIN
> again.
> 
> I use the latest 3.11 kernel and I don't think the kernel did go to suspend at all.
> 
> So it is still not really clear what is going on and why it depends so much on 2G vs. 3G.
> 
> The next step I want to try is to use the HSO driver with debugging enabled. 

Ok, here some snippet with CONFIG_USB_DEBUG=y debugging:

[  599.661346] usb usb1: usb auto-resume
[  599.665191] ehci-omap ehci-omap.0: resume root hub
[  599.679473] hub 1-0:1.0: hub_resume
[  599.683166] hub 1-0:1.0: port 2: status 0507 change 0000
[  599.688781] hub 1-0:1.0: state 7 ports 3 chg 0000 evt 0000
[  599.709472] usb 1-2: usb auto-resume
[  599.749481] ehci-omap ehci-omap.0: GetStatus port:2 status 001005 0  ACK POWER sig=se0 PE CONNECT
[  599.769470] usb 1-2: finish resume
[  601.993255] usb 1-2: usb auto-suspend, wakeup 1
[  602.009582] hub 1-0:1.0: hub_suspend
[  602.013366] usb usb1: bus auto-suspend, wakeup 1
[  602.018218] ehci-omap ehci-omap.0: suspend root hub

Such a sequence starts when you open a terminal emulator to /dev/ttyHS_Application
and type a command. As soon as you stop typing or response is done, there comes the
hub_suspend.

This means that the EHCI and USB3322 PHY are enabled/disabled on the fly
and each time the modem does not have data for some (small) delay.

A nice command to check what is going on is to run:

watch -d 'dmesg | tail -20'

in a separate terminal. Running 'evtest /dev/input/incoming' on the 3G wakeup event
in yet another terminal window is also useful.

And now comes the result (or problem): I was no longer able to produce a reenumeration in 3G.

I also cross-checked: I compiled the kernel again w/o CONFIG_USB_DEBUG,
installed, rebooted and the reenumerations were also lost.

So I additionally removed the #define DEBUG in hso.c so that I was back to the
original source and could not find reenumerations. So they must depend on the
current wind direction or humidity.

But if the dependence on USB_DEBUG would be is a valid observation, what
would this mean?

1. it would be created by some Linux kernel timing
2. it is most likely not the modem and its firmware
3. Slowing down some (interrupt?) handlers of the USB/EHCI code appears to remove the bug
4. 2G signalling may be generally slower than 3G so that the USB timing issue does not manifest itself

To really analyse this into details we would need to get some HS-USB tester,
and insert it into the connection between the USB3322 and the Modem. This
could be done by scratching open the internal USB bus (it is accessible under the display)
and soldering small wires/plugs to it in a way that does not harm the USB signals too much
(or the bus would no longer work in high-speed mode).

And then try to get the modem to reenumerate and store a trace of all events
on the USB bus. Then we can find out if it is really a software bug or some effect
of the USB3322 chip.

Phew. This is one of the most difficult to analyse problems we can have and may explain what
the difference between community developed hardware and million $$$ supported commercial
hardware is. Unfortunately, the commerial developed hardware has similar bugs, but since they
are cheaper we accept them...

So please don't expect a simple answer without joining efforts from the whole community.

BR,
Nikolaus