[Gta04-owner] [PATCH 3/3] tty/slaves: add a driver to power on/off UART attached devices.

Dr. H. Nikolaus Schaller hns at goldelico.com
Wed Mar 25 17:44:42 CET 2015


Hi,

Am 25.03.2015 um 16:21 schrieb Sebastian Reichel <sre at kernel.org>:

> Hi,
> 
> On Wed, Mar 25, 2015 at 08:59:14AM +0100, Dr. H. Nikolaus Schaller wrote:
>> Am 25.03.2015 um 02:45 schrieb Sebastian Reichel <sre at kernel.org>:
>>> On Tue, Mar 24, 2015 at 06:58:15PM +0100, Dr. H. Nikolaus Schaller wrote:
>>>> So you propose that the parent->child relationship is “control”? I.e. some
>>>> channel which allows to address some bus client (through <reg>) and
>>>> control that devices.
>>>> 
>>>> Makes sense. This is how i2c and spi clients are specified.
>>>> 
>>>>> 
>>>>> In the case of our GPS, it receives control over the serial connection from
>>>>> the UART,
>>>> 
>>>> Ahem - does it?
>>>> 
>>>> AFAIK the chip simply starts to emit NMEA records if powered on.
>>>> There is no command going over the serial interface to address it
>>>> or control it.
>>> 
>>> Right, since GPS basically doesn't need any configuration/control.
>>> That’s not true for other UART attached devices, though.
>> 
>> Do you have an example? Usually an UART data stream is transparently
>> presented to a /dev/tty - and user-space daemon can configure/control the
>> attached device. In most cases it can mix payload data and control command
>> by some AT command and escape sequences.
> 
> Yes, but the configuration is minimal. Anyways as you said there
> *is* some kind of control happening over the UART.

Control is happening on a higher network stack level than UART. It
control is done through AT commands.

> 
>>>>> also receives control via a GPIO to the on/off pin, and also needs
>>>>> a regulator to power the antenna.
>>>>> 
>>>>> So should the parent be the uart, the on/off GPIO, or the regulator?
>>>>> 
>>>>> I would much rather there wasn't a parent, and that each of these were listed
>>>>> as ad-hoc attribute assignments.  But device-tree says there must be a parent
>>>>> (where possible), and the parent is the thing that is “primarily" in control.
>>>> 
>>>> Well, IMHO the “parent” could also be the root. Representing the
>>>> whole board.
>>>> 
>>>> Nevertheless, I doubt your rule that “ability to control” defines
>>>> the parent>child relation (see below).
>>>> 
>>>>> 
>>>>> I think the GPS is “primarily" a uart-attached device.
>>>> 
>>>> But not in the same way as an I2C device.
>>>> 
>>>> Especially the serial interface is not a bus and not used for
>>>> signalling and power control. It is payload data (only).
>>> 
>>> Actually many I2C devices are also powered on/off via a GPIO and
>>> even use additional GPIOs for sending interrupts. Nevertheless they
>>> are normally identified as an I2C device.
>> 
>> Because I2C is a bus that can address multiple clients and gpio isn’t
>> a bus and a point-to-point connection.
>> 
>> But IMHO it is not because they (can) send payload data over i2c.
> 
> From my POV it's not because I2C is a bus, but because the primary
> function is happening via I2C (e.g. configuring sensor, gettings its
> data). The GPIOs are only needed to compensate some I2C shortcomings
> (missing irq feature in I2C) or reduce the complexity of the client
> (power GPIO). The actual system interaction with the I2C chip is
> going via I2C, though.

Yes, but this is in my new understanding irrelevant for proper DT description.

> 
>>> Also for non-GPS device the serial connection is used for
>>> controlling and configuring the device.
>> 
>> This assumes that “controls a device” is the criterion for making a device
>> a subnode. I doubt that.
> 
> For me the criterion always was "accessing the device's
> registers/configuration/data" from the system's point of view (so
> your video port does not count, since it models a connection between
> two components without system interaction).

> 
>>>>> So I propose a device-node which describes the GPS, which is a child of the
>>>>> UART, and explicitly identifies the GPIO it uses to power on/off, the
>>>>> regulator it uses to power the antenna, and how it receives "on or off"
>>>>> status indications from the GPS.
>>>> 
>>>> The more I think about that, you have given good arguments *not*
>>>> to use the parent->child relationship for the UART interface of
>>>> the GPS.
>>>> 
>>>> Let me give another example. The 3G Modem has 3 (or 4) interfaces:
>>>> 1. an USB-Interface for AT signalling and payload
>>>> 2. some GPIOs for power control.
>>>> 3. a PCM interface for digital voice. 
>>>> 4. it might also have a serial interface as alternate AT command and (GPRS
>>>>   low speed) payload.
>>>> 
>>>> So which one is the most prominent or most important to make it a child of?
>>>> 
>>>> If you use “control” you must make it a child of the USB phy and the serial interface
>>>> which requires multiple inheritance…
>>>> 
>>>> So I am not sure at all. None is IMHO prominent and unique enough to make it
>>>> a parent>child relation.
>>>> 
>>>> Threrefore, I would be happy to see it as multiple children of /. For example a
>>>> MFD with subnodes.
>>> 
>>> This scenario has already been seen before and can happen for
>>> non-UART based devices (e.g. SPI + I2C). So far the decision was to
>>> postpone the discussion about this kind of devices until one of them
>>> appears.
>>> 
>>>>> It is arguable that the "antenna" should be treated as a separate device - a
>>>>> child of the GPS - which controls the regulator and also provides a 'extcon'
>>>>> which reports whether an external GPS antenna is attached, or whether the
>>>>> internal on is in use.  I haven't made the time to properly explore that
>>>>> issue yet.
>>>> 
>>>> If you assume that parent>child relationship is about control (only). But if it
>>>> is about data flow, there is a different concept in DT. For example the CSI/DSI
>>>> (OMAP DSS) use the “port” concept.
>>> 
>>> Yes, this is about data not going to the cpu / system memory.
>> 
>> Really? The frame buffer data is in system memory and gets to the panel. Or a
>> camera image ends up in system memory.
> 
> Really? So the panel accesses the system memory? Check again. The
> panel is simply connected via DSI/SDI/... and does not access the
> system memory at all. The display controller OTH does. So let's have
> a look at the DT structure (simplified):
> 
> ocp -> dss -> { port = <&panel>; }
> ... -> spi -> panel { port = <&dss>; }
> 
> So the port models a device-to-device connection, which works
> independent of the remaining system. No system memory is involved.

Sorry, but I can’t exactly follow what you want to show.

Yes, the port mechanism describes a device to device connection.

But it does not describe system memory. Although it is the source of
the video (i.e. the framebuffer).

This shows that the DT does not necessarily describe data flow.

So why do you want to describe data flow from tty to uart to some connected
device?

> 
>>>> This means that indeed parent>child is used for control, e.g. I2C
>>>> or SPI to tell the panel controller to switch on.
>>> 
>>> well parent > child is not about power control, but about "primary"
>>> control.
>> 
>> Can you clearly and precisely define what “primary” control is? I think
>> two people will have three opinions about that.
>> 
>> For example I would see the w2sg0004 on/off gpio as the primary “control”
>> which makes the light (NMES records) turned on.
>> 
>> Therefore we should make it the subnode of some gpio-controller, shouldn’t we?
> 
> I never thing of enable gpios to be the main interface, since it can
> simply be connected to some voltage and be enabled constantly. It's
> just some kind of helper for the device. It does not really exchange
> data between the system and the device, though. (Note: This is
> not true, if the GPIOs are used for bitbanging some high-level
> protocol, in which case a virtual controller (like i2c-gpio) is
> modelled in DT).
> 
> So let's have a look at the UART link again. It is transporting
> non-trivial data from the UART to the system and it is handling
> the data I actually want. It's providing access to the device's
> “registers".

Well, this view can be followed, but is as strange and bent to fit
like my view of the w2sg0004 as a “regulator”.

> 
> For me its clear, that one would expect the GPS under the UART
> port, since when speaking with other developers I would describe
> the GPS as being a "UART attached GPS chip". From me the
> differences between a UART port and a SPI port from a system
> software’s POV can be reduced to missing multi-device support.

DT does not necessarily have to describe data flow as we have seen
in the video pipeline example.

> 
>>> Some I2C/SPI devices have additional gpios for power
>>> control.
>>> 
>>>> For example we have the GTA04 panel as a subnode of SPI (because SPI is a
>>>> bus and the panel is controlled through SPI). Other panels need no control
>>>> interface and are simply a child of /.
>>> 
>>> Yes, some devices do not have an interface worth of being called
>>> “primary control" :)
>> 
>> This is why I criticise this criterion to define when we need a
>> parent/child relation.  It is not strict and does not follow a
>> principle but opinions.
>> 
>> But before we stick to opinions, I have studied again:
>> 
>> 	http://www.devicetree.org/Device_Tree_Usage#Devices
>> 
>> In their examples (which unfortunately don’t cover our special cases)
>> they make clear that subnodes are needed for “addressable” subnodes
>> like CPUs, Memory banks, i2c clients, ethernet clients etc.
>> 
>> (well, the first sentence in “Interrupts” can be read to confirm
>> this interpretation: “Unlike address range translation which
>> follows the natural structure of the tree”)
>> 
>> In general I read from this (although it is not explicitly stated)
>> that parent>child hierarchy is about addressing bus clients and
>> mapping devices into memory address space.
>> 
>> Everything else is to be done through references.
> 
> Actually I think so far we had only one similar case, so its not
> obviously a corner case, which is underdocumented. The one similar
> case is the HSI subystem, which I maintain (so obviously I'm biased
> ;)). It's also describing a 1:1 dependency (also serial, but with
> multiple logical channels) and uses a child node.
> 
> Actually the nokia-modem binding is quite similar to your UART
> attached GPS device. It has some GPIOs for power control and even
> an additional GPIO for interrupting. The primary interface is still
> the HSI one, which exchanges all of the configuration and data.
> 
>>>> The video stream and its connections are arranged through such ports.
>>>> For the video out management we have added our OPA362 video amplifier into
>>>> that video pipeline which starts at the VENC (Video encode) in the SoC,
>>>> goes through the OPA362 and ends in the connector (which has its own
>>>> node). All this is connected by references. The opa362 node itself is a child
>>>> of /: 
>>>> 
>>>> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/arch/arm/boot/dts/omap3-gta04.dtsi?id=refs/tags/v4.0-rc5#n98
>>>> 
>>>> So to summarize I now see exactly two cases where parent>child is used:
>>>> a) for components of an aggregate device (OMAP3 > I2C controllers)
>>>> b) for components connected to a bus controller (I2C > TWL4030)
>>>> 
>>>> Since the twl regulators are components of the twl chip it completely explains
>>>> why we see
>>>> 
>>>> omap3 > i2c > twl4030 > vaux4
>>>> 
>>>> Everything else is done by references. Therefore we see references for
>>>> * gpios within a controller (independently of it’s relation on a bus)
>>>> * regulators
>>>> * clocks
>>>> * any function that is not connected by a bus
>>>> 
>>>> There is no need (and practice) to squeeze everything into a parent > child
>>>> relationship. And I can’t see your “control” or a “main interface” rule.
>>> 
>>> And I can't see your bus rule for me the motiviation behind the
>>> parent > child relationship is about the system's primary way for
>>> accessing a device.
>> 
>> It appears not to be “my bus rule”. I just formulated in that way from the
>> examples I have seen…
> 
> yes, me too ;)
> 
>>> It's coincidence, that this is currently (always?) a bus connection,
>>> since most devices are actually connected using some kind of bus.
>> 
>> I think the view must be reversed and then it is not a coincidence but a
>> consequence.
> 
> so show me some non-bus connections, which are actually used to
> exchange non-trivial data (with on/off being trivial). We currently
> do not really have them.

Are we talking about DT or data flow from user space through the kernel?

> 
>>>> At least this is my (a little limited) view from using and programming DT for
>>>> multiple boards.
>>>> 
>>>> Maybe this is what the DT list should comment or show a clear definition when
>>>> parent>child relations are wanted and when not.
>>>> 
>>>> Now back to the GPS chip connected to an UART.
>>>> 
>>>> Is it
>>>> a) a component of an aggregate device? no
>>>> b) is it connected to a bus (i2c, spi, address bus)? no
>>> 
>>>> This comes back to the question if UART is a bus.
>>>> 
>>>> I still think that it is NOT a bus and therefore it should not be modeled in DT
>>>> like busses are (enforcing clients to become subnodes).
>>> 
>>> If I use a SPI device with hardwired chip select its *very* similar
>>> to UART.
>> 
>> Yes, because you can just connect a single client to SPI and then
>> question if it is a bus. But that is a useage pattern but not a
>> principle.
>> 
>> And IMHO, DT definitions should follow principles and not current
>> need.
>> 
>> Otherwise I don’t understand why the virtual-gpio connected to
>> UART DTR or the regulator is *not* acceptable.
> 
> I have missed virtual-gpio UART DTR discussion and assume, that the
> regulator is about the regulator on the GPS chip?

Well, it was introduced in platform device kernels that it was possible
to define a dtr-gpio for the uart. And there was a driver for the w2sg0004
that presented itself as a gpio-controller with a single gpio. This “virtual”
gpio was used to power on the chip as soon as DTR was asserted.

Parts of this had been in mainline but have been removed some months
ago.

> 
> I don't think the regulator is not acceptable. I simply think its
> not useful. Basically the structure would be
> 
> uart {
>    /* stuff */
>    gps {
>        vdd-supply = <&gpsreg>;
>        /* stuff */
> 
>        gpsreg: regulator {
>            enable-gpio = <GPIO 42>;
>            /* stuff */
>        }
>    }
> }

Except that the gpsreg is a subnode of /

> 
> and could simply be
> 
> uart {
>    /* stuff */
>    gps {
>        enable-gpio = <GPIO 42>;
>        /* stuff */
>    }
> }
> 
> Without loosing important information.

Only if you assume that the gpsreg must be a subnode of the uart.

> The regulator would only be
> interesting if it is also used by other devices. Other devices also
> do not expose their internal clocks etc.
> 
>>>> The reason is there no well defined protocol to make multiple clients and
>>>> you can’t specify a <reg>.
>>>> 
>>>> So you are either abusing the notion of a bus or the gps chip must be a separate
>>>> DT node (child of something else).
>>>> 
>>>> I hope my gut feeling that we are trying to do something fundamentally wrong
>>>> with making the gps chip a subode of serial is now argued a little better.
>>>> 
>>>> For making the serial device control some regulator through a reference to
>>>> a regulator, I am very fine. It is what you do for the W2CBW bluetooth part.
>>> 
>>> NAK about referencing the regulator from the UART node. It should be
>>> referenced from the Bluetooth / GPS node. The UART port works
>>> without the regulator. It's the remote side, that does not work.
>>> We also do not add the regulators of I2C clients to the I2C host.
>> 
>> I think Pavel made a very good comment:
>> 
>> The uart node should be able to power a rs232 converter/level shifter directly
>> connected to it (through a regulator or gpio-regulator).
> 
> Yes and this may belong to the UART node.
> 
>> And there must be a separate mechanism to power the remote rs232 converter
>> and the remote device.
> 
> Yes and I think this goes to the remote device's node.
> 
>> The standard signalling for this are the RTS/DTR lines of an UART/RS232.
>> UART is not only RX+TX.
> 
> so?

You said NAK about referencing a regulator from the UART node and now?

> 
>>> If you want to talk with an I2C device you enable the I2C device
>>> *and* its parent (aka the I2C host controller). I guess the same
>>> should be done for the UART.
>>> 
>>>>>>>> All the following is very special logic for the w2sg0004 chip which should be
>>>>>>>> divided out into a separate driver.
>>>>>>>> 
>>>>>>>> Marek and me already had proposed such a chip specific driver (to be located
>>>>>>>> in drivers/misc) some months ago. It would encapsulate everything w2sg0004
>>>>>>>> specific and present itself as a regulator (because that is its main purpose:
>>>>>>>> control the LDO regulator inside the w2sg0004 chip).
>>>>>>> 
>>>>>>> Presenting itself as a regulator would be wrong because it isn’t a regulator.
>>>>>> 
>>>>>> It has a regulator that can be controlled by a gpio…
>>>>> 
>>>>> Does it?  I guess may it does.
>>>>> Maybe it also has an ARM core and some memory and assorted other bits and
>>>>> pieces.  But we don’t really need to describe them to device tree.
>>>> 
>>>> Well, if I look into the data sheet, it has indeed an LDO from 3.3V to 1.8V.
>>>> 
>>>> Not to make you confused: the w2sg0084 would have a WAKEUP output which - if
>>>> connected to a GPIO - would tell if the module is really active or not. Using this would
>>>> make monitoring the RX line through different pinmux states obsolete.
>>>> 
>>>> But the only Linux board using the w2sg00x4 appears to be the GTA04 and it has
>>>> not wired the WAKEUP line to a  GPIO.
>>>> 
>>>>> The w2sg0004 is primarily a GPS device, so that needs to be stated in the
>>>>> devices tree description.  If there are subcomponents that can usefully be
>>>>> described as well then there could be a place to describe those subcomponents.
>>>>> I notice that there is a "1V8-out" pin, so presumable the chip can deliver a
>>>>> 1V8 source based on its 3V3 input.  If a board made use of that, it could be
>>>>> useful to describe the regulator inside the GPS so it could be declare that
>>>>> some other device which needed 1V8 made use of that regulator….
>>>> 
>>>> Yes. Well, even if *we* do not use it in this way yet, we should not make it
>>>> difficult for others to do by enforcing the wrong description.
>>> 
>>> well this has nothing to do with the parent-child discussion, since
>>> you can obviously reference a regulator independently of its
>>> position in the DT.
>>> 
>>>>>> Another example to think about: the twl4030 is also not a regulator.
>>>>>> Nevertheless they present some regulator nodes.
>>>>> 
>>>>> The TWL4030 is a multifunction device which contains regulators and GPIOs and
>>>>> audio codec and  USB PHY etc etc etc.
>>>>> 
>>>>> So in device-tree there is a device-node for the TWL4030, and it has
>>>>> child-nodes for each sub-device.   They in turn can provide services to other
>>>>> devices on the board.
>>>> 
>>>> Exactly as discussed above. And there, the parent>child relation for the twl4030
>>>> and its subdevices is right.
>>>> 
>>>>> These sub-devices are much more independent of the whole than the regulator
>>>>> inside the w2sg0004 is.
>>>> 
>>>> Hm. That is probably argueable as well. At least a vague ciriterion.
>>>>> 
>>>>>> 
>>>>>> Or take the OMAP3 pbias_regulator. The OMAP3 isn’t a regulator as well
>>>>>> but has an internal pbias_regulator that needs to be controlled.
>>>>> 
>>>>> I don't know much about a "pbias_regulator", but the OMAP3 is a multifunction
>>>>> device which contains a CPU and multiple other controllers and  communicators
>>>>> and stuff.  One of the components in the OMAP3 is this pbias_regulator, and
>>>>> it has a device-node which is a child of the 'ocp' node - which is the main
>>>>> interconnect in the OMAP3 I think.
>>>>> 
>>>>> So I'm not against describing the regulator in the w2sg0004, but for
>>>>> consistency with everything else, it would need to be child-node of a
>>>>> device-node which describes the w2sg0004 as a whole.  I'm not sure that doing
>>>>> that would gain anything.
>>>> 
>>>> Well, it would gain consistency how chips with multiple components inside
>>>> are described.
>>> 
>>> I don't think so. Many devices do not expose all of their
>>> subdevices. e.g. they do not expose their internal clocks.
>>> Apart from that there are a few devices in the kernel, which
>>> do use enable-gpio(s).
>> 
>> After reading http://www.devicetree.org/Device_Tree_Usage#Devices I must
>> revise my formulation a little.
>> 
>> It is not that parent>child is used for describing “busses or grouping subdevices".
>> 
>> It is only for “grouping addressable components”, which implies some bus.
>> 
>> This still explains the ocp component of the OMAP3 - which represents the
>> whole internal “interconnect” bus (see comment in omap3.dtsi).
>> 
>> Therefore you are right. It is not correct to split up a device into arbitrary
>> components and group them again by the parent>child pattern.
>> 
>> Although the structure of twl4030.dtsi appears to simply list and group
>> subcomponents without telling explicitly how to address them.
> 
> Yes and this style is not only used for twl4030, but also for other
> multi function devices. I do not have a problem with splitting up a
> device into components as long as they are referenced by other
> components. There is no point in exposing all internal information.
> For the system its not relevant, that the GPS chip has a 30MHz
> clock if its part of the chip and nobody else is using it.
> 
>>>> And would make it easier to see the whole w2sg0004 a child of /.
>>> 
>>> Even with the regulator subdevice I would still make the w2sg0004 a
>>> child of the UART port.
>> 
>> This would mean that a tty / uart is able to address multiple clients. How
>> could it do that?
> 
> In theory it's possible:
> 
> ? -> uart -> multiplexer -> { device1, device2 }
> 
> I hope nobody will build such a device, though :)
> 
> I don't think addressability is important for a parent > child
> connection, though. It is important for a reg node, of course.

That is the key question I am trying to get founded answers, not opinions :)

> 
>>> Main reason is, that I would need to go
>>> through the UART to “communicate" with the w2sg0004.
>> 
>> You can always "communicate” through the UART. Even without DT. As
>> long as the connected chip is powered up by any means (could be
>> some fixed-regulator or hard wired).
>> 
>> Power control for open()/close() did even work with the DT based
>> DTR->virtual-gpio approach.
>> 
>> We just do not described in the DT that there is a data connection
>> from uart1 to the RX/TX pins. But you just open /dev/ttyO1.
>> 
>> Thus, describing a relation between the uart node and the GPS chip
>> in the DT is not even a requirement.
> 
> It is when the kernel should be able to handle the GPS. The DT is
> supposed to be independent from Linux and other operating systems
> may want to hide the tty if a GPS is attached to it and instead
> provide something like /dev/gps, which could automatically be picked
> up by userspace software.

Well, it is difficult to cover all future use cases.

But then you can still add a uart = <&uart1> reference to the DT node
of the driver. Because I understand that then the driver is the master
(and needs access to a hidden UART) and not the GPS chip a slave of
an exposed UART.

> 
>> Probably the main problem we have is that we want to describe this
>> data connection but the device tree is about describing how to
>> address components and data flow is left to internal APIs. Or
>> “ports” like in the video pipelines.
> 
> As I wrote the tree structure shows how to "reach" a component for
> accessing its configuration/data from the system.

Yes, you wrote and I already pointed out that I disagree. It contradicts
what

http://www.devicetree.org/Device_Tree_Usage#Devices

appears to tell me. So who is right?

> 
>> Let me raise the question:
>> 
>> Why do we need to describe in the DT (independently of Linux power
>> control structures and drivers!) that the GPS data interface is
>> connected to a specific UART?
>> Who needs this information and for which purpose?
> 
> Because an operating system can provide drivers, which completly
> hide away the UART connection. Example:
> 
> Let's assume we have an UART attached Bluetooth chip connected to
> the second UART.
> 
> In Linux, userspace currently must know, that the BT module is
> connected to this port and correctly setup the port for BT usage.
> Once this has been done the port is completly handled in the kernel
> and userspace can ignore the UART.

Ok, this is a reasonable example. But I still don’ see that it requires that the
bt chip must be a subnode of the uart.

> 
> So why must the user configure the port? Because the kernel does not
> know, that a BT chip is connected to the port.

Then, I would suggest to write it like you have done:

“the BT chip is connected to the port uart1”

This can be easily translated into DT language:

/ {
	bt {
		compatible = “vendor,chip”;
		uart = <&uart1>;	/* this chip is connected to uart1 */
		enable-gpio = <&gpio3 17 0>;
	};
};

This also resembles the “port” mechanism (just simplified because it is not
a backwards forward linkage).

Or exactly follows the pattern that a driver that wants to use some gpio simply
links to the gpio controller. Or a driver wants to control a gpio.

What I don’t understand is what is here so special to not use this DT pattern.

But It could also open a different solution for the tty open() -> w2sg power on problem.

/ {
	gps {
		compatible = “wi2wi,w2sg0004”;
		uart = <&uart1>;	/* this chip is connected to uart1 */
		enable-gpio = <&gpio3 17 0>;
	};
};

Here, the w2sg0004 probe function could register with uart1 to get notified
about open/close events and then do what it has to do.

In that case the w2sg0004 driver would not even need to present itself
as a virtual gpio (for DTR) or a regulator because it has a kernel internal
API to apply to tell the uart what it wants to do.

> If the kernel gets
> this information from DT it can do all the UART handling itself and
> provide a bluetooth device without any user interaction.

I think we should get a better understanding of the DT design patterns.

IMHO (or IMPoV):

* parent>child is not for describing “control” or “primary interface” but addressability
* if a driver needs access to another component, a reference property is used

BR,
Nikolaus



More information about the Gta04-owner mailing list