[Tinkerphones] makesd (was Re: [Gta04-owner] New LetuxOS Kernels and some tricks and thoughts)

H. Nikolaus Schaller hns at goldelico.com
Mon May 27 20:46:32 CEST 2019


> Am 27.05.2019 um 19:55 schrieb Paul Boddie <paul at boddie.org.uk>:
> 
> On Monday 27. May 2019 16.28.15 H. Nikolaus Schaller wrote:
>> 
>> Exactly. It is coming not from theory but practice and has evolved over
>> the years into a simple workhorse. Features were added where needed
>> and over the years some of the underlaying tools had changed.
>> 
>> So it basically is a macro processor which allows to describe how to
>> partition and fill an SD card with a bootable system for the most
>> common bootable SD card schemes we find for OMAP, i.MX, Broadcom processors.
> 
> I thought that the macro concept was a good one, allowing systems to be 
> derived from others and for different concerns to be mixed.
> 
>> It also knows abbreviations for many components for the SD card so that
>> nobody has to type http: links or run gunzip manually.
>> 
>> Key principle is: people need a useful SD card for first boot of a naked
>> device which may not have any external interfaces. So you can't bootstrap
>> in on the device itself.
> 
> I see that pre-built filesystems are downloaded, and I guess that they are 
> preconfigured. In my own efforts, I have tended to perform a debootstrap (or 
> equivalent) and then had the device perform the second stage itself, which 
> worked surprisingly well.

The pre-built images are just the result of a complete debootstrap in a fresh
chroot on some ARM or MIPS SBC. Either bare bone with some minimal set of required
or generally useful packages or with some more, e.g. a full GUI like Xfce or
QtMoko.

Configuration of the LetuxOS is now completely done by some letux-something.deb
packages. So no single file is touched (well, there is some fallback at the moment
that should be eliminated soon) besides through installing a .deb package. So
it can also be removed or replaced (there are usually preinst and postrm scripts
to make backups). The same for the kernel package.

It is also possible to take one of the preconfigured packages to boot a board
and then debootstrap something else on the initially running system. That can
be stored on some network server and the download link passed to makesd.

Even the SD card we use for the ARM or MIPS SBCs to debootstrap the images are
created by makesd... If such a card or SBC fails, we simply makesd a new one
from a stable version.

With this concept it did only need some initial and manual cross-debootstrap
and manual config (e.g. /etc/fstab) ca. 10 years ago to pull it up to where it
is now :)

> 
>> Most SBC vendors provide some .dd or .tgz files for this purpose which
>> need to be manually fetched, extracted and copied to the raw SD card.
>> In most cases the SD card size is fixed (e.g. 1GB) even if your destination
>> card is bigger. And for every board there is a different download location,
>> download and installation procedure - and finally every board has a
>> different password, different preinstalled packages, different kernel
>> version etc.
> 
> Yes, this kind of thing can be a mess, with mostly the same things being done 
> slightly differently every time.
> 
>> This is what makesd (and LetuxOS) wants to get rid of. Have a single OS
>> that runs everywhere (almost) the same. Like macOS or iOS or Windows can
>> be installed on any current hardware without need for "configuration".
> 
> Or GNU/Linux on Intel-based systems.

Yes. They are fortunately homogenous enough so that it is much easier to
achieve.

> 
>>> https://hg.boddie.org.uk/remakesd
>> 
>> I have looked into it but did not get the feeling that it is easier to
>> maintain than a single script file which is broken up into shell functions.
>> With multiple files, you have to open them separately to get the overall
>> structure. Maybe that depends on the source code editor you have...
> 
> I just use vim. :-) But it is something I try and do with shell scripts: to 
> separate out each of the different tasks and to formalise some interfaces 
> between them.
> 
> Typically, I am very cautious about big scripts that are downloaded and run, 
> particularly if they need to acquire elevated privileges. Separating things 
> into separate scripts allows the parts which need elevated privileges to be 
> confined, leaving the other parts to do their things under normal privileges.
> 
> Another thing that I wanted to explore was the macro expansion. By having a 
> separate script just concerned with that, it becomes possible to ask that 
> script to summarise its understanding of the board definition or macro given 
> to it. For example:
> 
> ./makesd-expand-def lc8
> 
> -f fat -s 5 -b Letux-Cortex-8/latest -k latest -d latest -f ext4 ...
> 
> And these arguments can be presented to another script to see what the 
> partitioning scheme will be for the chosen board:
> 
> ./makesd-expand-def lc8 | xargs ./makesd-partition-summary
> 
> -       5       fat
> -       95      ext4
> 
> Since this kind of summary is usable by the partitioning tool and the 
> formatting tool, those tools only need to handle such a summary. Consequently, 
> if someone prefers this kind of summary for whatever reason, they can ignore 
> the macro expansion and the different arguments it produces, writing their own 
> summaries by hand instead.
> 
> I haven't thought about how the kernel, device tree, root filesystem and other 
> arguments are handled, but they may end up in the summary format, too. The 
> partitioning and formatting tools will ignore such information, just as the 
> downloading tool will ignore the partitioning details.
> 
>> Makesd just defines a set of shell functions and then runs through the main
>> code in a predetermined sequence (parse arguments, setup partitions, format
>> the partitions, extract files into the partitions, fsck and umount
>> everything).
>> 
>> So it is just an SD card baking recipe with some common subroutines.
> 
> Indeed. I am not criticising the approach: there are plenty of advantages in 
> using a single script and by using shell functions instead of separate 
> scripts.
> 
>> And it is tied closely to a server infrastructure which provides the files
>> to be installed at certain well known locations.
> 
> Another motivation for using separate scripts is to be able to separate the 
> downloading activities from the deployment activities. Sometimes downloading 
> isn't necessary because the files are already present; sometimes the download 
> process needs to be attempted more than once. And it can be the case that the 
> downloading is the most interesting part for some people and that they just 
> use the tool to automate that process.

Yes, I noticed that you are trying to make it a little more object-oriented by
this approach, especially in the aspect of modularization and encapsulation
of implementations.

Unfortunately the bash isn't the best OO language :)

Although there are objects (input files), methods (scripts called to process the
input files) and return values (output files). But there are no classes (scripts
dependent on input file types?) and no polymorphism and inheritance.

Some years ago I had thought about some "oosh" where standard shell syntax is a
subset, but did not come to a nice and convincing concept. So I focussed on
other projects.

> 
>> BTW: maintaining makesd is a recreational walk compared to maintaining
>> kernel drivers that are not upstream and where others break in-kernel APIs
>> without documentations besides the code or similar drivers...
> 
> I completely agree with you here!
> 
>>> Eventually, I will get round to looking at more substantial things, of
>>> course.
>> The key question I have is what you think is missing or wrong with makesd
>> that you did spin your own or did you just want to learn something? Which
>> is mostly my motivation to do something myself, even if similar things
>> exist. We are tinkerers...
> 
> I probably gave most of my reasons above. I don't think there is anything 
> wrong with makesd as such, but I do tend to put an emphasis on having 
> collections of scripts that can be combined as desired.

Yes, that is basically a good idea.

> 
> In a previous job, I had to automate data processing pipelines, and it was 
> useful to break up the activities into pieces that could be expected to fail 
> and therefore be retried, potentially under supervision, or to run to 
> completion autonomously, only failing if a genuine error condition occurred.
> 
> The aim was to avoid awkward situations where anticipated unreliability 
> (downloading stuff) might infect parts of the system that were anticipated to 
> be reliable, leaving people with no other option but to just run things over 
> and over again and hope that something wouldn't fail at a critical point and 
> waste hours of processing time.
> 
> Of course, there is also the motivation to investigate the problem domain and 
> to learn things, even though I now feel that I know things about sfdisk that 
> no-one should really need to learn.

:)

BR and thanks,
Nikolaus



More information about the Community mailing list