[Tinkerphones] makesd (was Re: [Gta04-owner] New LetuxOS Kernels and some tricks and thoughts)
paul at boddie.org.uk
Mon May 27 19:55:01 CEST 2019
On Monday 27. May 2019 16.28.15 H. Nikolaus Schaller wrote:
> Exactly. It is coming not from theory but practice and has evolved over
> the years into a simple workhorse. Features were added where needed
> and over the years some of the underlaying tools had changed.
> So it basically is a macro processor which allows to describe how to
> partition and fill an SD card with a bootable system for the most
> common bootable SD card schemes we find for OMAP, i.MX, Broadcom processors.
I thought that the macro concept was a good one, allowing systems to be
derived from others and for different concerns to be mixed.
> It also knows abbreviations for many components for the SD card so that
> nobody has to type http: links or run gunzip manually.
> Key principle is: people need a useful SD card for first boot of a naked
> device which may not have any external interfaces. So you can't bootstrap
> in on the device itself.
I see that pre-built filesystems are downloaded, and I guess that they are
preconfigured. In my own efforts, I have tended to perform a debootstrap (or
equivalent) and then had the device perform the second stage itself, which
worked surprisingly well.
> Most SBC vendors provide some .dd or .tgz files for this purpose which
> need to be manually fetched, extracted and copied to the raw SD card.
> In most cases the SD card size is fixed (e.g. 1GB) even if your destination
> card is bigger. And for every board there is a different download location,
> download and installation procedure - and finally every board has a
> different password, different preinstalled packages, different kernel
> version etc.
Yes, this kind of thing can be a mess, with mostly the same things being done
slightly differently every time.
> This is what makesd (and LetuxOS) wants to get rid of. Have a single OS
> that runs everywhere (almost) the same. Like macOS or iOS or Windows can
> be installed on any current hardware without need for "configuration".
Or GNU/Linux on Intel-based systems.
> > https://hg.boddie.org.uk/remakesd
> I have looked into it but did not get the feeling that it is easier to
> maintain than a single script file which is broken up into shell functions.
> With multiple files, you have to open them separately to get the overall
> structure. Maybe that depends on the source code editor you have...
I just use vim. :-) But it is something I try and do with shell scripts: to
separate out each of the different tasks and to formalise some interfaces
Typically, I am very cautious about big scripts that are downloaded and run,
particularly if they need to acquire elevated privileges. Separating things
into separate scripts allows the parts which need elevated privileges to be
confined, leaving the other parts to do their things under normal privileges.
Another thing that I wanted to explore was the macro expansion. By having a
separate script just concerned with that, it becomes possible to ask that
script to summarise its understanding of the board definition or macro given
to it. For example:
-f fat -s 5 -b Letux-Cortex-8/latest -k latest -d latest -f ext4 ...
And these arguments can be presented to another script to see what the
partitioning scheme will be for the chosen board:
./makesd-expand-def lc8 | xargs ./makesd-partition-summary
- 5 fat
- 95 ext4
Since this kind of summary is usable by the partitioning tool and the
formatting tool, those tools only need to handle such a summary. Consequently,
if someone prefers this kind of summary for whatever reason, they can ignore
the macro expansion and the different arguments it produces, writing their own
summaries by hand instead.
I haven't thought about how the kernel, device tree, root filesystem and other
arguments are handled, but they may end up in the summary format, too. The
partitioning and formatting tools will ignore such information, just as the
downloading tool will ignore the partitioning details.
> Makesd just defines a set of shell functions and then runs through the main
> code in a predetermined sequence (parse arguments, setup partitions, format
> the partitions, extract files into the partitions, fsck and umount
> So it is just an SD card baking recipe with some common subroutines.
Indeed. I am not criticising the approach: there are plenty of advantages in
using a single script and by using shell functions instead of separate
> And it is tied closely to a server infrastructure which provides the files
> to be installed at certain well known locations.
Another motivation for using separate scripts is to be able to separate the
downloading activities from the deployment activities. Sometimes downloading
isn't necessary because the files are already present; sometimes the download
process needs to be attempted more than once. And it can be the case that the
downloading is the most interesting part for some people and that they just
use the tool to automate that process.
> BTW: maintaining makesd is a recreational walk compared to maintaining
> kernel drivers that are not upstream and where others break in-kernel APIs
> without documentations besides the code or similar drivers...
I completely agree with you here!
> > Eventually, I will get round to looking at more substantial things, of
> > course.
> The key question I have is what you think is missing or wrong with makesd
> that you did spin your own or did you just want to learn something? Which
> is mostly my motivation to do something myself, even if similar things
> exist. We are tinkerers...
I probably gave most of my reasons above. I don't think there is anything
wrong with makesd as such, but I do tend to put an emphasis on having
collections of scripts that can be combined as desired.
In a previous job, I had to automate data processing pipelines, and it was
useful to break up the activities into pieces that could be expected to fail
and therefore be retried, potentially under supervision, or to run to
completion autonomously, only failing if a genuine error condition occurred.
The aim was to avoid awkward situations where anticipated unreliability
(downloading stuff) might infect parts of the system that were anticipated to
be reliable, leaving people with no other option but to just run things over
and over again and hope that something wouldn't fail at a critical point and
waste hours of processing time.
Of course, there is also the motivation to investigate the problem domain and
to learn things, even though I now feel that I know things about sfdisk that
no-one should really need to learn.
More information about the Community