[Tinkerphones] makesd (was Re: [Gta04-owner] New LetuxOS Kernels and some tricks and thoughts)

Paul Boddie paul at boddie.org.uk
Mon May 27 19:55:01 CEST 2019


On Monday 27. May 2019 16.28.15 H. Nikolaus Schaller wrote:
> 
> Exactly. It is coming not from theory but practice and has evolved over
> the years into a simple workhorse. Features were added where needed
> and over the years some of the underlaying tools had changed.
> 
> So it basically is a macro processor which allows to describe how to
> partition and fill an SD card with a bootable system for the most
> common bootable SD card schemes we find for OMAP, i.MX, Broadcom processors.

I thought that the macro concept was a good one, allowing systems to be 
derived from others and for different concerns to be mixed.

> It also knows abbreviations for many components for the SD card so that
> nobody has to type http: links or run gunzip manually.
> 
> Key principle is: people need a useful SD card for first boot of a naked
> device which may not have any external interfaces. So you can't bootstrap
> in on the device itself.

I see that pre-built filesystems are downloaded, and I guess that they are 
preconfigured. In my own efforts, I have tended to perform a debootstrap (or 
equivalent) and then had the device perform the second stage itself, which 
worked surprisingly well.

> Most SBC vendors provide some .dd or .tgz files for this purpose which
> need to be manually fetched, extracted and copied to the raw SD card.
> In most cases the SD card size is fixed (e.g. 1GB) even if your destination
> card is bigger. And for every board there is a different download location,
> download and installation procedure - and finally every board has a
> different password, different preinstalled packages, different kernel
> version etc.

Yes, this kind of thing can be a mess, with mostly the same things being done 
slightly differently every time.

> This is what makesd (and LetuxOS) wants to get rid of. Have a single OS
> that runs everywhere (almost) the same. Like macOS or iOS or Windows can
> be installed on any current hardware without need for "configuration".

Or GNU/Linux on Intel-based systems.

> > https://hg.boddie.org.uk/remakesd
> 
> I have looked into it but did not get the feeling that it is easier to
> maintain than a single script file which is broken up into shell functions.
> With multiple files, you have to open them separately to get the overall
> structure. Maybe that depends on the source code editor you have...

I just use vim. :-) But it is something I try and do with shell scripts: to 
separate out each of the different tasks and to formalise some interfaces 
between them.

Typically, I am very cautious about big scripts that are downloaded and run, 
particularly if they need to acquire elevated privileges. Separating things 
into separate scripts allows the parts which need elevated privileges to be 
confined, leaving the other parts to do their things under normal privileges.

Another thing that I wanted to explore was the macro expansion. By having a 
separate script just concerned with that, it becomes possible to ask that 
script to summarise its understanding of the board definition or macro given 
to it. For example:

./makesd-expand-def lc8

-f fat -s 5 -b Letux-Cortex-8/latest -k latest -d latest -f ext4 ...

And these arguments can be presented to another script to see what the 
partitioning scheme will be for the chosen board:

./makesd-expand-def lc8 | xargs ./makesd-partition-summary

-       5       fat
-       95      ext4

Since this kind of summary is usable by the partitioning tool and the 
formatting tool, those tools only need to handle such a summary. Consequently, 
if someone prefers this kind of summary for whatever reason, they can ignore 
the macro expansion and the different arguments it produces, writing their own 
summaries by hand instead.

I haven't thought about how the kernel, device tree, root filesystem and other 
arguments are handled, but they may end up in the summary format, too. The 
partitioning and formatting tools will ignore such information, just as the 
downloading tool will ignore the partitioning details.

> Makesd just defines a set of shell functions and then runs through the main
> code in a predetermined sequence (parse arguments, setup partitions, format
> the partitions, extract files into the partitions, fsck and umount
> everything).
> 
> So it is just an SD card baking recipe with some common subroutines.

Indeed. I am not criticising the approach: there are plenty of advantages in 
using a single script and by using shell functions instead of separate 
scripts.

> And it is tied closely to a server infrastructure which provides the files
> to be installed at certain well known locations.

Another motivation for using separate scripts is to be able to separate the 
downloading activities from the deployment activities. Sometimes downloading 
isn't necessary because the files are already present; sometimes the download 
process needs to be attempted more than once. And it can be the case that the 
downloading is the most interesting part for some people and that they just 
use the tool to automate that process.

> BTW: maintaining makesd is a recreational walk compared to maintaining
> kernel drivers that are not upstream and where others break in-kernel APIs
> without documentations besides the code or similar drivers...

I completely agree with you here!

> > Eventually, I will get round to looking at more substantial things, of
> > course.
> The key question I have is what you think is missing or wrong with makesd
> that you did spin your own or did you just want to learn something? Which
> is mostly my motivation to do something myself, even if similar things
> exist. We are tinkerers...

I probably gave most of my reasons above. I don't think there is anything 
wrong with makesd as such, but I do tend to put an emphasis on having 
collections of scripts that can be combined as desired.

In a previous job, I had to automate data processing pipelines, and it was 
useful to break up the activities into pieces that could be expected to fail 
and therefore be retried, potentially under supervision, or to run to 
completion autonomously, only failing if a genuine error condition occurred.

The aim was to avoid awkward situations where anticipated unreliability 
(downloading stuff) might infect parts of the system that were anticipated to 
be reliable, leaving people with no other option but to just run things over 
and over again and hope that something wouldn't fail at a critical point and 
waste hours of processing time.

Of course, there is also the motivation to investigate the problem domain and 
to learn things, even though I now feel that I know things about sfdisk that 
no-one should really need to learn.

Paul


More information about the Community mailing list