OverlayFS Porteus

[neko]

The Porteus space is composed of compressed squash files (.XZM)
and uses the AUFS tool to configure the space.
This trial changes this tool from AUFS to OV (overlay file system).

In the case of AUFS, it is possible to dynamically incorporate/remove XZM files to/from the Porteus space.
However, this is difficult for OV.
In this trial, this function is provided by "symbolic link" (ln -sf).
However, there are restrictions and problems can occur.
Therefore, this feature should be used temporarily.

Porteus has a change inheritance function with "EXIT".
This feature writes variations to the file on shutdown / reboot.
Communication is required between the init RAM space (boot space) and the Porteus normal space.
Its control is complex and unstable.
Therefore, in the OV trial, this "EXIT" function is deleted.
--------------------------------------------------------------------------------------------------------

Trial version of OverlayFS Porteus

1.Restriction of specifications
1.1. The "EXIT" of cheatcode "changes=EXIT" will be ignored.
That is that "changes=EXIT" is same as "changes".
1.2. The "activate/deactivate" command is provided as disabled or incomplete.
1.3. Activated XZMs during booting can not be deactivated.

2.Implementation
This is achieved by replacing initrd.xz and adding the 001-overlayAct.xzm module.
The initrd.xz and 001-overlayAct.xzm modules are included in
"Porteus Kernel Builder" version 22.03.24-noarch-2 and later.
See the first post in the "Porteus Kernel Builder" thread.

Code: Select all

% su
# ls
mkKernel-22.03.24-noarch-2.xzm
# mloop mkKernel-22.03.24-noarch-2.xzm
# cp /mnt/loop/usr/local/share/mkKernel/lib/upkernel/base/001-overlayAct.xzm .
# cp /mnt/loop/usr/local/share/mkKernel/lib/upkernel/syslinux/v5.0rc3.initrd.xz initrd.xz
# uloop
# ls -1
001-overlayAct.xzm
initrd.xz
mkKernel-22.03.24-noarch-2.xzm

2.1. replacement (update)
[initrd.xz] (in ISO/boot/syslinux/)
This initrd.xz is an integrated initrd.xz that can be used for both AUFS and OV.

2.1. additional
[001-overlayAct.xzm] Option
This module is required when using the "activate/deactivate" command for temporary use in OV.
If you put it in ISO/porteus/base or modules,
it will always be usable.
If it is stored in /porteus/optional,
it can be used only at startup by specifying the cheat code "load=overlay".

3. Example
3.1 v5.0
[Update v5.0 Porteus 64Bit ISO with OV kernel 6.0-rc1]

OV.PorteusV5.0-XFCE_ja-22.08.16-x86_64.iso (526 M)
https://www.mediafire.com/file/yvzmyakz ... x86_64.iso
md5sum: 0e1121ae9ec5d824a0974c7b7d6942b0 OV.PorteusV5.0-XFCE_ja-22.08.16-x86_64.iso

Language: Japanese & English
Web browser: chromium 104.0.5112.79-x86_64-1alien


3.2 v5.0rc3
[Update v5.0rc3 Porteus 32Bit/64Bit ISO with OV kernel 5.19-rc3]

== Prepare ==
(1) "Kernel Builder" that is more recent version than 22.03.24-1
Please refer to the first article of "Porteus Kernel Builder".

(2) Built OV kernel (64Bit 5.19-rc3)
OV.v5.19-rc3-64bit.tar.xz (27 M)
https://www.mediafire.com/file/sne1j7z6 ... bit.tar.xz
md5sum: 9788a582de0c4adabd4d053387c73cff OV.v5.19-rc3-64bit.tar.xz

== Perform ==
(1) expand "Built OV kernel" tar on "Kernel Builder" work place.

(2) GUI menue --> System --> Kernel Builder

"Kernel Builder" Input Window


"Kernel Builder" Confirm Window

Thanks.

-----------------------------------------
[updated on 2020.08.31] kernel of Porteus-XFCE-v5.0rc2-k5.9rc3-x86_64.iso was updated to 5.9-rc3.
[updated on 2020.09.07] kernel of Porteus-XFCE-v5.0rc2-k5.9rc4-x86_64.iso was updated to 5.9-rc4.
[updated on 2020.09.15] kernel of Porteus-XXXX-v5.0rc2-k5.9rc5-x86_64.iso was updated to kernel 5.9-rc5.
[updated on 2020.09.21] kernel of Porteus-XXXX-v5.0rc2-k5.9rc6-x86_64.iso was updated to 5.9-rc6.
[updated on 2020.09.28] kernel of Porteus-XXXX-v5.0rc2-k5.9rc7-x86_64.iso was updated to 5.9-rc7.
[updated on 2020.10.05] kernel of Porteus-XXXX-v5.0rc2-k5.9rc8-x86_64.iso was updated to 5.9-rc8.
[updated on 2020.10.13] kernel of Porteus-XXXX-v5.0rc2-k5.9.OV-x86_64.iso was updated to 5.9.
[updated on 2020.10.18] kernel of Porteus-XXXX-v5.0rc2-k5.9.1-x86_64.iso was updated to 5.9.1.
[updated on 2020.10.26] kernel of Porteus-XXXX-v5.0rc2-k5.10rc1-x86_64.iso was updated to 5.10-rc1.
[updated on 2020.11.12] kernel of Porteus-XXXX-v5.0rc2-k5.9.8-x86_64.iso was updated to 5.9.8.
[updated on 2020.12.24] kernel of Porteus-XXXX-v5.0rc2-k5.9.8-x86_64.iso was updated to 5.10.2.
[updated on 2022.03.17] release new implement.
[updated on 2022.03.28] get initrd from "Kernel Builder".
[updated on 2022.04.12] Example kernel 5.18-rc2.
[updated on 2022.04.18] Example kernel 5.18-rc3.
[updated on 2022.04.25] Example kernel 5.18-rc4.
[updated on 2022.05.04] Example kernel 5.18-rc5.
[updated on 2022.05.09] Example kernel 5.18-rc6.
[updated on 2022.05.16] Example kernel 5.18-rc7.
[updated on 2022.05.23] Example kernel 5.18.
[updated on 2022.05.30] Example kernel 5.18.1.
[updated on 2022.06.06] Example kernel 5.19-rc1.
[updated on 2022.06.13] Example kernel 5.19-rc2.
[updated on 2022.06.20] Example kernel 5.19-rc3.

[babam]

Hi neko,
Does it also support Aufs?
If yes, which is better to activate new modules with Aufs or Overlayfs.

[neko]

@babam
"Does it also support Aufs?"
---->
No, this implementation is only for OverlayFS.

If a kernel can be built with AUFS, it is much better to use the AUFS kernel.


Thanks.
-------------------------
[integrate initrd]
1. initrd.xz of original Porteus 5.0rc2
is only for AUFS.

2. initrd.zstd of Porteus-XFCE-v5.0rc2-k5.9rcN-x86_64.iso (64Bit Kernel 5.9-rcN without AUFS)
is only for OVFS.

3. initrd.xz of CROSS.Porteus-MATE-v4.0-k5.8.13-i586.iso (followed exaple ISO)
is integrated with
one for AUFS Porteus,
one for OVFS Porteus,
one for AUFS APorteus,
and
one for OVFS APorteus.

------------------------------
[Integrated initrd can make "32Bit Porteus" running under "64Bit AUFS kernel"]
CROSS.Porteus-MATE-v4.0-k5.4.72-i586.iso (409 M)
http://www.mediafire.com/file/a9rgk04ao ... 2-i586.iso
md5sum: c1311b07478ddabec5e35dfefdf09105 CROSS.Porteus-MATE-v4.0-k5.4.72-i586.iso

This ISO is Porteus-MATE-v4.0-i586 running on 64Bit Kernel 5.4.72 with AUFS.
Firefox version 81.0.1 is included.


Thanks.


-----------------------------------
[updated on 2020.08.28] CROSS.Porteus-XFCE-v4.0-k5.8.5-i586.iso is updated from k5.8.4.
[updated on 2020.09.03] CROSS.Porteus-XFCE-v4.0-k5.8.6-i586.iso is updated from k5.8.5.
[updated on 2020.09.05] CROSS.Porteus-XFCE-v4.0-k5.8.7-i586.iso is updated from k5.8.6.
[updated on 2020.09.07] Porteus-XFCE-v5.0rc2-k5.9rc4-x86_64.iso is updated from k5.9-rc3.
[updated on 2020.09.10] CROSS.Porteus-XFCE-v4.0-k5.8.8-i586.iso is updated from k5.8.7.
[updated on 2020.09.13] CROSS.Porteus-XFCE-v4.0-k5.8.9-i586.iso is updated from k5.8.8.
[updated on 2020.09.15] Porteus-XFCE-v5.0rc2-k5.9rc5-x86_64.iso is updated from k5.9-rc4.
[updated on 2020.09.18] CROSS.Porteus-XFCE-v4.0-k5.8.10-i586.iso is updated from k5.8.9.
[updated on 2020.09.24] CROSS.Porteus-XFCE-v4.0-k5.8.11-i586.iso is updated from k5.8.10.
[updated on 2020.09.27] CROSS.Porteus-XFCE-v4.0-k5.8.12-i586.iso is updated from k5.8.11.
[updated on 2020.10.02] new add CROSS.Porteus-MATE-v4.0-k5.8.13-i586.iso, delete CROSS.Porteus-XFCE-v4.0-k5.8.12-i586.iso
[updated on 2020.10.08] CROSS.Porteus-MATE-v4.0-k5.8.14-i586.iso is updated to kernel 5.8.14.
[updated on 2020.10.18] CROSS.Porteus-MATE-v4.0-k5.4.72-i586.iso is updated to kernel 5.4.72.

[burdi01]

@neko:

If a kernel can be built with AUFS, it is much better to use the AUFS kernel.

Would you be so kind as to elaborate? I am using overlayfs for several years now and imho it is at least on par with aufs.

[neko]

@burdi01
The most advantage of AUFS is an activate/deactivate function.
I implement that funcion under OverlayFS with "ln -sf" command.
I could not implement that funcion under OverlayFS with "mount" commnad.

Please refer to the top article.
OverlayFS Porteus (Post by neko #78485)
001-overlay/opt/porteus-scripts/xorg/aufs-insert
001-overlay/opt/porteus-scripts/xorg/aufs-remove
001-overlay/opt/porteus-scripts/xorg/ov.act.sh
001-overlay/opt/porteus-scripts/xorg/ov.deact.sh

You are using overlayfs for several years.
Would you tell me how to implement the activate/deactivate function ?

Thanks.

[babam]

The most advantage of AUFS is an activate/deactivate function.
I implement that funcion under OverlayFS with "ln -sf" command.
I could not implement that funcion under OverlayFS with "mount" commnad.

Can you please explain how to activate/deactivate a module using the command "ln - sf" under OverlayFS.

Thanks.

[neko]

@babam
If you have a time to read shell scripts, please refer to the top article.
OverlayFS Porteus (Post by neko #78485)
001-overlay/opt/porteus-scripts/xorg/aufs-insert
001-overlay/opt/porteus-scripts/xorg/aufs-remove
001-overlay/opt/porteus-scripts/xorg/ov.act.sh
001-overlay/opt/porteus-scripts/xorg/ov.deact.sh

Thanks.

----------------------------------------------------------------
1.prepare tools

Code: Select all

% su
# ls
Porteus-XFCE-v5.0rc2-k5.9rc5-x86_64.iso
# mloop Porteus-XFCE-v5.0rc2-k5.9rc5-x86_64.iso
# cp /mnt/loop/porteus/modules/atool.xzm .
# activate atool.xzm

2.how to switch to OVFS or AUFS

Code: Select all

# dXZM /mnt/loop/porteus/base/001-core.xzm
# dXZM /mnt/loop/porteus/base/001-overlay.xzm
# uloop
#
#
# diff 001-core/opt/porteus-scripts/xorg/aufs-insert 001-overlay/opt/porteus-scripts/xorg/
23,26c23,26
< if [ "`grep '^aufs / ' /proc/mounts`" = "" ]; then
<     echo "Not in the live mode, can't continue. Try xzm2dir $1 /"
<     exit 2
< fi
---
> #if [ "`grep '^aufs / ' /proc/mounts`" = "" ]; then
> #    echo "Not in the live mode, can't continue. Try xzm2dir $1 /"
> #    exit 2
> #fi
28c28
< if [ ! -f "$targetmod" ]; then
---
> if [ ! -f $targetmod ]; then
33,37c33
< LOADEDMODULES=$(cat /proc/mounts | cut -d" " -f2)
< LOADEDMODULES=${LOADEDMODULES//040/}
< LOADEDMODULES=${LOADEDMODULES//\\/ } 
< 
< if [[ "$LOADEDMODULES" =~ "$MOD" ]]; then
---
> if cat /proc/mounts | cut -d" " -f2 | grep -q "$MOD"; then
63c59,63
< mount -no remount,add:1:"$MOD"=rr aufs /
---
> if [ "`grep '^aufs / ' /proc/mounts`" = "" ]; then
> 	/opt/porteus-scripts/xorg/ov.act.sh $BASE
> else
> 	mount -no remount,add:1:"$MOD"=rr aufs /
> fi

(1)delete "only AUFS case"

23,26c23,26
< if [ "`grep '^aufs / ' /proc/mounts`" = "" ]; then
< echo "Not in the live mode, can't continue. Try xzm2dir $1 /"
< exit 2
< fi
---
> #if [ "`grep '^aufs / ' /proc/mounts`" = "" ]; then
> # echo "Not in the live mode, can't continue. Try xzm2dir $1 /"
> # exit 2
> #fi

(2)OVFS case, call /opt/porteus-scripts/xorg/ov.act.sh.

33,37c33
< LOADEDMODULES=$(cat /proc/mounts | cut -d" " -f2)
< LOADEDMODULES=${LOADEDMODULES//040/}
< LOADEDMODULES=${LOADEDMODULES//\\/ }
<
< if [[ "$LOADEDMODULES" =~ "$MOD" ]]; then
---
> if cat /proc/mounts | cut -d" " -f2 | grep -q "$MOD"; then
63c59,63
< mount -no remount,add:1:"$MOD"=rr aufs /
---
> if [ "`grep '^aufs / ' /proc/mounts`" = "" ]; then
> /opt/porteus-scripts/xorg/ov.act.sh $BASE
> else
> mount -no remount,add:1:"$MOD"=rr aufs /
> fi

Code: Select all

# diff 001-core/opt/porteus-scripts/xorg/aufs-remove 001-overlay/opt/porteus-scripts/xorg/
38c38
< [ -e "$MOD"/usr/share/applications ] && desktop=`find "$MOD"/usr/share/applications -name "*.desktop" 2>/dev/null`
---
> [ -e "$MOD"/usr/share/applications ] && desktop=`find $MOD/usr/share/applications -name "*.desktop" 2>/dev/null`
49c49,53
< mount -t aufs -o remount,del:"$MOD" aufs / 2>/dev/null
---
> if [ "`grep '^aufs / ' /proc/mounts`" = "" ]; then
> 	/opt/porteus-scripts/xorg/ov.deact.sh $targetmod
> else
> 	mount -t aufs -o remount,del:"$MOD" aufs / 2>/dev/null
> fi
62c66
< [ `losetup -a | grep "$1" | cut -d: -f1` ] && losetup -d `losetup -a | grep "$1" | cut -d: -f1`
---
> [ `losetup -a | grep $1 | cut -d: -f1` ] && losetup -d `losetup -a | grep $1 | cut -d: -f1`

(1)OVFS case, call /opt/porteus-scripts/xorg/ov.deact.sh.

49c49,53
< mount -t aufs -o remount,del:"$MOD" aufs / 2>/dev/null
---
> if [ "`grep '^aufs / ' /proc/mounts`" = "" ]; then
> /opt/porteus-scripts/xorg/ov.deact.sh $targetmod
> else
> mount -t aufs -o remount,del:"$MOD" aufs / 2>/dev/null
> fi

3.how to implement activate function under OVFS by using "ln -sf" command

Code: Select all

# cat 001-overlay/opt/porteus-scripts/xorg/ov.act.sh
#!/bin/sh

LOOPMNT=/mnt/live/memory/images
#LOOPMNT=/mnt

STACK=/mnt/live/tmp/stack
#STACK=/mnt/sda3/init/stack

expand()
{
local SAVE=$LANG
export LANG=C

find $LOOPMNT/$1/ | sed -e "s@$LOOPMNT/$1/@@" | sed -e "/^$/d"

export LANG=$SAVE
}

#=== START ===#
PKG=$1

if [ ! -d $STACK ]
then
	mkdir -p $STACK
fi

if [ -f $STACK/$PKG ]
then
	echo "already activate $PKG"
	exit 1
fi

#=== STEP 1 対象ピックアップ ===#
SKIP=""
LEN=0
for i in `expand $PKG`
do
	DEST="/${i%/*}"
	OBJ=${i##*/}

	if [ -n "$SKIP" ]
	then
		if [ "${i:0:$LEN}" = $SKIP ]
		then
		# 新規ディレクトリ内実態 #
			continue
		fi
		SKIP=""
	fi

	if [ -f $DEST/.act.new.$OBJ ]
	then
	# 同一実態へ別ACT重なり #
	# 前のACTの当重なり実態を無効化 #
		rm /$i
		rm $DEST/.act.new.$OBJ
		if [ -e $DEST/.act.org.$OBJ ]
		then
			mv $DEST/.act.org.$OBJ /$i
		fi
	fi 

	if [ -d $LOOPMNT/$PKG/$i ]
	then
	# ディレクトリをシンボリックリンクする #
	#--- directory ---#
		if [ -d /$i ]
		then
		# 元の実態はディレクトリ #
			continue
		fi
		# 新規ディレクトリ処理 #
		echo $i >> $STACK/$PKG
		SKIP=$i
		LEN=${#SKIP}
		continue
	fi
	echo $i >> $STACK/$PKG
done
	
#=== STEP 2 シンボリックリンク設定 ===#
if [ ! -f $STACK/$PKG ]
then
	exit 0
fi
for i in `cat $STACK/$PKG`
do
	DEST="/${i%/*}"
	OBJ=${i##*/}

	if [ -e /$i ]
	then
	# シンボリックリンク対象先に既に実態有り #
		mv /$i $DEST/.act.org.$OBJ
	fi

	# シンボリックリンク設定 #
	touch $DEST/.act.new.$OBJ
	ln -sf $LOOPMNT/$PKG/$i $DEST/.

done
	
exit 0	

4.how to implement deactivate function under OVFS by using "rm" & "mv" commands

Code: Select all

# cat 001-overlay/opt/porteus-scripts/xorg/ov.deact.sh
#!/bin/sh

LOOPMNT=/mnt/live/memory/images
#LOOPMNT=/mnt

STACK=/mnt/live/tmp/stack
#STACK=/mnt/sda3/init/stack


PKG=$1
#PKG=loop

if [ ! -f $STACK/$PKG ]
then
	echo "activated $PKG at boot"
	exit 1
fi
for i in `cat $STACK/$PKG`
do
	DEST="/${i%/*}"
	OBJ=${i##*/}
	if [ -f $DEST/.act.new.$OBJ ]
	then
	# シンボリックリンク設定処理有り #
		if [ -h /$i ]
		then
		# 設定したシンボリックリンク有り #
			rm -fr /$i
			rm $DEST/.act.new.$OBJ
			if [ -e $DEST/.act.org.$OBJ ]
			then
			# 元実態が有りセーブされている #
				mv $DEST/.act.org.$OBJ /$i
			fi
			# 正常ケース #
			continue
		fi

		if [ ! -e /$i ]
		then
		# 不正ケース 1 #
		# 何も無い #
			rm $DEST/.act.new.$OBJ
			if [ -e $DEST/.act.org.$OBJ ]
			then
			# 元実態が有りセーブされている #
				mv $DEST/.act.org.$OBJ /$i
			fi
			continue
		fi
		
		# 不正ケース 2 #
		# 設定したシンボリックリンク以外の物が有り #
		if [ -e $DEST/.act.org.$OBJ ]
		then
		# 元実態が有りセーブされている #
			rm -fr /$i
			mv $DEST/.act.org.$OBJ /$i
		fi
		rm $DEST/.act.new.$OBJ
		continue
	fi

	# 不正ケース 3 #
	# シンボリックリンク設定処理無し #
	if [ -e $DEST/.act.org.$OBJ ]
	then
	# 元実態が有りセーブされている #
		rm -fr /$i
		mv $DEST/.act.org.$OBJ /$i
	fi
done

rm $STACK/$PKG
	
exit 0	

Note:
Please read original
001-core/opt/porteus-scripts/xorg/aufs-insert and 001-core/opt/porteus-scripts/xorg/aufs-remove
first.

[burdi01]

@neko: The following is an excerpt from my email to the PartedMagic BDFL:

Patrick:

Indeed the problem with overlayfs as compared to aufs is that it is not possible to add bundles (.sqfm) to a running overlay file system. Ideally therefore one should include all bundles when the overlay file system is created.

I subdivide the bundles in three categories: always, sometimes and ad-hoc:
-- always: the bundles in /pmagic/pmodules/ are always included by init;
-- sometimes: the bundles in /pmagic/optionals/ are included by init when they are referenced in the "oload" kernel boot parameter *);
-- ad-hoc: /usr/bin/pmagic_load_sqfm "untars" its arguments just as is done with packages (.t?z) **).

*) On memory restrained boxes one might not always want to load "biggies" like libreoffice, gimp or vlc. Note that init's features are rather limited, hence the /pmagic/optionals/ requirement.

**) One might want to try a "wild" bundle downloaded from somewhere -- e.g. a Porteus bundle (.xzm). Or test the bundle one is developing.

The above is implemented in:
-- initrd/init - see above;
-- /etc/rc.d/rcS - comment out the sqfmlist() invocations;
-- /usr/bin/pmagic_load - see above;
-- /usr/bin/pmagic_save_session - create the 099-saved-session.sqfm file.

Deactivation is not implemented - I do not see a reason for it, just reboot if applicable.

[neko]

@burdi01
First, please understand that I mention about Porteus(/APorteus).

1.OverlayFS Porteus (Post by neko #78517)
If a kernel can be built with AUFS, it is much better to use the AUFS kernel.
--->
If a kernel can be built with AUFS, it is much better to use the AUFS kernel for Porteus's usage.

2.OverlayFS Porteus (Post by neko #78607)
You are using overlayfs for several years.
Would you tell me how to implement the activate/deactivate function ?
---> (my misunderstood)
You are using overlayfs in Porteus system for several years.
Would you tell me how to implement the activate/deactivate function in Porteus ?

------------------------------
Your kind answer is helpfull for introducing overlayFS into Porteus.
It is that a changing usage specification might make overlayFS useful as same as AUFS.


Thanks.

[burdi01]

Aah, I see the misunderstanding now:
I understood your remark "If a kernel can be built with AUFS, it is much better to use the AUFS kernel" as a non-distribution-specific remark, so when I said "I am using overlayfs for for several years now" I meant that as a non-distribution-specific remark too ...

PartedMagic (PM for short) is a Slackware-based live CD just as Porteus is - though for a different use case. IMHO including the "sometimes" bundles at overlayfs creation time makes activating/loading the "ad-hoc" bundles a rare event. But YMMV ...

[babam]

neko,
I am afraid to activate/deactivate the modules under Overlayfs when in persistence mode.

I'm afraid it will overwrite (ln -sf) and delete (rm -fr) files/directories that are contained in the persistent file (save.dat) or persistent device (e.g. /dev/sdb2).

Thanks.

[neko]

@babam
Please try Porteus-XFCE-v5.0rc2-k5.9rc5-x86_64.iso with persistence mode.
It will make your afraid clear.

Thanks.

[babam]

neko,
If I have '/bin/foo' (located inside a persistent device) then I activate a module which also contains '/bin/foo' will that overwrite (ln -sf) '/bin/foo'?

Update:
That will rename '/bin/foo' to '/bin/.act.org.foo'

Don't forget to deactivate that module before shutdown/reboot to restore '/bin/.act.org.foo' to '/bin/foo'

Thanks.

[neko]

@babam
Would you tell me the result of "Please try Porteus-XFCE-v5.0rc2-k5.9rc5-x86_64.iso with persistence mode.".

Thanks.

[babam]

neko,
Activating modules in persistence mode is works fine.

But we have to deactivate the modules before shutdown/reboot to restore them to their original state (removing symlinks and .act.new.* and restore .act.org.* to original name).

Thanks.