mergexzm - merge folder of modules and slack packages

Here is a place for your projects which are not officially supported by the Porteus Team. For example: your own kernel patched with extra features; desktops not included in the standard ISO like Gnome; base modules that are different than the standard ISO, etc...
User avatar
freestyler
Contributor
Contributor
Posts: 384
Joined: 17 Oct 2013, 14:21
Distribution: Porteus XFCE

mergexzm - merge folder of modules and slack packages

Post#1 by freestyler » 01 Jun 2014, 01:03

This script merges a folder of modules and slackware packages into one module.
usage:
follow the prompts with:

Code: Select all

mergexzm

or

Code: Select all

mergexzm /path/to/folder/of/modules modulename 
example:

Code: Select all

mergexzm /home/guest/xfcemodules xfce
xfce.xzm will be created in /tmp/mergexzm

download: mergexzm-1.0-no-arch_fs.xzm
https://www.porteus-apps.org

User avatar
freestyler
Contributor
Contributor
Posts: 384
Joined: 17 Oct 2013, 14:21
Distribution: Porteus XFCE

Re: mergexzm - merge folder of modules and slack packages

Post#2 by freestyler » 02 Jun 2014, 00:12

Ive updated the script to include a gui. Here are some screenshots...

If you run 'mergexzm' it will ask for a folder or modules and a name.
Image

It can also be used as 'mergexzm /path/to/folder/of/modules modulename'
Image

The gui is run using 'mergeusm -g'
Image

You can choose modules from various locations and merge them.It also merges tgz and txz slackware packages
Image

The gui can also be used to merge a folder of modules and slackware packages.
Image

Download: mergexzm-1.1-no-arch_fs.xzm

Code: Select all

#!/bin/bash

# mergexzm merges a folder of xzm modules and slackware packages into one module
# for graphical user interface use: 'mergexzm -g'
# usage: mergexzm /path/to/folder/of/modules modulename
# example: mergexzm /home/guest/xfcemodules xfce
# xfce.xzm will be created in /tmp/mergexzm

# version 1.1

# script by freestyler <mail@porteus-apps.org>


# make sure user has root access

if [ $UID -ne 0 ]; then
	echo; echo "you must have root access to merge modules."; echo; exit; 
fi

helptext(){ 
	echo ""
	echo "==============================================================================="
	echo "!                                  mergexzm                                   !"
	echo "==============================================================================="
	echo ""
	echo "                                  version 1.1"
	echo ""
	echo "                  mergexzm merges a folder of xzm modules and" 
	echo "                       slackware packages into one module"
	echo ""
	echo "	        for graphical user interface use: 'mergexzm -g'"
	echo ""
	echo "             usage: 'mergexzm /path/to/folder/of/modules modulename'"
	echo ""
	echo "                example: mergexzm /home/guest/xfcemodules xfce"
	echo "                  xfce.xzm will be created in /tmp/mergexzm"
	echo ""
	echo "==============================================================================="
	echo "!                 script by freestyler <mail@porteus-apps.org>                !"
	echo "==============================================================================="
	echo ""
}

start(){
	echo ""
	echo "==============================================================================="
	echo "!                                  mergexzm                                   !"
	echo "==============================================================================="
	echo ""
	echo "                  mergexzm merges a folder of xzm modules and" 
	echo "                       slackware packages into one module"
	echo ""
	echo "                           for help use: 'mergexzm -h'"
	echo ""
	echo "	        for graphical user interface use: 'mergexzm -g'"
	echo ""
}

processfiles(){
	cd /tmp/mergexzm/merged
	echo; echo "Processing files from "$directory""
	echo ""
	for file in *; do
		case `echo $file |awk -F . '{if (NF>1) {print $NF}}'` in
		txz*)  # Convert and merge txz files
			echo "$file is being converted"
			txz2xzm $file &>/dev/null
 	      ;;
		tgz*)  # Convert and merge txz files
			echo "$file is being converted"
		        tgz2xzm $file &>/dev/null
              ;;		
		xzm*)  # xzm module		
 	      ;;
		*) # Anything else
			echo "$file is not a valid module or slackware package."
			continue
              ;;
	esac
	done
}

processxzm(){
	echo ""
	echo "Modules to be merged:"
	cd /tmp/mergexzm/merged
	for file in *; do
	case `echo $file |awk -F . '{if (NF>1) {print $NF}}'` in
		xzm*)  # xzm module
			echo "$file"
			xzm2dir $file /tmp/mergexzm/tmp/ &>/dev/null
              		;;
	esac
	done
}

merge(){
	modulename="$mergeNAME"
	echo "$mergeNAME"
	processfiles
	processxzm
	createmodule
	cleanup
}

dothatshit(){
	cleanup
	cat /tmp/mergexzm/list.txt |
	while read a; do
	cp $a /tmp/mergexzm/merged
	done
	gtkdialog --program=TERMINAL2
}

clearlist(){
	echo > /tmp/mergexzm/list.txt
	rm -rf /tmp/mergexzm/list.txt
	cat > /tmp/mergexzm/list.txt &
}

createmodule(){
	cd /tmp/mergexzm
	echo ""
	echo "==============================================================================="
	echo "!                               Creating module                               !"
	echo "==============================================================================="
	echo ""
	dir2xzm /tmp/mergexzm/tmp "$modulename".xzm
	echo ""
	echo "==============================================================================="
	echo "!                                  Complete!                                  !"
	echo "==============================================================================="
	echo ""
	echo ""$modulename".xzm has been created in /tmp/mergexzm"
	echo ""
}

cleanup(){
	if [[ ! -d "/tmp/mergexzm" ]]; then 
		mkdir /tmp/mergexzm
	fi

	if [[ -d "/tmp/mergexzm/tmp" ]]; then
		rm -rf /tmp/mergexzm/tmp
		mkdir /tmp/mergexzm/tmp
		else
		mkdir /tmp/mergexzm/tmp
	fi

	if [[ -d "/tmp/mergexzm/merged" ]]; then
		rm -rf /tmp/mergexzm/merged
		mkdir /tmp/mergexzm/merged
	fi
}

export MERGEGUI='
<window window_position="1" title="mergexzm" width_request="380" >
  <vbox>   
    <button space-expand="false" space-fill="false">
      <label>Select Modules To Merge</label>
      <action>gtkdialog --program=SELECTMODULES</action></button>
    <button space-expand="false" space-fill="false">
      <label>Select A Folder Of Modules to Merge</label>
      <action>gtkdialog --program=SELECTFOLDER</action></button>
    <button space-expand="false" 
      space-fill="false">
      <label>Exit</label></button>
  </vbox>
</window>
'

export SELECTMODULES='
<window window_position="1" title="Select modules to merge" width_request="540" height_request="380">
  <vbox>
    <tree auto-refresh="true">
      <label>Modules to be merged</label>
      <height>100</height>
      <width>200</width>
      <variable>mergeLIST</variable>
      <input file>/tmp/mergexzm/list.txt</input>
    </tree>
    <hbox>
      <button><label>Clear list</label>
        <action>clearlist</action></button>
      <entry fs-action="file"
	fs-title="Select a file"
	fs-filters="*.'xzm'|*.'tgz'|*.'txz'"
	fs-folder="'$HOME'"
	editable="true">
        <variable>selectMODULE</variable></entry>
      <button><label>Browse</label>
	<action function="fileselect">selectMODULE</action></button>
      <button><label>Add</label>
	<action>echo "$selectMODULE" >> /tmp/mergexzm/list.txt</action></button>
    </hbox>
    <hbox><text><label>Module Name:</label></text>
      <entry editable="true">
        <variable>mergeNAME</variable></entry>
      <button space-expand="false" space-fill="false">
        <label>Merge</label>
        <action>dothatshit</action></button>
      <button space-expand="false" space-fill="false">
        <label>Cancel</label></button>
    </hbox>
  </vbox>
</window>
'

export SELECTFOLDER='
<window window_position="1" title="Select folder to merge" width_request="380" >
  <vbox>   
    <text><label>This will merge all xzm modules and slackware packages contained in selected folder. Please choose a name for the module</label></text>
      <hbox>	
	<text space-expand="false" space-fill="false">
	  <label>Name:</label></text>
	<entry editable="true">
          <variable>mergeNAME</variable></entry>
      </hbox>
      <hbox>	
        <entry fs-action="folder"
	  space-expand="true"
	  space-fill="true"
	  fs-title="Select a folder"
	  fs-folder="'/home/guest/'"
	  editable="true">
          <default>/path/to/folder/containing/modules</default>
          <variable>mergeDIR</variable></entry>
        <button space-expand="false" space-fill="false">
	  <label>Browse</label>
	  <action function="fileselect">mergeDIR</action></button>
      </hbox>
    <button space-expand="false" space-fill="false">
      <label>Merge</label>
      <variable>butselFile</variable>
      <action>gtkdialog --program=TERMINAL</action></button>
    <button space-expand="false" space-fill="false">
      <label>Cancel</label></button>
  </vbox>
</window>
'

export TERMINAL='
<window window_position="1" title="Merging Modules" width_request="680" height_request="400" icon-name="media-playback-start">
  <vbox>
    <hbox>
      <terminal tag_attr="value" vscrollbar-policy="0">  
        <input>echo mergexzm "$mergeDIR" "$mergeNAME"</input>
      </terminal>
    </hbox>
  </vbox>
</window>
'

export TERMINAL2='
<window window_position="1" title="Merging Modules" width_request="680" height_request="400" icon-name="media-playback-start">
  <vbox>
    <hbox>
      <terminal tag_attr="value" vscrollbar-policy="0">  
        <input>echo merge</input>
      </terminal>
    </hbox>
  </vbox>
</window>
'

export -f merge
export -f clearlist
export -f dothatshit
export -f cleanup
export -f processfiles
export -f processxzm
export -f createmodule
export -f start
export -f helptext

# set help screen and start script

[[ $@ ]] || { start; }

while getopts "hg" opt; do
  case $opt in
    h) helptext >&2
      exit 1
    ;;
    g) gtkdialog --program=MERGEGUI >&2
      exit 1
    ;;
    \?) echo "for help use: mergexzm -h" >&2
      exit 1
    ;;
    :) echo "Option -$OPTARG requires an argument." >&2
      exit 1
    ;;
    *) start
      exit 1	    
    ;;
  esac
done

# check if user has entered variables

if [ -z "$1" ]; then
	echo "Enter location of folder containing modules you wish to merge:"
	read directory
	echo "Modules in "$directory" will be merged"
 	else
	export directory="$1"
fi

if [ -z "$2" ]; then
	echo "Please choose a name for your module:"
	read modulename
	echo "Module will be named "$modulename".xzm"
 	else
	export modulename="$2"
fi

# clean temp files and create needed temporary folders

cleanup

# copy contents of directory to temporary loctation

cp $(find $directory -type f | grep -iE '\.(xzm|tgz|txz)$') /tmp/mergexzm/merged

# convert slackware packages to modules and then extract modules to temp directory

processfiles
processxzm

# create xzm module

createmodule
cleanup
https://www.porteus-apps.org

User avatar
brokenman
Site Admin
Site Admin
Posts: 6105
Joined: 27 Dec 2010, 03:50
Distribution: Porteus v4 all desktops
Location: Brazil

Re: mergexzm - merge folder of modules and slack packages

Post#3 by brokenman » 02 Jun 2014, 02:49

Nice contribution. Will come in handy.
How do i become super user?
Wear your underpants on the outside and put on a cape.

User avatar
freestyler
Contributor
Contributor
Posts: 384
Joined: 17 Oct 2013, 14:21
Distribution: Porteus XFCE

Re: mergexzm - merge folder of modules and slack packages

Post#4 by freestyler » 02 Jun 2014, 14:42

Fixed an initialization problem (was not creating a necessary temp file when using gui)
I updated the download file
https://www.porteus-apps.org

Payoon
Black ninja
Black ninja
Posts: 88
Joined: 01 Mar 2013, 19:16
Distribution: Porteus 3.2 32 bit XFCE
Location: Duisburg, Germany

Re: mergexzm - merge folder of modules and slack packages

Post#5 by Payoon » 08 Jun 2014, 12:35

Hi Freestyler,
thanks a lot for this nice little programm. It's very useful. When I use the built-in merge function of porteus the path is to /porteus/modules, where other active modules are and to keep things separate is sometimes tricky. With merge2xzm I can create a folder with the new module name, throw all my converted dependency deb's and txz's into it, open a terminal inside this folder et voila! a new module_alldeps.xzm is born. Accordingly I have merged it into my own myutilites.xzm for everyday use and things work fine.
Payoon

User avatar
francois
Contributor
Contributor
Posts: 6438
Joined: 28 Dec 2010, 14:25
Distribution: xfce plank porteus nemesis
Location: Le printemps, le printemps, le printemps... ... l'hiver s'essoufle.

Re: mergexzm - merge folder of modules and slack packages

Post#6 by francois » 12 Jun 2014, 02:54

This is a nice tool. I will use it for sure.

Thanks.
Prendre son temps, profiter de celui qui passe.

User avatar
francois
Contributor
Contributor
Posts: 6438
Joined: 28 Dec 2010, 14:25
Distribution: xfce plank porteus nemesis
Location: Le printemps, le printemps, le printemps... ... l'hiver s'essoufle.

Re: mergexzm - merge folder of modules and slack packages

Post#7 by francois » 10 Jan 2016, 16:18

1) It does not work for me:

Code: Select all

root@porteus:~# mergexzm /mnt/sda5/64bit_v31rc1/porteus/optional/gimp-2.8.6 /tmp/gimp-2.8.6.xzm
I get a module but with nothing in it.

2) the gui is not functioning either:

Code: Select all

root@porteus:~# mergeusm -g
bash: mergeusm: command not found
Prendre son temps, profiter de celui qui passe.

donald
Full of knowledge
Full of knowledge
Posts: 2081
Joined: 17 Jun 2013, 13:17
Distribution: Porteus 3.2.2 XFCE 32bit
Location: Germany

Re: mergexzm - merge folder of modules and slack packages

Post#8 by donald » 10 Jan 2016, 17:41

^
# for graphical user interface use: 'mergexzm -g'
:)

port
Samurai
Samurai
Posts: 137
Joined: 18 Feb 2016, 09:25
Distribution: Linux porteus 3.2.2 KDE
Location: Spain

Re: mergexzm - merge folder of modules and slack packages

Post#9 by port » 12 Mar 2016, 02:26

A potencial problem I see is order of merging. It seems you merge the modules in the order you read them from disk but this may not be the right idea because you can't control which module hides which else and that's not good.

There's not a right solution for the problem as the process being automated but taking into account date would probably be a good idea and also a kind of sorting based in file names will give the opportunity to user to stablish a desired order of appliance, it would be great if this were available as parameter or config

For reasons like that I think a better solution is to manually unpack the files to dir in the order you wish (maybe after analyzing contents) and tweaking manually dir contents to be sure it's all ok and then build the module from dir. Maybe another good options for the tools is automate the process of unpacking without building the module after user validation, maybe dividing the script in several stateful stages in the *ctl way with commands to init each stage (i.e. git, mercurial, sysctl...)

rych
Warlord
Warlord
Posts: 685
Joined: 04 Jan 2014, 04:27
Distribution: Porteus 5.0 x64 OpenBox
Location: NZ
Contact:

mergexzm - merge folder of modules and slack packages

Post#10 by rych » 17 Jun 2024, 07:03

porteux, thanks for adding the mergexzm to PorteuX 1.4. From the performance (of SquashFS, aufs mounts, ...) point of view, is it batter to have 20 small .xzms or one large .XZM module?

Vic
Samurai
Samurai
Posts: 132
Joined: 10 Aug 2016, 05:36
Distribution: Porteux V-0.1 64 KDE
Location: Utopia in Tampa, Florida, USA

mergexzm - merge folder of modules and slack packages

Post#11 by Vic » 17 Jun 2024, 12:58

Good question Rych. I have quite a few (50) small modules around 4KiB in size and was wondering the same thing. Having individual modules makes it alot easier to organize and modify a single parameter.

Vic

roadie
Full of knowledge
Full of knowledge
Posts: 425
Joined: 02 Jan 2011, 18:41
Distribution: Porteus v5.01.....PorteuX v1.4
Location: In the bush now

mergexzm - merge folder of modules and slack packages

Post#12 by roadie » 17 Jun 2024, 18:08

I merged my collection of modules into one large module. The upshot was, from the standpoint of performance, the difference wasn't enough to be worth the effort. It also becomes problematic when the files contained in any certain module cause problems. A while back, I had problems with Filezilla, it wouldn't start if the keypassx module was activated. With merged modules, it's pretty hard to track down what's causing problems.

The performance difference amounted to no more than 2 seconds judging by boot times, which themselves vary. Even with individual modules, it cost 2 hours to find that keypassx was the problem......I can't imagine the headaches with a merged module.

Vic
Samurai
Samurai
Posts: 132
Joined: 10 Aug 2016, 05:36
Distribution: Porteux V-0.1 64 KDE
Location: Utopia in Tampa, Florida, USA

mergexzm - merge folder of modules and slack packages

Post#13 by Vic » 17 Jun 2024, 22:56

I just counted the modules that load everytime I boot Porteux: 119

Many are config files for Plasma and the programs I always use. Also application and service menu .desktop files.

I discovered that the number prefix not only works for .xzm files but also folders, so I have a hierarchy of numbered folders too.

With the cheatcodes I can load any folders to find a problem or just try something new out.

As Roadie posted, troubleshooting is much easier with individual modules.

Vic

porteux
Samurai
Samurai
Posts: 189
Joined: 12 Mar 2023, 22:10
Distribution: PorteuX

mergexzm - merge folder of modules and slack packages

Post#14 by porteux » 18 Jun 2024, 22:03

rych wrote:
17 Jun 2024, 07:03
porteux, thanks for adding the mergexzm to PorteuX 1.4. From the performance (of SquashFS, aufs mounts, ...) point of view, is it batter to have 20 small .xzms or one large .XZM module?
Considering the modules loading time, it's much faster to have a big one than many tiny ones.

The size of a module shouldn't impact its loading time, only the amount of files it contains inside. Still, it would be faster to load a merged one with everything than many little ones.

Anyway, we're probably talking about a few seconds difference here, depending on how many modules we're loading. Most likely not a big deal.

But, as Vic and roadie said, merging modules can be problematic for other reasons.

Post Reply