[BETA] prepatch a system-wide patching system (alternative to patchmanager)
 

Update:
Coderus is working on integrating the preload-method into patchmanger 3!

This will make prepatch pretty much redundant so I recommend every patch-dev to wait for pm3 instead of porting their patch as prepatch won't be maintained anymore when pm3 releases.

Patchmanager 3 is currently in closed alpha and you can request access by following the instructions here.

This is still in beta!
You may (or may not) still encounter issues. (please report them if you do)

What is this?

Prepatch is an alternative to patchmanager.
However unlike patchmanager prepatch doesn't modify any files on disk but instead replaces the "open"/"open64" function in order to make applications think that those files were modified.

Why should I use/not use prepatch instead of patchmanger?

Advantages

• Undoing patches is never a problem even if multiple patches were applied to a single file.
• Patches don't have to be reapplied if the patched file has been updated.
• System-Updates no longer break stuff if patches are applied and all patches which still work after the update will instantly be working again (no reenabling patches).
• An "apply-order" can be specified if patches which patch the same file need it.
• Prepatch can replace/add files if you're confident that the patched file should stay the same even if jolla updates it.
• It's generally "cleaner" to leave files of other applications alone.

Disadvantages

• Because of the way it works prepatch will slow down the opening of all files a (small) bit since it has to check if patches have to be applied and, if necessary, apply them. Patchmanager doesn't suffer from this, since it applies the patch to the real files.
• No gui (at the moment) to install/toggle patches. However this will likely change in the future.
• Patchmanager has a lot more patches availible. However porting them to prepatch should be trivial in the vast majority of cases.

How does it work?

Prepatch works by preloading a library into every binary (don't worry, the overhead is much smaller then it sounds) and replaces the open+open64 function in there.

If an application tries to open a file which doesn't have a patch (or tries to open the file for writing/as any user other then nemo) the replaced open function will just behave like the normal open function.

However if an application tries to open a file for which a patch has been installed (in most cases an qml-file) prepatch will apply the patch(es) to a temporary copy of the file and then open that one instead of the one the application requested.

That way the application will think that the file has been patched and behave that way while in reality the real file stays unmodified.

How do I install it?

Just install the prepatch package from openrepos.

How do I uninstall it?

To uninstall prepatch simply remove the prepatch package and you're done. No additional cleanup needed!

How do I install patches?

Just find the patch on openrepos and install it :)

For example here's my keyboard-swipe-patch.

In some cases restarting lipstick might be required.

How do I develop patches?

Developing patches for prepatch is easy :)

I'll explain it with my keyboard-swipe-patch as the example. Replace the names/path with yours as needed.

First we'll create the patch-folder (which will be placed into /usr/share/prepatch after we're done creating the patch).

The 050 is there to make it easy to specify the order in which the patches shall be loaded.

If for example an other patch modifies the same file as yours but only works if your patch is applied before the other one you simply choose an lower number and your patch will always be applied first.

Don't cd into it yet.

First create two copies of the file which you want to modify.
One will stay the same and the other one will be modified.

Please note that, if there's already a prepatch patch applied on this file you'll get the modified version when trying to copy it (as prepatch is loaded into cp).
If that's the case temporarily disable the other patch until you're done with this step.

Now edit the KeyboardBase-Modified.qml file according to your needs.

After that create the directory in which your patch will be located in.

In my case I have to run

Now run diff -u old new > 050-yourpatch/path/to/orig.qml.patch.

Im my case:

Repeat this process with all the files you want to patch (you can place multiple patches in the same 050-prepatch-folder).

If you want to add/completely replace a file instead of patching it you can simply place it inside the folder without the .patch extension. Please note that new files will not appear in directory-listings so ls /path/to/fakefile will not show the file but running cat /path/to/fakefile/file will print out its contents.

Now simply copy the 050-prepatch* folder to /usr/share/prepatch and your patch will be applied the next time the modified file is loaded!

If you want to package your patch you can take a look at my keyboard-swipe-repo, modify the folders+spec to your needs and package it by following this guide.

If you need to add an settings-page to your patch you can take the way it's done in this patch as a reference.

How do I convert patches from patchmanager to prepatch?

I've written a script which can automatically split patchmanager-patches into the prepatch-format.

Obviously the patch you're trying to port has to work with the SailfishOS-version you're on.

This script requires you to have patchutils installed on your computer (or your phone if you want to port this with your device :) ).

If you don't it's most likely availible in your package-sources on the os of your choice.
For example here's how you install patchutils on SailfishOS:
I'll explain it with my button-vibrate-patch which was originally made for patchmanager (that patch has already been ported to prepatch by me so if you want to use that you can do that already without following this tutorial).

First create a working directory and cd there (not necessary but recommended).
Now download my convert.sh script and mark it as executabe.
Now obtain the unified_diff.patch from the patch that you want to port to prepatch and place it in the working directory. In my case I can get it from the zip on the patchmanager-page .
Convert the unified_diff.patch to a patchmanager-patch like this:
Now you can activate the patch by copying it to /usr/share/prepatch.
The patch is now applied. Restart lipstick if necessary.
If you want to create an rpm for publishing you can follow my development-instructions and just skip until after creating the patch.

If the patch has any settings-pages included you'll need to manually install them by creating an settings-page entry information like
this one and place it in /usr/share/jolla-settings/entries and place the main.qml file in /usr/share/jolla-settings/pages/YOUR_PAGE_NAME/main.qml and the icon (if there is any) in /usr/share/jolla-settings/pages/YOUR_PAGE_NAME/ICON_NAME.png.

Is it possible to run this alongside with patchmanager?

Yes it is. However all patches that are applied using prepatch will be "applied" after the patchmanager-patches.

What bugs/issues are currently known?

• Fixed in 0.2.2-1 //For some reason lipstick crashes when trying to click on the weather-applet in the notification-center. (thanks to cy8aer for reporting this)
• Fixed in 0.2.0-1! //For some reason after starting/restarting lipstick it keeps loading for about 20 seconds.
  After that everything goes back to normal and performance doesn't seem to be affected in other apps so I suspect that this is caused by a bug in prepatch.

I want to help! How can I help?

• There's no graphical interface for toggeling patches yet (not particularly urgent).
• Update: There is one now! Many thanks to Fellfrosch! //Prepatch has no icon yet.

What patches are availible?

• keyboard-swipe-cursor A simple patch which allows you to move the cursor on the keyboard by swiping it to the left/right.
• silica-button-vibrate A patch which makes silica-buttons and icon-buttons vibrate when touched or released.

If you want your patch listed here please open an issue on github or ask in this thread.

UPDATES:

1. Added packaging for patches.
2. Added prepatch package!
3. Fixed the loading-screen bug! (0.2.0-1) Prepatch should now be usable on older devices!
4. Added script to convert patchmanager-patches to prepatch-patches.
5. Added icon! Many thanks to Fellfrosch for creating it!
6. Fixed the weather-applet-issue (0.2.2-1)

Re: [WIP] prepatch a system-wide patching system (alternative to patchmanager)
 

This is absolutely ingenious way of applying patches and beats patchmanager hands down :p
Congratulations, a fine idea indeed!

Re: [WIP] prepatch a system-wide patching system (alternative to patchmanager)
 

I don't want to think about the security implications of this; but
great idea!

Re: [WIP] prepatch a system-wide patching system (alternative to patchmanager)
 

Looking forward to trying this. Very good work on a very good idea. Taking a small hit in speed is a small price to pay weighted against the advantages.

Re: [WIP] prepatch a system-wide patching system (alternative to patchmanager)
 

Hello? Security? Am I the only one worried here? :(

Re: [WIP] prepatch a system-wide patching system (alternative to patchmanager)
 

If you have concerns please voice them.
I don't want to be shipping anything which compromises users security.

My code is obviously not perfect so there may be some attack-surface there but I don't see how it's worse than patchmanager in that regard.

For installing prepatch and patches for it you'll need root-permissions with which an attacker could do anything evil anyways.

Re: [WIP] prepatch a system-wide patching system (alternative to patchmanager)
 

I packaged everything (my patch and prepatch itself) and uploaded them to openrepos. Installing/Removing prepatch and prepatch-patches is now as easy as with every other package! :)

Re: [WIP] prepatch a system-wide patching system (alternative to patchmanager)
 

Very nice work :)
It would be even better if it could use coderus's Webcatalog, the patchfiles itself shouldn't be differnt, didn't they?
Maybe coderus can help integrating the api in your app and we all can call it patchmanger 2.5 or 3 ;)

Re: [WIP] prepatch a system-wide patching system (alternative to patchmanager)
 

Patchmanager patches have one patch file for all files that are going to be patched while prepatcher uses one patch-file for each file that's going to be patched in order to reduce the overhead that's caused by finding and applying a patch (which is done when the file loads so not spending too much time on that is vital).

However it should be trivial to write a tool which converts patchmanager-patches to prepatcher patches. (I might do so in the future).

Part of the point of prepatcher is that nothing special has to be done to enable/disable patches though so I don't see much of a reason not to simply distribute patches through openrepos since there's already a nice frontend for that (storeman).

Re: [WIP] prepatch a system-wide patching system (alternative to patchmanager)
 

I tried the packaged version and rebooted. But I had some difficulties:

1st try: Unlocking froze the device
2nd try: After shutdown and start I was able to unlock but then lipstick was veeery slow (waiting circle on startup). I had a frozen screen when swiping but it came back.

Then I uninstalled both packages (prepatcher and the keyboard patch) and rebooted and every went smooth again.

JollaC 2.1.4.14

Re: [WIP] prepatch a system-wide patching system (alternative to patchmanager)
 

Hmm :( I only tried it on my xperia x which is a good deal more powerful then your device.

Could you please try again and this time just let the phone sit for a few minutes after rebooting? I also have the circle-issue but after it goes away (20 seconds for me) everything goes smoothly.

I know this is not a permanent solution but I want to narrow down the issue.

I also released a new version which should fix a few memory-leaks but unfortunately doesn't completely fix the loading-bug so that may help.

Re: [WIP] prepatch a system-wide patching system (alternative to patchmanager)
 

I think I know where you're heading.

Patchmanager modifies files which I can check afterwards to be in place and doing what its was meant for. Checking can happen even in an environment like recovery shell.

Prepatch is injected in file system calls. You say yourself "be sure turn of this patch or that" to actually copy the correct file. So if you want to be sure, a patch does not contain malicious parts, you have to check that either before installing or has to happen in recovery shell were the prepatch is not loaded. Otherwise you can't be sure while prepatch is active. Because if someone wants to add malicious code, he just needs to also cover any file requests regarding his files with something innocent using the same architecture.

Re: [WIP] prepatch a system-wide patching system (alternative to patchmanager)
 

I see your point and it's a very valid concern but isn't that the case with essentially everything you install from an untrused source?

Any rpm you install could also setup a rootkit on your device while it's installing which makes sure that you can't notice it unless you're in recovery even without prepatch.

I don't see any way to prevent rpms from doing evil things (as they're being installed with a tool that I don't have any control over) but that's the same way with the original patchmanager.

If I ever add something like the pm2-webstore I'll make sure that users have to explicitly enable the patches after installing them so that they can check out the content but for now I don't see the point in creating something that would be trivial to bypass in the rpm postininstall script.

Re: [EARLY BETA] prepatch a system-wide patching system (alternative to patchmanager)
 

Well, it works on my Inoi R7, the first lipstick start after installation was about 30-35 seconds, but it works as fast as usual.

Re: [EARLY BETA] prepatch a system-wide patching system (alternative to patchmanager)
 

I fixed the loading-screen-bug! Prepatch should now be usable on older devices too!

Re: [EARLY BETA] prepatch a system-wide patching system (alternative to patchmanager)
 

I'm facin issue after installing prepatch, can't launch anymore the dialer.
App is loading and then is closing itself.
Tryed restarting lipstick and rebooting Xperia without any impact.
Removing prepatch and restarting give me the dialer back.

Maybe some parts from "old" patchmanager are still there and causing this issue ?
Do you want me to fill a bug on github ?

Re: [EARLY BETA] prepatch a system-wide patching system (alternative to patchmanager)
 

That's very weird. The dialer works perfectly for me.

Please make sure that you're running the newest version (I released an update a few minutes ago).

If the issue persists could you please open an issue on github with the output of running
from ssh. (I don't want to clutter this thread with the back and forth of trying to debug this)

Re: [EARLY BETA] prepatch a system-wide patching system (alternative to patchmanager)
 

Just reinstalled your new 0.2 version and issue is fixed, yay ! :D

Re: [WIP] prepatch a system-wide patching system (alternative to patchmanager)
 

So what? I do not buy any electronic crap every year. And it does not matter how fast it is...

Re: [BETA] prepatch a system-wide patching system (alternative to patchmanager)
 

Ok, the new version of prepatcher seems to work. Thank you.

Just a lesson: Never test on the fastest machine you can buy. Test it on the slowest oldest. If there will be some hardware bug some time and your processors get 30% slower all of your hardware - old and new - will still perform.

Re: [BETA] prepatch a system-wide patching system (alternative to patchmanager)
 

I would've done just that but buying an old phone just for testing an hobby-project of mine seems a bit of overkill. (If the issues with older devices persited I probably would've bought one in order to fix it but fortunately it works now :))

Re: [BETA] prepatch a system-wide patching system (alternative to patchmanager)
 

That seems to be a common attitude of all software developers.
Pity.

Re: [BETA] prepatch a system-wide patching system (alternative to patchmanager)
 

For real? Go and donate device for jakibaki if you want it to be tested on other devicel. WTF!!!

Re: [BETA] prepatch a system-wide patching system (alternative to patchmanager)
 

10chars...

https://media.giphy.com/media/j9djzcMmzg8ow/giphy.gif

Re: [BETA] prepatch a system-wide patching system (alternative to patchmanager)
 

Not beeing a software developer I anyway hope you ment this as a joke. :eek: :eek: :eek:

If not i'm wondering if your mum slapped you when you were a little boy and drew her a picture. And if she yelled at you, learn to paint properly if you want to make me presents. :rolleyes: :(

Re: [BETA] prepatch a system-wide patching system (alternative to patchmanager)
 

Yes, for real. I am not targeting bedroom developers like jakibaki but, for example, large, multi-milion-dollar companies like Canonical who assume everybody has huge screens and state-of-the-art CPUs that came out last week. And thus three quarters of dialog boxes do not fit on my screen and my PC does not boot after a kernel update because they turn some super-duper feature on by default.

If anyone is at least half serious about developing software, they really should be testing on more than one device. "It works for me" is just Not Good Enough™.

"More than one" = "at least two". More is better but if you have only two, then it makes perfect sense to make the second one the lowest spec you can find. Or include a disclaimer, "only works on 6GB deca-core or above, because I have not tested on anything smaller".

Yes, for real.

Re: [BETA] prepatch a system-wide patching system (alternative to patchmanager)
 

Don't look a gift horse in the mouth.

If you pay for software, I think you can justify your attitude. But getting it for free, I don't understand it. Doesn't matter if it's a bedroom developer or Canonical.

Re: [BETA] prepatch a system-wide patching system (alternative to patchmanager)
 

Jakibaki is developping a tool that may prove very useful to all of us Sailfish users, is being very active to answer testers in this thread, and according to his postcount, this tool is a significant part of his activity here. I bet we could welcome him a little better and be more grateful for his time, especially if it is assumed that he should buy extra devices just for testing his free non-profit project.

I don't want to appear naive, but I am a bit surprised by the hostility since he announced Prepatcher. Do we really want him to think "**** it." and abandon the project? He has an Xperia X and this is the device that made him work on Prepatcher, I think we could deal with it and still see it as a good addition to the SFOS ecosystem.

Re: [BETA] prepatch a system-wide patching system (alternative to patchmanager)
 

My take on this is what you can already see in the thread topic;
[BETA] prepatch a system-wide patching system (alternative to patchmanager)

it's already announced as beta which to me means "please test it, work in progress"

When something is in beta, you don't expect it to work on all devices, in all conditions, everywhere.

Re: [BETA] prepatch a system-wide patching system (alternative to patchmanager)
 

Even as a hobbyist developer with more devices than I'd like to admit, I still think jakibaki did everything right – he disclosed that there was problem with start up time which he intended to fix ASAP – and he did.
Releasing not-quite-ready things (he marked it as BETA, that's right, caps) is a great way to share the load of testing different circumstances. Including devices.

So, yes, in this context of this thread, at this time, "works for me" is totally enough. Please don't discourage hobbyist developers by being not nice – even if you later clarify that you actually talked about someone else. Canonical doesn't have anything to do with this thread.

It helps as much as: "Look, I built a nice bobby car, perhaps you find it useful" – "Oh, thats Bad for getting anywhere!" - "Yeah, I don't have the means to hire testers" – "Yes, exactly, I hate when trains are delayed, because that's what you did. Anyone interested in transportation should build helicopters".

Back on topic.
@jakibaki: As others mentioned, I'd be so happy if you and coderus got together on this and pull on the same end of the rope. While it's a bit scary such a preloader is possible, I think it's totally awesome that you actually did it. Thank you.

TLDR: More love, please. Keep up the good work!

Edit: I took a while to write my rant, juiceme said about the same thing, shorter. cheers!

Re: [BETA] prepatch a system-wide patching system (alternative to patchmanager)
 

So then: I am your slow device tester.

Oh oh, did not want to troll...

Re: [BETA] prepatch a system-wide patching system (alternative to patchmanager)
 

I added an script which can automatically convert patchmanager-patches into prepatch-patches and added instructions to the main-post :)

Re: [BETA] prepatch a system-wide patching system (alternative to patchmanager)
 

hm, I had problems with gpodder. The application got memory problems and killed other applications. After de- installing of prepatch it works smooth again.

Gpodder itself is not as stable as wanted. So please check it by yourself before blaming prepatch.

Re: [BETA] prepatch a system-wide patching system (alternative to patchmanager)
 

Thanks for reporting this issue!
I fixed a memory-leak in the new version I just pushed which I suspect to have been the problem.

Could you please update prepatch and then try again?

Re: [BETA] prepatch a system-wide patching system (alternative to patchmanager)
 

Just installed. Standby...

Re: [BETA] prepatch a system-wide patching system (alternative to patchmanager)
 

You are writing, that you need an icon.
Something like that?

Re: [BETA] prepatch a system-wide patching system (alternative to patchmanager)
 

This is great!
Thank you very much! :)

Re: [BETA] prepatch a system-wide patching system (alternative to patchmanager)
 

New one: when running analog clock on start screen patch with weather applet the weather applet clicking on the applet for opening and closing the forecast it crashes lipstick.

Re: [BETA] prepatch a system-wide patching system (alternative to patchmanager)
 

Obviously not related to patch

Re: [BETA] prepatch a system-wide patching system (alternative to patchmanager)
 

Oh, that's a weird one D:

I can confirm that it only happens when prepatch is enabled an am now working on finding out what causes it.