flat-manager serves and maintains a Flatpak repository

at an ostree repository and it will allow Flatpak clients to install

flat-manager

flat-manager serves and maintains a Flatpak repository. You point it at an ostree repository and it will allow Flatpak clients to install apps from the repository over HTTP. Additionally, it has an HTTP API that lets you upload new builds and manage the repository.

Building the server

The server is written in Rust, so you need to have Rust and Cargo installed. Everything works with the stable version of Rust, so you can get it from rustup or your distribution. On Fedora:

sudo dnf install cargo

PostgreSQL is used for the database, so the Postgres client libraries need to be installed first. On Fedora, this is done with:

sudo dnf install postgresql-devel

Then build the server by running:

cargo build

Building the client

flat-manager contains a Python-based client that can be used to talk to the server. To run this, you need Python 3 as well as the aiohttp packages, installed via pip or the distribution packages. On Fedora, this can be installed using:

sudo dnf install python3-aiohttp

Configuration

flat-manager reads the config.json file on startup in the current directory, although the REPO_CONFIG environment variable can be set to a different file. If you have a .env file in the current directory or one of its parents, it will be read and used to initialize environment variables.

The source repository contains an example.env and an example-config.json that can be used as a basis:

cp example.env .env
cp example-config.json config.json
# edit config.json

Database

flat-manager uses a PostgreSQL database to store information, and requires you to specify its address in the configuration file. The default example-config.json points this at:

"database-url": "#404",

This is a database called repo accessed via the default (at least on Fedora) UNIX domain socket. To install and start PostgreSQL, do something like:

sudo dnf install postgresql-server postgresql-contrib
sudo systemctl enable postgresql
sudo postgresql-setup --initdb --unit postgresql
sudo systemctl start postgresql

On Debian-based systems:

sudo apt install postgresql

And create the repo database owned by your user:

sudo -u postgres createuser $(whoami)
sudo -u postgres createdb --owner=$(whoami) repo

Note that if you're doing development work, it is important to also have DATABASE_URL=... set in the .env file for the Diesel command-line application to work. This is not required in production though.

Repositories

flat-manager maintains a set of repositories specified in the configuration, as well as a set of dynamically generated repositories beneath the configured build-repo-base path. For testing with the example configuration, these can be initialized by doing:

ostree --repo=repo init --mode=archive-z2
ostree --repo=beta-repo init --mode=archive-z2
mkdir build-repo

On a deployed system, these should be stored elsewhere, but make sure they are on the same filesystem so that hardlinks work between them as otherwise performance will be degraded.

Tokens

All requests to the API require a token. Token are signed with a secret that has to be stored on the server. The default configuration contains:

"secret": "c2VjcmV0"

This is base64 of "secret", so don't use this in production, but it works for local testing. Otherwise, you can generate one based on some random data:

dd bs=256 count=1 if=/dev/random of=/dev/stdout | base64 -w 0

Each token can have various levels of privileges. For example one could let you do everything, while another would only allow you to upload builds to a particular build. There is an API to subset your token for sharing with others (for example sending the above upload-only token to a builder), but you can also generate a token with the gentoken command:

echo -n "secret" | base64 | cargo run --bin gentoken -- --base64 --secret-file - --name testtoken

The above matches the default secret, so can be used for testing.

The client takes tokens via either the --token argument or in the REPO_TOKEN environment variable.

Running

To start the server, run:

cargo run --bin flat-manager

It will listen on port 8080 by default.

To test adding something to the repository, you can try building a simple app and exporting it to a repository. Use a recent version of flatpak and flatpak-builer to make sure you can build from Yaml files. This would normally happen on a different machine than the one serving the repository, but for testing we can just do it in a subdirectory:

git clone https://github.com/flathub/org.gnome.eog.git test-build
cd test-build
flatpak-builder --install-deps-from=flathub --repo=local-repo builddir org.gnome.eog.yml
cd ..

Then we can upload it to the repository by doing (assuming the default secret):

export REPO_TOKEN=$(echo -n "secret" | base64 | cargo run --bin gentoken -- --base64 --secret-file - --name test)
./flat-manager-client push --commit $(./flat-manager-client create http://127.0.0.1:8080 stable) test-build/local-repo

This will create a new "build", upload the build to it and then "commit" the build.

License

Licensed under either of

  • Apache License, Version 2.0 (LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0)
  • MIT license (LICENSE-MIT or https://opensource.org/licenses/MIT)

at your option.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

Issues

Collection of the latest Issues

jameswestman

jameswestman

0

As part of the Flathub donations & payments project, we would like to make it easier for developers to use build tokens to upload their apps. Currently, build tokens are given out on an ad-hoc basis, but we want to have a self-service UI accessible to all developers.

In order to do this, some changes are needed to the way builds and build tokens work in flat-manager. Most significantly, builds are currently not associated with an app. Build tokens can only be scoped to "all builds" or "a specific build by ID", and separately can be scoped to "build"/"upload"/"publish". This means we can't create tokens that allow developers to create builds for their app, then manage exactly that set of builds.

To fix this, I'd like to associate each build with an app prefix. There will be an endpoint to query builds by app, and tokens will have a field to restrict their scope to one or more app prefixes.

Another issue is that there is no authentication on downloading from build repos. This needs to be changed so that builds can be marked as public, and if they are not public, a build token is required to access them.

oleid

oleid

0

I'm using flat-manager to host self-made applications. This works fine, but I noticed downloads can be slow if the application is big.

Digging the docs, it turned out that static deltas are the way to go! And indeed, running

solves the issue.

While flat-manager actually runs flatpak build-update-repo, no static deltas are generated (=--generate-static-deltas is not set).

What is the recommended way to generate provide deltas? (i.e. for first-time app installations or updating via the flatpak command).

If it is --generate-static-deltas, would you accept a patch to enable the feature per repo on-demand?

nishp77

nishp77

0

Hi team, I have no clue till now how flatpak works nor how flatpak apps are installed? More interested in making a client hub for apps but could someone tell me how to access this apps? Just an overview would be fine

ckruse

ckruse

0

Hey there,

I set up a flatpak repo, but I just get a 400 error when trying to deploy via a github action using the actions at https://github.com/bilelmoussaoui/flatpak-github-actions

Api call to redacted/api/v1/build failed with status 400, details: {'status': 400, 'error-type': 'no-error', 'message': 'No json error details from server'}
Error: Failed to publish the build: Error: The process '/usr/bin/flat-manager-client' failed with exit code 1
Mar 21 11:35:43 flatpak flat-manager[172593]: [2022-03-21T10:35:43Z INFO  actix_server::builder] Starting 1 workers
Mar 21 11:35:43 flatpak flat-manager[172593]: [2022-03-21T10:35:43Z INFO  actix_server::builder] Starting server on 0.0.0.0:8080
Mar 21 11:35:43 flatpak flat-manager[172593]: [2022-03-21T10:35:43Z INFO  flatmanager::app] Started http server: 0.0.0.0:8080
Mar 21 11:45:15 flatpak flat-manager[172593]: [2022-03-21T10:45:15Z INFO  flatmanager::logger] 52.190.196.115 "POST /api/v1/build HTTP/1.1" github 400 24 Python/3.10 aiohttp/3.7.4 0.004536

Attaching via strace didn't reveal more information. What am I doing wrong? This is the relevant action code:

      - uses: actions/checkout@v2
      - uses: bilelmoussaoui/flatpak-github-actions/flatpak-builder@v4
        name: "Build"
        with:
          bundle: recordwatcher.flatpak
          manifest-path: de.termitel.recordwatcher.json
          cache-key: flatpak-builder-tt-record-watcher-${{ github.sha }}
  - <span class="hljs-string">uses:</span> bilelmoussaoui<span class="hljs-regexp">/flatpak-github-actions/</span>flat-manager<span class="hljs-meta">@v</span>3

name: "Deploy" with: repository: termitel flat-manager-url: redacted token: ${{ secrets.FLATPAK_KEY }}

This is the config.json:

{
  "repos": {
    "stable": {
      "path": "repo",
      "collection-id": "de.termitel.Stable",
      "suggested-repo-name": "Termitel",
      "runtime-repo-url": "redacted/repo/flathub.flatpakrepo",
      "gpg-key": null,
      "base-url": null,
      "subsets": {
        "all": {
          "collection-id": "de.termitel.Stable",
          "base-url": null
        }
      }
    }
  },
  "host": "0.0.0.0",
  "port": 8080,
  "delay-update-secs": 10,
  "database-url": "#404",
  "build-repo-base": "/home/flatpak/build-repo",
  "build-gpg-key": null,
  "gpg-homedir": null,
  "secret": "redacted"
}

I compiled flat-manager from HEAD on a Ubuntu box. Deploying manually via flat-manager-client works:

$ ./flat-manager-client push --commit $(./flat-manager-client create redacted stable) RecordWatcher/termitel
Uploading refs to redacted/api/v1/build/9: ['app/de.termitel.recordwatcher/aarch64/master', 'runtime/de.termitel.recordwatcher.Debug/aarch64/master']
Refs contain 33 metadata objects
Remote missing 33 of those
Has 16 file objects for those
Remote missing 16 of those
Uploading file objects
Uploading 1 files (214 bytes)
Uploading 1 files (39982483 bytes)
Uploading 3 files (1731 bytes)
Uploading 1 files (4946843 bytes)
Uploading 10 files (11266 bytes)
Uploading metadata objects
Uploading 33 files (4885 bytes)
Uploading deltas
Creating ref app/de.termitel.recordwatcher/aarch64/master with commit b308577a76134cf0d826ded174982d8e04b03569ef7a95b99b865da57a26ffb5
Creating ref runtime/de.termitel.recordwatcher.Debug/aarch64/master with commit ac2dfde8e35f9233eb0e29786cca79aae4be61e597bed000372964c7fb6be8c6
Committing build redacted/api/v1/build/9

What am I doing wrong?

Best regards, CK

bilelmoussaoui

bilelmoussaoui

0

Currently, in ostree.rs there is manual code to deserialize the GVariant format used by OSTree which could ideally be replaced with zvariant using the gvariant format.

upstream zbus has currently a test case of a ostree repo to test the gvariant feature. So normally it should work without issues

willyywt

willyywt

0

A link to https://github.com/flatpak/flatpak/issues/4383

I am a user from China and I am affected by https://github.com/flathub/flathub/issues/813, as GFW in China completely censored Fastly CDN. Currently flatpak repo are only intended to used by a CDN, but it bascially restricted the mirroring method to becoming a reverse proxy, which is proved very toxic to mirror sites: for tuna and sjtug

Rsync rationale

I don't know flatpak very well, but I think flatpak is a repo manager for an ostree repo, which itself seems to be easy for rsync to mirror: https://github.com/ostreedev/ostree-releng-scripts/blob/master/rsync-repos. I suggest flatpak should support an additional remote storage backend using rsync in parallel with the current CDN-like approach.

Rsync advantages

  • Commonly accepted protocol for mirroring
  • Incremental updates
  • Packed transfering of files
  • Safe resume from failed sync
  • Support both unauthorized client and authorized client

Rsync problems

  • Pressure on server resource
  • Still more costly than plain deb/rpm packages (but better than making large number of small http requests)

CDN advantage

  • Doesn't require a strong server
  • Approachable to many (but not more than mirror sites since CDN providers are not approachable everywhere) areas using a relatively low price

CDN-only problems

  • Effectively limits mirror methods to reverse proxy only, which is unacceptable for some mirror sites and irritating for the rest
  • Too many delta file increases the risk of failure because of increased number of http request round trip
  • No safe and incremental way of recovering from failed download

These are not flatpak-specific issues; many FOSS projects are affected by them.

Many mirror sites can support rsync in addition to plain http; for these sites rsync are a proved working way of mirroring and are way easier to manage. If remotes like flathub doesn't have much server resource, they can limit the bandwidth to unauthorized clients and reserve more resource to mirror sites.

btkostner

btkostner

0

Right now there is no nice way to delete an app from a flatpak repository. It would be awesome if there was an endpoint for this and a cli command with flat-manager-client.

btkostner

btkostner

0

Setting up the elementary flatpak repository with flat-manager I came across an error with publishing.

Our Setup

We have a simple server running flat-manager. It is setup with very similar ansible playbooks. We then have delta generator running on another server also setup from the ansible playbooks.

The flatpak repository itself is published with nginx with regular http and https enabled (both work, no redirects).

The relevant flat-manager config looks like this:

Console log

When flat-manager runs a publish job, it gets assigned to the delta generator worker successfully, but then errors out pretty fast with this:

All of those files do exist, and from what I can tell, it seems the http client used for delta generator is trying to access https://flatpak.elementary.io without ssl configured / enabled, and therefor getting an error.

As a sanity check, this is from the delta-generator server:

dbnicholson

dbnicholson

0

Currently flat-manager only uses the repo secret to verify the JWT used for authenticated requests. The token itself is created by the authorization server. It would be better if flat-manager did not require a secret for validation then since it just increases the risk that the secret could be leaked.

Since JWT supports public key signatures such as RSA and EDCSA, it would be nice if flat-manager supported verification with public keys rather then HMAC secrets. Then the private key could just live with the authorization server. Since jsonwebtoken supports many algorithms, this shouldn't be too hard to add.

davidmhewitt

davidmhewitt

1

Given that the title field is supposed to be a human readable name for the application it seems a bit strange to use the RDNN when the .flatpakref file is generated. I appreciate the correct information from the appdata file may be difficult to get at this stage though.

This is something I'd be interested in working on if anyone was able to give me any pointers in terms of the best approach.

Versions

Find the latest versions by id

0.3.7 - Nov 08, 2019

  • Client has been updated to use asyncio and retry more before failing
  • Server now generates flatpakref files for non-apps
  • The token validation now gives better error codes and debug messages

0.3.6 - Oct 25, 2019

  • Add various ostree helpers
  • Make flat-manager-client upload objects and deltas asynchronously with aiohttp

0.3.5 - Aug 16, 2019

Now supports --endoflife-rebase

0.3.4 - Jun 17, 2019

Now with a deploy-collection-id boolean config option per repo that will trigger the setting of DeployCollectionID in the created flatpakref files.

0.3.3 - Mar 18, 2019

This has a fix for the client backtraceing (imported into buildbot flathub8) and a delta generator timeout fix.

0.3.2 - Mar 15, 2019

Better fix for the crash in delta generation

0.3.1 - Mar 15, 2019

Hopefully this fixes a crash we're seeing on flathub.

0.3.0 - Mar 15, 2019

Major changesin this version:

  • flatpak >= 1.3.0 is required
  • update-repo job is split out from publish job and one update can be used for multiple publish
  • We now parallelize some jobs. All publish/update for a particular repo are serialized, but build commits can be handled in parallel.
  • Support for external delta generators
  • Log output is much nicer

0.2.2 - Feb 12, 2019

Added post-publish-script support.

0.2.1 - Feb 07, 2019

This release has various tweaks you can do to the generated flatpakref files.

0.2.0 - Feb 05, 2019

Now with multi-repo and extra-ids.

Information - Updated Jun 22, 2022

Stars: 89
Forks: 30
Issues: 29

serde-json for no_std programs

MIT license (LICENSE-MIT or

serde-json for no_std programs

macOS/iOS Security framework for Rust

MIT license (LICENSE-MIT or

macOS/iOS Security framework for Rust

OpenGliderNetwork client for Rust based on actix

MIT license (LICENSE-MIT or

OpenGliderNetwork client for Rust based on actix

A video game for programmers set in space

MIT license (LICENSE-MIT or

A video game for programmers set in space

Simple SQL migration manager for your project

MIT license (LICENSE-MIT or

Simple SQL migration manager for your project

TODO_README_DESCRIPTION

(LICENSE-MIT or Semantic Versioning 2

TODO_README_DESCRIPTION

CBOR Event library

MIT license (LICENSE-MIT or

CBOR Event library

HAL for the STM32WB55 family of microcontrollers

MIT license (LICENSE-MIT or

HAL for the STM32WB55 family of microcontrollers
CLI

123

write licenses to stdout

Write a license to standard output given its SPDX ID

write licenses to stdout

A simple easy to use wrapper around Ctrl-C signal

MIT license (LICENSE-MIT or

A simple easy to use wrapper around Ctrl-C signal
Facebook Instagram Twitter GitHub Dribbble
Privacy