mirror of
https://github.com/arcan1s/ahriman.git
synced 2025-06-28 06:41:43 +00:00
feat: allow to use one application for multiple repositories (#111)
* allow to use one application for multiple repositories * update tests * handle None append argument everywhere * rewrite repository definition logic * drop optional flags from docs * support of new schema in systemd units * add migration docs and ability to migrate tree automatically * use repostory id instead * verbose multiarchitectureerror * object path support for s3 sync * fix tests after rebase
This commit is contained in:
@ -172,6 +172,14 @@ ahriman.application.handlers.structure module
|
||||
:no-undoc-members:
|
||||
:show-inheritance:
|
||||
|
||||
ahriman.application.handlers.tree\_migrate module
|
||||
-------------------------------------------------
|
||||
|
||||
.. automodule:: ahriman.application.handlers.tree_migrate
|
||||
:members:
|
||||
:no-undoc-members:
|
||||
:show-inheritance:
|
||||
|
||||
ahriman.application.handlers.triggers module
|
||||
--------------------------------------------
|
||||
|
||||
|
||||
@ -92,6 +92,14 @@ ahriman.core.database.migrations.m010\_version\_based\_logs\_removal module
|
||||
:no-undoc-members:
|
||||
:show-inheritance:
|
||||
|
||||
ahriman.core.database.migrations.m011\_repository\_name module
|
||||
--------------------------------------------------------------
|
||||
|
||||
.. automodule:: ahriman.core.database.migrations.m011_repository_name
|
||||
:members:
|
||||
:no-undoc-members:
|
||||
:show-inheritance:
|
||||
|
||||
Module contents
|
||||
---------------
|
||||
|
||||
|
||||
@ -36,10 +36,10 @@ ahriman.core.log.lazy\_logging module
|
||||
:no-undoc-members:
|
||||
:show-inheritance:
|
||||
|
||||
ahriman.core.log.log module
|
||||
---------------------------
|
||||
ahriman.core.log.log\_loader module
|
||||
-----------------------------------
|
||||
|
||||
.. automodule:: ahriman.core.log.log
|
||||
.. automodule:: ahriman.core.log.log_loader
|
||||
:members:
|
||||
:no-undoc-members:
|
||||
:show-inheritance:
|
||||
|
||||
@ -164,6 +164,14 @@ ahriman.models.report\_settings module
|
||||
:no-undoc-members:
|
||||
:show-inheritance:
|
||||
|
||||
ahriman.models.repository\_id module
|
||||
------------------------------------
|
||||
|
||||
.. automodule:: ahriman.models.repository_id
|
||||
:members:
|
||||
:no-undoc-members:
|
||||
:show-inheritance:
|
||||
|
||||
ahriman.models.repository\_paths module
|
||||
---------------------------------------
|
||||
|
||||
|
||||
@ -102,6 +102,53 @@ All subcommands are divided into several groups depending on the role they are d
|
||||
|
||||
For historical reasons and in order to keep backward compatibility some subcommands have aliases to their shorter forms or even other groups, but the service doesn't guarantee that they will remain unchanged.
|
||||
|
||||
Filesystem tree
|
||||
---------------
|
||||
|
||||
The application supports two types of trees, one is for legacy configuration (when there were no repository name explicit configuration available) and another one is for new-style tree. This document describes only new-style tree in order to avoid deprecated structure.
|
||||
|
||||
Having default root as ``/var/lib/ahriman`` (differs from container though), the directory structure is the following:
|
||||
|
||||
.. code-block::
|
||||
|
||||
/var/lib/ahriman/
|
||||
├── ahriman.db
|
||||
├── cache
|
||||
├── chroot
|
||||
│ └── aur-clone
|
||||
├── packages
|
||||
│ └── aur-clone
|
||||
│ └── x86_64
|
||||
├── pacman
|
||||
│ └── aur-clone
|
||||
│ └── x86_64
|
||||
│ ├── local
|
||||
│ │ └── ALPM_DB_VERSION
|
||||
│ └── sync
|
||||
│ ├── core.db
|
||||
│ ├── extra.db
|
||||
│ └── multilib.db
|
||||
│
|
||||
└── repository
|
||||
└── aur-clone
|
||||
└── x86_64
|
||||
├── aur-clone.db -> aur-clone.db.tar.gz
|
||||
├── aur-clone.db.tar.gz
|
||||
├── aur-clone.files -> aur-clone.files.tar.gz
|
||||
└── aur-clone.files.tar.gz
|
||||
|
||||
There are multiple subdirectories, some of them are commons for any repository, but some of them are not.
|
||||
|
||||
* ``cache`` is a directory with locally stored PKGBUILD's and VCS packages. It is common for all repositories and architectures.
|
||||
* ``chroot/{repository}`` is a chroot directory for ``devtools``. It is specific for each repository, but shared for different architectures inside (the ``devtools`` handles architectures automatically).
|
||||
* ``packages/{repository}/{architecture}`` is a directory with prebuilt packages. When package is built, first it will be uploaded to this directory and later will be handled by update process. It is architecture and repository specific.
|
||||
* ``pacman/{repository}/{architecture}`` is repository and architecture specific caches for pacman's databases.
|
||||
* ``repository/{repository}/{architecture}`` is a repository packages directory.
|
||||
|
||||
Normally you should avoid direct interaction with the application tree.
|
||||
|
||||
For tree migration process refer to the :doc:`migration notes <migration>`.
|
||||
|
||||
Database
|
||||
--------
|
||||
|
||||
@ -274,7 +321,7 @@ There are several supported synchronization providers, currently they are ``rsyn
|
||||
|
||||
``rsync`` provider does not have any specific logic except for running external rsync application with configured arguments. The service does not handle SSH configuration, thus it has to be configured before running application manually.
|
||||
|
||||
``s3`` provider uses ``boto3`` package and implements sync feature. The files are stored in architecture directory (e.g. if bucket is ``repository``, packages will be stored in ``repository/x86_64`` for the ``x86_64`` architecture), bucket must be created before any action and API key must have permissions to write to the bucket. No external configuration required. In order to upload only changed files the service compares calculated hashes with the Amazon ETags, used realization is described `here <https://teppen.io/2018/10/23/aws_s3_verify_etags/>`_.
|
||||
``s3`` provider uses ``boto3`` package and implements sync feature. The files are stored in architecture directory (e.g. if bucket is ``repository``, packages will be stored in ``repository/aur-clone/x86_64`` for the ``aur-clone`` repository ``x86_64`` architecture), bucket must be created before any action and API key must have permissions to write to the bucket. No external configuration required. In order to upload only changed files the service compares calculated hashes with the Amazon ETags, used realization is described `here <https://teppen.io/2018/10/23/aws_s3_verify_etags/>`_.
|
||||
|
||||
``github`` provider authenticates through basic auth, API key with repository write permissions is required. There will be created a release with the name of the architecture in case if it does not exist; files will be uploaded to the release assets. It also stores array of files and their MD5 checksums in release body in order to upload only changed ones. According to the Github API in case if there is already uploaded asset with the same name (e.g. database files), asset will be removed first.
|
||||
|
||||
|
||||
@ -1,7 +1,12 @@
|
||||
Configuration
|
||||
=============
|
||||
|
||||
Some groups can be specified for each architecture separately. E.g. if there are ``build`` and ``build:x86_64`` groups it will use an option from ``build:x86_64`` for the ``x86_64`` architecture and ``build`` for any other (architecture specific group has higher priority). In case if both groups are presented, architecture specific options will be merged into global ones overriding them.
|
||||
Some groups can be specified for each architecture and/or repository separately. E.g. if there are ``build`` and ``build:x86_64`` groups it will use an option from ``build:x86_64`` for the ``x86_64`` architecture and ``build`` for any other (architecture specific group has higher priority). In case if both groups are presented, architecture specific options will be merged into global ones overriding them. The order which will be used for option resolution is the following:
|
||||
|
||||
#. Repository and architecture specific, e.g. ``build:aur-clone:x86_64``.
|
||||
#. Repository specific, e.g. ``build:aur-clone``.
|
||||
#. Architecture specific, e.g. ``build:x86_64``.
|
||||
#. Default section, e.g. ``build``.
|
||||
|
||||
There are two variable types which have been added to default ones, they are paths and lists. List values will be read in the same way as shell does:
|
||||
|
||||
@ -25,7 +30,7 @@ There is also additional subcommand which will allow to validate configuration a
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
ahriman -a x86_64 service-config-validate
|
||||
ahriman service-config-validate
|
||||
|
||||
It will check current settings on common errors and compare configuration with known schema.
|
||||
|
||||
@ -87,7 +92,6 @@ Build related configuration. Group name can refer to architecture, e.g. ``build:
|
||||
|
||||
Base repository settings.
|
||||
|
||||
* ``name`` - repository name, string, required.
|
||||
* ``root`` - root path for application, string, required.
|
||||
|
||||
``sign:*`` groups
|
||||
@ -292,20 +296,21 @@ Type will be read from several sources:
|
||||
``github`` type
|
||||
^^^^^^^^^^^^^^^
|
||||
|
||||
This feature requires Github key creation (see below). Section name must be either ``github`` (plus optional architecture name, e.g. ``github:x86_64``) or random name with ``type`` set.
|
||||
This feature requires GitHub key creation (see below). Section name must be either ``github`` (plus optional architecture name, e.g. ``github:x86_64``) or random name with ``type`` set.
|
||||
|
||||
* ``type`` - type of the upload, string, optional, must be set to ``github`` if exists.
|
||||
* ``owner`` - Github repository owner, string, required.
|
||||
* ``password`` - created Github API key. In order to create it do the following:
|
||||
* ``owner`` - GitHub repository owner, string, required.
|
||||
* ``password`` - created GitHub API key. In order to create it do the following:
|
||||
|
||||
#. Go to `settings page <https://github.com/settings/profile>`_.
|
||||
#. Switch to `developers settings <https://github.com/settings/apps>`_.
|
||||
#. Switch to `personal access tokens <https://github.com/settings/tokens>`_.
|
||||
#. Generate new token. Required scope is ``public_repo`` (or ``repo`` for private repository support).
|
||||
|
||||
* ``repository`` - Github repository name, string, required. Repository must be created before any action and must have active branch (e.g. with readme).
|
||||
* ``repository`` - GitHub repository name, string, required. Repository must be created before any action and must have active branch (e.g. with readme).
|
||||
* ``timeout`` - HTTP request timeout in seconds, int, optional, default is ``30``.
|
||||
* ``username`` - Github authorization user, string, required. Basically the same as ``owner``.
|
||||
* ``use_full_release_name`` - if set to ``yes``, the release will contain both repository name and architecture, and only architecture otherwise, boolean, optional, default ``no`` (legacy behavior).
|
||||
* ``username`` - GitHub authorization user, string, required. Basically the same as ``owner``.
|
||||
|
||||
``remote-service`` type
|
||||
^^^^^^^^^^^^^^^^^^^^^^^
|
||||
@ -329,9 +334,10 @@ Requires ``rsync`` package to be installed. Do not forget to configure ssh for u
|
||||
|
||||
Requires ``boto3`` library to be installed. Section name must be either ``s3`` (plus optional architecture name, e.g. ``s3:x86_64``) or random name with ``type`` set.
|
||||
|
||||
* ``type`` - type of the upload, string, optional, must be set to ``github`` if exists.
|
||||
* ``type`` - type of the upload, string, optional, must be set to ``s3`` if exists.
|
||||
* ``access_key`` - AWS access key ID, string, required.
|
||||
* ``bucket`` - bucket name (e.g. ``bucket``), string, required.
|
||||
* ``chunk_size`` - chunk size for calculating entity tags, int, optional, default 8 * 1024 * 1024.
|
||||
* ``object_path`` - path prefix for stored objects, string, optional. If none set, the prefix as in repository tree will be used.
|
||||
* ``region`` - bucket region (e.g. ``eu-central-1``), string, required.
|
||||
* ``secret_key`` - AWS secret access key, string, required.
|
||||
99
docs/faq.rst
99
docs/faq.rst
@ -17,8 +17,8 @@ TL;DR
|
||||
.. code-block:: shell
|
||||
|
||||
yay -S ahriman
|
||||
ahriman -a x86_64 service-setup --packager "ahriman bot <ahriman@example.com>" --repository "repository"
|
||||
systemctl enable --now ahriman@x86_64.timer
|
||||
ahriman -a x86_64 -r aur-clone service-setup --packager "ahriman bot <ahriman@example.com>"
|
||||
systemctl enable --now ahriman@x86_64-aur-clone.timer
|
||||
|
||||
Long answer
|
||||
"""""""""""
|
||||
@ -32,7 +32,7 @@ There is special command which can be used in order to validate current configur
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
ahriman -a x86_64 service-config-validate --exit-code
|
||||
ahriman service-config-validate --exit-code
|
||||
|
||||
This command will print found errors, based on `cerberus <https://docs.python-cerberus.org/>`_, e.g.:
|
||||
|
||||
@ -71,7 +71,7 @@ states that default build command is ``extra-x86_64-build``. But if there is sec
|
||||
[build:i686]
|
||||
build_command = extra-i686-build
|
||||
|
||||
the ``extra-i686-build`` command will be used for ``i686`` architecture.
|
||||
the ``extra-i686-build`` command will be used for ``i686`` architecture. You can also override settings for different repositories and architectures; in this case section names will be ``build:aur-clone`` (repository name only) and ``build:aur-clone:i686`` (both repository name and architecture).
|
||||
|
||||
How to generate build reports
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
@ -121,7 +121,7 @@ How do I add new package
|
||||
|
||||
sudo -u ahriman ahriman package-add ahriman --now
|
||||
|
||||
``--now`` flag is totally optional and just run ``repo-update`` subcommand after the registering the new package, Thus the extended flow is the following:
|
||||
``--now`` flag is totally optional and just run ``repo-update`` subcommand after the registering the new package. Thus the extended flow is the following:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
@ -209,7 +209,7 @@ So it is the same as adding any other package, but due to restrictions you must
|
||||
|
||||
sudo -u ahriman ahriman package-add pacman -s repository
|
||||
|
||||
This feature is heavily depends on local pacman cache. In order to use this feature it is recommended to either run ``pacman -Sy`` before the interaction or configure timer for this.
|
||||
This feature is heavily depends on local pacman cache. In order to use this feature it is recommended to either run ``pacman -Sy`` before the interaction or use internal application cache with ``--refresh`` flag.
|
||||
|
||||
Package build fails because it cannot validate PGP signature of source files
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
@ -317,7 +317,7 @@ Add the following lines to your ``pacman.conf``:
|
||||
.. code-block:: ini
|
||||
|
||||
[repository]
|
||||
Server = file:///var/lib/ahriman/repository/x86_64
|
||||
Server = file:///var/lib/ahriman/repository/$repo/$arch
|
||||
|
||||
(You might need to add ``SigLevel`` option according to the pacman documentation.)
|
||||
|
||||
@ -359,7 +359,14 @@ Example of the status page configuration is the following (status service is usi
|
||||
Docker image
|
||||
------------
|
||||
|
||||
We provide official images which can be found under ``arcan1s/ahriman`` repository. Docker image is being updated on each commit to master as well as on each version. If you would like to use last (probably unstable) build you can use ``edge`` tag or ``latest`` for any tagged versions; otherwise you can use any version tag available.
|
||||
We provide official images which can be found under:
|
||||
|
||||
* docker registry ``arcan1s/ahriman``;
|
||||
* ghcr.io registry ``ghcr.io/arcan1s/ahriman``;
|
||||
|
||||
These images are totally identical.
|
||||
|
||||
Docker image is being updated on each commit to master as well as on each version. If you would like to use last (probably unstable) build you can use ``edge`` tag or ``latest`` for any tagged versions; otherwise you can use any version tag available.
|
||||
|
||||
The default action (in case if no arguments provided) is ``repo-update``. Basically the idea is to run container, e.g.:
|
||||
|
||||
@ -456,22 +463,22 @@ Physical server setup
|
||||
In this example we are going to use files and packages which are provided by official repositories of the used architecture. Note, that versions might be different, thus you need to find correct versions on the distribution web site, e.g. `archlinux32 packages <https://www.archlinux32.org/packages/>`_.
|
||||
|
||||
#.
|
||||
First, considering having base Arch Linux system, we need to install keyring for the specified repositories:
|
||||
First, considering having base Arch Linux system, we need to install keyring for the specified repositories, e.g.:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
wget http://pool.mirror.archlinux32.org/i686/core/archlinux32-keyring-20220927-1.0-any.pkg.tar.zst
|
||||
pacman -U archlinux32-keyring-20220927-1.0-any.pkg.tar.zst
|
||||
wget http://pool.mirror.archlinux32.org/i686/core/archlinux32-keyring-20230705-1.0-any.pkg.tar.zst
|
||||
pacman -U archlinux32-keyring-20230705-1.0-any.pkg.tar.zst
|
||||
|
||||
#.
|
||||
In order to run ``devtools`` scripts for custom architecture they also need specific ``makepkg`` configuration, it can be retrieved by installing the ``devtools`` package of the distribution:
|
||||
In order to run ``devtools`` scripts for custom architecture they also need specific ``makepkg`` configuration, it can be retrieved by installing the ``devtools`` package of the distribution, e.g.:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
wget http://pool.mirror.archlinux32.org/i686/extra/devtools-20221208-1.0-any.pkg.tar.zst
|
||||
pacman -U devtools-20221208-1.0-any.pkg.tar.zst
|
||||
wget http://pool.mirror.archlinux32.org/i686/extra/devtools-20221208-1.2-any.pkg.tar.zst
|
||||
pacman -U devtools-20221208-1.2-any.pkg.tar.zst
|
||||
|
||||
Alternatively, you can create your own ``makepkg`` configuration and save it as ``/usr/share/devtools/makepkg-i686.conf``.
|
||||
Alternatively, you can create your own ``makepkg`` configuration and save it as ``/usr/share/devtools/makepkg.conf.d/i686.conf``.
|
||||
|
||||
#.
|
||||
Setup repository as usual:
|
||||
@ -485,6 +492,9 @@ In this example we are going to use files and packages which are provided by off
|
||||
* ``--mirror`` - link to the mirrors which will be used instead of official repositories.
|
||||
* ``--no-multilib`` - in the example we are using i686 architecture for which multilib repository doesn't exist.
|
||||
|
||||
#.
|
||||
That's all Folks!
|
||||
|
||||
Docker container setup
|
||||
^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
@ -510,8 +520,8 @@ There are two possible ways to achieve same setup, by using docker container. Th
|
||||
.. code-block:: dockerfile
|
||||
|
||||
RUN pacman --noconfirm -Sy wget
|
||||
RUN wget http://pool.mirror.archlinux32.org/i686/extra/devtools-20221208-1.0-any.pkg.tar.zst && pacman --noconfirm -U devtools-20221208-1.0-any.pkg.tar.zst
|
||||
RUN wget http://pool.mirror.archlinux32.org/i686/core/archlinux32-keyring-20220927-1.0-any.pkg.tar.zst && pacman --noconfirm -U archlinux32-keyring-20220927-1.0-any.pkg.tar.zst
|
||||
RUN wget http://pool.mirror.archlinux32.org/i686/extra/devtools-20221208-1.2-any.pkg.tar.zst && pacman --noconfirm -U devtools-20221208-1.2-any.pkg.tar.zst
|
||||
RUN wget http://pool.mirror.archlinux32.org/i686/core/archlinux32-keyring-20230705-1.0-any.pkg.tar.zst && pacman --noconfirm -U archlinux32-keyring-20230705-1.0-any.pkg.tar.zst
|
||||
|
||||
#.
|
||||
At that point you should have full ``Dockerfile`` like:
|
||||
@ -523,8 +533,8 @@ There are two possible ways to achieve same setup, by using docker container. Th
|
||||
RUN pacman-key --init
|
||||
|
||||
RUN pacman --noconfirm -Sy wget
|
||||
RUN wget http://pool.mirror.archlinux32.org/i686/extra/devtools-20221208-1.0-any.pkg.tar.zst && pacman --noconfirm -U devtools-20221208-1.0-any.pkg.tar.zst
|
||||
RUN wget http://pool.mirror.archlinux32.org/i686/core/archlinux32-keyring-20220927-1.0-any.pkg.tar.zst && pacman --noconfirm -U archlinux32-keyring-20220927-1.0-any.pkg.tar.zst
|
||||
RUN wget http://pool.mirror.archlinux32.org/i686/extra/devtools-20221208-1.2-any.pkg.tar.zst && pacman --noconfirm -U devtools-20221208-1.2-any.pkg.tar.zst
|
||||
RUN wget http://pool.mirror.archlinux32.org/i686/core/archlinux32-keyring-20230705-1.0-any.pkg.tar.zst && pacman --noconfirm -U archlinux32-keyring-20230705-1.0-any.pkg.tar.zst
|
||||
|
||||
#.
|
||||
After that you can build you own container, e.g.:
|
||||
@ -554,8 +564,8 @@ There are several choices:
|
||||
.. code-block::
|
||||
|
||||
server {
|
||||
location /x86_64 {
|
||||
root /var/lib/ahriman/repository/x86_64;
|
||||
location / {
|
||||
root /var/lib/ahriman/repository/;
|
||||
autoindex on;
|
||||
}
|
||||
}
|
||||
@ -571,7 +581,7 @@ There are several choices:
|
||||
[rsync]
|
||||
remote = 192.168.0.1:/srv/repo
|
||||
|
||||
After that just add ``/srv/repo`` to the ``pacman.conf`` as usual. You can also upload to S3 (e.g. ``Server = https://s3.eu-central-1.amazonaws.com/repository/x86_64``) or to Github (e.g. ``Server = https://github.com/ahriman/repository/releases/download/x86_64``).
|
||||
After that just add ``/srv/repo`` to the ``pacman.conf`` as usual. You can also upload to S3 (``Server = https://s3.eu-central-1.amazonaws.com/repository/aur-clone/x86_64``) or to Github (``Server = https://github.com/ahriman/repository/releases/download/aur-clone-x86_64``).
|
||||
|
||||
How to sync to S3
|
||||
^^^^^^^^^^^^^^^^^
|
||||
@ -632,6 +642,23 @@ How to sync to S3
|
||||
region = eu-central-1
|
||||
secret_key = ...
|
||||
|
||||
S3 with SSL
|
||||
"""""""""""
|
||||
|
||||
In order to configure S3 on custom domain with SSL (and some other features, like redirects), the CloudFront should be used.
|
||||
|
||||
#. Configure S3 as described above.
|
||||
#. In bucket properties, enable static website hosting with hosting type "Host a static website".
|
||||
#. Go to AWS Certificate Manager and create public ceritificate on your domain. Validate domain as suggested.
|
||||
#. Go to CloudFront and create distribution. The following settings are required:
|
||||
|
||||
* Origin domain choose S3 bucket.
|
||||
* Tick use website endpoint.
|
||||
* Disable caching.
|
||||
* Select issued certificate.
|
||||
|
||||
#. Point DNS record to CloudFront address.
|
||||
|
||||
How to sync to Github releases
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
@ -676,7 +703,7 @@ How to report by email
|
||||
|
||||
[email]
|
||||
host = smtp.example.com
|
||||
link_path = http://example.com/x86_64
|
||||
link_path = http://example.com/aur-clone/x86_64
|
||||
password = ...
|
||||
port = 465
|
||||
receivers = me@example.com
|
||||
@ -702,10 +729,10 @@ How to generate index page for S3
|
||||
target = html
|
||||
|
||||
[html]
|
||||
path = /var/lib/ahriman/repository/x86_64/index.html
|
||||
link_path = http://example.com/x86_64
|
||||
path = /var/lib/ahriman/repository/aur-clone/x86_64/index.html
|
||||
link_path = http://example.com/aur-clone/x86_64
|
||||
|
||||
After these steps ``index.html`` file will be automatically synced to S3
|
||||
After these steps ``index.html`` file will be automatically synced to S3.
|
||||
|
||||
How to post build report to telegram
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
@ -741,7 +768,7 @@ How to post build report to telegram
|
||||
[telegram]
|
||||
api_key = aaAAbbBBccCC
|
||||
chat_id = @ahriman
|
||||
link_path = http://example.com/x86_64
|
||||
link_path = http://example.com/aur-clone/x86_64
|
||||
|
||||
``api_key`` is the one sent by `@BotFather <https://t.me/botfather>`_, ``chat_id`` is the value retrieved from previous step.
|
||||
|
||||
@ -756,7 +783,7 @@ If you did everything fine you should receive the message with the next update.
|
||||
Distributed builds
|
||||
------------------
|
||||
|
||||
The service allows to run build on multiple machines and collect packages on main node. There are multiple ways to achieve it, this section describes officially supported methods.
|
||||
The service allows to run build on multiple machines and collect packages on main node. There are several ways to achieve it, this section describes officially supported methods.
|
||||
|
||||
Remote synchronization and remote server call
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
@ -933,7 +960,7 @@ Command to run worker node:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
docker run --privileged -v worker.ini:/etc/ahriman.ini.d/overrides.ini -it arcan1s/ahriman:latest package-add arhiman --now
|
||||
docker run --privileged -v worker.ini:/etc/ahriman.ini.d/overrides.ini -it arcan1s/ahriman:latest package-add ahriman --now
|
||||
|
||||
The command above will successfully build ``ahriman`` package, upload it on master node and, finally, will update master node repository.
|
||||
|
||||
@ -1046,7 +1073,7 @@ How to setup web service
|
||||
port = 8080
|
||||
|
||||
#.
|
||||
Start the web service ``systemctl enable --now ahriman-web@x86_64``.
|
||||
Start the web service ``systemctl enable --now ahriman-web@x86_64-aur-clone``.
|
||||
|
||||
How to enable basic authorization
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
@ -1059,7 +1086,7 @@ How to enable basic authorization
|
||||
yay -S --asdeps python-aiohttp-security python-aiohttp-session python-cryptography
|
||||
|
||||
#.
|
||||
Configure the service to enable authorization (``salt`` can be generated as any random string):
|
||||
Configure the service to enable authorization (``salt`` can be generated as any random string and optional):
|
||||
|
||||
.. code-block:: ini
|
||||
|
||||
@ -1087,7 +1114,7 @@ How to enable basic authorization
|
||||
|
||||
sudo -u ahriman ahriman user-add -r full api
|
||||
|
||||
This command will ask for the password, just type it in stdin; *do not* leave the field blank, user will not be able to authorize, and finally configure the application:
|
||||
This command will ask for the password, just type it in stdin; **do not** leave the field blank, user will not be able to authorize, and finally configure the application:
|
||||
|
||||
.. code-block:: ini
|
||||
|
||||
@ -1103,7 +1130,7 @@ How to enable basic authorization
|
||||
sudo -u ahriman ahriman user-add -r full my-first-user
|
||||
|
||||
#.
|
||||
Restart web service ``systemctl restart ahriman-web@x86_64``.
|
||||
Restart web service ``systemctl restart ahriman-web@x86_64-aur-clone``.
|
||||
|
||||
How to enable OAuth authorization
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
@ -1149,7 +1176,7 @@ How to enable OAuth authorization
|
||||
When it will ask for the password leave it blank.
|
||||
|
||||
#.
|
||||
Restart web service ``systemctl restart ahriman-web@x86_64``.
|
||||
Restart web service ``systemctl restart ahriman-web@x86_64-aur-clone``.
|
||||
|
||||
How to implement own interface
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
@ -1265,7 +1292,9 @@ You can also ask to forward logs to ``stderr``, just set ``--log-handler`` flag,
|
||||
|
||||
ahriman --log-handler console ...
|
||||
|
||||
You can even configure logging as you wish, but kindly refer to python ``logging`` module `configuration <https://docs.python.org/3/library/logging.config.html>`_. The application uses java concept to log messages, e.g. class ``Application`` imported from ``ahriman.application.application`` package will have logger called ``ahriman.application.application.Application``. In order to e.g. change logger name for whole application package it is possible to change values for ``ahriman.application`` package; thus editing ``ahriman`` logger configuration will change logging for whole application (unless there are overrides for another logger).
|
||||
You can even configure logging as you wish, but kindly refer to python ``logging`` module `configuration <https://docs.python.org/3/library/logging.config.html>`_.
|
||||
|
||||
The application uses java concept to log messages, e.g. class ``Application`` imported from ``ahriman.application.application`` package will have logger called ``ahriman.application.application.Application``. In order to e.g. change logger name for whole application package it is possible to change values for ``ahriman.application`` package; thus editing ``ahriman`` logger configuration will change logging for whole application (unless there are overrides for another logger).
|
||||
|
||||
Html customization
|
||||
^^^^^^^^^^^^^^^^^^
|
||||
|
||||
@ -34,6 +34,7 @@ Contents
|
||||
configuration
|
||||
command-line
|
||||
faq
|
||||
migration
|
||||
architecture
|
||||
advanced-usage
|
||||
triggers
|
||||
|
||||
55
docs/migration.rst
Normal file
55
docs/migration.rst
Normal file
@ -0,0 +1,55 @@
|
||||
Manual migrations
|
||||
=================
|
||||
|
||||
Normally most of migrations are handled automatically after application start. However, some upgrades require manual interventions; this document describes them.
|
||||
|
||||
Upgrades to breakpoints
|
||||
-----------------------
|
||||
|
||||
To 2.9.0
|
||||
^^^^^^^^
|
||||
|
||||
This release includes major upgrade for the newest devtools and archlinux repository structure. In order to upgrade package need to:
|
||||
|
||||
#. Upgrade to the latest major release of python (3.11) (required by other changes).
|
||||
#. Upgrade devtools to the latest release.
|
||||
#. Backup your settings, ``/etc/ahriman.ini.d/00-setup-overrides.ini`` by default.
|
||||
#. Run setup command (i.e. ``ahriman service-setup``) again with the same arguments as you used before. This step can be done manually by moving ``devtools`` configuration (something like ``/usr/share/devtools/pacman-ahriman*.conf``) to new location ``/usr/share/devtools/pacman.conf.d/`` under name ``ahriman.conf``. After that make sure to remove any ``community`` mentions from configurations (e.g. ``/usr/share/devtools/pacman.conf.d/ahriman.conf``, ``/etc/ahriman.ini``) if there were any. The only thing which will change is ``devtools`` configuration.
|
||||
#. Remove build chroot as it is incompatible, e.g. ``sudo ahriman service-clean --chroot``.
|
||||
#. Run ``sudo -u ahriman ahriman update --no-aur --no-local --no-manual -yy`` in order to update local databases.
|
||||
|
||||
To 2.12.0
|
||||
^^^^^^^^^
|
||||
|
||||
This release includes paths migration. Unlike usual case, no automatic migration is performed because it might break user configuration. The following noticeable changes have been made:
|
||||
|
||||
* Path to pre-built packages now includes repository name, i.e. it has been changed from ``/var/lib/ahriman/packages/x86_64`` to ``/var/lib/ahriman/packages/aur-clone/x86_64``.
|
||||
* Path to pacman databases now includes repository name too, it has been changed from ``/var/lib/ahriman/pacman/x86_64`` to ``/var/lib/ahriman/pacman/aur-clone/x86_64``.
|
||||
* Path to repository itself also includes repository name, from ``/var/lib/ahriman/repository/x86_64`` to ``/var/lib/ahriman/repository/aur-clone/x86_64``.
|
||||
|
||||
In order to migrate to new filesystem tree the following actions are required:
|
||||
|
||||
#.
|
||||
Stop and disable all services, e.g. timer and web service:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
sudo systemctl disable --now ahriman@x86_64.timer
|
||||
sudo systemctl disable --now ahriman-web@x86_64
|
||||
|
||||
#.
|
||||
Create directory tree. It can be done by running ``ahriman service-tree-migrate`` subcommand. It performs copying between the old repository tree and the new one. Alternatively you can copy directories by hands.
|
||||
|
||||
#.
|
||||
Edit configuration in case if anything is pointing to the old path, e.g. HTML report generation, in the way in which it will be pointed to directory inside repository specific one, e.g. ``/var/lib/ahriman/repository/x86_64`` to ``/var/lib/ahriman/repository/aur-clone/x86_64``.
|
||||
|
||||
#.
|
||||
Make sure to update remote synchronization services if any. Almost all of them rely on current repository tree by default, so you need to setup either redirects or configure to synchronize to the old locations (e.g. ``object_path`` option for S3 synchronization).
|
||||
|
||||
#.
|
||||
Enable and start services again. Unit template parameter should include both repository architecture and name, dash separated, e.g. ``x86_64-aur-clone``:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
sudo systemctl enable --now ahriman@x86_64-aur-clone.timer
|
||||
sudo systemctl enable --now ahriman-web@x86_64-aur-clone
|
||||
@ -10,7 +10,7 @@ Initial setup
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
sudo ahriman -a x86_64 service-setup ...
|
||||
sudo ahriman -a x86_64 -r aur-clone service-setup ...
|
||||
|
||||
``service-setup`` literally does the following steps:
|
||||
|
||||
@ -29,26 +29,26 @@ Initial setup
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
ln -s /usr/bin/archbuild /usr/local/bin/ahriman-x86_64-build
|
||||
ln -s /usr/bin/archbuild /usr/local/bin/aur-clone-x86_64-build
|
||||
|
||||
#.
|
||||
Create configuration file (same as previous ``{name}.conf``):
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
cp /usr/share/devtools/pacman.conf.d/{extra,ahriman}.conf
|
||||
cp /usr/share/devtools/pacman.conf.d/{extra,aur-clone}.conf
|
||||
|
||||
#.
|
||||
Change configuration file, add your own repository, add multilib repository etc:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
echo '[multilib]' | tee -a /usr/share/devtools/pacman-ahriman.conf
|
||||
echo 'Include = /etc/pacman.d/mirrorlist' | tee -a /usr/share/devtools/pacman.conf.d/ahriman.conf
|
||||
echo '[multilib]' | tee -a /usr/share/devtools/pacman.conf.d/aur-clone-x86_64.conf
|
||||
echo 'Include = /etc/pacman.d/mirrorlist' | tee -a /usr/share/devtools/pacman.conf.d/aur-clone-x86_64.conf
|
||||
|
||||
echo '[aur-clone]' | tee -a /usr/share/devtools/pacman-ahriman.conf
|
||||
echo 'SigLevel = Optional TrustAll' | tee -a /usr/share/devtools/pacman.conf.d/ahriman.conf
|
||||
echo 'Server = file:///var/lib/ahriman/repository/$arch' | tee -a /usr/share/devtools/pacman.conf.d/ahriman.conf
|
||||
echo '[aur-clone]' | tee -a /usr/share/devtools/pacman.conf.d/aur-clone-x86_64.conf
|
||||
echo 'SigLevel = Optional TrustAll' | tee -a /usr/share/devtools/pacman.conf.d/aur-clone-x86_64.conf
|
||||
echo 'Server = file:///var/lib/ahriman/repository/$repo/$arch' | tee -a /usr/share/devtools/pacman.conf.d/aur-clone-x86_64.conf
|
||||
|
||||
#.
|
||||
Set ``build_command`` option to point to your command:
|
||||
@ -56,14 +56,14 @@ Initial setup
|
||||
.. code-block:: shell
|
||||
|
||||
echo '[build]' | tee -a /etc/ahriman.ini.d/build.ini
|
||||
echo 'build_command = ahriman-x86_64-build' | tee -a /etc/ahriman.ini.d/build.ini
|
||||
echo 'build_command = aur-clone-x86_64-build' | tee -a /etc/ahriman.ini.d/build.ini
|
||||
|
||||
#.
|
||||
Configure ``/etc/sudoers.d/ahriman`` to allow running command without a password:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
echo 'Cmnd_Alias CARCHBUILD_CMD = /usr/local/bin/ahriman-x86_64-build *' | tee -a /etc/sudoers.d/ahriman
|
||||
echo 'Cmnd_Alias CARCHBUILD_CMD = /usr/local/bin/aur-clone-x86_64-build *' | tee -a /etc/sudoers.d/ahriman
|
||||
echo 'ahriman ALL=(ALL) NOPASSWD:SETENV: CARCHBUILD_CMD' | tee -a /etc/sudoers.d/ahriman
|
||||
chmod 400 /etc/sudoers.d/ahriman
|
||||
|
||||
@ -74,20 +74,20 @@ Initial setup
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
systemctl enable --now ahriman@x86_64.timer
|
||||
systemctl enable --now ahriman@x86_64-aur-clone.timer
|
||||
|
||||
#.
|
||||
Start and enable status page:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
systemctl enable --now ahriman-web@x86_64
|
||||
systemctl enable --now ahriman-web@x86_64-aur-clone
|
||||
|
||||
#.
|
||||
Add packages by using ``ahriman package-add {package}`` command:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
sudo -u ahriman ahriman -a x86_64 package-add ahriman --now --refresh
|
||||
sudo -u ahriman ahriman package-add ahriman --now --refresh
|
||||
|
||||
The ``--refresh`` flag is required in order to handle local database update.
|
||||
|
||||
Reference in New Issue
Block a user