arcanis/ahriman
1
0
Fork 0
mirror of https://github.com/arcan1s/ahriman.git synced 2025-07-16 07:19:57 +00:00
Code Issues Projects Releases Wiki Activity

add migration docs and ability to migrate tree automatically

Browse Source
This commit is contained in:
Evgenii Alekseev
2023-09-01 16:52:21 +03:00
parent 0eebcc5740
commit 1b5bfe5088
24 changed files with 469 additions and 131 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
docs/ahriman.application.handlers.rst
View File

@ -172,6 +172,14 @@ ahriman.application.handlers.structure module
:no-undoc-members:
:show-inheritance:
ahriman.application.handlers.tree\_migrate module
-------------------------------------------------
.. automodule:: ahriman.application.handlers.tree_migrate
:members:
:no-undoc-members:
:show-inheritance:
ahriman.application.handlers.triggers module
--------------------------------------------

47
docs/architecture.rst
View File

@ -102,6 +102,53 @@ All subcommands are divided into several groups depending on the role they are d
For historical reasons and in order to keep backward compatibility some subcommands have aliases to their shorter forms or even other groups, but the service doesn't guarantee that they will remain unchanged.
Filesystem tree
---------------
The application supports two types of trees, one is for legacy configuration (when there were no repository name explicit configuration available) and another one is for new-style tree. This document describes only new-style tree in order to avoid deprecated structure.
Having default root as ``/var/lib/ahriman`` (differs from container though), the directory structure is the following:
.. code-block::
/var/lib/ahriman/
├── ahriman.db
├── cache
├── chroot
│ └── aur-clone
├── packages
│ └── aur-clone
│ └── x86_64
├── pacman
│ └── aur-clone
│ └── x86_64
│ ├── local
│ │ └── ALPM_DB_VERSION
│ └── sync
│ ├── core.db
│ ├── extra.db
│ └── multilib.db
│
└── repository
└── aur-clone
└── x86_64
├── aur-clone.db -> aur-clone.db.tar.gz
├── aur-clone.db.tar.gz
├── aur-clone.files -> aur-clone.files.tar.gz
└── aur-clone.files.tar.gz
There are multiple subdirectories, some of them are commons for any repository, but some of them are not.
* ``cache`` is a directory with locally stored PKGBUILD's and VCS packages. It is common for all repositories and architectures.
* ``chroot/{repository}`` is a chroot directory for ``devtools``. It is specific for each repository, but shared for different architectures inside (the ``devtools`` handles architectures automatically).
* ``packages/{repository}/{architecture}`` is a directory with prebuilt packages. When package is built, first it will be uploaded to this directory and later will be handled by update process. It is architecture and repository specific.
* ``pacman/{repository}/{architecture}`` is repository and architecture specific caches for pacman's databases.
* ``repository/{repository}/{architecture}`` is a repository packages directory.
Normally you should avoid direct interaction with the application tree.
For tree migration process refer to the :doc:`migration notes <migration>`.
Database
--------

60
docs/faq.rst
View File

@ -71,7 +71,7 @@ states that default build command is ``extra-x86_64-build``. But if there is sec
[build:i686]
build_command = extra-i686-build
the ``extra-i686-build`` command will be used for ``i686`` architecture.
the ``extra-i686-build`` command will be used for ``i686`` architecture. You can also override settings for different repositories and architectures; in this case section names will be ``build:aur-clone`` (repository name only) and ``build:aur-clone:i686`` (both repository name and architecture).
How to generate build reports
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@ -121,7 +121,7 @@ How do I add new package
sudo -u ahriman ahriman package-add ahriman --now
``--now`` flag is totally optional and just run ``repo-update`` subcommand after the registering the new package, Thus the extended flow is the following:
``--now`` flag is totally optional and just run ``repo-update`` subcommand after the registering the new package. Thus the extended flow is the following:
.. code-block:: shell
@ -209,7 +209,7 @@ So it is the same as adding any other package, but due to restrictions you must
sudo -u ahriman ahriman package-add pacman -s repository
This feature is heavily depends on local pacman cache. In order to use this feature it is recommended to either run ``pacman -Sy`` before the interaction or configure timer for this.
This feature is heavily depends on local pacman cache. In order to use this feature it is recommended to either run ``pacman -Sy`` before the interaction or use internal application cache with ``--refresh`` flag.
Package build fails because it cannot validate PGP signature of source files
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@ -359,7 +359,14 @@ Example of the status page configuration is the following (status service is usi
Docker image
------------
We provide official images which can be found under ``arcan1s/ahriman`` repository. Docker image is being updated on each commit to master as well as on each version. If you would like to use last (probably unstable) build you can use ``edge`` tag or ``latest`` for any tagged versions; otherwise you can use any version tag available.
We provide official images which can be found under:
* docker registry ``arcan1s/ahriman``;
* ghcr.io registry ``ghcr.io/arcan1s/ahriman``;
These images are totally identical.
Docker image is being updated on each commit to master as well as on each version. If you would like to use last (probably unstable) build you can use ``edge`` tag or ``latest`` for any tagged versions; otherwise you can use any version tag available.
The default action (in case if no arguments provided) is ``repo-update``. Basically the idea is to run container, e.g.:
@ -456,22 +463,22 @@ Physical server setup
In this example we are going to use files and packages which are provided by official repositories of the used architecture. Note, that versions might be different, thus you need to find correct versions on the distribution web site, e.g. `archlinux32 packages <https://www.archlinux32.org/packages/>`_.
#.
First, considering having base Arch Linux system, we need to install keyring for the specified repositories:
First, considering having base Arch Linux system, we need to install keyring for the specified repositories, e.g.:
.. code-block:: shell
wget http://pool.mirror.archlinux32.org/i686/core/archlinux32-keyring-20220927-1.0-any.pkg.tar.zst
pacman -U archlinux32-keyring-20220927-1.0-any.pkg.tar.zst
wget http://pool.mirror.archlinux32.org/i686/core/archlinux32-keyring-20230705-1.0-any.pkg.tar.zst
pacman -U archlinux32-keyring-20230705-1.0-any.pkg.tar.zst
#.
In order to run ``devtools`` scripts for custom architecture they also need specific ``makepkg`` configuration, it can be retrieved by installing the ``devtools`` package of the distribution:
In order to run ``devtools`` scripts for custom architecture they also need specific ``makepkg`` configuration, it can be retrieved by installing the ``devtools`` package of the distribution, e.g.:
.. code-block:: shell
wget http://pool.mirror.archlinux32.org/i686/extra/devtools-20221208-1.0-any.pkg.tar.zst
pacman -U devtools-20221208-1.0-any.pkg.tar.zst
wget http://pool.mirror.archlinux32.org/i686/extra/devtools-20221208-1.2-any.pkg.tar.zst
pacman -U devtools-20221208-1.2-any.pkg.tar.zst
Alternatively, you can create your own ``makepkg`` configuration and save it as ``/usr/share/devtools/makepkg-i686.conf``.
Alternatively, you can create your own ``makepkg`` configuration and save it as ``/usr/share/devtools/makepkg.conf.d/i686.conf``.
#.
Setup repository as usual:
@ -485,6 +492,9 @@ In this example we are going to use files and packages which are provided by off
* ``--mirror`` - link to the mirrors which will be used instead of official repositories.
* ``--no-multilib`` - in the example we are using i686 architecture for which multilib repository doesn't exist.
#.
That's all Folks!
Docker container setup
^^^^^^^^^^^^^^^^^^^^^^
@ -510,8 +520,8 @@ There are two possible ways to achieve same setup, by using docker container. Th
.. code-block:: dockerfile
RUN pacman --noconfirm -Sy wget
RUN wget http://pool.mirror.archlinux32.org/i686/extra/devtools-20221208-1.0-any.pkg.tar.zst && pacman --noconfirm -U devtools-20221208-1.0-any.pkg.tar.zst
RUN wget http://pool.mirror.archlinux32.org/i686/core/archlinux32-keyring-20220927-1.0-any.pkg.tar.zst && pacman --noconfirm -U archlinux32-keyring-20220927-1.0-any.pkg.tar.zst
RUN wget http://pool.mirror.archlinux32.org/i686/extra/devtools-20221208-1.2-any.pkg.tar.zst && pacman --noconfirm -U devtools-20221208-1.2-any.pkg.tar.zst
RUN wget http://pool.mirror.archlinux32.org/i686/core/archlinux32-keyring-20230705-1.0-any.pkg.tar.zst && pacman --noconfirm -U archlinux32-keyring-20230705-1.0-any.pkg.tar.zst
#.
At that point you should have full ``Dockerfile`` like:
@ -523,8 +533,8 @@ There are two possible ways to achieve same setup, by using docker container. Th
RUN pacman-key --init
RUN pacman --noconfirm -Sy wget
RUN wget http://pool.mirror.archlinux32.org/i686/extra/devtools-20221208-1.0-any.pkg.tar.zst && pacman --noconfirm -U devtools-20221208-1.0-any.pkg.tar.zst
RUN wget http://pool.mirror.archlinux32.org/i686/core/archlinux32-keyring-20220927-1.0-any.pkg.tar.zst && pacman --noconfirm -U archlinux32-keyring-20220927-1.0-any.pkg.tar.zst
RUN wget http://pool.mirror.archlinux32.org/i686/extra/devtools-20221208-1.2-any.pkg.tar.zst && pacman --noconfirm -U devtools-20221208-1.2-any.pkg.tar.zst
RUN wget http://pool.mirror.archlinux32.org/i686/core/archlinux32-keyring-20230705-1.0-any.pkg.tar.zst && pacman --noconfirm -U archlinux32-keyring-20230705-1.0-any.pkg.tar.zst
#.
After that you can build you own container, e.g.:
@ -554,8 +564,8 @@ There are several choices:
.. code-block::
server {
location /aur-clone/x86_64 {
root /var/lib/ahriman/repository/aur-clone/x86_64;
location / {
root /var/lib/ahriman/repository/;
autoindex on;
}
}
@ -571,7 +581,7 @@ There are several choices:
[rsync]
remote = 192.168.0.1:/srv/repo
After that just add ``/srv/repo`` to the ``pacman.conf`` as usual. You can also upload to S3 (e.g. ``Server = https://s3.eu-central-1.amazonaws.com/repository/aur-clone/x86_64``) or to Github (e.g. ``Server = https://github.com/ahriman/repository/releases/download/aur-clone-x86_64``).
After that just add ``/srv/repo`` to the ``pacman.conf`` as usual. You can also upload to S3 (``Server = https://s3.eu-central-1.amazonaws.com/repository/aur-clone/x86_64``) or to Github (``Server = https://github.com/ahriman/repository/releases/download/aur-clone-x86_64``).
How to sync to S3
^^^^^^^^^^^^^^^^^
@ -705,7 +715,7 @@ How to generate index page for S3
path = /var/lib/ahriman/repository/aur-clone/x86_64/index.html
link_path = http://example.com/aur-clone/x86_64
After these steps ``index.html`` file will be automatically synced to S3
After these steps ``index.html`` file will be automatically synced to S3.
How to post build report to telegram
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@ -756,7 +766,7 @@ If you did everything fine you should receive the message with the next update.
Distributed builds
------------------
The service allows to run build on multiple machines and collect packages on main node. There are multiple ways to achieve it, this section describes officially supported methods.
The service allows to run build on multiple machines and collect packages on main node. There are several ways to achieve it, this section describes officially supported methods.
Remote synchronization and remote server call
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@ -933,7 +943,7 @@ Command to run worker node:
.. code-block:: shell
docker run --privileged -v worker.ini:/etc/ahriman.ini.d/overrides.ini -it arcan1s/ahriman:latest package-add arhiman --now
docker run --privileged -v worker.ini:/etc/ahriman.ini.d/overrides.ini -it arcan1s/ahriman:latest package-add ahriman --now
The command above will successfully build ``ahriman`` package, upload it on master node and, finally, will update master node repository.
@ -1059,7 +1069,7 @@ How to enable basic authorization
yay -S --asdeps python-aiohttp-security python-aiohttp-session python-cryptography
#.
Configure the service to enable authorization (``salt`` can be generated as any random string):
Configure the service to enable authorization (``salt`` can be generated as any random string and optional):
.. code-block:: ini
@ -1087,7 +1097,7 @@ How to enable basic authorization
sudo -u ahriman ahriman user-add -r full api
This command will ask for the password, just type it in stdin; *do not* leave the field blank, user will not be able to authorize, and finally configure the application:
This command will ask for the password, just type it in stdin; **do not** leave the field blank, user will not be able to authorize, and finally configure the application:
.. code-block:: ini
@ -1265,7 +1275,9 @@ You can also ask to forward logs to ``stderr``, just set ``--log-handler`` flag,
ahriman --log-handler console ...
You can even configure logging as you wish, but kindly refer to python ``logging`` module `configuration <https://docs.python.org/3/library/logging.config.html>`_. The application uses java concept to log messages, e.g. class ``Application`` imported from ``ahriman.application.application`` package will have logger called ``ahriman.application.application.Application``. In order to e.g. change logger name for whole application package it is possible to change values for ``ahriman.application`` package; thus editing ``ahriman`` logger configuration will change logging for whole application (unless there are overrides for another logger).
You can even configure logging as you wish, but kindly refer to python ``logging`` module `configuration <https://docs.python.org/3/library/logging.config.html>`_.
The application uses java concept to log messages, e.g. class ``Application`` imported from ``ahriman.application.application`` package will have logger called ``ahriman.application.application.Application``. In order to e.g. change logger name for whole application package it is possible to change values for ``ahriman.application`` package; thus editing ``ahriman`` logger configuration will change logging for whole application (unless there are overrides for another logger).
Html customization
^^^^^^^^^^^^^^^^^^

52
docs/migration.rst Normal file
View File

@ -0,0 +1,52 @@
Manual migrations
=================
Normally most of migrations are handled automatically after application start. However, some upgrades require manual interventions; this document describes them.
Upgrades from versions
----------------------
To 2.9.0
^^^^^^^^
This release includes major upgrade for the newest devtools and archlinux repository structure. In order to upgrade package need to:
#. Upgrade to the latest major release of python (3.11) (required by other changes).
#. Upgrade devtools to the latest release.
#. Backup your settings, ``/etc/ahriman.ini.d/00-setup-overrides.ini`` by default.
#. Run setup command (i.e. ``ahriman service-setup``) again with the same arguments as you used before. This step can be done manually by moving ``devtools`` configuration (something like ``/usr/share/devtools/pacman-ahriman*.conf``) to new location ``/usr/share/devtools/pacman.conf.d/`` under name ``ahriman.conf``. After that make sure to remove any ``community`` mentions from configurations (e.g. ``/usr/share/devtools/pacman.conf.d/ahriman.conf``, ``/etc/ahriman.ini``) if there were any. The only thing which will change is ``devtools`` configuration.
#. Remove build chroot as it is incompatible, e.g. ``sudo ahriman service-clean --chroot``.
#. Run ``sudo -u ahriman ahriman update --no-aur --no-local --no-manual -yy`` in order to update local databases.
To 2.12.0
^^^^^^^^^
This release includes paths migration. Unlike usual case, no automatic migration is performed because it might break user configuration. The following noticeable changes have been made:
* Path to pre-built packages now includes repository name, i.e. it has been changed from ``/var/lib/ahriman/packages/x86_64`` to ``/var/lib/ahriman/packages/aur-clone/x86_64``.
* Path to pacman databases now includes repository name too, it has been changed from ``/var/lib/ahriman/pacman/x86_64`` to ``/var/lib/ahriman/pacman/aur-clone/x86_64``.
* Path to repository itself also includes repository name, from ``/var/lib/ahriman/repository/x86_64`` to ``/var/lib/ahriman/repository/aur-clone/x86_64``.
In order to migrate to new filesystem tree the following actions are required:
#.
Stop and disable all services, e.g. timer and web service:
.. code-block:: shell
sudo systemctl disable --now ahriman@x86_64.timer
sudo systemctl disable --now ahriman-web@x86_64
#.
Create directory tree. It can be done by running ``ahriman service-tree-migrate`` subcommand. It performs copying between the old repository tree and the new one. Alternatively you can copy directories by hands.
#.
Edit configuration in case if anything is pointing to the old path, e.g. HTML report generation, in the way in which it will be pointed to directory inside repository specific one, e.g. ``/var/lib/ahriman/repository/x86_64`` to ``/var/lib/ahriman/repository/aur-clone/x86_64``.
#.
Enable and start services again. Unit template parameter should include both repository architecture and name, dash separated, e.g. ``x86_64-aur-clone``:
.. code-block:: shell
sudo systemctl enable --now ahriman@x86_64-aur-clone.timer
sudo systemctl enable --now ahriman-web@x86_64-aur-clone