feat: allow to use one application for multiple repositories (#111)

* allow to use one application for multiple repositories

* update tests

* handle None append argument everywhere

* rewrite repository definition logic

* drop optional flags from docs

* support of new schema in systemd units

* add migration docs and ability to migrate tree automatically

* use repostory id instead

* verbose multiarchitectureerror

* object path support for s3 sync

* fix tests after rebase

docs/architecture.rst

@@ -102,6 +102,53 @@ All subcommands are divided into several groups depending on the role they are d
 
 For historical reasons and in order to keep backward compatibility some subcommands have aliases to their shorter forms or even other groups, but the service doesn't guarantee that they will remain unchanged.
 
+Filesystem tree
+---------------
+
+The application supports two types of trees, one is for legacy configuration (when there were no repository name explicit configuration available) and another one is for new-style tree. This document describes only new-style tree in order to avoid deprecated structure.
+
+Having default root as ``/var/lib/ahriman`` (differs from container though), the directory structure is the following:
+
+.. code-block::
+
+   /var/lib/ahriman/
+   ├── ahriman.db
+   ├── cache
+   ├── chroot
+   │   └── aur-clone
+   ├── packages
+   │   └── aur-clone
+   │       └── x86_64
+   ├── pacman
+   │   └── aur-clone
+   │       └── x86_64
+   │           ├── local
+   │           │   └── ALPM_DB_VERSION
+   │           └── sync
+   │               ├── core.db
+   │               ├── extra.db
+   │               └── multilib.db
+   │
+   └── repository
+       └── aur-clone
+           └── x86_64
+               ├── aur-clone.db -> aur-clone.db.tar.gz
+               ├── aur-clone.db.tar.gz
+               ├── aur-clone.files -> aur-clone.files.tar.gz
+               └── aur-clone.files.tar.gz
+
+There are multiple subdirectories, some of them are commons for any repository, but some of them are not.
+
+* ``cache`` is a directory with locally stored PKGBUILD's and VCS packages. It is common for all repositories and architectures.
+* ``chroot/{repository}`` is a chroot directory for ``devtools``. It is specific for each repository, but shared for different architectures inside (the ``devtools`` handles architectures automatically).
+* ``packages/{repository}/{architecture}`` is a directory with prebuilt packages. When package is built, first it will be uploaded to this directory and later will be handled by update process. It is architecture and repository specific.
+* ``pacman/{repository}/{architecture}`` is repository and architecture specific caches for pacman's databases.
+* ``repository/{repository}/{architecture}`` is a repository packages directory.
+
+Normally you should avoid direct interaction with the application tree.
+
+For tree migration process refer to the :doc:`migration notes <migration>`.
+
 Database
 --------
 
@@ -274,7 +321,7 @@ There are several supported synchronization providers, currently they are ``rsyn
 
 ``rsync`` provider does not have any specific logic except for running external rsync application with configured arguments. The service does not handle SSH configuration, thus it has to be configured before running application manually.
 
-``s3`` provider uses ``boto3`` package and implements sync feature. The files are stored in architecture directory (e.g. if bucket is ``repository``, packages will be stored in ``repository/x86_64`` for the ``x86_64`` architecture), bucket must be created before any action and API key must have permissions to write to the bucket. No external configuration required. In order to upload only changed files the service compares calculated hashes with the Amazon ETags, used realization is described `here <https://teppen.io/2018/10/23/aws_s3_verify_etags/>`_.
+``s3`` provider uses ``boto3`` package and implements sync feature. The files are stored in architecture directory (e.g. if bucket is ``repository``, packages will be stored in ``repository/aur-clone/x86_64`` for the ``aur-clone`` repository ``x86_64`` architecture), bucket must be created before any action and API key must have permissions to write to the bucket. No external configuration required. In order to upload only changed files the service compares calculated hashes with the Amazon ETags, used realization is described `here <https://teppen.io/2018/10/23/aws_s3_verify_etags/>`_.
 
 ``github`` provider authenticates through basic auth, API key with repository write permissions is required. There will be created a release with the name of the architecture in case if it does not exist; files will be uploaded to the release assets. It also stores array of files and their MD5 checksums in release body in order to upload only changed ones. According to the Github API in case if there is already uploaded asset with the same name (e.g. database files), asset will be removed first.