Release 1.4.1

according to the source code defaults always updates the values
dictionary. This in this specific case it is impossible to override the
value it will be always empty list.

In order to handle it we are adding another property to the Handler
class which allows to run with None architecture list.

This particular set_defaults behaviour is still useful for other cases
when we have to run command without any specific architecture

.bandit-test.yml

@@ -0,0 +1 @@
+skips: ['B101', 'B404']

.bandit.yml

@@ -0,0 +1 @@
+skips: ['B404', 'B603']

.github/ISSUE_TEMPLATE/bug-report.md

@@ -0,0 +1,24 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: ''
+labels: bug
+assignees: ''
+
+---
+
+## Summary
+
+A clear and concise description of what the bug is.
+
+### Steps to Reproduce
+
+Steps to reproduce the behavior (commands, environment etc)
+
+### Expected behavior
+
+A clear and concise description of what you expected to happen.
+
+### Logs
+
+Add logs to help explain your problem. Logs to stderr can be generated by using `--no-log` command line option.

.github/ISSUE_TEMPLATE/config.yml

@@ -0,0 +1 @@
+blank_issues_enabled: false

.github/ISSUE_TEMPLATE/discussion.md

@@ -0,0 +1,12 @@
+---
+name: Question
+about: Create an issue to get help with project
+title: ''
+labels: question
+assignees: ''
+
+---
+
+## Describe your question below
+
+A clear and concise description of your issue for which you would like to get help.

.github/ISSUE_TEMPLATE/feature-request.md

@@ -0,0 +1,20 @@
+---
+name: Feature request
+about: Suggest an idea for this project
+title: ''
+labels: enhancement
+assignees: ''
+
+---
+
+## Feature summary
+
+Brief description of the feature required
+
+### Cause of the feature request
+
+A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+
+### Proposed changes and/or features
+
+A clear and concise description of what you want to happen.

.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,13 @@
+## Summary
+
+Brief description of the pull request. Try to provide clear explanation for major changes.
+
+Please make sure that branch called either `feature/feature-name` for feature-related pull requests or `bug/bug-name` for bug-related ones.
+
+Put `closes #ISSUE` in case if the pull requests solves one of the opened issues.
+
+### Checklist
+
+- [ ] Tests to cover new code
+- [ ] `make check` passed
+- [ ] `make tests` passed

.github/workflows/release.yml

@@ -0,0 +1,37 @@
+name: release
+
+on:
+  push:
+    tags:
+      - '*.*.*'
+
+jobs:
+  make-release:
+
+    runs-on: ubuntu-latest
+
+    steps:
+      - uses: actions/checkout@v2
+      - name: extract version
+        id: version
+        run: echo ::set-output name=VERSION::${GITHUB_REF#refs/tags/}
+      - name: create changelog
+        id: changelog
+        uses: jaywcjlove/changelog-generator@main
+        with:
+          token: ${{ secrets.GITHUB_TOKEN }}
+          filter: 'Release \d+\.\d+\.\d+'
+      - name: create archive
+        run: make archive
+        env:
+          VERSION: ${{ steps.version.outputs.VERSION }}
+      - name: Release
+        uses: softprops/action-gh-release@v1
+        with:
+          body: |
+            ${{ steps.changelog.outputs.compareurl }}
+            ${{ steps.changelog.outputs.changelog }}
+          files: ahriman-*-src.tar.xz
+          fail_on_unmatched_files: true
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

.github/workflows/run-tests.yml

@@ -0,0 +1,26 @@
+name: tests
+
+on:
+  push:
+    branches: [ master ]
+  pull_request:
+    branches: [ master ]
+
+jobs:
+  run-tests:
+
+    runs-on: ubuntu-latest
+
+    steps:
+      - uses: actions/checkout@v2
+      - name: run check and tests in arch linux container
+        run: |
+          docker run \
+          -v  ${{ github.workspace }}:/build -w /build \
+          archlinux:latest \
+          /bin/bash -c "pacman --noconfirm -Syu base-devel python-argparse-manpage python-pip && \
+          pip install -e .[web] && \
+          pip install -e .[check] && \
+          pip install -e .[s3] && \
+          pip install -e .[test] && \
+          make check tests"

.pylintrc

@@ -22,7 +22,7 @@ ignore-patterns=
 
 # Use multiple processes to speed up Pylint. Specifying 0 will auto-detect the
 # number of processors available to use.
-jobs=1
+jobs=0
 
 # Control the amount of potential inferred values when inferring a single
 # object. This can help the performance when dealing with large functions or
@@ -149,7 +149,6 @@ disable=print-statement,
         too-few-public-methods,
         too-many-instance-attributes,
         broad-except,
-        logging-fstring-interpolation,
         too-many-ancestors,
         fixme,
         too-many-arguments,

AUTHORS

@@ -0,0 +1,2 @@
+Current developers:
+Evgenii Alekseev aka arcanis <esalexeev (at) gmail (dot) com>

CONFIGURING.md

@@ -1,85 +0,0 @@
-# ahriman configuration
-
-Some groups can be specified for each architecture separately. E.g. if there are `build` and `build_x86_64` groups it will use the `build_x86_64` for the `x86_64` architecture and `build` for any other (architecture specific group has higher priority).
-
-## `settings` group
-
-Base configuration settings.
-
-* `include` - path to directory with configuration files overrides, string, required.
-* `logging` - path to logging configuration, string, required. Check `logging.ini` for reference.
-
-## `alpm` group
-
-libalpm and AUR related configuration.
-
-* `aur_url` - base url for AUR, string, required.
-* `database` - path to pacman local database cache, string, required.
-* `repositories` - list of pacman repositories, space separated list of strings, required.
-* `root` - root for alpm library, string, required.
-
-## `build_*` groups
-
-Build related configuration. Group name must refer to architecture, e.g. it should be `build_x86_64` for x86_64 architecture.
-
-* `archbuild_flags` - additional flags passed to `archbuild` command, space separated list of strings, optional.
-* `build_command` - default build command, string, required.
-* `ignore_packages` - list packages to ignore during a regular update (manual update will still work), space separated list of strings, optional.
-* `makepkg_flags` - additional flags passed to `makepkg` command, space separated list of strings, optional.
-* `makechrootpkg_flags` - additional flags passed to `makechrootpkg` command, space separated list of strings, optional.
-
-## `repository` group
-
-Base repository settings.
-
-* `name` - repository name, string, required.
-* `root` - root path for application, string, required.
-
-## `sign_*` groups
-
-Settings for signing packages or repository. Group name must refer to architecture, e.g. it should be `sign_x86_64` for x86_64 architecture.
-
-* `target` - configuration flag to enable signing, space separated list of strings, required. Allowed values are `package` (sign each package separately), `repository` (sign repository database file).
-* `key` - default PGP key, string, required. This key will also be used for database signing if enabled.
-* `key_*` settings - PGP key which will be used for specific packages, string, optional. For example, if there is `key_yay` option the specified key will be used for yay package and default key for others.
-
-## `report` group
-
-Report generation settings.
-
-* `target` - list of reports to be generated, space separated list of strings, optional. Allowed values are `html`.
-
-### `html_*` groups
-
-Group name must refer to architecture, e.g. it should be `html_x86_64` for x86_64 architecture.
-
-* `path` - path to html report file, string, required.
-* `homepage` - link to homepage, string, optional.
-* `link_path` - prefix for HTML links, string, required.
-* `template_path` - path to Jinja2 template, string, required.
-
-## `upload` group
-
-Remote synchronization settings.
-
-* `target` - list of synchronizations to be used, space separated list of strings, optional. Allowed values are `rsync`, `s3`.
-
-### `rsync_*` groups
-
-Group name must refer to architecture, e.g. it should be `rsync_x86_64` for x86_64 architecture. Requires `rsync` package to be installed. Do not forget to configure ssh for user `ahriman`.
-
-* `remote` - remote server to rsync (e.g. `1.2.3.4:5678:path/to/sync`), string, required.
-
-### `s3_*` groups
-
-Group name must refer to architecture, e.g. it should be `s3_x86_64` for x86_64 architecture. Requires `aws-cli` package to be installed. Do not forget to configure it for user `ahriman`.
-
-* `bucket` - bucket name (e.g. `s3://bucket/path`), string, required.
-
-## `web_*` groups
-
-Web server settings. If any of `host`/`port` is not set, web integration will be disabled. Group name must refer to architecture, e.g. it should be `web_x86_64` for x86_64 architecture.
-
-* `host` - host to bind, string, optional.
-* `port` - port to bind, int, optional.
-* `templates` - path to templates directory, string, required.

Makefile

@@ -1,50 +1,63 @@
-.PHONY: archive archive_directory archlinux check clean directory push version
+.PHONY: architecture archive archive_directory archlinux check clean directory man push tests version
 .DEFAULT_GOAL := archlinux
 
 PROJECT := ahriman
 
-FILES := COPYING CONFIGURING.md README.md package src setup.py
+FILES := AUTHORS COPYING README.md docs package src setup.cfg setup.py
 TARGET_FILES := $(addprefix $(PROJECT)/, $(FILES))
 IGNORE_FILES := package/archlinux src/.mypy_cache
 
 $(TARGET_FILES) : $(addprefix $(PROJECT), %) : $(addprefix ., %) directory version
 	@cp -rp $< $@
 
+architecture:
+	cd src && pydeps ahriman -o ../docs/ahriman-architecture.svg --no-show --cluster
+
 archive: archive_directory
 	tar cJf "$(PROJECT)-$(VERSION)-src.tar.xz" "$(PROJECT)"
 	rm -rf "$(PROJECT)"
 
 archive_directory: $(TARGET_FILES)
 	rm -fr $(addprefix $(PROJECT)/, $(IGNORE_FILES))
-	find $(PROJECT) -type f -name '*.pyc' -delete
-	find $(PROJECT) -depth -type d -name '__pycache__' -execdir rm -rf {} +
-	find $(PROJECT) -depth -type d -name '*.egg-info' -execdir rm -rf {} +
+	find "$(PROJECT)" -type f -name "*.pyc" -delete
+	find "$(PROJECT)" -depth -type d -name "__pycache__" -execdir rm -rf {} +
+	find "$(PROJECT)" -depth -type d -name "*.egg-info" -execdir rm -rf {} +
 
 archlinux: archive
-	sed -i "/sha512sums=('[0-9A-Fa-f]*/s/[^'][^)]*/sha512sums=('$$(sha512sum $(PROJECT)-$(VERSION)-src.tar.xz | awk '{print $$1}')'/" package/archlinux/PKGBUILD
 	sed -i "s/pkgver=[0-9.]*/pkgver=$(VERSION)/" package/archlinux/PKGBUILD
 
-check:
-	cd src && mypy --implicit-reexport --strict -p $(PROJECT)
-	cd src && find $(PROJECT) -name '*.py' -execdir autopep8 --max-line-length 120 -aa -i {} +
-	cd src && pylint --rcfile=../.pylintrc $(PROJECT)
+check: clean mypy
+	autopep8 --exit-code --max-line-length 120 -aa -i -j 0 -r "src/$(PROJECT)" "tests/$(PROJECT)"
+	pylint --rcfile=.pylintrc "src/$(PROJECT)"
+	bandit -c .bandit.yml -r "src/$(PROJECT)"
+	bandit -c .bandit-test.yml -r "tests/$(PROJECT)"
 
 clean:
-	find . -type f -name '$(PROJECT)-*-src.tar.xz' -delete
+	find . -type f -name "$(PROJECT)-*-src.tar.xz" -delete
 	rm -rf "$(PROJECT)"
 
 directory: clean
 	mkdir "$(PROJECT)"
 
-push: archlinux
-	git add package/archlinux/PKGBUILD src/ahriman/version.py
+man:
+	cd src &&  PYTHONPATH=. argparse-manpage --module ahriman.application.ahriman --function _parser --author "ahriman team" --project-name ahriman --author-email "" --url https://github.com/arcan1s/ahriman --output ../docs/ahriman.1
+
+mypy:
+	cd src && mypy --implicit-reexport --strict -p "$(PROJECT)" --install-types --non-interactive || true
+	cd src && mypy --implicit-reexport --strict -p "$(PROJECT)"
+
+push: architecture man archlinux
+	git add package/archlinux/PKGBUILD src/ahriman/version.py docs/ahriman-architecture.svg docs/ahriman.1
 	git commit -m "Release $(VERSION)"
-	git push
 	git tag "$(VERSION)"
+	git push
 	git push --tags
 
+tests: clean
+	python setup.py test
+
 version:
 ifndef VERSION
 	$(error VERSION is required, but not set)
 endif
-	sed -i "/__version__ = '[0-9.]*/s/[^'][^)]*/__version__ = '$(VERSION)'/" src/ahriman/version.py
+	sed -i '/__version__ = "[0-9.]*/s/[^"][^)]*/__version__ = "$(VERSION)"/' src/ahriman/version.py

README.md

@@ -1,68 +1,72 @@
-# ArcHlinux ReposItory MANager
+# ArcH Linux ReposItory MANager
+
+[![build status](https://github.com/arcan1s/ahriman/actions/workflows/run-tests.yml/badge.svg)](https://github.com/arcan1s/ahriman/actions/workflows/run-tests.yml)
+[![CodeFactor](https://www.codefactor.io/repository/github/arcan1s/ahriman/badge)](https://www.codefactor.io/repository/github/arcan1s/ahriman)
 
 Wrapper for managing custom repository inspired by [repo-scripts](https://github.com/arcan1s/repo-scripts).
 
 ## Features
 
-* Install-configure-forget manager for own repository
-* Multi-architecture support
-* VCS packages support
-* Sign support with gpg (repository, package, per package settings)
-* Synchronization to remote services (rsync, s3) and report generation (html)
-* Dependency manager
-* Repository status interface
+* Install-configure-forget manager for own repository.
+* Multi-architecture support.
+* VCS packages support.
+* Sign support with gpg (repository, package, per package settings).
+* Synchronization to remote services (rsync, s3) and report generation (html).
+* Dependency manager.
+* Repository status interface with optional authorization and control options:
+
+    ![web interface](web.png)
 
 ## Installation and run
 
-* Install package as usual.
-* Change settings if required, see [CONFIGURING](CONFIGURING.md) for more details.
-* Create `/var/lib/ahriman/.makepkg.conf` with `makepkg.conf` overrides if required (at least you might want to set `PACKAGER`):
+For installation details please refer to the [documentation](docs/setup.md). For command help, `--help` subcommand must be used, e.g.:
 
-    ```shell
-    echo 'PACKAGES="John Doe <john@doe.com>"' | sudo -u ahriman tee -a /var/lib/ahriman/.makepkg.conf
-    ```
+```shell
+$ ahriman --help
+usage: ahriman [-h] [-a ARCHITECTURE] [-c CONFIGURATION] [--force] [-l LOCK] [--no-log] [--no-report] [--unsafe] [-v]
+               {add,check,clean,config,create-user,init,key-import,rebuild,remove,remove-unknown,report,search,setup,sign,status,status-update,sync,update,web} ...
 
-* Configure build tools (it is required for correct dependency management system):
+ArcH Linux ReposItory MANager
 
-    * create build command, e.g. `ln -s /usr/bin/archbuild /usr/local/bin/ahriman-x86_64-build` (you can choose any name for command, basically it should be `{name}-{arch}-build`);
-    * create configuration file, e.g. `cp /usr/share/devtools/pacman-{extra,ahriman}.conf` (same as previous `pacman-{name}.conf`);
-    * change configuration file, add your own repository, add multilib repository etc. Hint: you can use `Include` option as well;
-    * set `build_command` option to point to your command;
-    * configure `/etc/sudoers.d/ahriman` to allow running command without a password.
+optional arguments:
+  -h, --help            show this help message and exit
+  -a ARCHITECTURE, --architecture ARCHITECTURE
+                        target architectures (can be used multiple times) (default: None)
+  -c CONFIGURATION, --configuration CONFIGURATION
+                        configuration path (default: /etc/ahriman.ini)
+  --force               force run, remove file lock (default: False)
+  -l LOCK, --lock LOCK  lock file (default: /tmp/ahriman.lock)
+  --no-log              redirect all log messages to stderr (default: False)
+  --no-report           force disable reporting to web service (default: False)
+  --unsafe              allow to run ahriman as non-ahriman user (default: False)
+  -v, --version         show program's version number and exit
 
-    ```shell
-    ln -s /usr/bin/archbuild /usr/local/bin/ahriman-x86_64-build
-    cp /usr/share/devtools/pacman-{extra,ahriman}.conf
+command:
+  {add,check,clean,config,create-user,init,key-import,rebuild,remove,remove-unknown,report,search,setup,sign,status,status-update,sync,update,web}
+                        command to run
+    add                 add package
+    check               check for updates
+    clean               clean local caches
+    config              dump configuration
+    create-user         create user for web services
+    init                create repository tree
+    key-import          import PGP key
+    rebuild             rebuild repository
+    remove              remove package
+    remove-unknown      remove unknown packages
+    report              generate report
+    search              search for package
+    setup               initial service configuration
+    sign                sign packages
+    status              get package status
+    status-update       update package status
+    sync                sync repository
+    update              update packages
+    web                 start web server
+```
 
-    echo '[multilib]' | tee -a /usr/share/devtools/pacman-ahriman.conf
-    echo 'Include = /etc/pacman.d/mirrorlist' | tee -a /usr/share/devtools/pacman-ahriman.conf
+Subcommands have own help message as well.
 
-    echo '[aur-clone]' | tee -a /usr/share/devtools/pacman-ahriman.conf
-    echo 'SigLevel = Optional TrustAll' | tee -a /usr/share/devtools/pacman-ahriman.conf
-    echo 'Server = file:///var/lib/ahriman/repository/$arch' | tee -a /usr/share/devtools/pacman-ahriman.conf
+## Configuration
 
-    echo '[build]' | tee -a /etc/ahriman.ini.d/build.ini
-    echo 'build_command = ahriman-x86_64-build' | tee -a /etc/ahriman.ini.d/build.ini
-
-    echo 'Cmnd_Alias CARCHBUILD_CMD = /usr/local/bin/ahriman-x86_64-build *' | tee -a /etc/sudoers.d/ahriman
-    echo 'ahriman ALL=(ALL) NOPASSWD: CARCHBUILD_CMD' | tee -a /etc/sudoers.d/ahriman
-    chmod 400 /etc/sudoers.d/ahriman
-    ```
-
-* Start and enable `ahriman@.timer` via `systemctl`:
-
-    ```shell
-    systemctl enable --now ahriman@x86_64.timer
-    ```
-
-* Start and enable status page:
-
-    ```shell
-    systemctl enable --now ahriman-web@x86_64
-    ```
-
-* Add packages by using `ahriman add {package}` command:
-
-    ```shell
-    sudo -u ahriman ahriman -a x86_64 add yay
-    ```
+Every available option is described in the [documentation](docs/configuration.md).

docs/ahriman.1

@@ -0,0 +1,401 @@
+.TH ahriman "1" Manual
+.SH NAME
+ahriman
+.SH SYNOPSIS
+.B ahriman
+[-h] [-a ARCHITECTURE] [-c CONFIGURATION] [--force] [-l LOCK] [--no-log] [--no-report] [--unsafe] [-v] {add,check,clean,config,init,key-import,rebuild,remove,remove-unknown,report,search,setup,sign,status,status-update,sync,update,user,web} ...
+.SH DESCRIPTION
+ArcH Linux ReposItory MANager
+.SH OPTIONS
+
+.TP
+\fB\-a\fR \fI\,ARCHITECTURE\/\fR, \fB\-\-architecture\fR \fI\,ARCHITECTURE\/\fR
+target architectures (can be used multiple times)
+
+.TP
+\fB\-c\fR \fI\,CONFIGURATION\/\fR, \fB\-\-configuration\fR \fI\,CONFIGURATION\/\fR
+configuration path
+
+.TP
+\fB\-\-force\fR
+force run, remove file lock
+
+.TP
+\fB\-l\fR \fI\,LOCK\/\fR, \fB\-\-lock\fR \fI\,LOCK\/\fR
+lock file
+
+.TP
+\fB\-\-no\-log\fR
+redirect all log messages to stderr
+
+.TP
+\fB\-\-no\-report\fR
+force disable reporting to web service
+
+.TP
+\fB\-\-unsafe\fR
+allow to run ahriman as non\-ahriman user
+
+.TP
+\fB\-v\fR, \fB\-\-version\fR
+show program's version number and exit
+
+.SS
+\fBSub-commands\fR
+.TP
+\fBahriman\fR \fI\,add\/\fR
+add package
+.TP
+\fBahriman\fR \fI\,check\/\fR
+check for updates
+.TP
+\fBahriman\fR \fI\,clean\/\fR
+clean local caches
+.TP
+\fBahriman\fR \fI\,config\/\fR
+dump configuration
+.TP
+\fBahriman\fR \fI\,init\/\fR
+create repository tree
+.TP
+\fBahriman\fR \fI\,key-import\/\fR
+import PGP key
+.TP
+\fBahriman\fR \fI\,rebuild\/\fR
+rebuild repository
+.TP
+\fBahriman\fR \fI\,remove\/\fR
+remove package
+.TP
+\fBahriman\fR \fI\,remove-unknown\/\fR
+remove unknown packages
+.TP
+\fBahriman\fR \fI\,report\/\fR
+generate report
+.TP
+\fBahriman\fR \fI\,search\/\fR
+search for package
+.TP
+\fBahriman\fR \fI\,setup\/\fR
+initial service configuration
+.TP
+\fBahriman\fR \fI\,sign\/\fR
+sign packages
+.TP
+\fBahriman\fR \fI\,status\/\fR
+get package status
+.TP
+\fBahriman\fR \fI\,status-update\/\fR
+update package status
+.TP
+\fBahriman\fR \fI\,sync\/\fR
+sync repository
+.TP
+\fBahriman\fR \fI\,update\/\fR
+update packages
+.TP
+\fBahriman\fR \fI\,user\/\fR
+manage users for web services
+.TP
+\fBahriman\fR \fI\,web\/\fR
+start web server
+.SH OPTIONS 'ahriman add'
+usage: ahriman add [-h] [--now] [--source {PackageSource.Auto,PackageSource.Archive,PackageSource.Directory,PackageSource.AUR}] [--without-dependencies] package [package ...]
+
+add package
+
+.TP
+\fBpackage\fR
+package base/name or archive path
+
+.TP
+\fB\-\-now\fR
+run update function after
+
+.TP
+\fB\-\-source\fR {PackageSource.Auto,PackageSource.Archive,PackageSource.Directory,PackageSource.AUR}
+package source
+
+.TP
+\fB\-\-without\-dependencies\fR
+do not add dependencies
+
+.SH OPTIONS 'ahriman check'
+usage: ahriman check [-h] [--no-vcs] [package ...]
+
+check for updates. Same as update \-\-dry\-run \-\-no\-manual
+
+.TP
+\fBpackage\fR
+filter check by package base
+
+.TP
+\fB\-\-no\-vcs\fR
+do not check VCS packages
+
+.SH OPTIONS 'ahriman clean'
+usage: ahriman clean [-h] [--no-build] [--no-cache] [--no-chroot] [--no-manual] [--no-packages]
+
+clear local caches
+
+
+.TP
+\fB\-\-no\-build\fR
+do not clear directory with package sources
+
+.TP
+\fB\-\-no\-cache\fR
+do not clear directory with package caches
+
+.TP
+\fB\-\-no\-chroot\fR
+do not clear build chroot
+
+.TP
+\fB\-\-no\-manual\fR
+do not clear directory with manually added packages
+
+.TP
+\fB\-\-no\-packages\fR
+do not clear directory with built packages
+
+.SH OPTIONS 'ahriman config'
+usage: ahriman config [-h]
+
+dump configuration for specified architecture
+
+
+
+.SH OPTIONS 'ahriman init'
+usage: ahriman init [-h]
+
+create empty repository tree. Optional command for auto architecture support
+
+
+
+.SH OPTIONS 'ahriman key-import'
+usage: ahriman key-import [-h] [--key-server KEY_SERVER] key
+
+import PGP key from public sources to repository user
+
+.TP
+\fBkey\fR
+PGP key to import from public server
+
+.TP
+\fB\-\-key\-server\fR \fI\,KEY_SERVER\/\fR
+key server for key import
+
+.SH OPTIONS 'ahriman rebuild'
+usage: ahriman rebuild [-h] [--depends-on DEPENDS_ON]
+
+rebuild whole repository
+
+
+.TP
+\fB\-\-depends\-on\fR \fI\,DEPENDS_ON\/\fR
+only rebuild packages that depend on specified package
+
+.SH OPTIONS 'ahriman remove'
+usage: ahriman remove [-h] package [package ...]
+
+remove package
+
+.TP
+\fBpackage\fR
+package name or base
+
+
+.SH OPTIONS 'ahriman remove-unknown'
+usage: ahriman remove-unknown [-h] [--dry-run]
+
+remove packages which are missing in AUR
+
+
+.TP
+\fB\-\-dry\-run\fR
+just perform check for packages without removal
+
+.SH OPTIONS 'ahriman report'
+usage: ahriman report [-h] [target ...]
+
+generate report
+
+.TP
+\fBtarget\fR
+target to generate report
+
+
+.SH OPTIONS 'ahriman search'
+usage: ahriman search [-h] search [search ...]
+
+search for package in AUR using API
+
+.TP
+\fBsearch\fR
+search terms, can be specified multiple times
+
+
+.SH OPTIONS 'ahriman setup'
+usage: ahriman setup [-h] [--build-command BUILD_COMMAND] [--from-configuration FROM_CONFIGURATION] [--no-multilib] --packager PACKAGER --repository REPOSITORY [--sign-key SIGN_KEY]
+                     [--sign-target {SignSettings.Packages,SignSettings.Repository}] [--web-port WEB_PORT]
+
+create initial service configuration, requires root
+
+
+.TP
+\fB\-\-build\-command\fR \fI\,BUILD_COMMAND\/\fR
+build command prefix
+
+.TP
+\fB\-\-from\-configuration\fR \fI\,FROM_CONFIGURATION\/\fR
+path to default devtools pacman configuration
+
+.TP
+\fB\-\-no\-multilib\fR
+do not add multilib repository
+
+.TP
+\fB\-\-packager\fR \fI\,PACKAGER\/\fR
+packager name and email
+
+.TP
+\fB\-\-repository\fR \fI\,REPOSITORY\/\fR
+repository name
+
+.TP
+\fB\-\-sign\-key\fR \fI\,SIGN_KEY\/\fR
+sign key id
+
+.TP
+\fB\-\-sign\-target\fR {SignSettings.Packages,SignSettings.Repository}
+sign options
+
+.TP
+\fB\-\-web\-port\fR \fI\,WEB_PORT\/\fR
+port of the web service
+
+.SH OPTIONS 'ahriman sign'
+usage: ahriman sign [-h] [package ...]
+
+(re\-)sign packages and repository database
+
+.TP
+\fBpackage\fR
+sign only specified packages
+
+
+.SH OPTIONS 'ahriman status'
+usage: ahriman status [-h] [--ahriman] [--status {BuildStatusEnum.Unknown,BuildStatusEnum.Pending,BuildStatusEnum.Building,BuildStatusEnum.Failed,BuildStatusEnum.Success}] [package ...]
+
+request status of the package
+
+.TP
+\fBpackage\fR
+filter status by package base
+
+.TP
+\fB\-\-ahriman\fR
+get service status itself
+
+.TP
+\fB\-\-status\fR {BuildStatusEnum.Unknown,BuildStatusEnum.Pending,BuildStatusEnum.Building,BuildStatusEnum.Failed,BuildStatusEnum.Success}
+filter packages by status
+
+.SH OPTIONS 'ahriman status-update'
+usage: ahriman status-update [-h] [--status {BuildStatusEnum.Unknown,BuildStatusEnum.Pending,BuildStatusEnum.Building,BuildStatusEnum.Failed,BuildStatusEnum.Success}] [--remove] [package ...]
+
+request status of the package
+
+.TP
+\fBpackage\fR
+set status for specified packages. If no packages supplied, service status will be updated
+
+.TP
+\fB\-\-status\fR {BuildStatusEnum.Unknown,BuildStatusEnum.Pending,BuildStatusEnum.Building,BuildStatusEnum.Failed,BuildStatusEnum.Success}
+new status
+
+.TP
+\fB\-\-remove\fR
+remove package status page
+
+.SH OPTIONS 'ahriman sync'
+usage: ahriman sync [-h] [target ...]
+
+sync packages to remote server
+
+.TP
+\fBtarget\fR
+target to sync
+
+
+.SH OPTIONS 'ahriman update'
+usage: ahriman update [-h] [--dry-run] [--no-aur] [--no-manual] [--no-vcs] [package ...]
+
+run updates
+
+.TP
+\fBpackage\fR
+filter check by package base
+
+.TP
+\fB\-\-dry\-run\fR
+just perform check for updates, same as check command
+
+.TP
+\fB\-\-no\-aur\fR
+do not check for AUR updates. Implies \-\-no\-vcs
+
+.TP
+\fB\-\-no\-manual\fR
+do not include manual updates
+
+.TP
+\fB\-\-no\-vcs\fR
+do not check VCS packages
+
+.SH OPTIONS 'ahriman user'
+usage: ahriman user [-h] [--as-service] [-a {UserAccess.Safe,UserAccess.Read,UserAccess.Write}] [--no-reload] [-p PASSWORD] [-r] [--secure] username
+
+manage users for web services with password and role. In case if password was not entered it will be asked interactively
+
+.TP
+\fBusername\fR
+username for web service
+
+.TP
+\fB\-\-as\-service\fR
+add user as service user
+
+.TP
+\fB\-a\fR {UserAccess.Safe,UserAccess.Read,UserAccess.Write}, \fB\-\-access\fR {UserAccess.Safe,UserAccess.Read,UserAccess.Write}
+user access level
+
+.TP
+\fB\-\-no\-reload\fR
+do not reload authentication module
+
+.TP
+\fB\-p\fR \fI\,PASSWORD\/\fR, \fB\-\-password\fR \fI\,PASSWORD\/\fR
+user password
+
+.TP
+\fB\-r\fR, \fB\-\-remove\fR
+remove user from configuration
+
+.TP
+\fB\-\-secure\fR
+set file permissions to user\-only
+
+.SH OPTIONS 'ahriman web'
+usage: ahriman web [-h]
+
+start web server
+
+.SH AUTHORS
+.B ahriman
+was written by ahriman team <>.
+.SH DISTRIBUTION
+The latest version of ahriman may be downloaded from
+.UR https://github.com/arcan1s/ahriman
+.UE

docs/architecture.md

@@ -0,0 +1,185 @@
+# Package structure
+
+Packages have strict rules of importing:
+
+* `ahriman.application` package must not be used anywhere except for itself.
+* `ahriman.core` and `ahriman.models` packages don't have any import restriction. Actually we would like to totally restrict importing of `core` package from `models`, but it is impossible at the moment.
+* `ahriman.web` package is allowed to be imported from `ahriman.application` (web handler only, only `ahriman.web.web` methods). It also must not be imported globally, only local import is allowed. 
+
+Full dependency diagram:
+
+![architecture](ahriman-architecture.svg)
+
+## `ahriman.application` package
+
+This package contains application (aka executable) related classes and everything for that. It also contains package called `ahriman.application.handlers` in which all available subcommands are described as separated classes derived from base `ahriman.application.handlers.handler.Handler` class. `ahriman.application.ahriman` contains only command line parses and executes specified `Handler` on success, `ahriman.application.application.Application` is a god class which provides interfaces for all repository related actions. `ahriman.application.lock.Lock` is additional class which provides file-based lock and also performs some common checks.
+
+## `ahriman.core` package
+
+This package contains everything which is required for any time of application run and separated to several packages:
+
+* `ahriman.core.alpm` package controls pacman related functions. It provides wrappers for `pyalpm` library and safe calls for repository tools (`repo-add` and `repo-remove`).
+* `ahriman.core.auth` package provides classes for authorization methods used by web mostly. Base class is `ahriman.core.auth.auth.Auth` which must be called by `load` method.
+* `ahriman.core.build_tools` is a package which provides wrapper for `devtools` commands.
+* `ahriman.core.report` is a package with reporting classes. Usually it must be called by `ahriman.core.report.report.Report.load` method.
+* `ahriman.core.repository` contains several traits and base repository (`ahriman.core.repository.repository.Repository` class) implementation.
+* `ahriman.core.sign` package provides sign feature (only gpg calls are available).
+* `ahriman.core.status` contains helpers and watcher class which are required for web application. Reporter must be initialized by using `ahriman.core.status.client.Client.load` method.
+* `ahriman.core.upload` package provides sync feature, must be called by `ahriman.core.upload.upload.Upload.load` method.
+
+This package also provides some generic functions and classes which may be used by other packages:
+
+* `ahriman.core.configuration.Configuration` is an extension for standard `configparser` library.
+* `ahriman.core.exceptions` provides custom exceptions.
+* `ahriman.core.spawn.Spawn` is a tool which can spawn another `ahriman` process. This feature is used by web application.
+* `ahriman.core.tree` is a dependency tree implementation.
+
+## `ahriman.models` package
+
+It provides models for any other part of application. Unlike `ahriman.core` package classes from here provides only conversion methods (e.g. create class from another or convert to). Mostly case classes and enumerations.
+
+## `ahriman.web` package
+
+Web application. It is important that this package is isolated from any other to allow it to be optional feature (i.e. dependencies which are required by the package are optional).
+
+* `ahriman.web.middlewares` provides middlewares for request handlers.
+* `ahriman.web.views` contains web views derived from aiohttp view class.
+* `ahriman.web.routes` creates routes for web application.
+* `ahriman.web.web` provides main web application functions (e.g. start, initialization).
+
+# Application run
+
+* Parse command line arguments, find command and related handler which is set by parser.
+* Call `Handler.execute` method.
+* Define list of architectures to run. In case if there is more than one architecture specified run several subprocesses or process in current process otherwise. Class attribute `ALLOW_MULTI_ARCHITECTURE_RUN` controls whether application can be run in multiple processes or not - this feature is required for some handlers (e.g. `Web`) which should be able to spawn child process in daemon mode (it is impossible to do for daemonic processes).
+* In each child process call lock functions.
+* After success checks pass control to `Handler.run` method defined by specific handler class.
+* Return result (success or failure) of each subprocess and exit from application.
+
+In most cases handlers spawn god class `ahriman.application.application.Application` class and call required methods.
+
+Application is designed to run from `systemd` services and provides parametrized by architecture timer and service file for that.
+
+# Basic flows
+
+## Add new packages or rebuild existing
+
+Idea is to copy package to the directory from which it will be handled at the next update run. Different variants are supported:
+
+* If supplied argument is file then application moves the file to the directory with built packages. Same rule applies for directory, but in this case it copies every package-like file from the specified directory.
+* If supplied argument iis not file then application tries to lookup for the specified name in AUR and clones it into the directory with manual updates. This scenario can also handle package dependencies which are missing in repositories.
+
+## Rebuild packages
+
+Same as add function for every package in repository. Optional filter by reverse dependency can be supplied.
+
+## Remove packages
+
+This flow removes package from filesystem, updates repository database and also runs synchronization and reporting methods.
+
+## Update packages
+
+This feature is divided into to stages: check AUR for updates and run rebuild for required packages. Whereas check does not do anything except for check itself, update flow is the following:
+
+1. Process every built package first. Those packages are usually added manually.
+2. Run sync and report methods.
+3. Generate dependency tree for packages to be built.
+4. For each level of tree it does:
+   1. Download package data from AUR.
+   2. Build every package in clean chroot.
+   3. Sign packages if required.
+   4. Add packages to database and sign database if required.
+   5. Process sync and report methods.
+
+After any step any package data is being removed.
+
+# Core functions reference
+
+## Configuration 
+
+`ahriman.core.configuration.Configuration` class provides some additional methods (e.g. `getpath` and `getlist`) and also combines multiple files into single configuration dictionary using architecture overrides. It is recommended to read class related settings from the class, not outside.
+
+## Utils
+
+For every external command run (which is actually not recommended if possible) custom wrapper for `subprocess` is used. Additional functions `ahriman.core.auth.helpers` provide safe calls for `aiohttp_security` methods and are required to make this dependency optional.
+
+## Submodules
+
+Some packages provide different behaviour depending on configuration settings. In this cases inheritance is used and recommended way to deal with them is to call class method `load` from base classes.
+
+## Authorization
+
+The package provides several authorization methods: disabled, based on configuration and OAuth2. 
+
+Disabled (default) authorization provider just allows everything for everyone and does not have any specific configuration (it uses some default configuration parameters though). It also provides generic interface for derived classes.
+
+Mapping (aka configuration) provider uses hashed passwords with salt from configuration file in order to authenticate users. This provider also enables user permission checking (read/write) (authorization). Thus, it defines the following methods:
+
+* `check_credentials` - user password validation (authentication).
+* `verify_access` - user permission validation (authorization).
+
+Passwords must be stored in configuration as `hash(password + salt)`, where `password` is user defined password (taken from user input), `salt` is random string (any length) defined globally in configuration and `hash` is secure hash function. Thus, the following configuration
+
+```ini
+[auth:read]
+username = $6$rounds=656000$mWBiecMPrHAL1VgX$oU4Y5HH8HzlvMaxwkNEJjK13ozElyU1wAHBoO/WW5dAaE4YEfnB0X3FxbynKMl4FBdC3Ovap0jINz4LPkNADg0
+```
+
+means that there is user `username` with `read` access and password `password` hashed by `sha512` with salt `salt`.
+
+OAuth provider uses library definitions (`aioauth-client`) in order _authenticate_ users. It still requires user permission to be set in configuration, thus it inherits mapping provider without any changes. Whereas we could override `check_credentials` (authentication method) by something custom, OAuth flow is a bit more complex than just forward request, thus we have to implement the flow in login form.
+
+OAuth's implementation also allows authenticating users via username + password (in the same way as mapping does) though it is not recommended for end-users and password must be left blank. In particular this feature is used by service reporting (aka robots).
+
+In order to configure users there is special command.
+
+## Additional features
+
+Some features require optional dependencies to be installed:
+
+* Version control executables (e.g. `git`, `svn`) for VCS packages.
+* `gnupg` application for package and repository sign feature.
+* `rsync` application for rsync based repository sync.
+* `boto3` python package for `S3` sync.
+* `Jinja2` python package for HTML report generation (it is also used by web application).
+
+# Web application
+
+Web application requires the following python packages to be installed:
+
+* Core part requires `aiohttp` (application itself), `aiohttp_jinja2` and `Jinja2` (HTML generation from templates).
+* In addition, `aiohttp_debugtoolbar` is required for debug panel. Please note that this option does not work together with authorization and basically must not be used in production.
+* In addition, authorization feature requires `aiohttp_security`, `aiohttp_session` and `cryptography`.
+* In addition to base authorization dependencies, OAuth2 also requires `aioauth-client` library.
+
+## Middlewares
+
+Service provides some custom middlewares, e.g. logging every exception (except for user ones) and user authorization.
+
+## Web views
+
+All web views are defined in separated package and derived from `ahriman.web.views.base.Base` class which provides typed interfaces for web application. 
+
+REST API supports both form and JSON data, but the last one is recommended. 
+
+Different APIs are separated into different packages:
+
+* `ahriman.web.views.service` provides views for application controls.
+* `ahriman.web.views.status` package provides REST API for application reporting.
+* `ahriman.web.views.user` package provides login and logout methods which can be called without authorization.
+
+## Templating
+
+Package provides base jinja templates which can be overridden by settings. Vanilla templates are actively using bootstrap library.
+
+## Requests and scopes
+
+Service provides optional authorization which can be turned on in settings. In order to control user access there are two levels of authorization - read-only (only GET-like requests) and write (anything) which are provided by each web view directly.
+
+If this feature is configured any request will be prohibited without authentication. In addition, configuration flag `auth.safe_build_status` can be used in order to allow seeing main page without authorization.
+
+For authenticated users it uses encrypted session cookies to store tokens; encryption key is generated each time at the start of the application. It also stores expiration time of the session inside.
+
+## External calls
+
+Web application provides external calls to control main service. It spawns child process with specific arguments and waits for its termination. This feature must be used either with authorization or in safe (i.e. when status page is not available world-wide) environment.

docs/configuration.md

@@ -0,0 +1,137 @@
+# ahriman configuration
+
+Some groups can be specified for each architecture separately. E.g. if there are `build` and `build:x86_64` groups it will use the option from `build:x86_64` for the `x86_64` architecture and `build` for any other (architecture specific group has higher priority). In case if both groups are presented, architecture specific options will be merged into global ones overriding them. 
+
+## `settings` group
+
+Base configuration settings.
+
+* `include` - path to directory with configuration files overrides, string, required.
+* `logging` - path to logging configuration, string, required. Check `logging.ini` for reference.
+
+## `alpm` group
+
+libalpm and AUR related configuration.
+
+* `aur_url` - base url for AUR, string, required.
+* `database` - path to pacman local database cache, string, required.
+* `repositories` - list of pacman repositories, space separated list of strings, required.
+* `root` - root for alpm library, string, required.
+
+## `auth` group
+
+Base authorization settings. `OAuth` provider requires `aioauth-client` library to be installed.
+
+* `target` - specifies authorization provider, string, optional, default `disabled`. Allowed values are `disabled`, `configuration`, `oauth`.
+* `client_id` - OAuth2 application client ID, string, required in case if `oauth` is used.
+* `client_secret` - OAuth2 application client secret key, string, required in case if `oauth` is used.
+* `max_age` - parameter which controls both cookie expiration and token expiration inside the service, integer, optional, default is 7 days.
+* `oauth_provider` - OAuth2 provider class name as is in `aioauth-client` (e.g. `GoogleClient`, `GithubClient` etc), string, required in case if `oauth` is used.
+* `oauth_scopes` - scopes list for OAuth2 provider, which will allow retrieving user email (which is used for checking user permissions), e.g. `https://www.googleapis.com/auth/userinfo.email` for `GoogleClient` or `user:email` for `GithubClient`, space separated list of strings, required in case if `oauth` is used.
+* `safe_build_status` - allow requesting status page without authorization, boolean, required.
+* `salt` - password hash salt, string, required in case if authorization enabled (automatically generated by `create-user` subcommand).
+
+## `auth:*` groups
+
+Authorization mapping. Group name must refer to user access level, i.e. it should be one of `auth:read` (read hidden pages), `auth:write` (everything is allowed).
+
+Key is always username (case-insensitive), option value depends on authorization provider:
+
+* `OAuth` - by default requires only usernames and ignores values. But in case of direct login method call (via POST request) it will act as `Mapping` authorization method.
+* `Mapping` (default) - reads salted password hashes from values, uses SHA512 in order to hash passwords. Password can be set by using `create-user` subcommand.
+
+## `build:*` groups
+
+Build related configuration. Group name must refer to architecture, e.g. it should be `build:x86_64` for x86_64 architecture.
+
+* `archbuild_flags` - additional flags passed to `archbuild` command, space separated list of strings, optional.
+* `build_command` - default build command, string, required.
+* `ignore_packages` - list packages to ignore during a regular update (manual update will still work), space separated list of strings, optional.
+* `makepkg_flags` - additional flags passed to `makepkg` command, space separated list of strings, optional.
+* `makechrootpkg_flags` - additional flags passed to `makechrootpkg` command, space separated list of strings, optional.
+
+## `repository` group
+
+Base repository settings.
+
+* `name` - repository name, string, required.
+* `root` - root path for application, string, required.
+
+## `sign:*` groups
+
+Settings for signing packages or repository. Group name must refer to architecture, e.g. it should be `sign:x86_64` for x86_64 architecture.
+
+* `target` - configuration flag to enable signing, space separated list of strings, required. Allowed values are `package` (sign each package separately), `repository` (sign repository database file).
+* `key` - default PGP key, string, required. This key will also be used for database signing if enabled.
+* `key_*` settings - PGP key which will be used for specific packages, string, optional. For example, if there is `key_yay` option the specified key will be used for yay package and default key for others.
+
+## `report` group
+
+Report generation settings.
+
+* `target` - list of reports to be generated, space separated list of strings, required. Allowed values are `html`, `email`.
+
+### `email:*` groups
+
+Group name must refer to architecture, e.g. it should be `email:x86_64` for x86_64 architecture.
+
+* `full_template_path` - path to Jinja2 template for full package description index, string, optional.
+* `homepage` - link to homepage, string, optional.
+* `host` - SMTP host for sending emails, string, required.
+* `link_path` - prefix for HTML links, string, required.
+* `no_empty_report` - skip report generation for empty packages list, boolean, optional, default `yes`.
+* `password` - SMTP password to authenticate, string, optional.
+* `port` - SMTP port for sending emails, int, required.
+* `receivers` - SMTP receiver addresses, space separated list of strings, required.
+* `sender` - SMTP sender address, string, required.
+* `ssl` - SSL mode for SMTP connection, one of `ssl`, `starttls`, `disabled`, optional, default `disabled`.
+* `template_path` - path to Jinja2 template, string, required.
+* `user` - SMTP user to authenticate, string, optional.
+
+### `html:*` groups
+
+Group name must refer to architecture, e.g. it should be `html:x86_64` for x86_64 architecture.
+
+* `path` - path to html report file, string, required.
+* `homepage` - link to homepage, string, optional.
+* `link_path` - prefix for HTML links, string, required.
+* `template_path` - path to Jinja2 template, string, required.
+
+## `upload` group
+
+Remote synchronization settings.
+
+* `target` - list of synchronizations to be used, space separated list of strings, required. Allowed values are `rsync`, `s3`.
+
+### `rsync:*` groups
+
+Group name must refer to architecture, e.g. it should be `rsync:x86_64` for x86_64 architecture. Requires `rsync` package to be installed. Do not forget to configure ssh for user `ahriman`.
+
+* `command` - rsync command to run, space separated list of string, required.
+* `remote` - remote server to rsync (e.g. `1.2.3.4:5678:path/to/sync`), string, required.
+
+### `s3:*` groups
+
+Group name must refer to architecture, e.g. it should be `s3:x86_64` for x86_64 architecture.
+
+* `access_key` - AWS access key ID, string, required.
+* `bucket` - bucket name (e.g. `bucket`), string, required.
+* `chunk_size` - chunk size for calculating entity tags, int, optional, default 8 * 1024 * 1024.
+* `region` - bucket region (e.g. `eu-central-1`), string, required.
+* `secret_key` - AWS secret access key, string, required.
+
+## `web:*` groups
+
+Web server settings. If any of `host`/`port` is not set, web integration will be disabled. Group name must refer to architecture, e.g. it should be `web:x86_64` for x86_64 architecture.
+
+* `address` - optional address in form `proto://host:port` (`port` can be omitted in case of default `proto` ports), will be used instead of `http://{host}:{port}` in case if set, string, optional. This option is required in case if `OAuth` provider is used.
+* `debug` - enable debug toolbar, boolean, optional, default `no`.
+* `debug_check_host` - check hosts to access debug toolbar, boolean, optional, default `no`.
+* `debug_allowed_hosts` - allowed hosts to get access to debug toolbar, space separated list of string, optional.
+* `host` - host to bind, string, optional.
+* `index_url` - full url of the repository index page, string, optional.
+* `password` - password to authorize in web service in order to update service status, string, required in case if authorization enabled.  
+* `port` - port to bind, int, optional.
+* `static_path` - path to directory with static files, string, required.
+* `templates` - path to templates directory, string, required.
+* `username` - username to authorize in web service in order to update service status, string, required in case if authorization enabled.  

docs/setup.md

@@ -0,0 +1,60 @@
+# Setup instructions
+
+1. Install package as usual.
+2. Change settings if required, see [configuration reference](configuration.md) for more details.
+3. Create `/var/lib/ahriman/.makepkg.conf` with `makepkg.conf` overrides if required (at least you might want to set `PACKAGER`):
+
+    ```shell
+    echo 'PACKAGER="John Doe <john@doe.com>"' | sudo -u ahriman tee -a /var/lib/ahriman/.makepkg.conf
+    ```
+
+4. Configure build tools (it is required for correct dependency management system):
+
+    1. Create build command, e.g. `ln -s /usr/bin/archbuild /usr/local/bin/ahriman-x86_64-build` (you can choose any name for command, basically it should be `{name}-{arch}-build`).
+    2. Create configuration file, e.g. `cp /usr/share/devtools/pacman-{extra,ahriman}.conf` (same as previous `pacman-{name}.conf`).
+    3. Change configuration file, add your own repository, add multilib repository etc;
+    4. Set `build_command` option to point to your command.
+    5. Configure `/etc/sudoers.d/ahriman` to allow running command without a password.
+
+    ```shell
+    ln -s /usr/bin/archbuild /usr/local/bin/ahriman-x86_64-build
+    cp /usr/share/devtools/pacman-{extra,ahriman}.conf
+
+    echo '[multilib]' | tee -a /usr/share/devtools/pacman-ahriman.conf
+    echo 'Include = /etc/pacman.d/mirrorlist' | tee -a /usr/share/devtools/pacman-ahriman.conf
+
+    echo '[aur-clone]' | tee -a /usr/share/devtools/pacman-ahriman.conf
+    echo 'SigLevel = Optional TrustAll' | tee -a /usr/share/devtools/pacman-ahriman.conf
+    echo 'Server = file:///var/lib/ahriman/repository/$arch' | tee -a /usr/share/devtools/pacman-ahriman.conf
+
+    echo '[build]' | tee -a /etc/ahriman.ini.d/build.ini
+    echo 'build_command = ahriman-x86_64-build' | tee -a /etc/ahriman.ini.d/build.ini
+
+    echo 'Cmnd_Alias CARCHBUILD_CMD = /usr/local/bin/ahriman-x86_64-build *' | tee -a /etc/sudoers.d/ahriman
+    echo 'ahriman ALL=(ALL) NOPASSWD: CARCHBUILD_CMD' | tee -a /etc/sudoers.d/ahriman
+    chmod 400 /etc/sudoers.d/ahriman
+    ```
+
+5. Start and enable `ahriman@.timer` via `systemctl`:
+
+    ```shell
+    systemctl enable --now ahriman@x86_64.timer
+    ```
+
+6. Start and enable status page:
+
+    ```shell
+    systemctl enable --now ahriman-web@x86_64
+    ```
+
+7. Add packages by using `ahriman add {package}` command:
+
+    ```shell
+    sudo -u ahriman ahriman -a x86_64 add yay --now
+    ```
+
+Note that initial service configuration can be done by running `ahriman setup` with specific arguments.
+
+## User creation
+
+`create-user` subcommand is recommended for new user creation.

package/archlinux/PKGBUILD

@@ -1,31 +1,32 @@
 # Maintainer: Evgeniy Alekseev
 
 pkgname='ahriman'
-pkgver=0.15.0
+pkgver=1.4.1
 pkgrel=1
-pkgdesc="ArcHlinux ReposItory MANager"
+pkgdesc="ArcH Linux ReposItory MANager"
 arch=('any')
 url="https://github.com/arcan1s/ahriman"
 license=('GPL3')
-depends=('devtools' 'git' 'pyalpm' 'python-aur' 'python-srcinfo')
+depends=('devtools' 'git' 'pyalpm' 'python-aur' 'python-passlib' 'python-srcinfo')
 makedepends=('python-pip')
-optdepends=('aws-cli: sync to s3'
-            'breezy: -bzr packages support'
+optdepends=('breezy: -bzr packages support'
             'darcs: -darcs packages support'
             'gnupg: package and repository sign'
             'mercurial: -hg packages support'
+            'python-aioauth-client: web server with OAuth2 authorization'
             'python-aiohttp: web server'
+            'python-aiohttp-debugtoolbar: web server with enabled debug panel'
             'python-aiohttp-jinja2: web server'
+            'python-aiohttp-security: web server with authorization'
+            'python-aiohttp-session: web server with authorization'
+            'python-boto3: sync to s3'
+            'python-cryptography: web server with authorization'
             'python-jinja: html report generation'
-            'python-requests: web server'
             'rsync: sync by using rsync'
             'subversion: -svn packages support')
 source=("https://github.com/arcan1s/ahriman/releases/download/$pkgver/$pkgname-$pkgver-src.tar.xz"
         'ahriman.sysusers'
         'ahriman.tmpfiles')
-sha512sums=('a1db44390ce1785da3d535e3cfd2242d8d56070228eb9b3c1d5629163b65941d60753c481c0fdc69e475e534a828ceea39568dc6711abeee092616dac08e31a9'
-            '13718afec2c6786a18f0b223ef8e58dccf0688bca4cdbe203f14071f5031ed20120eb0ce38b52c76cfd6e8b6581a9c9eaa2743eb11abbaca637451a84c33f075'
-            '55b20f6da3d66e7bbf2add5d95a3b60632df121717d25a993e56e737d14f51fe063eb6f1b38bd81cc32e05db01c0c1d80aaa720c45cde87f238d8b46cdb8cbc4')
 backup=('etc/ahriman.ini'
         'etc/ahriman.ini.d/logging.ini')
 
@@ -43,3 +44,7 @@ package() {
   install -Dm644 "$srcdir/$pkgname.sysusers" "$pkgdir/usr/lib/sysusers.d/$pkgname.conf"
   install -Dm644 "$srcdir/$pkgname.tmpfiles" "$pkgdir/usr/lib/tmpfiles.d/$pkgname.conf"
 }
+
+sha512sums=('6ab741bfb42f92ab00d1b6ecfc44426c00e5c433486e014efbdb585715d9a12dbbafc280e5a9f85b941c8681b13a9dad41327a3e3c44a9683ae30c1d6f017f50'
+            '13718afec2c6786a18f0b223ef8e58dccf0688bca4cdbe203f14071f5031ed20120eb0ce38b52c76cfd6e8b6581a9c9eaa2743eb11abbaca637451a84c33f075'
+            '55b20f6da3d66e7bbf2add5d95a3b60632df121717d25a993e56e737d14f51fe063eb6f1b38bd81cc32e05db01c0c1d80aaa720c45cde87f238d8b46cdb8cbc4')

package/archlinux/ahriman.sysusers

@@ -1 +1 @@
-u ahriman 643 "ArcHlinux ReposItory MANager" /var/lib/ahriman
+u ahriman 643 "ArcH Linux ReposItory MANager" /var/lib/ahriman

package/archlinux/ahriman.tmpfiles

@@ -1,2 +1 @@
-d /var/lib/ahriman 0775 ahriman log
 d /var/log/ahriman 0755 ahriman ahriman

package/etc/ahriman.ini

@@ -8,12 +8,19 @@ database = /var/lib/pacman
 repositories = core extra community multilib
 root = /
 
+[auth]
+target = disabled
+max_age = 604800
+oauth_provider = GoogleClient
+oauth_scopes = https://www.googleapis.com/auth/userinfo.email
+safe_build_status = yes
+
 [build]
 archbuild_flags =
 build_command = extra-x86_64-build
 ignore_packages =
 makechrootpkg_flags =
-makepkg_flags = --skippgpcheck
+makepkg_flags =
 
 [repository]
 name = aur-clone
@@ -21,27 +28,32 @@ root = /var/lib/ahriman
 
 [sign]
 target =
-key =
 
 [report]
 target =
 
+[email]
+full_template_path = /usr/share/ahriman/repo-index.jinja2
+no_empty_report = yes
+template_path = /usr/share/ahriman/email-index.jinja2
+ssl = disabled
+
 [html]
-path =
-homepage =
-link_path =
 template_path = /usr/share/ahriman/repo-index.jinja2
 
 [upload]
 target =
 
 [rsync]
-remote =
+command = rsync --archive --compress --partial --delete
 
 [s3]
-bucket =
+chunk_size = 8388608
 
 [web]
-host =
-port =
+debug = no
+debug_check_host = no
+debug_allowed_hosts =
+host = 127.0.0.1
+static_path = /usr/share/ahriman/static
 templates = /usr/share/ahriman

package/etc/ahriman.ini.d/logging.ini

@@ -2,10 +2,10 @@
 keys = root,builder,build_details,http
 
 [handlers]
-keys = console_handler,build_file_handler,file_handler,http_handler
+keys = console_handler,syslog_handler
 
 [formatters]
-keys = generic_format
+keys = generic_format,syslog_format
 
 [handler_console_handler]
 class = StreamHandler
@@ -13,47 +13,39 @@ level = DEBUG
 formatter = generic_format
 args = (sys.stderr,)
 
-[handler_file_handler]
-class = logging.handlers.RotatingFileHandler
+[handler_syslog_handler]
+class = logging.handlers.SysLogHandler
 level = DEBUG
-formatter = generic_format
-args = ('/var/log/ahriman/ahriman.log', 'a', 20971520, 20)
-
-[handler_build_file_handler]
-class = logging.handlers.RotatingFileHandler
-level = DEBUG
-formatter = generic_format
-args = ('/var/log/ahriman/build.log', 'a', 20971520, 20)
-
-[handler_http_handler]
-class = logging.handlers.RotatingFileHandler
-level = DEBUG
-formatter = generic_format
-args = ('/var/log/ahriman/http.log', 'a', 20971520, 20)
+formatter = syslog_format
+args = ("/dev/log",)
 
 [formatter_generic_format]
 format = [%(levelname)s %(asctime)s] [%(filename)s:%(lineno)d] [%(funcName)s]: %(message)s
 datefmt =
 
+[formatter_syslog_format]
+format = [%(levelname)s] [%(filename)s:%(lineno)d] [%(funcName)s]: %(message)s
+datefmt =
+
 [logger_root]
 level = DEBUG
-handlers = file_handler
+handlers = syslog_handler
 qualname = root
 
 [logger_builder]
 level = DEBUG
-handlers = file_handler
+handlers = syslog_handler
 qualname = builder
 propagate = 0
 
 [logger_build_details]
 level = DEBUG
-handlers = build_file_handler
+handlers = syslog_handler
 qualname = build_details
 propagate = 0
 
 [logger_http]
 level = DEBUG
-handlers = http_handler
+handlers = syslog_handler
 qualname = http
 propagate = 0

package/lib/systemd/system/ahriman-web@.service

@@ -1,5 +1,5 @@
 [Unit]
-Description=ArcHlinux ReposItory MANager web server (%I architecture)
+Description=ArcH Linux ReposItory MANager web server (%I architecture)
 After=network.target
 
 [Service]
@@ -8,8 +8,5 @@ ExecStart=/usr/bin/ahriman --architecture %i web
 User=ahriman
 Group=ahriman
 
-KillSignal=SIGQUIT
-SuccessExitStatus=SIGQUIT
-
 [Install]
 WantedBy=multi-user.target

package/lib/systemd/system/ahriman@.service

@@ -1,5 +1,5 @@
 [Unit]
-Description=ArcHlinux ReposItory MANager (%I architecture)
+Description=ArcH Linux ReposItory MANager (%I architecture)
 
 [Service]
 ExecStart=/usr/bin/ahriman --architecture %i update

package/lib/systemd/system/ahriman@.timer

@@ -1,5 +1,5 @@
 [Unit]
-Description=ArcHlinux ReposItory MANager timer (%I architecture)
+Description=ArcH Linux ReposItory MANager timer (%I architecture)
 
 [Timer]
 OnCalendar=daily

package/share/ahriman/build-status.jinja2

@@ -1,54 +1,133 @@
 <!doctype html>
 <html lang="en">
     <head>
-        <title>{{ repository|e }}</title>
+        <title>{{ repository }}</title>
 
-        {% include "style.jinja2" %}
+        <meta name="viewport" content="width=device-width, initial-scale=1">
 
-        {% include "sorttable.jinja2" %}
-        {% include "search.jinja2" %}
+        <link rel="shortcut icon" href="/static/favicon.ico">
+
+        {% include "utils/style.jinja2" %}
     </head>
 
     <body>
-        <div class="root">
+
+        <div class="container">
             <h1>ahriman
-                <img src="https://img.shields.io/badge/version-{{ version|e }}-informational" alt="{{ version|e }}">
-                <img src="https://img.shields.io/badge/architecture-{{ architecture|e }}-informational" alt="{{ architecture|e }}">
-                <img src="https://img.shields.io/badge/service%20status-{{ service.status|e }}-{{ service.status_color|e }}" alt="{{ service.status|e }}" title="{{ service.timestamp|e }}">
+                {% if auth.authenticated %}
+                    <img src="https://img.shields.io/badge/version-{{ version }}-informational" alt="{{ version }}">
+                    <img src="https://img.shields.io/badge/repository-{{ repository }}-informational" alt="{{ repository }}">
+                    <img src="https://img.shields.io/badge/architecture-{{ architecture }}-informational" alt="{{ architecture }}">
+                    <img src="https://img.shields.io/badge/service%20status-{{ service.status }}-{{ service.status_color }}" alt="{{ service.status }}" title="{{ service.timestamp }}">
+                {% endif %}
             </h1>
+        </div>
 
-            {% include "search-line.jinja2" %}
+        <div class="container">
+            <div id="toolbar">
+                {% if not auth.enabled or auth.username is not none %}
+                    <button id="add" class="btn btn-primary" data-bs-toggle="modal" data-bs-target="#addForm">
+                        <i class="fa fa-plus"></i> Add
+                    </button>
+                    <button id="update" class="btn btn-secondary" onclick="updatePackages()" disabled>
+                        <i class="fa fa-play"></i> Update
+                    </button>
+                    <button id="remove" class="btn btn-danger" onclick="removePackages()" disabled>
+                        <i class="fa fa-trash"></i> Remove
+                    </button>
+                {% endif %}
+            </div>
 
-            <section class="element">
-                <table class="sortable search-table">
-                    <tr class="header">
-                        <th>package base</th>
-                        <th>packages</th>
-                        <th>version</th>
-                        <th>last update</th>
-                        <th>status</th>
+            <table id="packages" class="table table-striped table-hover"
+                   data-click-to-select="true"
+                   data-export-options='{"fileName": "packages"}'
+                   data-page-list="[10, 25, 50, 100, all]"
+                   data-page-size="10"
+                   data-pagination="true"
+                   data-resizable="true"
+                   data-search="true"
+                   data-show-columns="true"
+                   data-show-columns-search="true"
+                   data-show-columns-toggle-all="true"
+                   data-show-export="true"
+                   data-show-fullscreen="true"
+                   data-show-search-clear-button="true"
+                   data-sortable="true"
+                   data-sort-reset="true"
+                   data-toggle="table"
+                   data-toolbar="#toolbar">
+                <thead class="table-primary">
+                    <tr>
+                        <th data-checkbox="true"></th>
+                        <th data-sortable="true" data-switchable="false">package base</th>
+                        <th data-sortable="true">version</th>
+                        <th data-sortable="true">packages</th>
+                        <th data-sortable="true" data-visible="false">groups</th>
+                        <th data-sortable="true" data-visible="false">licenses</th>
+                        <th data-sortable="true">last update</th>
+                        <th data-sortable="true">status</th>
                     </tr>
+                </thead>
 
+                <tbody>
+                {% if auth.authenticated %}
                     {% for package in packages %}
-                        <tr class="package">
-                            <td class="include-search"><a href="{{ package.web_url|e }}" title="{{ package.base|e }}">{{ package.base|e }}</a></td>
-                            <td class="include-search">{{ package.packages|join("<br>"|safe) }}</td>
-                            <td>{{ package.version|e }}</td>
-                            <td>{{ package.timestamp|e }}</td>
-                            <td class="status package-{{ package.status|e }}">{{ package.status|e }}</td>
+                        <tr data-package-base="{{ package.base }}">
+                            <td data-checkbox="true"></td>
+                            <td><a href="{{ package.web_url }}" title="{{ package.base }}">{{ package.base }}</a></td>
+                            <td>{{ package.version }}</td>
+                            <td>{{ package.packages|join("<br>"|safe) }}</td>
+                            <td>{{ package.groups|join("<br>"|safe) }}</td>
+                            <td>{{ package.licenses|join("<br>"|safe) }}</td>
+                            <td>{{ package.timestamp }}</td>
+                            <td class="table-{{ package.status_color }}">{{ package.status }}</td>
                         </tr>
                     {% endfor %}
-                </table>
-            </section>
+                {% else %}
+                    <tr>
+                        <td colspan="100%">In order to see statuses you must login first.</td>
+                    </tr>
+                {% endif %}
+                </tbody>
+            </table>
+        </div>
 
-            <footer>
-                <ul class="navigation">
-                    <li><a href="https://github.com/arcan1s/ahriman" title="sources">ahriman</a></li>
-                    <li><a href="https://github.com/arcan1s/ahriman/releases" title="releases list">releases</a></li>
-                    <li><a href="https://github.com/arcan1s/ahriman/issues" title="issues tracker">report a bug</a></li>
+        <div class="container">
+            <footer class="d-flex flex-wrap justify-content-between align-items-center border-top">
+                <ul class="nav">
+                    <li><a class="nav-link" href="https://github.com/arcan1s/ahriman" title="sources">ahriman</a></li>
+                    <li><a class="nav-link" href="https://github.com/arcan1s/ahriman/releases" title="releases list">releases</a></li>
+                    <li><a class="nav-link" href="https://github.com/arcan1s/ahriman/issues" title="issues tracker">report a bug</a></li>
                 </ul>
+
+                {% if index_url is not none %}
+                    <ul class="nav">
+                        <li><a class="nav-link" href="{{ index_url }}" title="repo index">repo index</a></li>
+                    </ul>
+                {% endif %}
+
+                {% if auth.enabled %}
+                    {% if auth.username is none %}
+                        {{ auth.control|safe }}
+                    {% else %}
+                        <form action="/user-api/v1/logout" method="post">
+                            <button class="btn btn-link" style="text-decoration: none">logout ({{ auth.username }})</button>
+                        </form>
+                    {% endif %}
+                {% endif %}
             </footer>
         </div>
+
+        {% if auth.enabled %}
+            {% include "build-status/login-modal.jinja2" %}
+        {% endif %}
+
+        {% include "build-status/package-actions-modals.jinja2" %}
+
+        {% include "utils/bootstrap-scripts.jinja2" %}
+
+        {% include "build-status/package-actions-script.jinja2" %}
+
     </body>
 
 </html>

package/share/ahriman/build-status/login-modal.jinja2

@@ -0,0 +1,29 @@
+<div id="loginForm" tabindex="-1" role="dialog" class="modal fade">
+    <div class="modal-dialog" role="document">
+        <div class="modal-content">
+            <form action="/user-api/v1/login" method="post">
+                <div class="modal-header">
+                    <h4 class="modal-title">Login</h4>
+                    <button type="button" class="btn-close" data-bs-dismiss="modal" aria-label="close"></button>
+                </div>
+                <div class="modal-body">
+                    <div class="form-group row">
+                        <label for="username" class="col-sm-2 col-form-label">Username</label>
+                        <div class="col-sm-10">
+                            <input id="username" type="text" class="form-control" placeholder="enter username" name="username" required>
+                        </div>
+                    </div>
+                    <div class="form-group row">
+                        <label for="password" class="col-sm-2 col-form-label">Password</label>
+                        <div class="col-sm-10">
+                            <input id="password" type="password" class="form-control" placeholder="enter password" name="password" required>
+                        </div>
+                    </div>
+                </div>
+                <div class="modal-footer">
+                    <button class="btn btn-primary">Login</button>
+                </div>
+            </form>
+        </div>
+    </div>
+</div>

package/share/ahriman/build-status/package-actions-modals.jinja2

@@ -0,0 +1,60 @@
+<div id="addForm" tabindex="-1" role="dialog" class="modal fade">
+    <div class="modal-dialog" role="document">
+        <div class="modal-content">
+            <div class="modal-header">
+                <h4 class="modal-title">Add new packages</h4>
+                <button type="button" class="btn-close" data-bs-dismiss="modal" aria-label="close"></button>
+            </div>
+            <div class="modal-body">
+                <div class="form-group row">
+                    <label for="package" class="col-sm-2 col-form-label">Package</label>
+                    <div class="col-sm-10">
+                        <input id="package" type="text" list="knownPackages" class="form-control" placeholder="AUR package" name="package" required>
+                        <datalist id="knownPackages"></datalist>
+                    </div>
+                </div>
+            </div>
+            <div class="modal-footer">
+                <button type="button" class="btn btn-secondary" data-bs-dismiss="modal">Close</button>
+                <button type="button" class="btn btn-success" data-bs-dismiss="modal" onclick="requestPackages()">Request</button>
+                <button type="button" class="btn btn-primary" data-bs-dismiss="modal" onclick="addPackages()">Add</button>
+            </div>
+        </div>
+    </div>
+</div>
+
+<div id="failedForm" tabindex="-1" role="dialog" class="modal fade">
+    <div class="modal-dialog" role="document">
+        <div class="modal-content">
+            <div class="modal-header bg-danger">
+                <h4 class="modal-title">Failed</h4>
+                <button type="button" class="btn-close" data-bs-dismiss="modal" aria-label="close"></button>
+            </div>
+            <div class="modal-body">
+                <p>Packages update has failed.</p>
+                <p id="errorDetails"></p>
+            </div>
+            <div class="modal-footer">
+                <button type="button" class="btn btn-primary" data-bs-dismiss="modal">Close</button>
+            </div>
+        </div>
+    </div>
+</div>
+
+<div id="successForm" tabindex="-1" role="dialog" class="modal fade">
+    <div class="modal-dialog" role="document">
+        <div class="modal-content">
+            <div class="modal-header bg-success">
+                <h4 class="modal-title">Success</h4>
+                <button type="button" class="btn-close" data-bs-dismiss="modal" aria-label="close"></button>
+            </div>
+            <div class="modal-body">
+                <p>Packages update has been run.</p>
+                <ul id="successDetails"></ul>
+            </div>
+            <div class="modal-footer">
+                <button type="button" class="btn btn-primary" data-bs-dismiss="modal">Close</button>
+            </div>
+        </div>
+    </div>
+</div>

package/share/ahriman/build-status/package-actions-script.jinja2

@@ -0,0 +1,95 @@
+<script>
+    const $remove = $("#remove");
+    const $update = $("#update");
+
+    const $table = $("#packages");
+    $table.on("check.bs.table uncheck.bs.table check-all.bs.table uncheck-all.bs.table",
+        function () {
+            $remove.prop("disabled", !$table.bootstrapTable("getSelections").length);
+            $update.prop("disabled", !$table.bootstrapTable("getSelections").length);
+        })
+
+    const $successForm = $("#successForm");
+    const $successDetails = $("#successDetails");
+    $successForm.on("hidden.bs.modal", function() { window.location.reload(); });
+
+    const $failedForm = $("#failedForm");
+    const $errorDetails = $("#errorDetails");
+    $failedForm.on("hidden.bs.modal", function() { window.location.reload(); });
+
+    const $package = $("#package");
+    const $knownPackages = $("#knownPackages");
+    $package.keyup(function () {
+        const $this = $(this);
+        clearTimeout($this.data("timeout"));
+
+        $this.data("timeout", setTimeout($.proxy(function () {
+            const $value = $package.val();
+
+            $.ajax({
+                url: "/service-api/v1/search",
+                data: {"for": $value},
+                type: "GET",
+                dataType: "json",
+                success: function (resp) {
+                    const $options = resp.map(function (pkg) {
+                        const $option = document.createElement("option");
+                        $option.value = pkg.package;
+                        $option.innerText = `${pkg.package} (${pkg.description})`;
+                        return $option;
+                    });
+                    $knownPackages.empty().append($options);
+                    $this.focus();
+                },
+            })
+        }, this), 500));
+    })
+
+    function doPackageAction($uri, $packages) {
+        if ($packages.length === 0)
+            return;
+        $.ajax({
+            url: $uri,
+            data: JSON.stringify({packages: $packages}),
+            type: "POST",
+            contentType: "application/json",
+            success: function (_) {
+                const $details = $packages.map(function (pkg) {
+                    const $li = document.createElement("li");
+                    $li.innerText = pkg;
+                    return $li;
+                });
+                $successDetails.empty().append($details);
+                $successForm.modal("show");
+            },
+            error: function (jqXHR, textStatus, errorThrown) {
+                $errorDetails.text(errorThrown);
+                $failedForm.modal("show");
+            },
+        })
+    }
+
+    function getSelection() {
+        return $.map($table.bootstrapTable("getSelections"), function(row) {
+            return row._data["package-base"];
+        })
+    }
+
+    function addPackages() {
+        const $packages = [$package.val()]
+        doPackageAction("/service-api/v1/add", $packages);
+    }
+
+    function requestPackages() {
+        const $packages = [$package.val()]
+        doPackageAction("/service-api/v1/request", $packages);
+    }
+
+    function removePackages() { doPackageAction("/service-api/v1/remove", getSelection()); }
+
+    function updatePackages() { doPackageAction("/service-api/v1/add", getSelection()); }
+
+    $(function () {
+        $table.bootstrapTable("uncheckAll");
+    })
+</script>

package/share/ahriman/email-index.jinja2

@@ -0,0 +1,42 @@
+{#simplified version of full report#}
+<!doctype html>
+<html lang="en">
+    <head>
+        <title>{{ repository }}</title>
+
+        <meta name="viewport" content="width=device-width, initial-scale=1">
+
+        {% include "utils/style.jinja2" %}
+    </head>
+
+    <body>
+
+        <div class="container">
+            <table id="packages" class="table table-striped">
+                <thead class="table-primary">
+                    <tr>
+                        <th>package</th>
+                        <th>version</th>
+                        <th>archive size</th>
+                        <th>installed size</th>
+                        <th>build date</th>
+                    </tr>
+                </thead>
+
+                <tbody>
+                {% for package in packages %}
+                    <tr>
+                        <td><a href="{{ link_path }}/{{ package.filename }}" title="{{ package.name }}">{{ package.name }}</a></td>
+                        <td>{{ package.version }}</td>
+                        <td>{{ package.archive_size }}</td>
+                        <td>{{ package.installed_size }}</td>
+                        <td>{{ package.build_date }}</td>
+                    </tr>
+                {% endfor %}
+                </tbody>
+            </table>
+        </div>
+
+    </body>
+
+</html>

package/share/ahriman/repo-index.jinja2

@@ -1,62 +1,95 @@
 <!doctype html>
 <html lang="en">
     <head>
-        <title>{{ repository|e }}</title>
+        <title>{{ repository }}</title>
 
-        {% include "style.jinja2" %}
+        <meta name="viewport" content="width=device-width, initial-scale=1">
 
-        {% include "sorttable.jinja2" %}
-        {% include "search.jinja2" %}
+        {% include "utils/style.jinja2" %}
     </head>
 
     <body>
-        <div class="root">
-            <h1>Archlinux user repository</h1>
 
-            <section class="element">
-                {% if pgp_key is not none %}
-                    <p>This repository is signed with <a href="http://keys.gnupg.net/pks/lookup?search=0x{{ pgp_key|e }}&fingerprint=on&op=index" title="key search">{{ pgp_key|e }}</a> by default.</p>
-                {% endif %}
+        <div class="container">
+            <h1>Arch Linux user repository</h1>
+        </div>
 
-                <code>
-                    $ cat /etc/pacman.conf<br>
-                    [{{ repository|e }}]<br>
-                    Server = {{ link_path|e }}<br>
-                    SigLevel = Database{% if has_repo_signed %}Required{% else %}Never{% endif %} Package{% if has_package_signed %}Required{% else %}Never{% endif %} TrustedOnly
-                </code>
-            </section>
+        <div class="container">
+            {% if pgp_key is not none %}
+                <p>This repository is signed with <a href="https://pgp.mit.edu/pks/lookup?search=0x{{ pgp_key }}&fingerprint=on&op=index" title="key search">{{ pgp_key }}</a> by default.</p>
+            {% endif %}
 
-            {% include "search-line.jinja2" %}
+<pre>$ cat /etc/pacman.conf
+[{{ repository }}]
+Server = {{ link_path }}
+SigLevel = Database{% if has_repo_signed %}Required{% else %}Never{% endif %} Package{% if has_package_signed %}Required{% else %}Never{% endif %} TrustedOnly</pre>
+        </div>
 
-            <section class="element">
-                <table class="sortable search-table">
-                    <tr class="header">
-                        <th>package</th>
-                        <th>version</th>
-                        <th>archive size</th>
-                        <th>installed size</th>
-                        <th>build date</th>
+        <div class="container">
+            <table id="packages" class="table table-striped table-hover"
+                   data-export-options='{"fileName": "packages"}'
+                   data-page-list="[10, 25, 50, 100, all]"
+                   data-page-size="10"
+                   data-pagination="true"
+                   data-resizable="true"
+                   data-search="true"
+                   data-show-columns="true"
+                   data-show-columns-search="true"
+                   data-show-columns-toggle-all="true"
+                   data-show-export="true"
+                   data-show-fullscreen="true"
+                   data-show-search-clear-button="true"
+                   data-sortable="true"
+                   data-sort-reset="true"
+                   data-toggle="table">
+                <thead class="table-primary">
+                    <tr>
+                        <th data-sortable="true" data-switchable="false">package</th>
+                        <th data-sortable="true">version</th>
+                        <th data-sortable="true" data-visible="false">architecture</th>
+                        <th data-sortable="true" data-visible="false">description</th>
+                        <th data-sortable="true" data-visible="false">upstream url</th>
+                        <th data-sortable="true" data-visible="false">licenses</th>
+                        <th data-sortable="true" data-visible="false">groups</th>
+                        <th data-sortable="true" data-visible="false">depends</th>
+                        <th data-sortable="true">archive size</th>
+                        <th data-sortable="true">installed size</th>
+                        <th data-sortable="true">build date</th>
                     </tr>
+                </thead>
 
-                    {% for package in packages %}
-                        <tr class="package">
-                            <td class="include-search"><a href="{{ link_path|e }}/{{ package.filename|e }}" title="{{ package.name|e }}">{{ package.name|e }}</a></td>
-                            <td>{{ package.version|e }}</td>
-                            <td>{{ package.archive_size|e }}</td>
-                            <td>{{ package.installed_size|e }}</td>
-                            <td>{{ package.build_date|e }}</td>
-                        </tr>
-                    {% endfor %}
-                </table>
-            </section>
+                <tbody>
+                {% for package in packages %}
+                    <tr>
+                        <td><a href="{{ link_path }}/{{ package.filename }}" title="{{ package.name }}">{{ package.name }}</a></td>
+                        <td>{{ package.version }}</td>
+                        <td>{{ package.architecture }}</td>
+                        <td>{{ package.description }}</td>
+                        <td><a href="{{ package.url }}" title="{{ package.name }} upstream url">{{ package.url }}</a></td>
+                        <td>{{ package.licenses|join("<br>"|safe) }}</td>
+                        <td>{{ package.groups|join("<br>"|safe) }}</td>
+                        <td>{{ package.depends|join("<br>"|safe) }}</td>
+                        <td>{{ package.archive_size }}</td>
+                        <td>{{ package.installed_size }}</td>
+                        <td>{{ package.build_date }}</td>
+                    </tr>
+                {% endfor %}
+                </tbody>
+            </table>
+        </div>
 
-            <footer>
-                <ul class="navigation">
+        <div class="container">
+            <footer class="d-flex flex-wrap justify-content-between align-items-center border-top">
+                <ul class="nav">
                     {% if homepage is not none %}
-                        <li><a href="{{ homepage|e }}" title="homepage">Homepage</a></li>
+                        <li><a class="nav-link" href="{{ homepage }}" title="homepage">Homepage</a></li>
                     {% endif %}
                 </ul>
             </footer>
         </div>
+
+        {% include "utils/bootstrap-scripts.jinja2" %}
+
     </body>
+
 </html>

package/share/ahriman/search-line.jinja2

@@ -1,3 +0,0 @@
-<section class="element">
-    <input type="search" id="search" onkeyup="searchInTable()" placeholder="search for package" title="search for package"/>
-</section>

package/share/ahriman/search.jinja2

@@ -1,25 +0,0 @@
-<script type="text/javascript">
-    function searchInTable() {
-        const input = document.getElementById("search");
-        const filter = input.value.toLowerCase();
-        const tables = document.getElementsByClassName("search-table");
-
-        for (let i = 0; i < tables.length; i++) {
-            const tr = tables[i].getElementsByTagName("tr");
-            // from 1 coz of header
-            for (let i = 1; i < tr.length; i++) {
-                let td = tr[i].getElementsByClassName("include-search");
-                let display = "none";
-                for (let j = 0; j < td.length; j++) {
-                    if (td[j].tagName.toLowerCase() === "td") {
-                        if (td[j].innerHTML.toLowerCase().indexOf(filter) > -1) {
-                            display = "";
-                            break;
-                        }
-                    }
-                }
-                tr[i].style.display = display;
-            }
-        }
-    }
-</script>

package/share/ahriman/sorttable.jinja2

@@ -1 +0,0 @@
-<script src="https://www.kryogenix.org/code/browser/sorttable/sorttable.js"></script>

package/share/ahriman/style.jinja2

@@ -1,136 +0,0 @@
-<style>
-    :root {
-        --color-building: 255, 255, 146;
-        --color-failed: 255, 94, 94;
-        --color-pending: 255, 255, 146;
-        --color-success: 94, 255, 94;
-        --color-unknown: 225, 225, 225;
-
-        --color-header: 200, 200, 255;
-        --color-hover: 255, 255, 225;
-        --color-line-blue: 235, 235, 255;
-        --color-line-white: 255, 255, 255;
-    }
-
-    @keyframes blink-building {
-        0% { background-color: rgba(var(--color-building), 1.0); }
-        10% { background-color: rgba(var(--color-building), 0.9); }
-        20% { background-color: rgba(var(--color-building), 0.8); }
-        30% { background-color: rgba(var(--color-building), 0.7); }
-        40% { background-color: rgba(var(--color-building), 0.6); }
-        50% { background-color: rgba(var(--color-building), 0.5); }
-        60% { background-color: rgba(var(--color-building), 0.4); }
-        70% { background-color: rgba(var(--color-building), 0.3); }
-        80% { background-color: rgba(var(--color-building), 0.2); }
-        90% { background-color: rgba(var(--color-building), 0.1); }
-        100% { background-color: rgba(var(--color-building), 0.0); }
-    }
-
-    div.root {
-        width: 70%;
-        padding: 15px 15% 0;
-    }
-
-    section.element, footer {
-        width: 100%;
-        padding: 10px 0;
-    }
-
-    code, input, table {
-        width: inherit;
-    }
-
-    th, td {
-        padding: 5px;
-    }
-
-    tr.package:nth-child(odd) {
-        background-color: rgba(var(--color-line-white), 1.0);
-    }
-
-    tr.package:nth-child(even) {
-        background-color: rgba(var(--color-line-blue), 1.0);
-    }
-
-    tr.package:hover {
-        background-color: rgba(var(--color-hover), 1.0);
-    }
-
-    tr.header{
-        background-color: rgba(var(--color-header), 1.0);
-    }
-
-    td.status {
-        text-align: center;
-    }
-
-    td.package-unknown {
-        background-color: rgba(var(--color-unknown), 1.0);
-    }
-    td.package-pending {
-        background-color: rgba(var(--color-pending), 1.0);
-    }
-    td.package-building {
-        background-color: rgba(var(--color-building), 1.0);
-        animation-name: blink-building;
-        animation-duration: 1s;
-        animation-timing-function: linear;
-        animation-iteration-count: infinite;
-        animation-direction: alternate;
-    }
-    td.package-failed {
-        background-color: rgba(var(--color-failed), 1.0);
-    }
-    td.package-success {
-        background-color: rgba(var(--color-success), 1.0);
-    }
-
-    li.service-unknown {
-        background-color: rgba(var(--color-unknown), 1.0);
-    }
-    li.service-building {
-        background-color: rgba(var(--color-building), 1.0);
-        animation-name: blink-building;
-        animation-duration: 1s;
-        animation-timing-function: linear;
-        animation-iteration-count: infinite;
-        animation-direction: alternate;
-    }
-    li.service-failed {
-        background-color: rgba(var(--color-failed), 1.0);
-    }
-    li.service-success {
-        background-color: rgba(var(--color-success), 1.0);
-    }
-
-    ul.navigation {
-        list-style-type: none;
-        margin: 0;
-        padding: 0;
-        overflow: hidden;
-        background-color: rgba(var(--color-header), 1.0);
-    }
-
-    ul.navigation li {
-        float: left;
-    }
-
-    ul.navigation li.status {
-        display: block;
-        text-align: center;
-        text-decoration: none;
-        padding: 14px 16px;
-    }
-
-    ul.navigation li a {
-        display: block;
-        color: black;
-        text-align: center;
-        text-decoration: none;
-        padding: 14px 16px;
-    }
-
-    ul.navigation li a:hover {
-        background-color: rgba(var(--color-hover), 1.0);
-    }
-</style>

package/share/ahriman/utils/bootstrap-scripts.jinja2

@@ -0,0 +1,12 @@
+<script src="https://cdn.jsdelivr.net/npm/jquery/dist/jquery.min.js"></script>
+
+<script src="https://unpkg.com/tableexport.jquery.plugin/tableExport.min.js"></script>
+
+<script src="https://unpkg.com/jquery-resizable-columns@0.2.3/dist/jquery.resizableColumns.min.js"></script>
+
+<script src="https://cdn.jsdelivr.net/npm/bootstrap@5.1.0/dist/js/bootstrap.bundle.min.js" integrity="sha384-U1DAWAznBHeqEIlVSCgzq+c9gqGAJn5c/t99JyeKa9xxaYpSvHU5awsuZVVFIhvj" crossorigin="anonymous"></script>
+<script src="https://unpkg.com/bootstrap-table@1.18.3/dist/bootstrap-table.min.js"></script>
+
+<script src="https://unpkg.com/bootstrap-table@1.18.3/dist/extensions/export/bootstrap-table-export.min.js"></script>
+
+<script src="https://unpkg.com/bootstrap-table@1.18.3/dist/extensions/resizable/bootstrap-table-resizable.js"></script>

package/share/ahriman/utils/style.jinja2

@@ -0,0 +1,9 @@
+<script src="https://kit.fontawesome.com/0d6d6d5226.js" crossorigin="anonymous"></script>
+<link href="https://cdn.jsdelivr.net/npm/bootstrap@5.1.0/dist/css/bootstrap.min.css" rel="stylesheet" integrity="sha384-KyZXEAg3QhqLMpG8r+8fhAXLRk2vvoC2f3B09zVXn8CA5QIVfZOJ3BCsw2P0p/We" crossorigin="anonymous">
+
+<link href="https://unpkg.com/bootstrap-table@1.18.3/dist/bootstrap-table.min.css" rel="stylesheet">
+
+<link href="https://unpkg.com/jquery-resizable-columns@0.2.3/dist/jquery.resizableColumns.css" rel="stylesheet">
+
+<style>
+</style>

setup.cfg

@@ -0,0 +1,5 @@
+[aliases]
+test = pytest
+
+[tool:pytest]
+addopts = --cov=ahriman --cov-report term-missing:skip-covered --pspec

setup.py

@@ -1,72 +1,119 @@
-from distutils.util import convert_path
+from pathlib import Path
 from setuptools import setup, find_packages
-from os import path
+from typing import Any, Dict
+
+
+metadata_path = Path(__file__).resolve().parent / "src/ahriman/version.py"
+metadata: Dict[str, Any] = {}
+with metadata_path.open() as metadata_file:
+    exec(metadata_file.read(), metadata)  # pylint: disable=exec-used
 
-here = path.abspath(path.dirname(__file__))
-metadata = dict()
-with open(convert_path('src/ahriman/version.py')) as metadata_file:
-    exec(metadata_file.read(), metadata)
 
 setup(
-    name='ahriman',
+    name="ahriman",
 
-    version=metadata['__version__'],
+    version=metadata["__version__"],
     zip_safe=False,
 
-    description='ArcHlinux ReposItory MANager',
+    description="ArcH Linux ReposItory MANager",
 
-    author='arcanis',
-    author_email='',
-    url='https://github.com/arcan1s/ahriman',
+    author="arcanis",
+    author_email="",
+    url="https://github.com/arcan1s/ahriman",
 
-    license='GPL3',
+    license="GPL3",
 
-    packages=find_packages('src'),
-    package_dir={'': 'src'},
+    packages=find_packages("src"),
+    package_dir={"": "src"},
 
     dependency_links=[
     ],
     install_requires=[
-        'aur',
-        'pyalpm',
-        'srcinfo',
+        "aur",
+        "passlib",
+        "pyalpm",
+        "requests",
+        "srcinfo",
     ],
     setup_requires=[
-        'pytest-runner',
+        "pytest-runner",
     ],
     tests_require=[
-        'pytest',
+        "pytest",
+        "pytest-aiohttp",
+        "pytest-cov",
+        "pytest-helpers-namespace",
+        "pytest-mock",
+        "pytest-pspec",
+        "pytest-resource-path",
     ],
 
     include_package_data=True,
     scripts=[
-        'package/bin/ahriman',
+        "package/bin/ahriman",
     ],
     data_files=[
-        ('/etc', [
-            'package/etc/ahriman.ini',
+        ("/etc", [
+            "package/etc/ahriman.ini",
         ]),
-        ('/etc/ahriman.ini.d', [
-            'package/etc/ahriman.ini.d/logging.ini',
+        ("/etc/ahriman.ini.d", [
+            "package/etc/ahriman.ini.d/logging.ini",
         ]),
-        ('lib/systemd/system', [
-            'package/lib/systemd/system/ahriman@.service',
-            'package/lib/systemd/system/ahriman@.timer',
-            'package/lib/systemd/system/ahriman-web@.service',
+        ("lib/systemd/system", [
+            "package/lib/systemd/system/ahriman@.service",
+            "package/lib/systemd/system/ahriman@.timer",
+            "package/lib/systemd/system/ahriman-web@.service",
         ]),
-        ('share/ahriman', [
-            'package/share/ahriman/build-status.jinja2',
-            'package/share/ahriman/repo-index.jinja2',
-            'package/share/ahriman/search.jinja2',
-            'package/share/ahriman/search-line.jinja2',
-            'package/share/ahriman/sorttable.jinja2',
-            'package/share/ahriman/style.jinja2',
+        ("share/ahriman", [
+            "package/share/ahriman/build-status.jinja2",
+            "package/share/ahriman/email-index.jinja2",
+            "package/share/ahriman/repo-index.jinja2",
         ]),
+        ("share/ahriman/build-status", [
+            "package/share/ahriman/build-status/login-modal.jinja2",
+            "package/share/ahriman/build-status/package-actions-modals.jinja2",
+            "package/share/ahriman/build-status/package-actions-script.jinja2",
+        ]),
+        ("share/ahriman/static", [
+            "package/share/ahriman/static/favicon.ico",
+        ]),
+        ("share/ahriman/utils", [
+            "package/share/ahriman/utils/bootstrap-scripts.jinja2",
+            "package/share/ahriman/utils/style.jinja2",
+        ]),
+        ("share/man/man1", [
+            "docs/ahriman.1",
+        ])
     ],
 
     extras_require={
-        'html-templates': ['Jinja2'],
-        'test': ['coverage', 'pytest'],
-        'web': ['Jinja2', 'aiohttp', 'aiohttp_jinja2', 'requests'],
+        "check": [
+            "autopep8",
+            "bandit",
+            "mypy",
+            "pylint",
+        ],
+        "s3": [
+            "boto3",
+        ],
+        "test": [
+            "pytest",
+            "pytest-aiohttp",
+            "pytest-cov",
+            "pytest-helpers-namespace",
+            "pytest-mock",
+            "pytest-pspec",
+            "pytest-resource-path",
+        ],
+        "web": [
+            "Jinja2",
+            "aiohttp",
+            "aiohttp_jinja2",
+            "aioauth-client",
+            "aiohttp_debugtoolbar",
+            "aiohttp_session",
+            "aiohttp_security",
+            "cryptography",
+        ],
     },
 )

src/ahriman/__init__.py

@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2021 Evgenii Alekseev.
+# Copyright (c) 2021 ahriman team.
 #
 # This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).

src/ahriman/application/__init__.py

@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2021 Evgenii Alekseev.
+# Copyright (c) 2021 ahriman team.
 #
 # This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).

src/ahriman/application/ahriman.py

@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2021 Evgenii Alekseev.
+# Copyright (c) 2021 ahriman team.
 #
 # This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).
@@ -19,89 +19,376 @@
 #
 import argparse
 import sys
+import tempfile
 
-import ahriman.application.handlers as handlers
-import ahriman.version as version
+from pathlib import Path
+
+from ahriman import version
+from ahriman.application import handlers
+from ahriman.models.build_status import BuildStatusEnum
+from ahriman.models.package_source import PackageSource
+from ahriman.models.sign_settings import SignSettings
+from ahriman.models.user_access import UserAccess
 
 
-if __name__ == '__main__':
-    parser = argparse.ArgumentParser(prog='ahriman', description='ArcHlinux ReposItory MANager')
+# pylint thinks it is bad idea, but get the fuck off
+SubParserAction = argparse._SubParsersAction  # pylint: disable=protected-access
+
+
+def _parser() -> argparse.ArgumentParser:
+    """
+    command line parser generator
+    :return: command line parser for the application
+    """
+    parser = argparse.ArgumentParser(prog="ahriman", description="ArcH Linux ReposItory MANager",
+                                     formatter_class=argparse.ArgumentDefaultsHelpFormatter)
+    parser.add_argument("-a", "--architecture", help="target architectures (can be used multiple times)",
+                        action="append")
+    parser.add_argument("-c", "--configuration", help="configuration path", type=Path, default=Path("/etc/ahriman.ini"))
+    parser.add_argument("--force", help="force run, remove file lock", action="store_true")
     parser.add_argument(
-        '-a',
-        '--architecture',
-        help='target architectures (can be used multiple times)',
-        action='append')
-    parser.add_argument('-c', '--config', help='configuration path', default='/etc/ahriman.ini')
-    parser.add_argument('--force', help='force run, remove file lock', action='store_true')
-    parser.add_argument('--lock', help='lock file', default='/tmp/ahriman.lock')
-    parser.add_argument('--no-log', help='redirect all log messages to stderr', action='store_true')
-    parser.add_argument('--no-report', help='force disable reporting to web service', action='store_true')
-    parser.add_argument('--unsafe', help='allow to run ahriman as non-ahriman user', action='store_true')
-    parser.add_argument('-v', '--version', action='version', version=version.__version__)
-    subparsers = parser.add_subparsers(title='command')
+        "-l",
+        "--lock",
+        help="lock file",
+        type=Path,
+        default=Path(tempfile.gettempdir()) / "ahriman.lock")
+    parser.add_argument("--no-log", help="redirect all log messages to stderr", action="store_true")
+    parser.add_argument("--no-report", help="force disable reporting to web service", action="store_true")
+    parser.add_argument("--unsafe", help="allow to run ahriman as non-ahriman user", action="store_true")
+    parser.add_argument("-v", "--version", action="version", version=version.__version__)
 
-    add_parser = subparsers.add_parser('add', description='add package')
-    add_parser.add_argument('package', help='package base/name or archive path', nargs='+')
-    add_parser.add_argument('--without-dependencies', help='do not add dependencies', action='store_true')
-    add_parser.set_defaults(handler=handlers.Add)
+    subparsers = parser.add_subparsers(title="command", help="command to run", dest="command", required=True)
 
-    check_parser = subparsers.add_parser('check', description='check for updates. Same as update --dry-run --no-manual')
-    check_parser.add_argument('package', help='filter check by package base', nargs='*')
-    check_parser.add_argument('--no-vcs', help='do not check VCS packages', action='store_true')
-    check_parser.set_defaults(handler=handlers.Update, no_aur=False, no_manual=True, dry_run=True)
+    _set_add_parser(subparsers)
+    _set_check_parser(subparsers)
+    _set_clean_parser(subparsers)
+    _set_config_parser(subparsers)
+    _set_init_parser(subparsers)
+    _set_key_import_parser(subparsers)
+    _set_rebuild_parser(subparsers)
+    _set_remove_parser(subparsers)
+    _set_remove_unknown_parser(subparsers)
+    _set_report_parser(subparsers)
+    _set_search_parser(subparsers)
+    _set_setup_parser(subparsers)
+    _set_sign_parser(subparsers)
+    _set_status_parser(subparsers)
+    _set_status_update_parser(subparsers)
+    _set_sync_parser(subparsers)
+    _set_update_parser(subparsers)
+    _set_user_parser(subparsers)
+    _set_web_parser(subparsers)
 
-    clean_parser = subparsers.add_parser('clean', description='clear all local caches')
-    clean_parser.add_argument('--no-build', help='do not clear directory with package sources', action='store_true')
-    clean_parser.add_argument('--no-cache', help='do not clear directory with package caches', action='store_true')
-    clean_parser.add_argument('--no-chroot', help='do not clear build chroot', action='store_true')
-    clean_parser.add_argument(
-        '--no-manual',
-        help='do not clear directory with manually added packages',
-        action='store_true')
-    clean_parser.add_argument('--no-packages', help='do not clear directory with built packages', action='store_true')
-    clean_parser.set_defaults(handler=handlers.Clean)
+    return parser
 
-    config_parser = subparsers.add_parser('config', description='dump configuration for specified architecture')
-    config_parser.set_defaults(handler=handlers.Dump, lock=None, no_report=True, unsafe=True)
 
-    rebuild_parser = subparsers.add_parser('rebuild', description='rebuild whole repository')
-    rebuild_parser.set_defaults(handler=handlers.Rebuild)
+def _set_add_parser(root: SubParserAction) -> argparse.ArgumentParser:
+    """
+    add parser for add subcommand
+    :param root: subparsers for the commands
+    :return: created argument parser
+    """
+    parser = root.add_parser("add", help="add package", description="add package",
+                             formatter_class=argparse.ArgumentDefaultsHelpFormatter)
+    parser.add_argument("package", help="package base/name or archive path", nargs="+")
+    parser.add_argument("--now", help="run update function after", action="store_true")
+    parser.add_argument("--source", help="package source", choices=PackageSource, type=PackageSource,
+                        default=PackageSource.Auto)
+    parser.add_argument("--without-dependencies", help="do not add dependencies", action="store_true")
+    parser.set_defaults(handler=handlers.Add)
+    return parser
 
-    remove_parser = subparsers.add_parser('remove', description='remove package')
-    remove_parser.add_argument('package', help='package name or base', nargs='+')
-    remove_parser.set_defaults(handler=handlers.Remove)
 
-    report_parser = subparsers.add_parser('report', description='generate report')
-    report_parser.add_argument('target', help='target to generate report', nargs='*')
-    report_parser.set_defaults(handler=handlers.Report)
+def _set_check_parser(root: SubParserAction) -> argparse.ArgumentParser:
+    """
+    add parser for check subcommand
+    :param root: subparsers for the commands
+    :return: created argument parser
+    """
+    parser = root.add_parser("check", help="check for updates",
+                             description="check for updates. Same as update --dry-run --no-manual",
+                             formatter_class=argparse.ArgumentDefaultsHelpFormatter)
+    parser.add_argument("package", help="filter check by package base", nargs="*")
+    parser.add_argument("--no-vcs", help="do not check VCS packages", action="store_true")
+    parser.set_defaults(handler=handlers.Update, no_aur=False, no_manual=True, dry_run=True)
+    return parser
 
-    status_parser = subparsers.add_parser('status', description='request status of the package')
-    status_parser.add_argument('--ahriman', help='get service status itself', action='store_true')
-    status_parser.add_argument('package', help='filter status by package base', nargs='*')
-    status_parser.set_defaults(handler=handlers.Status, lock=None, no_report=True, unsafe=True)
 
-    sync_parser = subparsers.add_parser('sync', description='sync packages to remote server')
-    sync_parser.add_argument('target', help='target to sync', nargs='*')
-    sync_parser.set_defaults(handler=handlers.Sync)
+def _set_clean_parser(root: SubParserAction) -> argparse.ArgumentParser:
+    """
+    add parser for clean subcommand
+    :param root: subparsers for the commands
+    :return: created argument parser
+    """
+    parser = root.add_parser("clean", help="clean local caches", description="clear local caches",
+                             formatter_class=argparse.ArgumentDefaultsHelpFormatter)
+    parser.add_argument("--no-build", help="do not clear directory with package sources", action="store_true")
+    parser.add_argument("--no-cache", help="do not clear directory with package caches", action="store_true")
+    parser.add_argument("--no-chroot", help="do not clear build chroot", action="store_true")
+    parser.add_argument("--no-manual", help="do not clear directory with manually added packages", action="store_true")
+    parser.add_argument("--no-packages", help="do not clear directory with built packages", action="store_true")
+    parser.set_defaults(handler=handlers.Clean, no_log=True, unsafe=True)
+    return parser
 
-    update_parser = subparsers.add_parser('update', description='run updates')
-    update_parser.add_argument('package', help='filter check by package base', nargs='*')
-    update_parser.add_argument(
-        '--dry-run', help='just perform check for updates, same as check command', action='store_true')
-    update_parser.add_argument('--no-aur', help='do not check for AUR updates. Implies --no-vcs', action='store_true')
-    update_parser.add_argument('--no-manual', help='do not include manual updates', action='store_true')
-    update_parser.add_argument('--no-vcs', help='do not check VCS packages', action='store_true')
-    update_parser.set_defaults(handler=handlers.Update)
 
-    web_parser = subparsers.add_parser('web', description='start web server')
-    web_parser.set_defaults(handler=handlers.Web, lock=None, no_report=True)
+def _set_config_parser(root: SubParserAction) -> argparse.ArgumentParser:
+    """
+    add parser for config subcommand
+    :param root: subparsers for the commands
+    :return: created argument parser
+    """
+    parser = root.add_parser("config", help="dump configuration",
+                             description="dump configuration for specified architecture",
+                             formatter_class=argparse.ArgumentDefaultsHelpFormatter)
+    parser.set_defaults(handler=handlers.Dump, lock=None, no_log=True, no_report=True, unsafe=True)
+    return parser
 
-    args = parser.parse_args()
-    if 'handler' not in args:
-        parser.print_help()
-        sys.exit(1)
 
-    handler: handlers.Handler = args.handler
-    status = handler.execute(args)
+def _set_init_parser(root: SubParserAction) -> argparse.ArgumentParser:
+    """
+    add parser for init subcommand
+    :param root: subparsers for the commands
+    :return: created argument parser
+    """
+    parser = root.add_parser("init", help="create repository tree",
+                             description="create empty repository tree. Optional command for auto architecture support",
+                             formatter_class=argparse.ArgumentDefaultsHelpFormatter)
+    parser.set_defaults(handler=handlers.Init, no_report=True)
+    return parser
 
-    sys.exit(status)
+
+def _set_key_import_parser(root: SubParserAction) -> argparse.ArgumentParser:
+    """
+    add parser for key import subcommand
+    :param root: subparsers for the commands
+    :return: created argument parser
+    """
+    parser = root.add_parser("key-import", help="import PGP key",
+                             description="import PGP key from public sources to repository user",
+                             formatter_class=argparse.ArgumentDefaultsHelpFormatter)
+    parser.add_argument("--key-server", help="key server for key import", default="pgp.mit.edu")
+    parser.add_argument("key", help="PGP key to import from public server")
+    parser.set_defaults(handler=handlers.KeyImport, architecture=[""], lock=None, no_report=True)
+    return parser
+
+
+def _set_rebuild_parser(root: SubParserAction) -> argparse.ArgumentParser:
+    """
+    add parser for rebuild subcommand
+    :param root: subparsers for the commands
+    :return: created argument parser
+    """
+    parser = root.add_parser("rebuild", help="rebuild repository", description="rebuild whole repository",
+                             formatter_class=argparse.ArgumentDefaultsHelpFormatter)
+    parser.add_argument("--depends-on", help="only rebuild packages that depend on specified package", action="append")
+    parser.set_defaults(handler=handlers.Rebuild)
+    return parser
+
+
+def _set_remove_parser(root: SubParserAction) -> argparse.ArgumentParser:
+    """
+    add parser for remove subcommand
+    :param root: subparsers for the commands
+    :return: created argument parser
+    """
+    parser = root.add_parser("remove", help="remove package", description="remove package",
+                             formatter_class=argparse.ArgumentDefaultsHelpFormatter)
+    parser.add_argument("package", help="package name or base", nargs="+")
+    parser.set_defaults(handler=handlers.Remove)
+    return parser
+
+
+def _set_remove_unknown_parser(root: SubParserAction) -> argparse.ArgumentParser:
+    """
+    add parser for remove unknown packages subcommand
+    :param root: subparsers for the commands
+    :return: created argument parser
+    """
+    parser = root.add_parser("remove-unknown", help="remove unknown packages",
+                             description="remove packages which are missing in AUR",
+                             formatter_class=argparse.ArgumentDefaultsHelpFormatter)
+    parser.add_argument("--dry-run", help="just perform check for packages without removal", action="store_true")
+    parser.set_defaults(handler=handlers.RemoveUnknown)
+    return parser
+
+
+def _set_report_parser(root: SubParserAction) -> argparse.ArgumentParser:
+    """
+    add parser for report subcommand
+    :param root: subparsers for the commands
+    :return: created argument parser
+    """
+    parser = root.add_parser("report", help="generate report", description="generate report",
+                             formatter_class=argparse.ArgumentDefaultsHelpFormatter)
+    parser.add_argument("target", help="target to generate report", nargs="*")
+    parser.set_defaults(handler=handlers.Report)
+    return parser
+
+
+def _set_search_parser(root: SubParserAction) -> argparse.ArgumentParser:
+    """
+    add parser for search subcommand
+    :param root: subparsers for the commands
+    :return: created argument parser
+    """
+    parser = root.add_parser("search", help="search for package", description="search for package in AUR using API")
+    parser.add_argument("search", help="search terms, can be specified multiple times", nargs="+")
+    parser.set_defaults(handler=handlers.Search, architecture=[""], lock=None, no_log=True, no_report=True, unsafe=True)
+    return parser
+
+
+def _set_setup_parser(root: SubParserAction) -> argparse.ArgumentParser:
+    """
+    add parser for setup subcommand
+    :param root: subparsers for the commands
+    :return: created argument parser
+    """
+    parser = root.add_parser("setup", help="initial service configuration",
+                             description="create initial service configuration, requires root",
+                             formatter_class=argparse.ArgumentDefaultsHelpFormatter)
+    parser.add_argument("--build-command", help="build command prefix", default="ahriman")
+    parser.add_argument("--from-configuration", help="path to default devtools pacman configuration",
+                        type=Path, default=Path("/usr/share/devtools/pacman-extra.conf"))
+    parser.add_argument("--no-multilib", help="do not add multilib repository", action="store_true")
+    parser.add_argument("--packager", help="packager name and email", required=True)
+    parser.add_argument("--repository", help="repository name", required=True)
+    parser.add_argument("--sign-key", help="sign key id")
+    parser.add_argument("--sign-target", help="sign options", type=SignSettings.from_option,
+                        choices=SignSettings, action="append")
+    parser.add_argument("--web-port", help="port of the web service", type=int)
+    parser.set_defaults(handler=handlers.Setup, lock=None, no_log=True, no_report=True, unsafe=True)
+    return parser
+
+
+def _set_sign_parser(root: SubParserAction) -> argparse.ArgumentParser:
+    """
+    add parser for sign subcommand
+    :param root: subparsers for the commands
+    :return: created argument parser
+    """
+    parser = root.add_parser("sign", help="sign packages", description="(re-)sign packages and repository database",
+                             formatter_class=argparse.ArgumentDefaultsHelpFormatter)
+    parser.add_argument("package", help="sign only specified packages", nargs="*")
+    parser.set_defaults(handler=handlers.Sign)
+    return parser
+
+
+def _set_status_parser(root: SubParserAction) -> argparse.ArgumentParser:
+    """
+    add parser for status subcommand
+    :param root: subparsers for the commands
+    :return: created argument parser
+    """
+    parser = root.add_parser("status", help="get package status", description="request status of the package",
+                             formatter_class=argparse.ArgumentDefaultsHelpFormatter)
+    parser.add_argument("--ahriman", help="get service status itself", action="store_true")
+    parser.add_argument("--status", help="filter packages by status", choices=BuildStatusEnum, type=BuildStatusEnum)
+    parser.add_argument("package", help="filter status by package base", nargs="*")
+    parser.set_defaults(handler=handlers.Status, lock=None, no_log=True, no_report=True, unsafe=True)
+    return parser
+
+
+def _set_status_update_parser(root: SubParserAction) -> argparse.ArgumentParser:
+    """
+    add parser for status update subcommand
+    :param root: subparsers for the commands
+    :return: created argument parser
+    """
+    parser = root.add_parser("status-update", help="update package status", description="request status of the package",
+                             formatter_class=argparse.ArgumentDefaultsHelpFormatter)
+    parser.add_argument(
+        "package",
+        help="set status for specified packages. If no packages supplied, service status will be updated",
+        nargs="*")
+    parser.add_argument("--status", help="new status", choices=BuildStatusEnum,
+                        type=BuildStatusEnum, default=BuildStatusEnum.Success)
+    parser.add_argument("--remove", help="remove package status page", action="store_true")
+    parser.set_defaults(handler=handlers.StatusUpdate, lock=None, no_log=True, no_report=True, unsafe=True)
+    return parser
+
+
+def _set_sync_parser(root: SubParserAction) -> argparse.ArgumentParser:
+    """
+    add parser for sync subcommand
+    :param root: subparsers for the commands
+    :return: created argument parser
+    """
+    parser = root.add_parser("sync", help="sync repository", description="sync packages to remote server",
+                             formatter_class=argparse.ArgumentDefaultsHelpFormatter)
+    parser.add_argument("target", help="target to sync", nargs="*")
+    parser.set_defaults(handler=handlers.Sync)
+    return parser
+
+
+def _set_update_parser(root: SubParserAction) -> argparse.ArgumentParser:
+    """
+    add parser for update subcommand
+    :param root: subparsers for the commands
+    :return: created argument parser
+    """
+    parser = root.add_parser("update", help="update packages", description="run updates",
+                             formatter_class=argparse.ArgumentDefaultsHelpFormatter)
+    parser.add_argument("package", help="filter check by package base", nargs="*")
+    parser.add_argument("--dry-run", help="just perform check for updates, same as check command", action="store_true")
+    parser.add_argument("--no-aur", help="do not check for AUR updates. Implies --no-vcs", action="store_true")
+    parser.add_argument("--no-manual", help="do not include manual updates", action="store_true")
+    parser.add_argument("--no-vcs", help="do not check VCS packages", action="store_true")
+    parser.set_defaults(handler=handlers.Update)
+    return parser
+
+
+def _set_user_parser(root: SubParserAction) -> argparse.ArgumentParser:
+    """
+    add parser for create user subcommand
+    :param root: subparsers for the commands
+    :return: created argument parser
+    """
+    parser = root.add_parser(
+        "user",
+        help="manage users for web services",
+        description="manage users for web services with password and role. In case if password was not entered it will be asked interactively",
+        formatter_class=argparse.ArgumentDefaultsHelpFormatter)
+    parser.add_argument("username", help="username for web service")
+    parser.add_argument("--as-service", help="add user as service user", action="store_true")
+    parser.add_argument(
+        "-a",
+        "--access",
+        help="user access level",
+        type=UserAccess,
+        choices=UserAccess,
+        default=UserAccess.Read)
+    parser.add_argument("--no-reload", help="do not reload authentication module", action="store_true")
+    parser.add_argument("-p", "--password", help="user password")
+    parser.add_argument("-r", "--remove", help="remove user from configuration", action="store_true")
+    parser.add_argument("--secure", help="set file permissions to user-only", action="store_true")
+    parser.set_defaults(handler=handlers.User, architecture=[""], lock=None, no_log=True, no_report=True, unsafe=True)
+    return parser
+
+
+def _set_web_parser(root: SubParserAction) -> argparse.ArgumentParser:
+    """
+    add parser for web subcommand
+    :param root: subparsers for the commands
+    :return: created argument parser
+    """
+    parser = root.add_parser("web", help="start web server", description="start web server",
+                             formatter_class=argparse.ArgumentDefaultsHelpFormatter)
+    parser.set_defaults(handler=handlers.Web, lock=None, no_report=True, parser=_parser)
+    return parser
+
+
+def run() -> None:
+    """
+    run application instance
+    """
+    if __name__ == "__main__":
+        args_parser = _parser()
+        args = args_parser.parse_args()
+
+        handler: handlers.Handler = args.handler
+        status = handler.execute(args)
+
+        sys.exit(status)
+
+
+run()

src/ahriman/application/application.py

@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2021 Evgenii Alekseev.
+# Copyright (c) 2021 ahriman team.
 #
 # This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).
@@ -18,10 +18,10 @@
 # along with this program. If not, see <http://www.gnu.org/licenses/>.
 #
 import logging
-import os
 import shutil
 
-from typing import Callable, Iterable, List, Optional, Set
+from pathlib import Path
+from typing import Callable, Iterable, List, Set
 
 from ahriman.core.build_tools.task import Task
 from ahriman.core.configuration import Configuration
@@ -29,50 +29,54 @@ from ahriman.core.repository.repository import Repository
 from ahriman.core.tree import Tree
 from ahriman.core.util import package_like
 from ahriman.models.package import Package
+from ahriman.models.package_source import PackageSource
 
 
 class Application:
-    '''
+    """
     base application class
     :ivar architecture: repository architecture
-    :ivar config: configuration instance
+    :ivar configuration: configuration instance
     :ivar logger: application logger
     :ivar repository: repository instance
-    '''
+    """
 
-    def __init__(self, architecture: str, config: Configuration) -> None:
-        '''
+    def __init__(self, architecture: str, configuration: Configuration, no_report: bool) -> None:
+        """
         default constructor
         :param architecture: repository architecture
-        :param config: configuration instance
-        '''
-        self.logger = logging.getLogger('root')
-        self.config = config
+        :param configuration: configuration instance
+        :param no_report: force disable reporting
+        """
+        self.logger = logging.getLogger("root")
+        self.configuration = configuration
         self.architecture = architecture
-        self.repository = Repository(architecture, config)
+        self.repository = Repository(architecture, configuration, no_report)
+
+    def _finalize(self, built_packages: Iterable[Package]) -> None:
+        """
+        generate report and sync to remote server
+        """
+        self.report([], built_packages)
+        self.sync([], built_packages)
 
     def _known_packages(self) -> Set[str]:
-        '''
+        """
         load packages from repository and pacman repositories
         :return: list of known packages
-        '''
+        """
         known_packages: Set[str] = set()
         # local set
-        for package in self.repository.packages():
-            known_packages.update(package.packages.keys())
+        for base in self.repository.packages():
+            for package, properties in base.packages.items():
+                known_packages.add(package)
+                known_packages.update(properties.provides)
         known_packages.update(self.repository.pacman.all_packages())
         return known_packages
 
-    def _finalize(self) -> None:
-        '''
-        generate report and sync to remote server
-        '''
-        self.report()
-        self.sync()
-
     def get_updates(self, filter_packages: List[str], no_aur: bool, no_manual: bool, no_vcs: bool,
                     log_fn: Callable[[str], None]) -> List[Package]:
-        '''
+        """
         get list of packages to run update process
         :param filter_packages: do not check every package just specified in the list
         :param no_aur: do not check for aur updates
@@ -80,7 +84,7 @@ class Application:
         :param no_vcs: do not check VCS packages
         :param log_fn: logger function to log updates
         :return: list of out-of-dated packages
-        '''
+        """
         updates = []
 
         if not no_aur:
@@ -89,60 +93,61 @@ class Application:
             updates.extend(self.repository.updates_manual())
 
         for package in updates:
-            log_fn(f'{package.base} = {package.version}')
+            log_fn(f"{package.base} = {package.version}")
 
         return updates
 
-    def add(self, names: Iterable[str], without_dependencies: bool) -> None:
-        '''
+    def add(self, names: Iterable[str], source: PackageSource, without_dependencies: bool) -> None:
+        """
         add packages for the next build
         :param names: list of package bases to add
+        :param source: package source to add
         :param without_dependencies: if set, dependency check will be disabled
-        '''
+        """
         known_packages = self._known_packages()
 
-        def add_directory(path: str) -> None:
-            for package in filter(package_like, os.listdir(path)):
-                full_path = os.path.join(path, package)
-                add_manual(full_path)
+        def add_directory(path: Path) -> None:
+            for full_path in filter(package_like, path.iterdir()):
+                add_archive(full_path)
 
-        def add_manual(name: str) -> str:
-            package = Package.load(name, self.repository.pacman, self.config.get('alpm', 'aur_url'))
-            path = os.path.join(self.repository.paths.manual, package.base)
+        def add_manual(src: str) -> Path:
+            package = Package.load(src, self.repository.pacman, self.configuration.get("alpm", "aur_url"))
+            path = self.repository.paths.manual / package.base
             Task.fetch(path, package.git_url)
             return path
 
-        def add_archive(src: str) -> None:
-            dst = os.path.join(self.repository.paths.packages, os.path.basename(src))
+        def add_archive(src: Path) -> None:
+            dst = self.repository.paths.packages / src.name
             shutil.move(src, dst)
 
-        def process_dependencies(path: str) -> None:
+        def process_dependencies(path: Path) -> None:
             if without_dependencies:
                 return
             dependencies = Package.dependencies(path)
-            self.add(dependencies.difference(known_packages), without_dependencies)
+            self.add(dependencies.difference(known_packages), PackageSource.AUR, without_dependencies)
 
-        def process_single(name: str) -> None:
-            if os.path.isdir(name):
-                add_directory(name)
-            elif os.path.isfile(name):
-                add_archive(name)
+        def process_single(src: str) -> None:
+            resolved_source = source.resolve(src)
+            if resolved_source == PackageSource.Directory:
+                add_directory(Path(src))
+            elif resolved_source == PackageSource.Archive:
+                add_archive(Path(src))
             else:
-                path = add_manual(name)
+                path = add_manual(src)
                 process_dependencies(path)
 
         for name in names:
             process_single(name)
 
     def clean(self, no_build: bool, no_cache: bool, no_chroot: bool, no_manual: bool, no_packages: bool) -> None:
-        '''
+        """
         run all clean methods. Warning: some functions might not be available under non-root
         :param no_build: do not clear directory with package sources
         :param no_cache: do not clear directory with package caches
         :param no_chroot: do not clear build chroot
         :param no_manual: do not clear directory with manually added packages
         :param no_packages: do not clear directory with built packages
-        '''
+        """
         if not no_build:
             self.repository.clear_build()
         if not no_cache:
@@ -155,46 +160,86 @@ class Application:
             self.repository.clear_packages()
 
     def remove(self, names: Iterable[str]) -> None:
-        '''
+        """
         remove packages from repository
         :param names: list of packages (either base or name) to remove
-        '''
+        """
         self.repository.process_remove(names)
-        self._finalize()
+        self._finalize([])
 
-    def report(self, target: Optional[Iterable[str]] = None) -> None:
-        '''
+    def report(self, target: Iterable[str], built_packages: Iterable[Package]) -> None:
+        """
         generate report
         :param target: list of targets to run (e.g. html)
-        '''
+        :param built_packages: list of packages which has just been built
+        """
         targets = target or None
-        self.repository.process_report(targets)
+        self.repository.process_report(targets, built_packages)
 
-    def sync(self, target: Optional[Iterable[str]] = None) -> None:
-        '''
+    def sign(self, packages: Iterable[str]) -> None:
+        """
+        sign packages and repository
+        :param packages: only sign specified packages
+        """
+        # copy to prebuilt directory
+        for package in self.repository.packages():
+            # no one requested this package
+            if packages and package.base not in packages:
+                continue
+            for archive in package.packages.values():
+                if archive.filepath is None:
+                    self.logger.warning("filepath is empty for %s", package.base)
+                    continue  # avoid mypy warning
+                src = self.repository.paths.repository / archive.filepath
+                dst = self.repository.paths.packages / archive.filepath
+                shutil.copy(src, dst)
+        # run generic update function
+        self.update([])
+        # sign repository database if set
+        self.repository.sign.sign_repository(self.repository.repo.repo_path)
+        self._finalize([])
+
+    def sync(self, target: Iterable[str], built_packages: Iterable[Package]) -> None:
+        """
         sync to remote server
         :param target: list of targets to run (e.g. s3)
-        '''
+        :param built_packages: list of packages which has just been built
+        """
         targets = target or None
-        self.repository.process_sync(targets)
+        self.repository.process_sync(targets, built_packages)
+
+    def unknown(self) -> List[Package]:
+        """
+        get packages which were not found in AUR
+        :return: unknown package list
+        """
+        packages = []
+        for base in self.repository.packages():
+            try:
+                _ = Package.from_aur(base.base, base.aur_url)
+            except Exception:
+                packages.append(base)
+        return packages
 
     def update(self, updates: Iterable[Package]) -> None:
-        '''
+        """
         run package updates
         :param updates: list of packages to update
-        '''
-        def process_update(paths: Iterable[str]) -> None:
+        """
+        def process_update(paths: Iterable[Path]) -> None:
+            if not paths:
+                return  # don't need to process if no update supplied
+            updated = [Package.load(path, self.repository.pacman, self.repository.aur_url) for path in paths]
             self.repository.process_update(paths)
-            self._finalize()
+            self._finalize(updated)
 
         # process built packages
         packages = self.repository.packages_built()
         process_update(packages)
 
         # process manual packages
-        tree = Tree()
-        tree.load(updates)
+        tree = Tree.load(updates)
         for num, level in enumerate(tree.levels()):
-            self.logger.info(f'processing level #{num} {[package.base for package in level]}')
+            self.logger.info("processing level #%i %s", num, [package.base for package in level])
             packages = self.repository.process_build(level)
             process_update(packages)

src/ahriman/application/handlers/__init__.py

@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2021 Evgenii Alekseev.
+# Copyright (c) 2021 ahriman team.
 #
 # This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).
@@ -22,10 +22,18 @@ from ahriman.application.handlers.handler import Handler
 from ahriman.application.handlers.add import Add
 from ahriman.application.handlers.clean import Clean
 from ahriman.application.handlers.dump import Dump
+from ahriman.application.handlers.init import Init
+from ahriman.application.handlers.key_import import KeyImport
 from ahriman.application.handlers.rebuild import Rebuild
 from ahriman.application.handlers.remove import Remove
+from ahriman.application.handlers.remove_unknown import RemoveUnknown
 from ahriman.application.handlers.report import Report
+from ahriman.application.handlers.search import Search
+from ahriman.application.handlers.setup import Setup
+from ahriman.application.handlers.sign import Sign
 from ahriman.application.handlers.status import Status
+from ahriman.application.handlers.status_update import StatusUpdate
 from ahriman.application.handlers.sync import Sync
 from ahriman.application.handlers.update import Update
+from ahriman.application.handlers.user import User
 from ahriman.application.handlers.web import Web

src/ahriman/application/handlers/add.py

@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2021 Evgenii Alekseev.
+# Copyright (c) 2021 ahriman team.
 #
 # This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).
@@ -27,16 +27,24 @@ from ahriman.core.configuration import Configuration
 
 
 class Add(Handler):
-    '''
+    """
     add packages handler
-    '''
+    """
 
     @classmethod
-    def run(cls: Type[Handler], args: argparse.Namespace, architecture: str, config: Configuration) -> None:
-        '''
+    def run(cls: Type[Handler], args: argparse.Namespace, architecture: str,
+            configuration: Configuration, no_report: bool) -> None:
+        """
         callback for command line
         :param args: command line args
         :param architecture: repository architecture
-        :param config: configuration instance
-        '''
-        Application(architecture, config).add(args.package, args.without_dependencies)
+        :param configuration: configuration instance
+        :param no_report: force disable reporting
+        """
+        application = Application(architecture, configuration, no_report)
+        application.add(args.package, args.source, args.without_dependencies)
+        if not args.now:
+            return
+
+        packages = application.get_updates(args.package, True, False, True, application.logger.info)
+        application.update(packages)

src/ahriman/application/handlers/clean.py

@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2021 Evgenii Alekseev.
+# Copyright (c) 2021 ahriman team.
 #
 # This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).
@@ -27,17 +27,19 @@ from ahriman.core.configuration import Configuration
 
 
 class Clean(Handler):
-    '''
+    """
     clean caches handler
-    '''
+    """
 
     @classmethod
-    def run(cls: Type[Handler], args: argparse.Namespace, architecture: str, config: Configuration) -> None:
-        '''
+    def run(cls: Type[Handler], args: argparse.Namespace, architecture: str,
+            configuration: Configuration, no_report: bool) -> None:
+        """
         callback for command line
         :param args: command line args
         :param architecture: repository architecture
-        :param config: configuration instance
-        '''
-        Application(architecture, config).clean(args.no_build, args.no_cache, args.no_chroot,
-                                                args.no_manual, args.no_packages)
+        :param configuration: configuration instance
+        :param no_report: force disable reporting
+        """
+        Application(architecture, configuration, no_report).clean(args.no_build, args.no_cache, args.no_chroot,
+                                                                  args.no_manual, args.no_packages)

src/ahriman/application/handlers/dump.py

@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2021 Evgenii Alekseev.
+# Copyright (c) 2021 ahriman team.
 #
 # This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).
@@ -26,21 +26,27 @@ from ahriman.core.configuration import Configuration
 
 
 class Dump(Handler):
-    '''
-    dump config handler
-    '''
+    """
+    dump configuration handler
+    """
+
+    ALLOW_AUTO_ARCHITECTURE_RUN = False
+
+    _print = print
 
     @classmethod
-    def run(cls: Type[Handler], args: argparse.Namespace, architecture: str, config: Configuration) -> None:
-        '''
+    def run(cls: Type[Handler], args: argparse.Namespace, architecture: str,
+            configuration: Configuration, no_report: bool) -> None:
+        """
         callback for command line
         :param args: command line args
         :param architecture: repository architecture
-        :param config: configuration instance
-        '''
-        config_dump = config.dump(architecture)
-        for section, values in sorted(config_dump.items()):
-            print(f'[{section}]')
+        :param configuration: configuration instance
+        :param no_report: force disable reporting
+        """
+        dump = configuration.dump()
+        for section, values in sorted(dump.items()):
+            Dump._print(f"[{section}]")
             for key, value in sorted(values.items()):
-                print(f'{key} = {value}')
-            print()
+                Dump._print(f"{key} = {value}")
+            Dump._print()

src/ahriman/application/handlers/handler.py

@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2021 Evgenii Alekseev.
+# Copyright (c) 2021 ahriman team.
 #
 # This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).
@@ -21,55 +21,97 @@ from __future__ import annotations
 
 import argparse
 import logging
-from multiprocessing import Pool
 
-from typing import Type
+from multiprocessing import Pool
+from typing import Set, Type
 
 from ahriman.application.lock import Lock
 from ahriman.core.configuration import Configuration
+from ahriman.core.exceptions import MissingArchitecture, MultipleArchitecture
+from ahriman.models.repository_paths import RepositoryPaths
 
 
 class Handler:
-    '''
+    """
     base handler class for command callbacks
-    '''
+    :cvar ALLOW_AUTO_ARCHITECTURE_RUN: allow to define architecture from existing repositories
+    :cvar ALLOW_MULTI_ARCHITECTURE_RUN: allow to run with multiple architectures
+    """
+
+    ALLOW_AUTO_ARCHITECTURE_RUN = True
+    ALLOW_MULTI_ARCHITECTURE_RUN = True
 
     @classmethod
-    def _call(cls: Type[Handler], args: argparse.Namespace, architecture: str, config: Configuration) -> bool:
-        '''
+    def call(cls: Type[Handler], args: argparse.Namespace, architecture: str) -> bool:
+        """
         additional function to wrap all calls for multiprocessing library
         :param args: command line args
         :param architecture: repository architecture
-        :param config: configuration instance
         :return: True on success, False otherwise
-        '''
+        """
         try:
-            with Lock(args, architecture, config):
-                cls.run(args, architecture, config)
+            configuration = Configuration.from_path(args.configuration, architecture, not args.no_log)
+            with Lock(args, architecture, configuration):
+                cls.run(args, architecture, configuration, args.no_report)
             return True
         except Exception:
-            logging.getLogger('root').exception('process exception', exc_info=True)
+            logging.getLogger("root").exception("process exception")
             return False
 
     @classmethod
     def execute(cls: Type[Handler], args: argparse.Namespace) -> int:
-        '''
+        """
         execute function for all aru
         :param args: command line args
         :return: 0 on success, 1 otherwise
-        '''
-        configuration = Configuration.from_path(args.config, not args.no_log)
-        with Pool(len(args.architecture)) as pool:
-            result = pool.starmap(
-                cls._call, [(args, architecture, configuration) for architecture in args.architecture])
+        """
+        architectures = cls.extract_architectures(args)
+
+        # actually we do not have to spawn another process if it is single-process application, do we?
+        if len(architectures) > 1:
+            if not cls.ALLOW_MULTI_ARCHITECTURE_RUN:
+                raise MultipleArchitecture(args.command)
+
+            with Pool(len(architectures)) as pool:
+                result = pool.starmap(
+                    cls.call, [(args, architecture) for architecture in architectures])
+        else:
+            result = [cls.call(args, architectures.pop())]
+
         return 0 if all(result) else 1
 
     @classmethod
-    def run(cls: Type[Handler], args: argparse.Namespace, architecture: str, config: Configuration) -> None:
-        '''
+    def extract_architectures(cls: Type[Handler], args: argparse.Namespace) -> Set[str]:
+        """
+        get known architectures
+        :param args: command line args
+        :return: list of architectures for which tree is created
+        """
+        if not cls.ALLOW_AUTO_ARCHITECTURE_RUN and args.architecture is None:
+            # for some parsers (e.g. config) we need to run with specific architecture
+            # for those cases architecture must be set explicitly
+            raise MissingArchitecture(args.command)
+        if args.architecture:  # architecture is specified explicitly
+            return set(args.architecture)
+
+        config = Configuration()
+        config.load(args.configuration)
+        # wtf???
+        root = config.getpath("repository", "root")  # pylint: disable=assignment-from-no-return
+        architectures = RepositoryPaths.known_architectures(root)
+
+        if not architectures:  # well we did not find anything
+            raise MissingArchitecture(args.command)
+        return architectures
+
+    @classmethod
+    def run(cls: Type[Handler], args: argparse.Namespace, architecture: str,
+            configuration: Configuration, no_report: bool) -> None:
+        """
         callback for command line
         :param args: command line args
         :param architecture: repository architecture
-        :param config: configuration instance
-        '''
+        :param configuration: configuration instance
+        :param no_report: force disable reporting
+        """
         raise NotImplementedError

src/ahriman/application/handlers/init.py

@@ -0,0 +1,46 @@
+#
+# Copyright (c) 2021 ahriman team.
+#
+# This file is part of ahriman
+# (see https://github.com/arcan1s/ahriman).
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+import argparse
+
+from typing import Type
+
+from ahriman.application.application import Application
+from ahriman.application.handlers.handler import Handler
+from ahriman.core.configuration import Configuration
+
+
+class Init(Handler):
+    """
+    repository init handler
+    """
+
+    ALLOW_AUTO_ARCHITECTURE_RUN = False
+
+    @classmethod
+    def run(cls: Type[Handler], args: argparse.Namespace, architecture: str,
+            configuration: Configuration, no_report: bool) -> None:
+        """
+        callback for command line
+        :param args: command line args
+        :param architecture: repository architecture
+        :param configuration: configuration instance
+        :param no_report: force disable reporting
+        """
+        Application(architecture, configuration, no_report).repository.repo.init()

src/ahriman/application/handlers/key_import.py

@@ -0,0 +1,46 @@
+#
+# Copyright (c) 2021 ahriman team.
+#
+# This file is part of ahriman
+# (see https://github.com/arcan1s/ahriman).
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+import argparse
+
+from typing import Type
+
+from ahriman.application.application import Application
+from ahriman.application.handlers.handler import Handler
+from ahriman.core.configuration import Configuration
+
+
+class KeyImport(Handler):
+    """
+    key import packages handler
+    """
+
+    ALLOW_AUTO_ARCHITECTURE_RUN = False  # it should be called only as "no-architecture"
+
+    @classmethod
+    def run(cls: Type[Handler], args: argparse.Namespace, architecture: str,
+            configuration: Configuration, no_report: bool) -> None:
+        """
+        callback for command line
+        :param args: command line args
+        :param architecture: repository architecture
+        :param configuration: configuration instance
+        :param no_report: force disable reporting
+        """
+        Application(architecture, configuration, no_report).repository.sign.import_key(args.key_server, args.key)

src/ahriman/application/handlers/rebuild.py

@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2021 Evgenii Alekseev.
+# Copyright (c) 2021 ahriman team.
 #
 # This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).
@@ -27,18 +27,26 @@ from ahriman.core.configuration import Configuration
 
 
 class Rebuild(Handler):
-    '''
+    """
     make world handler
-    '''
+    """
 
     @classmethod
-    def run(cls: Type[Handler], args: argparse.Namespace, architecture: str, config: Configuration) -> None:
-        '''
+    def run(cls: Type[Handler], args: argparse.Namespace, architecture: str,
+            configuration: Configuration, no_report: bool) -> None:
+        """
         callback for command line
         :param args: command line args
         :param architecture: repository architecture
-        :param config: configuration instance
-        '''
-        application = Application(architecture, config)
-        packages = application.repository.packages()
+        :param configuration: configuration instance
+        :param no_report: force disable reporting
+        """
+        depends_on = set(args.depends_on) if args.depends_on else None
+
+        application = Application(architecture, configuration, no_report)
+        packages = [
+            package
+            for package in application.repository.packages()
+            if depends_on is None or depends_on.intersection(package.depends)
+        ]  # we have to use explicit list here for testing purpose
         application.update(packages)

src/ahriman/application/handlers/remove.py

@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2021 Evgenii Alekseev.
+# Copyright (c) 2021 ahriman team.
 #
 # This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).
@@ -27,16 +27,18 @@ from ahriman.core.configuration import Configuration
 
 
 class Remove(Handler):
-    '''
+    """
     remove packages handler
-    '''
+    """
 
     @classmethod
-    def run(cls: Type[Handler], args: argparse.Namespace, architecture: str, config: Configuration) -> None:
-        '''
+    def run(cls: Type[Handler], args: argparse.Namespace, architecture: str,
+            configuration: Configuration, no_report: bool) -> None:
+        """
         callback for command line
         :param args: command line args
         :param architecture: repository architecture
-        :param config: configuration instance
-        '''
-        Application(architecture, config).remove(args.package)
+        :param configuration: configuration instance
+        :param no_report: force disable reporting
+        """
+        Application(architecture, configuration, no_report).remove(args.package)

src/ahriman/application/handlers/remove_unknown.py

@@ -0,0 +1,61 @@
+#
+# Copyright (c) 2021 ahriman team.
+#
+# This file is part of ahriman
+# (see https://github.com/arcan1s/ahriman).
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+import argparse
+
+from typing import Type
+
+from ahriman.application.application import Application
+from ahriman.application.handlers.handler import Handler
+from ahriman.core.configuration import Configuration
+from ahriman.models.package import Package
+
+
+class RemoveUnknown(Handler):
+    """
+    remove unknown packages handler
+    """
+
+    @classmethod
+    def run(cls: Type[Handler], args: argparse.Namespace, architecture: str,
+            configuration: Configuration, no_report: bool) -> None:
+        """
+        callback for command line
+        :param args: command line args
+        :param architecture: repository architecture
+        :param configuration: configuration instance
+        :param no_report: force disable reporting
+        """
+        application = Application(architecture, configuration, no_report)
+        unknown_packages = application.unknown()
+        if args.dry_run:
+            for package in unknown_packages:
+                RemoveUnknown.log_fn(package)
+            return
+
+        application.remove(package.base for package in unknown_packages)
+
+    @staticmethod
+    def log_fn(package: Package) -> None:
+        """
+        log package information
+        :param package: package object to log
+        """
+        print(f"=> {package.base} {package.version}")
+        print(f"   {package.web_url}")

src/ahriman/application/handlers/report.py

@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2021 Evgenii Alekseev.
+# Copyright (c) 2021 ahriman team.
 #
 # This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).
@@ -27,16 +27,18 @@ from ahriman.core.configuration import Configuration
 
 
 class Report(Handler):
-    '''
+    """
     generate report handler
-    '''
+    """
 
     @classmethod
-    def run(cls: Type[Handler], args: argparse.Namespace, architecture: str, config: Configuration) -> None:
-        '''
+    def run(cls: Type[Handler], args: argparse.Namespace, architecture: str,
+            configuration: Configuration, no_report: bool) -> None:
+        """
         callback for command line
         :param args: command line args
         :param architecture: repository architecture
-        :param config: configuration instance
-        '''
-        Application(architecture, config).report(args.target)
+        :param configuration: configuration instance
+        :param no_report: force disable reporting
+        """
+        Application(architecture, configuration, no_report).report(args.target, [])

src/ahriman/application/handlers/search.py

@@ -0,0 +1,62 @@
+#
+# Copyright (c) 2021 ahriman team.
+#
+# This file is part of ahriman
+# (see https://github.com/arcan1s/ahriman).
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+import argparse
+import aur  # type: ignore
+
+from typing import Callable, Type
+
+from ahriman.application.handlers.handler import Handler
+from ahriman.core.configuration import Configuration
+
+
+class Search(Handler):
+    """
+    packages search handler
+    """
+
+    ALLOW_AUTO_ARCHITECTURE_RUN = False  # it should be called only as "no-architecture"
+
+    @classmethod
+    def run(cls: Type[Handler], args: argparse.Namespace, architecture: str,
+            configuration: Configuration, no_report: bool) -> None:
+        """
+        callback for command line
+        :param args: command line args
+        :param architecture: repository architecture
+        :param configuration: configuration instance
+        :param no_report: force disable reporting
+        """
+        search = " ".join(args.search)
+        packages = aur.search(search)
+
+        # it actually always should return string
+        # explicit cast to string just to avoid mypy warning for untyped library
+        comparator: Callable[[aur.Package], str] = lambda item: str(item.package_base)
+        for package in sorted(packages, key=comparator):
+            Search.log_fn(package)
+
+    @staticmethod
+    def log_fn(package: aur.Package) -> None:
+        """
+        log package information
+        :param package: package object as from AUR
+        """
+        print(f"=> {package.package_base} {package.version}")
+        print(f"   {package.description}")

src/ahriman/application/handlers/setup.py

@@ -0,0 +1,169 @@
+#
+# Copyright (c) 2021 ahriman team.
+#
+# This file is part of ahriman
+# (see https://github.com/arcan1s/ahriman).
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+import argparse
+
+from pathlib import Path
+from typing import Type
+
+from ahriman.application.application import Application
+from ahriman.application.handlers.handler import Handler
+from ahriman.core.configuration import Configuration
+from ahriman.models.repository_paths import RepositoryPaths
+
+
+class Setup(Handler):
+    """
+    setup handler
+    :cvar ARCHBUILD_COMMAND_PATH: default devtools command
+    :cvar BIN_DIR_PATH: directory for custom binaries
+    :cvar MIRRORLIST_PATH: path to pacman default mirrorlist (used by multilib repository)
+    :cvar SUDOERS_PATH: path to sudoers.d include configuration
+    """
+
+    ALLOW_AUTO_ARCHITECTURE_RUN = False
+
+    ARCHBUILD_COMMAND_PATH = Path("/usr/bin/archbuild")
+    BIN_DIR_PATH = Path("/usr/local/bin")
+    MIRRORLIST_PATH = Path("/etc/pacman.d/mirrorlist")
+    SUDOERS_PATH = Path("/etc/sudoers.d/ahriman")
+
+    @classmethod
+    def run(cls: Type[Handler], args: argparse.Namespace, architecture: str,
+            configuration: Configuration, no_report: bool) -> None:
+        """
+        callback for command line
+        :param args: command line args
+        :param architecture: repository architecture
+        :param configuration: configuration instance
+        :param no_report: force disable reporting
+        """
+        application = Application(architecture, configuration, no_report)
+        Setup.create_makepkg_configuration(args.packager, application.repository.paths)
+        Setup.create_executable(args.build_command, architecture)
+        Setup.create_devtools_configuration(args.build_command, architecture, args.from_configuration,
+                                            args.no_multilib, args.repository, application.repository.paths)
+        Setup.create_ahriman_configuration(args, architecture, args.repository, configuration.include)
+        Setup.create_sudo_configuration(args.build_command, architecture)
+
+    @staticmethod
+    def build_command(prefix: str, architecture: str) -> Path:
+        """
+        generate build command name
+        :param prefix: command prefix in {prefix}-{architecture}-build
+        :param architecture: repository architecture
+        :return: valid devtools command name
+        """
+        return Setup.BIN_DIR_PATH / f"{prefix}-{architecture}-build"
+
+    @staticmethod
+    def create_ahriman_configuration(args: argparse.Namespace, architecture: str, repository: str,
+                                     include_path: Path) -> None:
+        """
+        create service specific configuration
+        :param args: command line args
+        :param architecture: repository architecture
+        :param repository: repository name
+        :param include_path: path to directory with configuration includes
+        """
+        configuration = Configuration()
+
+        section = Configuration.section_name("build", architecture)
+        configuration.set_option(section, "build_command", str(Setup.build_command(args.build_command, architecture)))
+        configuration.set_option("repository", "name", repository)
+
+        if args.sign_key is not None:
+            section = Configuration.section_name("sign", architecture)
+            configuration.set_option(section, "target", " ".join([target.name.lower() for target in args.sign_target]))
+            configuration.set_option(section, "key", args.sign_key)
+
+        if args.web_port is not None:
+            section = Configuration.section_name("web", architecture)
+            configuration.set_option(section, "port", str(args.web_port))
+
+        target = include_path / "setup-overrides.ini"
+        with target.open("w") as ahriman_configuration:
+            configuration.write(ahriman_configuration)
+
+    @staticmethod
+    def create_devtools_configuration(prefix: str, architecture: str, source: Path,
+                                      no_multilib: bool, repository: str, paths: RepositoryPaths) -> None:
+        """
+        create configuration for devtools based on `source` configuration
+        :param prefix: command prefix in {prefix}-{architecture}-build
+        :param architecture: repository architecture
+        :param source: path to source configuration file
+        :param no_multilib: do not add multilib repository
+        :param repository: repository name
+        :param paths: repository paths instance
+        """
+        configuration = Configuration()
+        # preserve case
+        # stupid mypy thinks that it is impossible
+        configuration.optionxform = lambda key: key  # type: ignore
+
+        # load default configuration first
+        # we cannot use Include here because it will be copied to new chroot, thus no includes there
+        configuration.read(source)
+
+        # set our architecture now
+        configuration.set_option("options", "Architecture", architecture)
+
+        # add multilib
+        if not no_multilib:
+            configuration.set_option("multilib", "Include", str(Setup.MIRRORLIST_PATH))
+
+        # add repository itself
+        configuration.set_option(repository, "SigLevel", "Optional TrustAll")  # we don't care
+        configuration.set_option(repository, "Server", f"file://{paths.repository}")
+
+        target = source.parent / f"pacman-{prefix}-{architecture}.conf"
+        with target.open("w") as devtools_configuration:
+            configuration.write(devtools_configuration)
+
+    @staticmethod
+    def create_makepkg_configuration(packager: str, paths: RepositoryPaths) -> None:
+        """
+        create configuration for makepkg
+        :param packager: packager identifier (e.g. name, email)
+        :param paths: repository paths instance
+        """
+        (paths.root / ".makepkg.conf").write_text(f"PACKAGER='{packager}'\n")
+
+    @staticmethod
+    def create_sudo_configuration(prefix: str, architecture: str) -> None:
+        """
+        create configuration to run build command with sudo without password
+        :param prefix: command prefix in {prefix}-{architecture}-build
+        :param architecture: repository architecture
+        """
+        command = Setup.build_command(prefix, architecture)
+        Setup.SUDOERS_PATH.write_text(f"ahriman ALL=(ALL) NOPASSWD: {command} *\n", encoding="utf8")
+        Setup.SUDOERS_PATH.chmod(0o400)  # security!
+
+    @staticmethod
+    def create_executable(prefix: str, architecture: str) -> None:
+        """
+        create executable for the service
+        :param prefix: command prefix in {prefix}-{architecture}-build
+        :param architecture: repository architecture
+        """
+        command = Setup.build_command(prefix, architecture)
+        command.unlink(missing_ok=True)
+        command.symlink_to(Setup.ARCHBUILD_COMMAND_PATH)

src/ahriman/application/handlers/sign.py

@@ -0,0 +1,44 @@
+#
+# Copyright (c) 2021 ahriman team.
+#
+# This file is part of ahriman
+# (see https://github.com/arcan1s/ahriman).
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+import argparse
+
+from typing import Type
+
+from ahriman.application.application import Application
+from ahriman.application.handlers.handler import Handler
+from ahriman.core.configuration import Configuration
+
+
+class Sign(Handler):
+    """
+    (re-)sign handler
+    """
+
+    @classmethod
+    def run(cls: Type[Handler], args: argparse.Namespace, architecture: str,
+            configuration: Configuration, no_report: bool) -> None:
+        """
+        callback for command line
+        :param args: command line args
+        :param architecture: repository architecture
+        :param configuration: configuration instance
+        :param no_report: force disable reporting
+        """
+        Application(architecture, configuration, no_report).sign(args.package)

src/ahriman/application/handlers/status.py

@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2021 Evgenii Alekseev.
+# Copyright (c) 2021 ahriman team.
 #
 # This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).
@@ -19,7 +19,7 @@
 #
 import argparse
 
-from typing import Iterable, Tuple, Type
+from typing import Callable, Iterable, Tuple, Type
 
 from ahriman.application.application import Application
 from ahriman.application.handlers.handler import Handler
@@ -29,30 +29,39 @@ from ahriman.models.package import Package
 
 
 class Status(Handler):
-    '''
+    """
     package status handler
-    '''
+    """
+
+    ALLOW_AUTO_ARCHITECTURE_RUN = False
 
     @classmethod
-    def run(cls: Type[Handler], args: argparse.Namespace, architecture: str, config: Configuration) -> None:
-        '''
+    def run(cls: Type[Handler], args: argparse.Namespace, architecture: str,
+            configuration: Configuration, no_report: bool) -> None:
+        """
         callback for command line
         :param args: command line args
         :param architecture: repository architecture
-        :param config: configuration instance
-        '''
-        application = Application(architecture, config)
+        :param configuration: configuration instance
+        :param no_report: force disable reporting
+        """
+        # we are using reporter here
+        client = Application(architecture, configuration, no_report=False).repository.reporter
         if args.ahriman:
-            ahriman = application.repository.reporter.get_self()
+            ahriman = client.get_self()
             print(ahriman.pretty_print())
             print()
         if args.package:
             packages: Iterable[Tuple[Package, BuildStatus]] = sum(
-                [application.repository.reporter.get(base) for base in args.package],
+                [client.get(base) for base in args.package],
                 start=[])
         else:
-            packages = application.repository.reporter.get(None)
-        for package, package_status in sorted(packages, key=lambda item: item[0].base):
+            packages = client.get(None)
+
+        comparator: Callable[[Tuple[Package, BuildStatus]], str] = lambda item: item[0].base
+        filter_fn: Callable[[Tuple[Package, BuildStatus]], bool] =\
+            lambda item: args.status is None or item[1].status == args.status
+        for package, package_status in sorted(filter(filter_fn, packages), key=comparator):
             print(package.pretty_print())
-            print(f'\t{package.version}')
-            print(f'\t{package_status.pretty_print()}')
+            print(f"\t{package.version}")
+            print(f"\t{package_status.pretty_print()}")

src/ahriman/application/handlers/status_update.py

@@ -0,0 +1,58 @@
+#
+# Copyright (c) 2021 ahriman team.
+#
+# This file is part of ahriman
+# (see https://github.com/arcan1s/ahriman).
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+import argparse
+
+from typing import Callable, Type
+
+from ahriman.application.application import Application
+from ahriman.application.handlers.handler import Handler
+from ahriman.core.configuration import Configuration
+from ahriman.core.exceptions import InvalidCommand
+
+
+class StatusUpdate(Handler):
+    """
+    status update handler
+    """
+
+    ALLOW_AUTO_ARCHITECTURE_RUN = False
+
+    @classmethod
+    def run(cls: Type[Handler], args: argparse.Namespace, architecture: str,
+            configuration: Configuration, no_report: bool) -> None:
+        """
+        callback for command line
+        :param args: command line args
+        :param architecture: repository architecture
+        :param configuration: configuration instance
+        :param no_report: force disable reporting
+        """
+        # we are using reporter here
+        client = Application(architecture, configuration, no_report=False).repository.reporter
+        callback: Callable[[str], None] = lambda p: client.remove(p) if args.remove else client.update(p, args.status)
+        if args.package:
+            # update packages statuses
+            for package in args.package:
+                callback(package)
+        elif args.remove:
+            raise InvalidCommand("Remove option is supplied, but no packages set")
+        else:
+            # update service status
+            client.update_self(args.status)

src/ahriman/application/handlers/sync.py

@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2021 Evgenii Alekseev.
+# Copyright (c) 2021 ahriman team.
 #
 # This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).
@@ -27,16 +27,18 @@ from ahriman.core.configuration import Configuration
 
 
 class Sync(Handler):
-    '''
+    """
     remove sync handler
-    '''
+    """
 
     @classmethod
-    def run(cls: Type[Handler], args: argparse.Namespace, architecture: str, config: Configuration) -> None:
-        '''
+    def run(cls: Type[Handler], args: argparse.Namespace, architecture: str,
+            configuration: Configuration, no_report: bool) -> None:
+        """
         callback for command line
         :param args: command line args
         :param architecture: repository architecture
-        :param config: configuration instance
-        '''
-        Application(architecture, config).sync(args.target)
+        :param configuration: configuration instance
+        :param no_report: force disable reporting
+        """
+        Application(architecture, configuration, no_report).sync(args.target, [])

src/ahriman/application/handlers/update.py

@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2021 Evgenii Alekseev.
+# Copyright (c) 2021 ahriman team.
 #
 # This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).
@@ -19,7 +19,7 @@
 #
 import argparse
 
-from typing import Type
+from typing import Callable, Type
 
 from ahriman.application.application import Application
 from ahriman.application.handlers.handler import Handler
@@ -27,25 +27,36 @@ from ahriman.core.configuration import Configuration
 
 
 class Update(Handler):
-    '''
+    """
     package update handler
-    '''
+    """
 
     @classmethod
-    def run(cls: Type[Handler], args: argparse.Namespace, architecture: str, config: Configuration) -> None:
-        '''
+    def run(cls: Type[Handler], args: argparse.Namespace, architecture: str,
+            configuration: Configuration, no_report: bool) -> None:
+        """
         callback for command line
         :param args: command line args
         :param architecture: repository architecture
-        :param config: configuration instance
-        '''
-        # typing workaround
-        def log_fn(line: str) -> None:
-            return print(line) if args.dry_run else application.logger.info(line)
-
-        application = Application(architecture, config)
-        packages = application.get_updates(args.package, args.no_aur, args.no_manual, args.no_vcs, log_fn)
+        :param configuration: configuration instance
+        :param no_report: force disable reporting
+        """
+        application = Application(architecture, configuration, no_report)
+        packages = application.get_updates(args.package, args.no_aur, args.no_manual, args.no_vcs,
+                                           Update.log_fn(application, args.dry_run))
         if args.dry_run:
             return
 
         application.update(packages)
+
+    @staticmethod
+    def log_fn(application: Application, dry_run: bool) -> Callable[[str], None]:
+        """
+        package updates log function
+        :param application: application instance
+        :param dry_run: do not perform update itself
+        :return: in case if dry_run is set it will return print, logger otherwise
+        """
+        def inner(line: str) -> None:
+            return print(line) if dry_run else application.logger.info(line)
+        return inner

src/ahriman/application/handlers/user.py

@@ -0,0 +1,143 @@
+#
+# Copyright (c) 2021 ahriman team.
+#
+# This file is part of ahriman
+# (see https://github.com/arcan1s/ahriman).
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+import argparse
+import getpass
+
+from pathlib import Path
+from typing import Type
+
+from ahriman.application.application import Application
+from ahriman.application.handlers.handler import Handler
+from ahriman.core.configuration import Configuration
+from ahriman.models.user import User as MUser
+from ahriman.models.user_access import UserAccess
+
+
+class User(Handler):
+    """
+    user management handler
+    """
+
+    ALLOW_AUTO_ARCHITECTURE_RUN = False  # it should be called only as "no-architecture"
+
+    @classmethod
+    def run(cls: Type[Handler], args: argparse.Namespace, architecture: str,
+            configuration: Configuration, no_report: bool) -> None:
+        """
+        callback for command line
+        :param args: command line args
+        :param architecture: repository architecture
+        :param configuration: configuration instance
+        :param no_report: force disable reporting
+        """
+        salt = User.get_salt(configuration)
+        user = User.create_user(args)
+        auth_configuration = User.get_auth_configuration(configuration.include)
+
+        User.clear_user(auth_configuration, user)
+        if not args.remove:
+            User.create_configuration(auth_configuration, user, salt, args.as_service)
+        User.write_configuration(auth_configuration, args.secure)
+
+        if not args.no_reload:
+            client = Application(architecture, configuration, no_report=False).repository.reporter
+            client.reload_auth()
+
+    @staticmethod
+    def clear_user(configuration: Configuration, user: MUser) -> None:
+        """
+        remove user user from configuration file in case if it exists
+        :param configuration: configuration instance
+        :param user: user descriptor
+        """
+        for role in UserAccess:
+            section = Configuration.section_name("auth", role.value)
+            if not configuration.has_option(section, user.username):
+                continue
+            configuration.remove_option(section, user.username)
+
+    @staticmethod
+    def create_configuration(configuration: Configuration, user: MUser, salt: str, as_service_user: bool) -> None:
+        """
+        put new user to configuration
+        :param configuration: configuration instance
+        :param user: user descriptor
+        :param salt: password hash salt
+        :param as_service_user: add user as service user, also set password and user to configuration
+        """
+        section = Configuration.section_name("auth", user.access.value)
+        configuration.set_option("auth", "salt", salt)
+        configuration.set_option(section, user.username, user.hash_password(salt))
+
+        if as_service_user:
+            configuration.set_option("web", "username", user.username)
+            configuration.set_option("web", "password", user.password)
+
+    @staticmethod
+    def create_user(args: argparse.Namespace) -> MUser:
+        """
+        create user descriptor from arguments
+        :param args: command line args
+        :return: built user descriptor
+        """
+        user = MUser(args.username, args.password, args.access)
+        if user.password is None:
+            user.password = getpass.getpass()
+        return user
+
+    @staticmethod
+    def get_auth_configuration(include_path: Path) -> Configuration:
+        """
+        create configuration instance
+        :param include_path: path to directory with configuration includes
+        :return: configuration instance. In case if there are local settings they will be loaded
+        """
+        target = include_path / "auth.ini"
+        configuration = Configuration()
+        configuration.load(target)
+
+        return configuration
+
+    @staticmethod
+    def get_salt(configuration: Configuration, salt_length: int = 20) -> str:
+        """
+        get salt from configuration or create new string
+        :param configuration: configuration instance
+        :param salt_length: salt length
+        :return: current salt
+        """
+        salt = configuration.get("auth", "salt", fallback=None)
+        if salt:
+            return salt
+        return MUser.generate_password(salt_length)
+
+    @staticmethod
+    def write_configuration(configuration: Configuration, secure: bool) -> None:
+        """
+        write configuration file
+        :param configuration: configuration instance
+        :param secure: if true then set file permissions to 0o600
+        """
+        if configuration.path is None:
+            return  # should never happen actually
+        with configuration.path.open("w") as ahriman_configuration:
+            configuration.write(ahriman_configuration)
+        if secure:
+            configuration.path.chmod(0o600)

src/ahriman/application/handlers/web.py

@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2021 Evgenii Alekseev.
+# Copyright (c) 2021 ahriman team.
 #
 # This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).
@@ -23,21 +23,32 @@ from typing import Type
 
 from ahriman.application.handlers.handler import Handler
 from ahriman.core.configuration import Configuration
+from ahriman.core.spawn import Spawn
 
 
 class Web(Handler):
-    '''
+    """
     web server handler
-    '''
+    """
+
+    ALLOW_AUTO_ARCHITECTURE_RUN = False
+    ALLOW_MULTI_ARCHITECTURE_RUN = False  # required to be able to spawn external processes
 
     @classmethod
-    def run(cls: Type[Handler], args: argparse.Namespace, architecture: str, config: Configuration) -> None:
-        '''
+    def run(cls: Type[Handler], args: argparse.Namespace, architecture: str,
+            configuration: Configuration, no_report: bool) -> None:
+        """
         callback for command line
         :param args: command line args
         :param architecture: repository architecture
-        :param config: configuration instance
-        '''
+        :param configuration: configuration instance
+        :param no_report: force disable reporting
+        """
+        # we are using local import for optional dependencies
         from ahriman.web.web import run_server, setup_service
-        application = setup_service(architecture, config)
-        run_server(application, architecture)
+
+        spawner = Spawn(args.parser(), architecture, configuration)
+        spawner.start()
+
+        application = setup_service(architecture, configuration, spawner)
+        run_server(application)

src/ahriman/application/lock.py

@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2021 Evgenii Alekseev.
+# Copyright (c) 2021 ahriman team.
 #
 # This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).
@@ -20,106 +20,111 @@
 from __future__ import annotations
 
 import argparse
+import logging
 import os
 
+from pathlib import Path
 from types import TracebackType
 from typing import Literal, Optional, Type
 
+from ahriman import version
 from ahriman.core.configuration import Configuration
 from ahriman.core.exceptions import DuplicateRun, UnsafeRun
-from ahriman.core.watcher.client import Client
+from ahriman.core.status.client import Client
 from ahriman.models.build_status import BuildStatusEnum
 
 
 class Lock:
-    '''
+    """
     wrapper for application lock file
     :ivar force: remove lock file on start if any
     :ivar path: path to lock file if any
     :ivar reporter: build status reporter instance
     :ivar root: repository root (i.e. ahriman home)
     :ivar unsafe: skip user check
-    '''
+    """
 
-    def __init__(self, args: argparse.Namespace, architecture: str, config: Configuration) -> None:
-        '''
+    def __init__(self, args: argparse.Namespace, architecture: str, configuration: Configuration) -> None:
+        """
         default constructor
         :param args: command line args
         :param architecture: repository architecture
-        :param config: configuration instance
-        '''
-        self.path = f'{args.lock}_{architecture}' if args.lock is not None else None
+        :param configuration: configuration instance
+        """
+        self.path = Path(f"{args.lock}_{architecture}") if args.lock is not None else None
         self.force = args.force
         self.unsafe = args.unsafe
 
-        self.root = config.get('repository', 'root')
-        self.reporter = Client() if args.no_report else Client.load(architecture, config)
+        self.root = Path(configuration.get("repository", "root"))
+        self.reporter = Client() if args.no_report else Client.load(configuration)
 
     def __enter__(self) -> Lock:
-        '''
+        """
         default workflow is the following:
 
             check user UID
-            remove lock file if force flag is set
             check if there is lock file
+            check web status watcher status
             create lock file
             report to web if enabled
-        '''
+        """
         self.check_user()
-        if self.force:
-            self.remove()
-        self.check()
+        self.check_version()
         self.create()
         self.reporter.update_self(BuildStatusEnum.Building)
         return self
 
     def __exit__(self, exc_type: Optional[Type[Exception]], exc_val: Optional[Exception],
                  exc_tb: TracebackType) -> Literal[False]:
-        '''
+        """
         remove lock file when done
         :param exc_type: exception type name if any
         :param exc_val: exception raised if any
         :param exc_tb: exception traceback if any
         :return: always False (do not suppress any exception)
-        '''
-        self.remove()
+        """
+        self.clear()
         status = BuildStatusEnum.Success if exc_val is None else BuildStatusEnum.Failed
         self.reporter.update_self(status)
         return False
 
-    def check(self) -> None:
-        '''
-        check if lock file exists, raise exception if it does
-        '''
-        if self.path is None:
-            return
-        if os.path.exists(self.path):
-            raise DuplicateRun()
+    def check_version(self) -> None:
+        """
+        check web server version
+        """
+        status = self.reporter.get_internal()
+        if status.version is not None and status.version != version.__version__:
+            logging.getLogger("root").warning(
+                "status watcher version mismatch, our %s, their %s",
+                version.__version__,
+                status.version)
 
     def check_user(self) -> None:
-        '''
+        """
         check if current user is actually owner of ahriman root
-        '''
+        """
         if self.unsafe:
             return
         current_uid = os.getuid()
-        root_uid = os.stat(self.root).st_uid
+        root_uid = self.root.stat().st_uid
         if current_uid != root_uid:
             raise UnsafeRun(current_uid, root_uid)
 
-    def create(self) -> None:
-        '''
-        create lock file
-        '''
-        if self.path is None:
-            return
-        open(self.path, 'w').close()
-
-    def remove(self) -> None:
-        '''
+    def clear(self) -> None:
+        """
         remove lock file
-        '''
+        """
         if self.path is None:
             return
-        if os.path.exists(self.path):
-            os.remove(self.path)
+        self.path.unlink(missing_ok=True)
+
+    def create(self) -> None:
+        """
+        create lock file
+        """
+        if self.path is None:
+            return
+        try:
+            self.path.touch(exist_ok=self.force)
+        except FileExistsError:
+            raise DuplicateRun()

src/ahriman/core/__init__.py

@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2021 Evgenii Alekseev.
+# Copyright (c) 2021 ahriman team.
 #
 # This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).

src/ahriman/core/alpm/__init__.py

@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2021 Evgenii Alekseev.
+# Copyright (c) 2021 ahriman team.
 #
 # This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).

src/ahriman/core/alpm/pacman.py

@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2021 Evgenii Alekseev.
+# Copyright (c) 2021 ahriman team.
 #
 # This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).
@@ -18,35 +18,37 @@
 # along with this program. If not, see <http://www.gnu.org/licenses/>.
 #
 from pyalpm import Handle  # type: ignore
-from typing import List, Set
+from typing import Set
 
 from ahriman.core.configuration import Configuration
 
 
 class Pacman:
-    '''
+    """
     alpm wrapper
     :ivar handle: pyalpm root `Handle`
-    '''
+    """
 
-    def __init__(self, config: Configuration) -> None:
-        '''
+    def __init__(self, configuration: Configuration) -> None:
+        """
         default constructor
-        :param config: configuration instance
-        '''
-        root = config.get('alpm', 'root')
-        pacman_root = config.get('alpm', 'database')
-        self.handle = Handle(root, pacman_root)
-        for repository in config.getlist('alpm', 'repositories'):
+        :param configuration: configuration instance
+        """
+        root = configuration.get("alpm", "root")
+        pacman_root = configuration.getpath("alpm", "database")
+        self.handle = Handle(root, str(pacman_root))
+        for repository in configuration.getlist("alpm", "repositories"):
             self.handle.register_syncdb(repository, 0)  # 0 is pgp_level
 
-    def all_packages(self) -> List[str]:
-        '''
+    def all_packages(self) -> Set[str]:
+        """
         get list of packages known for alpm
         :return: list of package names
-        '''
+        """
         result: Set[str] = set()
         for database in self.handle.get_syncdbs():
-            result.update({package.name for package in database.pkgcache})
+            for package in database.pkgcache:
+                result.add(package.name)  # package itself
+                result.update(package.provides)  # provides list for meta-packages
 
-        return list(result)
+        return result

src/ahriman/core/alpm/repo.py

@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2021 Evgenii Alekseev.
+# Copyright (c) 2021 ahriman team.
 #
 # This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).
@@ -18,8 +18,8 @@
 # along with this program. If not, see <http://www.gnu.org/licenses/>.
 #
 import logging
-import os
 
+from pathlib import Path
 from typing import List
 
 from ahriman.core.exceptions import BuildFailed
@@ -28,56 +28,69 @@ from ahriman.models.repository_paths import RepositoryPaths
 
 
 class Repo:
-    '''
+    """
     repo-add and repo-remove wrapper
     :ivar logger: class logger
     :ivar name: repository name
     :ivar paths: repository paths instance
     :ivar sign_args: additional args which have to be used to sign repository archive
-    '''
+    """
+
+    _check_output = check_output
 
     def __init__(self, name: str, paths: RepositoryPaths, sign_args: List[str]) -> None:
-        '''
+        """
         default constructor
         :param name: repository name
         :param paths: repository paths instance
         :param sign_args: additional args which have to be used to sign repository archive
-        '''
-        self.logger = logging.getLogger('build_details')
+        """
+        self.logger = logging.getLogger("build_details")
         self.name = name
         self.paths = paths
         self.sign_args = sign_args
 
     @property
-    def repo_path(self) -> str:
-        '''
+    def repo_path(self) -> Path:
+        """
         :return: path to repository database
-        '''
-        return os.path.join(self.paths.repository, f'{self.name}.db.tar.gz')
+        """
+        return self.paths.repository / f"{self.name}.db.tar.gz"
 
-    def add(self, path: str) -> None:
-        '''
+    def add(self, path: Path) -> None:
+        """
         add new package to repository
         :param path: path to archive to add
-        '''
-        check_output(
-            'repo-add', *self.sign_args, '-R', self.repo_path, path,
-            exception=BuildFailed(path),
+        """
+        Repo._check_output(
+            "repo-add", *self.sign_args, "-R", str(self.repo_path), str(path),
+            exception=BuildFailed(path.name),
             cwd=self.paths.repository,
             logger=self.logger)
 
-    def remove(self, package: str) -> None:
-        '''
+    def init(self) -> None:
+        """
+        create empty repository database
+        """
+        Repo._check_output(
+            "repo-add", *self.sign_args, str(self.repo_path),
+            exception=None,
+            cwd=self.paths.repository,
+            logger=self.logger)
+
+    def remove(self, package: str, filename: Path) -> None:
+        """
         remove package from repository
         :param package: package name to remove
-        '''
+        :param filename: package filename to remove
+        """
         # remove package and signature (if any) from filesystem
-        for fn in filter(lambda f: f.startswith(package), os.listdir(self.paths.repository)):
-            full_path = os.path.join(self.paths.repository, fn)
-            os.remove(full_path)
+        for full_path in self.paths.repository.glob(f"{filename}*"):
+            full_path.unlink()
+
         # remove package from registry
-        check_output(
-            'repo-remove', *self.sign_args, self.repo_path, package,
+        Repo._check_output(
+            "repo-remove", *self.sign_args, str(self.repo_path), package,
             exception=BuildFailed(package),
             cwd=self.paths.repository,
             logger=self.logger)

src/ahriman/core/auth/__init__.py

@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2021 Evgenii Alekseev.
+# Copyright (c) 2021 ahriman team.
 #
 # This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).

src/ahriman/core/auth/auth.py

@@ -0,0 +1,128 @@
+#
+# Copyright (c) 2021 ahriman team.
+#
+# This file is part of ahriman
+# (see https://github.com/arcan1s/ahriman).
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+from __future__ import annotations
+
+import logging
+
+from typing import Dict, Optional, Type
+
+from ahriman.core.configuration import Configuration
+from ahriman.core.exceptions import DuplicateUser
+from ahriman.models.auth_settings import AuthSettings
+from ahriman.models.user import User
+from ahriman.models.user_access import UserAccess
+
+
+class Auth:
+    """
+    helper to deal with user authorization
+    :ivar enabled: indicates if authorization is enabled
+    :ivar max_age: session age in seconds. It will be used for both client side and server side checks
+    :ivar safe_build_status: allow read only access to the index page
+    """
+
+    def __init__(self, configuration: Configuration, provider: AuthSettings = AuthSettings.Disabled) -> None:
+        """
+        default constructor
+        :param configuration: configuration instance
+        :param provider: authorization type definition
+        """
+        self.logger = logging.getLogger("http")
+
+        self.safe_build_status = configuration.getboolean("auth", "safe_build_status")
+
+        self.enabled = provider.is_enabled
+        self.max_age = configuration.getint("auth", "max_age", fallback=7 * 24 * 3600)
+
+    @property
+    def auth_control(self) -> str:
+        """
+        This workaround is required to make different behaviour for login interface.
+        In case of internal authentication it must provide an interface (modal form) to login with button sends POST
+        request. But for an external providers behaviour can be different: e.g. OAuth provider requires sending GET
+        request to external resource
+        :return: login control as html code to insert
+        """
+        return """<button type="button" class="btn btn-link" data-bs-toggle="modal" data-bs-target="#loginForm" style="text-decoration: none">login</button>"""
+
+    @classmethod
+    def load(cls: Type[Auth], configuration: Configuration) -> Auth:
+        """
+        load authorization module from settings
+        :param configuration: configuration instance
+        :return: authorization module according to current settings
+        """
+        provider = AuthSettings.from_option(configuration.get("auth", "target", fallback="disabled"))
+        if provider == AuthSettings.Configuration:
+            from ahriman.core.auth.mapping import Mapping
+            return Mapping(configuration)
+        if provider == AuthSettings.OAuth:
+            from ahriman.core.auth.oauth import OAuth
+            return OAuth(configuration)
+        return cls(configuration)
+
+    @staticmethod
+    def get_users(configuration: Configuration) -> Dict[str, User]:
+        """
+        load users from settings
+        :param configuration: configuration instance
+        :return: map of username to its descriptor
+        """
+        users: Dict[str, User] = {}
+        for role in UserAccess:
+            section = configuration.section_name("auth", role.value)
+            if not configuration.has_section(section):
+                continue
+            for user, password in configuration[section].items():
+                normalized_user = user.lower()
+                if normalized_user in users:
+                    raise DuplicateUser(normalized_user)
+                users[normalized_user] = User(normalized_user, password, role)
+        return users
+
+    async def check_credentials(self, username: Optional[str], password: Optional[str]) -> bool:  # pylint: disable=no-self-use
+        """
+        validate user password
+        :param username: username
+        :param password: entered password
+        :return: True in case if password matches, False otherwise
+        """
+        del username, password
+        return True
+
+    async def known_username(self, username: Optional[str]) -> bool:  # pylint: disable=no-self-use
+        """
+        check if user is known
+        :param username: username
+        :return: True in case if user is known and can be authorized and False otherwise
+        """
+        del username
+        return True
+
+    async def verify_access(self, username: str, required: UserAccess, context: Optional[str]) -> bool:  # pylint: disable=no-self-use
+        """
+        validate if user has access to requested resource
+        :param username: username
+        :param required: required access level
+        :param context: URI request path
+        :return: True in case if user is allowed to do this request and False otherwise
+        """
+        del username, required, context
+        return True

src/ahriman/core/auth/helpers.py

@@ -0,0 +1,70 @@
+#
+# Copyright (c) 2021 ahriman team.
+#
+# This file is part of ahriman
+# (see https://github.com/arcan1s/ahriman).
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+from typing import Any
+
+try:
+    import aiohttp_security  # type: ignore
+    _has_aiohttp_security = True
+except ImportError:
+    _has_aiohttp_security = False
+
+
+async def authorized_userid(*args: Any) -> Any:
+    """
+    handle aiohttp security methods
+    :param args: argument list as provided by authorized_userid function
+    :return: None in case if no aiohttp_security module found and function call otherwise
+    """
+    if _has_aiohttp_security:
+        return await aiohttp_security.authorized_userid(*args)  # pylint: disable=no-value-for-parameter
+    return None
+
+
+async def check_authorized(*args: Any) -> Any:
+    """
+    handle aiohttp security methods
+    :param args: argument list as provided by check_authorized function
+    :return: None in case if no aiohttp_security module found and function call otherwise
+    """
+    if _has_aiohttp_security:
+        return await aiohttp_security.check_authorized(*args)  # pylint: disable=no-value-for-parameter
+    return None
+
+
+async def forget(*args: Any) -> Any:
+    """
+    handle aiohttp security methods
+    :param args: argument list as provided by forget function
+    :return: None in case if no aiohttp_security module found and function call otherwise
+    """
+    if _has_aiohttp_security:
+        return await aiohttp_security.forget(*args)  # pylint: disable=no-value-for-parameter
+    return None
+
+
+async def remember(*args: Any) -> Any:
+    """
+    handle disabled auth
+    :param args: argument list as provided by remember function
+    :return: None in case if no aiohttp_security module found and function call otherwise
+    """
+    if _has_aiohttp_security:
+        return await aiohttp_security.remember(*args)  # pylint: disable=no-value-for-parameter
+    return None

src/ahriman/core/auth/mapping.py

@@ -0,0 +1,84 @@
+#
+# Copyright (c) 2021 ahriman team.
+#
+# This file is part of ahriman
+# (see https://github.com/arcan1s/ahriman).
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+from typing import Optional
+
+from ahriman.core.auth.auth import Auth
+from ahriman.core.configuration import Configuration
+from ahriman.models.auth_settings import AuthSettings
+from ahriman.models.user import User
+from ahriman.models.user_access import UserAccess
+
+
+class Mapping(Auth):
+    """
+    user authorization based on mapping from configuration file
+    :ivar salt: random generated string to salt passwords
+    :ivar _users: map of username to its descriptor
+    """
+
+    def __init__(self, configuration: Configuration, provider: AuthSettings = AuthSettings.Configuration) -> None:
+        """
+        default constructor
+        :param configuration: configuration instance
+        :param provider: authorization type definition
+        """
+        Auth.__init__(self, configuration, provider)
+        self.salt = configuration.get("auth", "salt")
+        self._users = self.get_users(configuration)
+
+    async def check_credentials(self, username: Optional[str], password: Optional[str]) -> bool:
+        """
+        validate user password
+        :param username: username
+        :param password: entered password
+        :return: True in case if password matches, False otherwise
+        """
+        if username is None or password is None:
+            return False  # invalid data supplied
+        user = self.get_user(username)
+        return user is not None and user.check_credentials(password, self.salt)
+
+    def get_user(self, username: str) -> Optional[User]:
+        """
+        retrieve user from in-memory mapping
+        :param username: username
+        :return: user descriptor if username is known and None otherwise
+        """
+        normalized_user = username.lower()
+        return self._users.get(normalized_user)
+
+    async def known_username(self, username: Optional[str]) -> bool:
+        """
+        check if user is known
+        :param username: username
+        :return: True in case if user is known and can be authorized and False otherwise
+        """
+        return username is not None and self.get_user(username) is not None
+
+    async def verify_access(self, username: str, required: UserAccess, context: Optional[str]) -> bool:
+        """
+        validate if user has access to requested resource
+        :param username: username
+        :param required: required access level
+        :param context: URI request path
+        :return: True in case if user is allowed to do this request and False otherwise
+        """
+        user = self.get_user(username)
+        return user is not None and user.verify_access(required)

src/ahriman/core/auth/oauth.py

@@ -0,0 +1,113 @@
+#
+# Copyright (c) 2021 ahriman team.
+#
+# This file is part of ahriman
+# (see https://github.com/arcan1s/ahriman).
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+import aioauth_client
+
+from typing import Optional, Type
+
+from ahriman.core.auth.mapping import Mapping
+from ahriman.core.configuration import Configuration
+from ahriman.core.exceptions import InvalidOption
+from ahriman.models.auth_settings import AuthSettings
+
+
+class OAuth(Mapping):
+    """
+    OAuth user authorization.
+    It is required to create application first and put application credentials.
+    :ivar client_id: application client id
+    :ivar client_secret: application client secret key
+    :ivar provider: provider class, should be one of aiohttp-client provided classes
+    :ivar redirect_uri: redirect URI registered in provider
+    :ivar scopes: list of scopes required by the application
+    """
+
+    def __init__(self, configuration: Configuration, provider: AuthSettings = AuthSettings.OAuth) -> None:
+        """
+        default constructor
+        :param configuration: configuration instance
+        :param provider: authorization type definition
+        """
+        Mapping.__init__(self, configuration, provider)
+        self.client_id = configuration.get("auth", "client_id")
+        self.client_secret = configuration.get("auth", "client_secret")
+        # in order to use OAuth feature the service must be publicity available
+        # thus we expect that address is set
+        self.redirect_uri = f"""{configuration.get("web", "address")}/user-api/v1/login"""
+        self.provider = self.get_provider(configuration.get("auth", "oauth_provider"))
+        # it is list but we will have to convert to string it anyway
+        self.scopes = configuration.get("auth", "oauth_scopes")
+
+    @property
+    def auth_control(self) -> str:
+        """
+        :return: login control as html code to insert
+        """
+        return """<a class="nav-link" href="/user-api/v1/login" title="login via OAuth2">login</a>"""
+
+    @staticmethod
+    def get_provider(name: str) -> Type[aioauth_client.OAuth2Client]:
+        """
+        load OAuth2 provider by name
+        :param name: name of the provider. Must be valid class defined in aioauth-client library
+        :return: loaded provider type
+        """
+        provider: Type[aioauth_client.OAuth2Client] = getattr(aioauth_client, name)
+        try:
+            is_oauth2_client = issubclass(provider, aioauth_client.OAuth2Client)
+        except TypeError:  # what if it is random string?
+            is_oauth2_client = False
+        if not is_oauth2_client:
+            raise InvalidOption(name)
+        return provider
+
+    def get_client(self) -> aioauth_client.OAuth2Client:
+        """
+        load client from parameters
+        :return: generated client according to current settings
+        """
+        return self.provider(client_id=self.client_id, client_secret=self.client_secret)
+
+    def get_oauth_url(self) -> str:
+        """
+        get authorization URI for the specified settings
+        :return: authorization URI as a string
+        """
+        client = self.get_client()
+        uri: str = client.get_authorize_url(scope=self.scopes, redirect_uri=self.redirect_uri)
+        return uri
+
+    async def get_oauth_username(self, code: str) -> Optional[str]:
+        """
+        extract OAuth username from remote
+        :param code: authorization code provided by external service
+        :return: username as is in OAuth provider
+        """
+        try:
+            client = self.get_client()
+            access_token, _ = await client.get_access_token(code, redirect_uri=self.redirect_uri)
+            client.access_token = access_token
+
+            print(f"HEEELOOOO {client}")
+            user, _ = await client.user_info()
+            username: str = user.email  # type: ignore
+            return username
+        except Exception:
+            self.logger.exception("got exception while performing request")
+            return None

src/ahriman/core/build_tools/__init__.py

@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2021 Evgenii Alekseev.
+# Copyright (c) 2021 ahriman team.
 #
 # This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).

src/ahriman/core/build_tools/task.py

@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2021 Evgenii Alekseev.
+# Copyright (c) 2021 ahriman team.
 #
 # This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).
@@ -17,10 +17,10 @@
 # You should have received a copy of the GNU General Public License
 # along with this program. If not, see <http://www.gnu.org/licenses/>.
 #
-import os
 import logging
 import shutil
 
+from pathlib import Path
 from typing import List, Optional
 
 from ahriman.core.configuration import Configuration
@@ -31,94 +31,96 @@ from ahriman.models.repository_paths import RepositoryPaths
 
 
 class Task:
-    '''
+    """
     base package build task
     :ivar build_logger: logger for build process
     :ivar logger: class logger
     :ivar package: package definitions
     :ivar paths: repository paths instance
-    '''
+    """
 
-    def __init__(self, package: Package, architecture: str, config: Configuration, paths: RepositoryPaths) -> None:
-        '''
+    _check_output = check_output
+
+    def __init__(self, package: Package, configuration: Configuration, paths: RepositoryPaths) -> None:
+        """
         default constructor
         :param package: package definitions
-        :param architecture: repository architecture
-        :param config: configuration instance
+        :param configuration: configuration instance
         :param paths: repository paths instance
-        '''
-        self.logger = logging.getLogger('builder')
-        self.build_logger = logging.getLogger('build_details')
+        """
+        self.logger = logging.getLogger("builder")
+        self.build_logger = logging.getLogger("build_details")
         self.package = package
         self.paths = paths
 
-        section = config.get_section_name('build', architecture)
-        self.archbuild_flags = config.getlist(section, 'archbuild_flags')
-        self.build_command = config.get(section, 'build_command')
-        self.makepkg_flags = config.getlist(section, 'makepkg_flags')
-        self.makechrootpkg_flags = config.getlist(section, 'makechrootpkg_flags')
+        self.archbuild_flags = configuration.getlist("build", "archbuild_flags", fallback=[])
+        self.build_command = configuration.get("build", "build_command")
+        self.makepkg_flags = configuration.getlist("build", "makepkg_flags", fallback=[])
+        self.makechrootpkg_flags = configuration.getlist("build", "makechrootpkg_flags", fallback=[])
 
     @property
-    def cache_path(self) -> str:
-        '''
+    def cache_path(self) -> Path:
+        """
         :return: path to cached packages
-        '''
-        return os.path.join(self.paths.cache, self.package.base)
+        """
+        return self.paths.cache / self.package.base
 
     @property
-    def git_path(self) -> str:
-        '''
+    def git_path(self) -> Path:
+        """
         :return: path to clone package from git
-        '''
-        return os.path.join(self.paths.sources, self.package.base)
+        """
+        return self.paths.sources / self.package.base
 
     @staticmethod
-    def fetch(local: str, remote: str, branch: str = 'master') -> None:
-        '''
+    def fetch(local: Path, remote: str, branch: str = "master") -> None:
+        """
         either clone repository or update it to origin/`branch`
         :param local: local path to fetch
         :param remote: remote target (from where to fetch)
         :param branch: branch name to checkout, master by default
-        '''
-        logger = logging.getLogger('build_details')
+        """
+        logger = logging.getLogger("build_details")
         # local directory exists and there is .git directory
-        if os.path.isdir(os.path.join(local, '.git')):
-            check_output('git', 'fetch', 'origin', branch, exception=None, cwd=local, logger=logger)
+        if (local / ".git").is_dir():
+            Task._check_output("git", "fetch", "origin", branch, exception=None, cwd=local, logger=logger)
         else:
-            check_output('git', 'clone', remote, local, exception=None, logger=logger)
+            Task._check_output("git", "clone", remote, str(local), exception=None, logger=logger)
         # and now force reset to our branch
-        check_output('git', 'reset', '--hard', f'origin/{branch}', exception=None, cwd=local, logger=logger)
+        Task._check_output("git", "checkout", "--force", branch, exception=None, cwd=local, logger=logger)
+        Task._check_output("git", "reset", "--hard", f"origin/{branch}", exception=None, cwd=local, logger=logger)
 
-    def build(self) -> List[str]:
-        '''
+    def build(self) -> List[Path]:
+        """
         run package build
         :return: paths of produced packages
-        '''
-        cmd = [self.build_command, '-r', self.paths.chroot]
-        cmd.extend(self.archbuild_flags)
-        cmd.extend(['--'] + self.makechrootpkg_flags)
-        cmd.extend(['--'] + self.makepkg_flags)
-        self.logger.info(f'using {cmd} for {self.package.base}')
+        """
+        command = [self.build_command, "-r", str(self.paths.chroot)]
+        command.extend(self.archbuild_flags)
+        command.extend(["--"] + self.makechrootpkg_flags)
+        command.extend(["--"] + self.makepkg_flags)
+        self.logger.info("using %s for %s", command, self.package.base)
 
-        check_output(
-            *cmd,
+        Task._check_output(
+            *command,
             exception=BuildFailed(self.package.base),
             cwd=self.git_path,
             logger=self.build_logger)
 
         # well it is not actually correct, but we can deal with it
-        return check_output('makepkg', '--packagelist',
-                            exception=BuildFailed(self.package.base),
-                            cwd=self.git_path,
-                            logger=self.build_logger).splitlines()
+        packages = Task._check_output("makepkg", "--packagelist",
+                                      exception=BuildFailed(self.package.base),
+                                      cwd=self.git_path,
+                                      logger=self.build_logger).splitlines()
+        return [Path(package) for package in packages]
 
-    def init(self, path: Optional[str] = None) -> None:
-        '''
+    def init(self, path: Optional[Path] = None) -> None:
+        """
         fetch package from git
         :param path: optional local path to fetch. If not set default path will be used
-        '''
+        """
         git_path = path or self.git_path
-        if os.path.isdir(self.cache_path):
+        if self.cache_path.is_dir():
             # no need to clone whole repository, just copy from cache first
             shutil.copytree(self.cache_path, git_path)
-        return Task.fetch(git_path, self.package.git_url)
+        return self.fetch(git_path, self.package.git_url)

src/ahriman/core/configuration.py

@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2021 Evgenii Alekseev.
+# Copyright (c) 2021 ahriman team.
 #
 # This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).
@@ -21,132 +21,186 @@ from __future__ import annotations
 
 import configparser
 import logging
-import os
 
 from logging.config import fileConfig
-from typing import Dict, List, Optional, Type
+from pathlib import Path
+from typing import Any, Dict, List, Optional, Type
+
+from ahriman.core.exceptions import InitializeException
 
 
 class Configuration(configparser.RawConfigParser):
-    '''
+    """
     extension for built-in configuration parser
+    :ivar architecture: repository architecture
     :ivar path: path to root configuration file
     :cvar ARCHITECTURE_SPECIFIC_SECTIONS: known sections which can be architecture specific (required by dump)
     :cvar DEFAULT_LOG_FORMAT: default log format (in case of fallback)
     :cvar DEFAULT_LOG_LEVEL: default log level (in case of fallback)
-    :cvar STATIC_SECTIONS: known sections which are not architecture specific (required by dump)
-    '''
+    """
 
-    DEFAULT_LOG_FORMAT = '[%(levelname)s %(asctime)s] [%(filename)s:%(lineno)d] [%(funcName)s]: %(message)s'
+    DEFAULT_LOG_FORMAT = "[%(levelname)s %(asctime)s] [%(filename)s:%(lineno)d] [%(funcName)s]: %(message)s"
     DEFAULT_LOG_LEVEL = logging.DEBUG
 
-    STATIC_SECTIONS = ['alpm', 'report', 'repository', 'settings', 'upload']
-    ARCHITECTURE_SPECIFIC_SECTIONS = ['build', 'html', 'rsync', 's3', 'sign', 'web']
+    ARCHITECTURE_SPECIFIC_SECTIONS = ["build", "html", "rsync", "s3", "sign", "web"]
 
     def __init__(self) -> None:
-        '''
+        """
         default constructor. In the most cases must not be called directly
-        '''
-        configparser.RawConfigParser.__init__(self, allow_no_value=True)
-        self.path: Optional[str] = None
+        """
+        configparser.RawConfigParser.__init__(self, allow_no_value=True, converters={
+            "list": lambda value: value.split(),
+            "path": self.__convert_path,
+        })
+        self.architecture: Optional[str] = None
+        self.path: Optional[Path] = None
 
     @property
-    def include(self) -> str:
-        '''
+    def include(self) -> Path:
+        """
         :return: path to directory with configuration includes
-        '''
-        return self.get('settings', 'include')
+        """
+        return self.getpath("settings", "include")
+
+    @property
+    def logging_path(self) -> Path:
+        """
+        :return: path to logging configuration
+        """
+        return self.getpath("settings", "logging")
 
     @classmethod
-    def from_path(cls: Type[Configuration], path: str, logfile: bool) -> Configuration:
-        '''
+    def from_path(cls: Type[Configuration], path: Path, architecture: str, logfile: bool) -> Configuration:
+        """
         constructor with full object initialization
         :param path: path to root configuration file
+        :param architecture: repository architecture
         :param logfile: use log file to output messages
         :return: configuration instance
-        '''
+        """
         config = cls()
         config.load(path)
+        config.merge_sections(architecture)
         config.load_logging(logfile)
         return config
 
-    def dump(self, architecture: str) -> Dict[str, Dict[str, str]]:
-        '''
-        dump configuration to dictionary
-        :param architecture: repository architecture
-        :return: configuration dump for specific architecture
-        '''
-        result: Dict[str, Dict[str, str]] = {}
-        for section in Configuration.STATIC_SECTIONS:
-            if not self.has_section(section):
-                continue
-            result[section] = dict(self[section])
-        for group in Configuration.ARCHITECTURE_SPECIFIC_SECTIONS:
-            section = self.get_section_name(group, architecture)
-            if not self.has_section(section):
-                continue
-            result[section] = dict(self[section])
-
-        return result
-
-    def getlist(self, section: str, key: str) -> List[str]:
-        '''
-        get space separated string list option
+    @staticmethod
+    def section_name(section: str, suffix: str) -> str:
+        """
+        generate section name for sections which depends on context
         :param section: section name
-        :param key: key name
-        :return: list of string if option is set, empty list otherwise
-        '''
-        raw = self.get(section, key, fallback=None)
-        if not raw:  # empty string or none
-            return []
-        return raw.split()
+        :param suffix: session suffix, e.g. repository architecture
+        :return: correct section name for repository specific section
+        """
+        return f"{section}:{suffix}"
 
-    def get_section_name(self, prefix: str, suffix: str) -> str:
-        '''
-        check if there is `prefix`_`suffix` section and return it on success. Return `prefix` otherwise
-        :param prefix: section name prefix
-        :param suffix: section name suffix (e.g. architecture name)
-        :return: found section name
-        '''
-        probe = f'{prefix}_{suffix}'
-        return probe if self.has_section(probe) else prefix
+    def __convert_path(self, value: str) -> Path:
+        """
+        convert string value to path object
+        :param value: string configuration value
+        :return: path object which represents the configuration value
+        """
+        path = Path(value)
+        if self.path is None or path.is_absolute():
+            return path
+        return self.path.parent / path
 
-    def load(self, path: str) -> None:
-        '''
+    def dump(self) -> Dict[str, Dict[str, str]]:
+        """
+        dump configuration to dictionary
+        :return: configuration dump for specific architecture
+        """
+        return {
+            section: dict(self[section])
+            for section in self.sections()
+        }
+
+    # pylint and mypy are too stupid to find these methods
+    # pylint: disable=missing-function-docstring,multiple-statements,unused-argument,no-self-use
+    def getlist(self, *args: Any, **kwargs: Any) -> List[str]: ...
+
+    def getpath(self, *args: Any, **kwargs: Any) -> Path: ...
+
+    def load(self, path: Path) -> None:
+        """
         fully load configuration
         :param path: path to root configuration file
-        '''
+        """
         self.path = path
         self.read(self.path)
         self.load_includes()
 
     def load_includes(self) -> None:
-        '''
+        """
         load configuration includes
-        '''
+        """
         try:
-            for conf in filter(lambda p: p.endswith('.ini'), sorted(os.listdir(self.include))):
-                self.read(os.path.join(self.include, conf))
-        except (FileNotFoundError, configparser.NoOptionError):
+            for path in sorted(self.include.glob("*.ini")):
+                if path == self.logging_path:
+                    continue  # we don't want to load logging explicitly
+                self.read(path)
+        except (FileNotFoundError, configparser.NoOptionError, configparser.NoSectionError):
             pass
 
     def load_logging(self, logfile: bool) -> None:
-        '''
+        """
         setup logging settings from configuration
         :param logfile: use log file to output messages
-        '''
+        """
         def file_logger() -> None:
             try:
-                fileConfig(self.get('settings', 'logging'))
-            except PermissionError:
+                path = self.logging_path
+                fileConfig(path)
+            except (FileNotFoundError, PermissionError):
                 console_logger()
-                logging.error('could not create logfile, fallback to stderr', exc_info=True)
+                logging.exception("could not create logfile, fallback to stderr")
 
         def console_logger() -> None:
-            logging.basicConfig(filename=None, format=Configuration.DEFAULT_LOG_FORMAT,
-                                level=Configuration.DEFAULT_LOG_LEVEL)
+            logging.basicConfig(filename=None, format=self.DEFAULT_LOG_FORMAT,
+                                level=self.DEFAULT_LOG_LEVEL)
 
         if logfile:
             file_logger()
         else:
             console_logger()
+
+    def merge_sections(self, architecture: str) -> None:
+        """
+        merge architecture specific sections into main configuration
+        :param architecture: repository architecture
+        """
+        self.architecture = architecture
+        for section in self.ARCHITECTURE_SPECIFIC_SECTIONS:
+            # get overrides
+            specific = self.section_name(section, architecture)
+            if self.has_section(specific):
+                # if there is no such section it means that there is no overrides for this arch
+                # but we anyway will have to delete sections for others archs
+                for key, value in self[specific].items():
+                    self.set_option(section, key, value)
+            # remove any arch specific section
+            for foreign in self.sections():
+                # we would like to use lambda filter here, but pylint is too dumb
+                if not foreign.startswith(f"{section}:"):
+                    continue
+                self.remove_section(foreign)
+
+    def reload(self) -> None:
+        """
+        reload configuration if possible or raise exception otherwise
+        """
+        if self.path is None or self.architecture is None:
+            raise InitializeException("Configuration path and/or architecture are not set")
+        self.load(self.path)
+        self.merge_sections(self.architecture)
+
+    def set_option(self, section: str, option: str, value: Optional[str]) -> None:
+        """
+        set option. Unlike default `configparser.RawConfigParser.set` it also creates section if it does not exist
+        :param section: section name
+        :param option: option name
+        :param value: option value as string in parsable format
+        """
+        if not self.has_section(section):
+            self.add_section(section)
+        self.set(section, option, value)

src/ahriman/core/exceptions.py

@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2021 Evgenii Alekseev.
+# Copyright (c) 2021 ahriman team.
 #
 # This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).
@@ -20,104 +20,166 @@
 from typing import Any
 
 
-class BuildFailed(Exception):
-    '''
+class BuildFailed(RuntimeError):
+    """
     base exception for failed builds
-    '''
+    """
 
     def __init__(self, package: str) -> None:
-        '''
+        """
         default constructor
         :param package: package base raised exception
-        '''
-        Exception.__init__(self, f'Package {package} build failed, check logs for details')
+        """
+        RuntimeError.__init__(self, f"Package {package} build failed, check logs for details")
 
 
-class DuplicateRun(Exception):
-    '''
+class DuplicateRun(RuntimeError):
+    """
     exception which will be raised if there is another application instance
-    '''
+    """
 
     def __init__(self) -> None:
-        '''
+        """
         default constructor
-        '''
-        Exception.__init__(self, 'Another application instance is run')
+        """
+        RuntimeError.__init__(self, "Another application instance is run")
 
 
-class InitializeException(Exception):
-    '''
+class DuplicateUser(ValueError):
+    """
+    exception which will be thrown in case if there are two users with different settings
+    """
+
+    def __init__(self, username: str) -> None:
+        """
+        default constructor
+        :param username: username with duplicates
+        """
+        ValueError.__init__(self, f"Found duplicate user with username {username}")
+
+
+class InitializeException(RuntimeError):
+    """
     base service initialization exception
-    '''
+    """
 
-    def __init__(self) -> None:
-        '''
+    def __init__(self, details: str) -> None:
+        """
         default constructor
-        '''
-        Exception.__init__(self, 'Could not load service')
+        :param details: details of the exception
+        """
+        RuntimeError.__init__(self, f"Could not load service: {details}")
 
 
-class InvalidOption(Exception):
-    '''
-    exception which will be raised on configuration errors
-    '''
-
-    def __init__(self, value: Any) -> None:
-        '''
-        default constructor
-        :param value: option value
-        '''
-        Exception.__init__(self, f'Invalid or unknown option value `{value}`')
-
-
-class InvalidPackageInfo(Exception):
-    '''
-    exception which will be raised on package load errors
-    '''
+class InvalidCommand(ValueError):
+    """
+    exception raised on invalid command line options
+    """
 
     def __init__(self, details: Any) -> None:
-        '''
+        """
+        default constructor
+        :param details" error details
+        """
+        ValueError.__init__(self, details)
+
+
+class InvalidOption(ValueError):
+    """
+    exception which will be raised on configuration errors
+    """
+
+    def __init__(self, value: Any) -> None:
+        """
+        default constructor
+        :param value: option value
+        """
+        ValueError.__init__(self, f"Invalid or unknown option value `{value}`")
+
+
+class InvalidPackageInfo(RuntimeError):
+    """
+    exception which will be raised on package load errors
+    """
+
+    def __init__(self, details: Any) -> None:
+        """
         default constructor
         :param details: error details
-        '''
-        Exception.__init__(self, f'There are errors during reading package information: `{details}`')
+        """
+        RuntimeError.__init__(self, f"There are errors during reading package information: `{details}`")
 
 
-class ReportFailed(Exception):
-    '''
+class MissingArchitecture(ValueError):
+    """
+    exception which will be raised if architecture is required, but missing
+    """
+
+    def __init__(self, command: str) -> None:
+        """
+        default constructor
+        :param command: command name which throws exception
+        """
+        ValueError.__init__(self, f"Architecture required for subcommand {command}, but missing")
+
+
+class MultipleArchitecture(ValueError):
+    """
+    exception which will be raised if multiple architectures are not supported by the handler
+    """
+
+    def __init__(self, command: str) -> None:
+        """
+        default constructor
+        :param command: command name which throws exception
+        """
+        ValueError.__init__(self, f"Multiple architectures are not supported by subcommand {command}")
+
+
+class ReportFailed(RuntimeError):
+    """
     report generation exception
-    '''
+    """
 
     def __init__(self) -> None:
-        '''
+        """
         default constructor
-        '''
-        Exception.__init__(self, 'Report failed')
+        """
+        RuntimeError.__init__(self, "Report failed")
 
 
-class SyncFailed(Exception):
-    '''
+class SyncFailed(RuntimeError):
+    """
     remote synchronization exception
-    '''
+    """
 
     def __init__(self) -> None:
-        '''
+        """
         default constructor
-        '''
-        Exception.__init__(self, 'Sync failed')
+        """
+        RuntimeError.__init__(self, "Sync failed")
 
 
-class UnsafeRun(Exception):
-    '''
+class UnknownPackage(ValueError):
+    """
+    exception for status watcher which will be thrown on unknown package
+    """
+
+    def __init__(self, base: str) -> None:
+        ValueError.__init__(self, f"Package base {base} is unknown")
+
+
+class UnsafeRun(RuntimeError):
+    """
     exception which will be raised in case if user is not owner of repository
-    '''
+    """
 
     def __init__(self, current_uid: int, root_uid: int) -> None:
-        '''
+        """
         default constructor
-        '''
-        Exception.__init__(
+        """
+        RuntimeError.__init__(
             self,
-            f'''Current UID {current_uid} differs from root owner {root_uid}.
+            f"""Current UID {current_uid} differs from root owner {root_uid}.
 Note that for the most actions it is unsafe to run application as different user.
-If you are 100% sure that it must be there try --unsafe option''')
+If you are 100% sure that it must be there try --unsafe option""")

src/ahriman/core/report/__init__.py

@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2021 Evgenii Alekseev.
+# Copyright (c) 2021 ahriman team.
 #
 # This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).

src/ahriman/core/report/email.py

@@ -0,0 +1,111 @@
+#
+# Copyright (c) 2021 ahriman team.
+#
+# This file is part of ahriman
+# (see https://github.com/arcan1s/ahriman).
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+import datetime
+import smtplib
+
+from email.mime.multipart import MIMEMultipart
+from email.mime.text import MIMEText
+from typing import Dict, Iterable
+
+from ahriman.core.configuration import Configuration
+from ahriman.core.report.jinja_template import JinjaTemplate
+from ahriman.core.report.report import Report
+from ahriman.core.util import pretty_datetime
+from ahriman.models.package import Package
+from ahriman.models.smtp_ssl_settings import SmtpSSLSettings
+
+
+class Email(Report, JinjaTemplate):
+    """
+    email report generator
+    :ivar host: SMTP host to connect
+    :ivar no_empty_report: skip empty report generation
+    :ivar password: password to authenticate via SMTP
+    :ivar port: SMTP port to connect
+    :ivar receivers: list of receivers emails
+    :ivar sender: sender email address
+    :ivar ssl: SSL mode for SMTP connection
+    :ivar user: username to authenticate via SMTP
+    """
+
+    def __init__(self, architecture: str, configuration: Configuration) -> None:
+        """
+        default constructor
+        :param architecture: repository architecture
+        :param configuration: configuration instance
+        """
+        Report.__init__(self, architecture, configuration)
+        JinjaTemplate.__init__(self, "email", configuration)
+
+        self.full_template_path = configuration.getpath("email", "full_template_path", fallback=None)
+        self.template_path = configuration.getpath("email", "template_path")
+
+        # base smtp settings
+        self.host = configuration.get("email", "host")
+        self.no_empty_report = configuration.getboolean("email", "no_empty_report", fallback=True)
+        self.password = configuration.get("email", "password", fallback=None)
+        self.port = configuration.getint("email", "port")
+        self.receivers = configuration.getlist("email", "receivers")
+        self.sender = configuration.get("email", "sender")
+        self.ssl = SmtpSSLSettings.from_option(configuration.get("email", "ssl", fallback="disabled"))
+        self.user = configuration.get("email", "user", fallback=None)
+
+    def _send(self, text: str, attachment: Dict[str, str]) -> None:
+        """
+        send email callback
+        :param text: email body text
+        :param attachment: map of attachment filename to attachment content
+        """
+        message = MIMEMultipart()
+        message["From"] = self.sender
+        message["To"] = ", ".join(self.receivers)
+        message["Subject"] = f"{self.name} build report at {pretty_datetime(datetime.datetime.utcnow().timestamp())}"
+
+        message.attach(MIMEText(text, "html"))
+        for filename, content in attachment.items():
+            attach = MIMEText(content, "html")
+            attach.add_header("Content-Disposition", "attachment", filename=filename)
+            message.attach(attach)
+
+        if self.ssl != SmtpSSLSettings.SSL:
+            session = smtplib.SMTP(self.host, self.port)
+            if self.ssl == SmtpSSLSettings.STARTTLS:
+                session.starttls()
+        else:
+            session = smtplib.SMTP_SSL(self.host, self.port)
+        if self.user is not None and self.password is not None:
+            session.login(self.user, self.password)
+        session.sendmail(self.sender, self.receivers, message.as_string())
+        session.quit()
+
+    def generate(self, packages: Iterable[Package], built_packages: Iterable[Package]) -> None:
+        """
+        generate report for the specified packages
+        :param packages: list of packages to generate report
+        :param built_packages: list of packages which has just been built
+        """
+        if self.no_empty_report and not built_packages:
+            return
+        text = self.make_html(built_packages, self.template_path)
+        if self.full_template_path is not None:
+            attachments = {"index.html": self.make_html(packages, self.full_template_path)}
+        else:
+            attachments = {}
+        self._send(text, attachments)

src/ahriman/core/report/html.py

@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2021 Evgenii Alekseev.
+# Copyright (c) 2021 ahriman team.
 #
 # This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).
@@ -17,92 +17,37 @@
 # You should have received a copy of the GNU General Public License
 # along with this program. If not, see <http://www.gnu.org/licenses/>.
 #
-import jinja2
-import os
-
-from typing import Callable, Dict, Iterable
+from typing import Iterable
 
 from ahriman.core.configuration import Configuration
+from ahriman.core.report.jinja_template import JinjaTemplate
 from ahriman.core.report.report import Report
-from ahriman.core.util import pretty_size, pretty_datetime
 from ahriman.models.package import Package
-from ahriman.models.sign_settings import SignSettings
 
 
-class HTML(Report):
-    '''
+class HTML(Report, JinjaTemplate):
+    """
     html report generator
-
-    It uses jinja2 templates for report generation, the following variables are allowed:
-
-        homepage - link to homepage, string, optional
-        link_path - prefix fo packages to download, string, required
-        has_package_signed - True in case if package sign enabled, False otherwise, required
-        has_repo_signed - True in case if repository database sign enabled, False otherwise, required
-        packages - sorted list of packages properties: archive_size, build_date, filename, installed_size, name, version. Required
-        pgp_key - default PGP key ID, string, optional
-        repository - repository name, string, required
-
-    :ivar homepage: homepage link if any (for footer)
-    :ivar link_path: prefix fo packages to download
-    :ivar name: repository name
-    :ivar pgp_key: default PGP key
     :ivar report_path: output path to html report
-    :ivar sign_targets: targets to sign enabled in configuration
-    :ivar tempate_path: path to directory with jinja templates
-    '''
+    """
 
-    def __init__(self, architecture: str, config: Configuration) -> None:
-        '''
+    def __init__(self, architecture: str, configuration: Configuration) -> None:
+        """
         default constructor
         :param architecture: repository architecture
-        :param config: configuration instance
-        '''
-        Report.__init__(self, architecture, config)
-        section = config.get_section_name('html', architecture)
-        self.report_path = config.get(section, 'path')
-        self.link_path = config.get(section, 'link_path')
-        self.template_path = config.get(section, 'template_path')
+        :param configuration: configuration instance
+        """
+        Report.__init__(self, architecture, configuration)
+        JinjaTemplate.__init__(self, "html", configuration)
 
-        # base template vars
-        self.homepage = config.get(section, 'homepage', fallback=None)
-        self.name = config.get('repository', 'name')
+        self.report_path = configuration.getpath("html", "path")
+        self.template_path = configuration.getpath("html", "template_path")
 
-        sign_section = config.get_section_name('sign', architecture)
-        self.sign_targets = [SignSettings.from_option(opt) for opt in config.getlist(sign_section, 'target')]
-        self.pgp_key = config.get(sign_section, 'key') if self.sign_targets else None
-
-    def generate(self, packages: Iterable[Package]) -> None:
-        '''
+    def generate(self, packages: Iterable[Package], built_packages: Iterable[Package]) -> None:
+        """
         generate report for the specified packages
         :param packages: list of packages to generate report
-        '''
-        # idea comes from https://stackoverflow.com/a/38642558
-        templates_dir, template_name = os.path.split(self.template_path)
-        loader = jinja2.FileSystemLoader(searchpath=templates_dir)
-        environment = jinja2.Environment(loader=loader)
-        template = environment.get_template(template_name)
-
-        content = [
-            {
-                'archive_size': pretty_size(properties.archive_size),
-                'build_date': pretty_datetime(properties.build_date),
-                'filename': properties.filename,
-                'installed_size': pretty_size(properties.installed_size),
-                'name': package,
-                'version': base.version
-            } for base in packages for package, properties in base.packages.items()
-        ]
-        comparator: Callable[[Dict[str, str]], str] = lambda item: item['filename']
-
-        html = template.render(
-            homepage=self.homepage,
-            link_path=self.link_path,
-            has_package_signed=SignSettings.SignPackages in self.sign_targets,
-            has_repo_signed=SignSettings.SignRepository in self.sign_targets,
-            packages=sorted(content, key=comparator),
-            pgp_key=self.pgp_key,
-            repository=self.name)
-
-        with open(self.report_path, 'w') as out:
-            out.write(html)
+        :param built_packages: list of packages which has just been built
+        """
+        html = self.make_html(packages, self.template_path)
+        self.report_path.write_text(html)

src/ahriman/core/report/jinja_template.py

@@ -0,0 +1,115 @@
+#
+# Copyright (c) 2021 ahriman team.
+#
+# This file is part of ahriman
+# (see https://github.com/arcan1s/ahriman).
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+import jinja2
+
+from pathlib import Path
+from typing import Callable, Dict, Iterable
+
+from ahriman.core.configuration import Configuration
+from ahriman.core.sign.gpg import GPG
+from ahriman.core.util import pretty_datetime, pretty_size
+from ahriman.models.package import Package
+from ahriman.models.sign_settings import SignSettings
+
+
+class JinjaTemplate:
+    """
+    jinja based report generator
+
+    It uses jinja2 templates for report generation, the following variables are allowed:
+
+        homepage - link to homepage, string, optional
+        link_path - prefix fo packages to download, string, required
+        has_package_signed - True in case if package sign enabled, False otherwise, required
+        has_repo_signed - True in case if repository database sign enabled, False otherwise, required
+        packages - sorted list of packages properties, required
+                   * architecture, string
+                   * archive_size, pretty printed size, string
+                   * build_date, pretty printed datetime, string
+                   * depends, sorted list of strings
+                   * description, string
+                   * filename, string,
+                   * groups, sorted list of strings
+                   * installed_size, pretty printed datetime, string
+                   * licenses, sorted list of strings
+                   * name, string
+                   * url, string
+                   * version, string
+        pgp_key - default PGP key ID, string, optional
+        repository - repository name, string, required
+
+    :ivar homepage: homepage link if any (for footer)
+    :ivar link_path: prefix fo packages to download
+    :ivar name: repository name
+    :ivar default_pgp_key: default PGP key
+    :ivar sign_targets: targets to sign enabled in configuration
+    """
+
+    def __init__(self, section: str, configuration: Configuration) -> None:
+        """
+        default constructor
+        :param section: settings section name
+        :param configuration: configuration instance
+        """
+        self.link_path = configuration.get(section, "link_path")
+
+        # base template vars
+        self.homepage = configuration.get(section, "homepage", fallback=None)
+        self.name = configuration.get("repository", "name")
+
+        self.sign_targets, self.default_pgp_key = GPG.sign_options(configuration)
+
+    def make_html(self, packages: Iterable[Package], template_path: Path) -> str:
+        """
+        generate report for the specified packages
+        :param packages: list of packages to generate report
+        :param template_path: path to jinja template
+        """
+        # idea comes from https://stackoverflow.com/a/38642558
+        loader = jinja2.FileSystemLoader(searchpath=template_path.parent)
+        environment = jinja2.Environment(loader=loader, autoescape=True)
+        template = environment.get_template(template_path.name)
+
+        content = [
+            {
+                "architecture": properties.architecture or "",
+                "archive_size": pretty_size(properties.archive_size),
+                "build_date": pretty_datetime(properties.build_date),
+                "depends": properties.depends,
+                "description": properties.description or "",
+                "filename": properties.filename,
+                "groups": properties.groups,
+                "installed_size": pretty_size(properties.installed_size),
+                "licenses": properties.licenses,
+                "name": package,
+                "url": properties.url or "",
+                "version": base.version
+            } for base in packages for package, properties in base.packages.items()
+        ]
+        comparator: Callable[[Dict[str, str]], str] = lambda item: item["filename"]
+
+        return template.render(
+            homepage=self.homepage,
+            link_path=self.link_path,
+            has_package_signed=SignSettings.Packages in self.sign_targets,
+            has_repo_signed=SignSettings.Repository in self.sign_targets,
+            packages=sorted(content, key=comparator),
+            pgp_key=self.default_pgp_key,
+            repository=self.name)

src/ahriman/core/report/report.py

@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2021 Evgenii Alekseev.
+# Copyright (c) 2021 ahriman team.
 #
 # This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).
@@ -17,9 +17,11 @@
 # You should have received a copy of the GNU General Public License
 # along with this program. If not, see <http://www.gnu.org/licenses/>.
 #
+from __future__ import annotations
+
 import logging
 
-from typing import Iterable
+from typing import Iterable, Type
 
 from ahriman.core.configuration import Configuration
 from ahriman.core.exceptions import ReportFailed
@@ -28,47 +30,56 @@ from ahriman.models.report_settings import ReportSettings
 
 
 class Report:
-    '''
+    """
     base report generator
     :ivar architecture: repository architecture
-    :ivar config: configuration instance
+    :ivar configuration: configuration instance
     :ivar logger: class logger
-    '''
+    """
 
-    def __init__(self, architecture: str, config: Configuration) -> None:
-        '''
+    def __init__(self, architecture: str, configuration: Configuration) -> None:
+        """
         default constructor
         :param architecture: repository architecture
-        :param config: configuration instance
-        '''
-        self.logger = logging.getLogger('builder')
+        :param configuration: configuration instance
+        """
+        self.logger = logging.getLogger("builder")
         self.architecture = architecture
-        self.config = config
+        self.configuration = configuration
 
-    @staticmethod
-    def run(architecture: str, config: Configuration, target: str, packages: Iterable[Package]) -> None:
-        '''
-        run report generation
+    @classmethod
+    def load(cls: Type[Report], architecture: str, configuration: Configuration, target: str) -> Report:
+        """
+        load client from settings
         :param architecture: repository architecture
-        :param config: configuration instance
+        :param configuration: configuration instance
         :param target: target to generate report (e.g. html)
-        :param packages: list of packages to generate report
-        '''
+        :return: client according to current settings
+        """
         provider = ReportSettings.from_option(target)
         if provider == ReportSettings.HTML:
             from ahriman.core.report.html import HTML
-            report: Report = HTML(architecture, config)
-        else:
-            report = Report(architecture, config)
+            return HTML(architecture, configuration)
+        if provider == ReportSettings.Email:
+            from ahriman.core.report.email import Email
+            return Email(architecture, configuration)
+        return cls(architecture, configuration)  # should never happen
 
-        try:
-            report.generate(packages)
-        except Exception:
-            report.logger.exception('report generation failed', exc_info=True)
-            raise ReportFailed()
-
-    def generate(self, packages: Iterable[Package]) -> None:
-        '''
+    def generate(self, packages: Iterable[Package], built_packages: Iterable[Package]) -> None:
+        """
         generate report for the specified packages
         :param packages: list of packages to generate report
-        '''
+        :param built_packages: list of packages which has just been built
+        """
+
+    def run(self, packages: Iterable[Package], built_packages: Iterable[Package]) -> None:
+        """
+        run report generation
+        :param packages: list of packages to generate report
+        :param built_packages: list of packages which has just been built
+        """
+        try:
+            self.generate(packages, built_packages)
+        except Exception:
+            self.logger.exception("report generation failed")
+            raise ReportFailed()

src/ahriman/core/repository/__init__.py

@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2021 Evgenii Alekseev.
+# Copyright (c) 2021 ahriman team.
 #
 # This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).

src/ahriman/core/repository/cleaner.py

@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2021 Evgenii Alekseev.
+# Copyright (c) 2021 ahriman team.
 #
 # This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).
@@ -17,62 +17,62 @@
 # You should have received a copy of the GNU General Public License
 # along with this program. If not, see <http://www.gnu.org/licenses/>.
 #
-import os
 import shutil
 
+from pathlib import Path
 from typing import List
 
 from ahriman.core.repository.properties import Properties
 
 
 class Cleaner(Properties):
-    '''
+    """
     trait to clean common repository objects
-    '''
+    """
 
-    def packages_built(self) -> List[str]:
-        '''
+    def packages_built(self) -> List[Path]:
+        """
         get list of files in built packages directory
         :return: list of filenames from the directory
-        '''
+        """
         raise NotImplementedError
 
     def clear_build(self) -> None:
-        '''
+        """
         clear sources directory
-        '''
-        self.logger.info('clear package sources directory')
-        for package in os.listdir(self.paths.sources):
-            shutil.rmtree(os.path.join(self.paths.sources, package))
+        """
+        self.logger.info("clear package sources directory")
+        for package in self.paths.sources.iterdir():
+            shutil.rmtree(package)
 
     def clear_cache(self) -> None:
-        '''
+        """
         clear cache directory
-        '''
-        self.logger.info('clear packages sources cache directory')
-        for package in os.listdir(self.paths.cache):
-            shutil.rmtree(os.path.join(self.paths.cache, package))
+        """
+        self.logger.info("clear packages sources cache directory")
+        for package in self.paths.cache.iterdir():
+            shutil.rmtree(package)
 
     def clear_chroot(self) -> None:
-        '''
+        """
         clear cache directory. Warning: this method is architecture independent and will clear every chroot
-        '''
-        self.logger.info('clear build chroot directory')
-        for chroot in os.listdir(self.paths.chroot):
-            shutil.rmtree(os.path.join(self.paths.chroot, chroot))
+        """
+        self.logger.info("clear build chroot directory")
+        for chroot in self.paths.chroot.iterdir():
+            shutil.rmtree(chroot)
 
     def clear_manual(self) -> None:
-        '''
+        """
         clear directory with manual package updates
-        '''
-        self.logger.info('clear manual packages')
-        for package in os.listdir(self.paths.manual):
-            shutil.rmtree(os.path.join(self.paths.manual, package))
+        """
+        self.logger.info("clear manual packages")
+        for package in self.paths.manual.iterdir():
+            shutil.rmtree(package)
 
     def clear_packages(self) -> None:
-        '''
+        """
         clear directory with built packages (NOT repository itself)
-        '''
-        self.logger.info('clear built packages directory')
+        """
+        self.logger.info("clear built packages directory")
         for package in self.packages_built():
-            os.remove(package)
+            package.unlink()

src/ahriman/core/repository/executor.py

@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2021 Evgenii Alekseev.
+# Copyright (c) 2021 ahriman team.
 #
 # This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).
@@ -17,126 +17,140 @@
 # You should have received a copy of the GNU General Public License
 # along with this program. If not, see <http://www.gnu.org/licenses/>.
 #
-import os
 import shutil
 
+from pathlib import Path
 from typing import Dict, Iterable, List, Optional
 
 from ahriman.core.build_tools.task import Task
 from ahriman.core.report.report import Report
 from ahriman.core.repository.cleaner import Cleaner
-from ahriman.core.upload.uploader import Uploader
+from ahriman.core.upload.upload import Upload
 from ahriman.models.package import Package
 
 
 class Executor(Cleaner):
-    '''
+    """
     trait for common repository update processes
-    '''
+    """
 
     def packages(self) -> List[Package]:
-        '''
+        """
         generate list of repository packages
         :return: list of packages properties
-        '''
+        """
         raise NotImplementedError
 
-    def process_build(self, updates: Iterable[Package]) -> List[str]:
-        '''
+    def process_build(self, updates: Iterable[Package]) -> List[Path]:
+        """
         build packages
         :param updates: list of packages properties to build
         :return: `packages_built`
-        '''
+        """
         def build_single(package: Package) -> None:
             self.reporter.set_building(package.base)
-            task = Task(package, self.architecture, self.config, self.paths)
+            task = Task(package, self.configuration, self.paths)
             task.init()
             built = task.build()
             for src in built:
-                dst = os.path.join(self.paths.packages, os.path.basename(src))
+                dst = self.paths.packages / src.name
                 shutil.move(src, dst)
 
-        for package in updates:
+        for single in updates:
             try:
-                build_single(package)
+                build_single(single)
             except Exception:
-                self.reporter.set_failed(package.base)
-                self.logger.exception(f'{package.base} ({self.architecture}) build exception', exc_info=True)
-                continue
+                self.reporter.set_failed(single.base)
+                self.logger.exception("%s (%s) build exception", single.base, self.architecture)
         self.clear_build()
 
         return self.packages_built()
 
-    def process_remove(self, packages: Iterable[str]) -> str:
-        '''
+    def process_remove(self, packages: Iterable[str]) -> Path:
+        """
         remove packages from list
-        :param packages: list of package names or bases to rmeove
+        :param packages: list of package names or bases to remove
         :return: path to repository database
-        '''
-        def remove_single(package: str) -> None:
+        """
+        def remove_single(package: str, fn: Path) -> None:
             try:
-                self.repo.remove(package)
+                self.repo.remove(package, fn)
             except Exception:
-                self.logger.exception(f'could not remove {package}', exc_info=True)
+                self.logger.exception("could not remove %s", package)
 
         requested = set(packages)
         for local in self.packages():
-            if local.base in packages:
-                to_remove = set(local.packages.keys())
+            if local.base in packages or all(package in requested for package in local.packages):
+                to_remove = {
+                    package: Path(properties.filename)
+                    for package, properties in local.packages.items()
+                    if properties.filename is not None
+                }
                 self.reporter.remove(local.base)  # we only update status page in case of base removal
             elif requested.intersection(local.packages.keys()):
-                to_remove = requested.intersection(local.packages.keys())
+                to_remove = {
+                    package: Path(properties.filename)
+                    for package, properties in local.packages.items()
+                    if package in requested and properties.filename is not None
+                }
             else:
-                to_remove = set()
-            for package in to_remove:
-                remove_single(package)
+                to_remove = {}
+            for package, filename in to_remove.items():
+                remove_single(package, filename)
 
         return self.repo.repo_path
 
-    def process_report(self, targets: Optional[Iterable[str]]) -> None:
-        '''
+    def process_report(self, targets: Optional[Iterable[str]], built_packages: Iterable[Package]) -> None:
+        """
         generate reports
         :param targets: list of targets to generate reports. Configuration option will be used if it is not set
-        '''
+        :param built_packages: list of packages which has just been built
+        """
         if targets is None:
-            targets = self.config.getlist('report', 'target')
+            targets = self.configuration.getlist("report", "target")
         for target in targets:
-            Report.run(self.architecture, self.config, target, self.packages())
+            runner = Report.load(self.architecture, self.configuration, target)
+            runner.run(self.packages(), built_packages)
 
-    def process_sync(self, targets: Optional[Iterable[str]]) -> None:
-        '''
+    def process_sync(self, targets: Optional[Iterable[str]], built_packages: Iterable[Package]) -> None:
+        """
         process synchronization to remote servers
         :param targets: list of targets to sync. Configuration option will be used if it is not set
-        '''
+        :param built_packages: list of packages which has just been built
+        """
         if targets is None:
-            targets = self.config.getlist('upload', 'target')
+            targets = self.configuration.getlist("upload", "target")
         for target in targets:
-            Uploader.run(self.architecture, self.config, target, self.paths.repository)
+            runner = Upload.load(self.architecture, self.configuration, target)
+            runner.run(self.paths.repository, built_packages)
 
-    def process_update(self, packages: Iterable[str]) -> str:
-        '''
+    def process_update(self, packages: Iterable[Path]) -> Path:
+        """
         sign packages, add them to repository and update repository database
         :param packages: list of filenames to run
         :return: path to repository database
-        '''
+        """
         def update_single(fn: Optional[str], base: str) -> None:
             if fn is None:
-                self.logger.warning(f'received empty package name for base {base}')
+                self.logger.warning("received empty package name for base %s", base)
                 return  # suppress type checking, it never can be none actually
             # in theory it might be NOT packages directory, but we suppose it is
-            full_path = os.path.join(self.paths.packages, fn)
+            full_path = self.paths.packages / fn
             files = self.sign.sign_package(full_path, base)
             for src in files:
-                dst = os.path.join(self.paths.repository, os.path.basename(src))
+                dst = self.paths.repository / src.name
                 shutil.move(src, dst)
-            package_path = os.path.join(self.paths.repository, fn)
+            package_path = self.paths.repository / fn
             self.repo.add(package_path)
 
         # we are iterating over bases, not single packages
         updates: Dict[str, Package] = {}
-        for fn in packages:
-            local = Package.load(fn, self.pacman, self.aur_url)
-            updates.setdefault(local.base, local).packages.update(local.packages)
+        for filename in packages:
+            try:
+                local = Package.load(filename, self.pacman, self.aur_url)
+                updates.setdefault(local.base, local).packages.update(local.packages)
+            except Exception:
+                self.logger.exception("could not load package from %s", filename)
 
         for local in updates.values():
             try:
@@ -145,7 +159,7 @@ class Executor(Cleaner):
                 self.reporter.set_success(local)
             except Exception:
                 self.reporter.set_failed(local.base)
-                self.logger.exception(f'could not process {local.base}', exc_info=True)
+                self.logger.exception("could not process %s", local.base)
         self.clear_packages()
 
         return self.repo.repo_path

src/ahriman/core/repository/properties.py

@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2021 Evgenii Alekseev.
+# Copyright (c) 2021 ahriman team.
 #
 # This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).
@@ -23,16 +23,17 @@ from ahriman.core.alpm.pacman import Pacman
 from ahriman.core.alpm.repo import Repo
 from ahriman.core.configuration import Configuration
 from ahriman.core.sign.gpg import GPG
-from ahriman.core.watcher.client import Client
+from ahriman.core.status.client import Client
 from ahriman.models.repository_paths import RepositoryPaths
 
 
 class Properties:
-    '''
+    """
     repository internal objects holder
     :ivar architecture: repository architecture
     :ivar aur_url: base AUR url
-    :ivar config: configuration instance
+    :ivar configuration: configuration instance
+    :ivar ignore_list: package bases which will be ignored during auto updates
     :ivar logger: class logger
     :ivar name: repository name
     :ivar pacman: alpm wrapper instance
@@ -40,20 +41,27 @@ class Properties:
     :ivar repo: repo commands wrapper instance
     :ivar reporter: build status reporter instance
     :ivar sign: GPG wrapper instance
-    '''
+    """
 
-    def __init__(self, architecture: str, config: Configuration) -> None:
-        self.logger = logging.getLogger('builder')
+    def __init__(self, architecture: str, configuration: Configuration, no_report: bool) -> None:
+        """
+        default constructor
+        :param architecture: repository architecture
+        :param configuration: configuration instance
+        :param no_report: force disable reporting
+        """
+        self.logger = logging.getLogger("builder")
         self.architecture = architecture
-        self.config = config
+        self.configuration = configuration
 
-        self.aur_url = config.get('alpm', 'aur_url')
-        self.name = config.get('repository', 'name')
+        self.aur_url = configuration.get("alpm", "aur_url")
+        self.name = configuration.get("repository", "name")
 
-        self.paths = RepositoryPaths(config.get('repository', 'root'), architecture)
+        self.paths = RepositoryPaths(configuration.getpath("repository", "root"), architecture)
         self.paths.create_tree()
 
-        self.pacman = Pacman(config)
-        self.sign = GPG(architecture, config)
+        self.ignore_list = configuration.getlist("build", "ignore_packages", fallback=[])
+        self.pacman = Pacman(configuration)
+        self.sign = GPG(architecture, configuration)
         self.repo = Repo(self.name, self.paths, self.sign.repository_sign_args)
-        self.reporter = Client.load(architecture, config)
+        self.reporter = Client() if no_report else Client.load(configuration)

src/ahriman/core/repository/repository.py

@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2021 Evgenii Alekseev.
+# Copyright (c) 2021 ahriman team.
 #
 # This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).
@@ -17,8 +17,7 @@
 # You should have received a copy of the GNU General Public License
 # along with this program. If not, see <http://www.gnu.org/licenses/>.
 #
-import os
-
+from pathlib import Path
 from typing import Dict, List
 
 from ahriman.core.repository.executor import Executor
@@ -28,34 +27,28 @@ from ahriman.models.package import Package
 
 
 class Repository(Executor, UpdateHandler):
-    '''
+    """
     base repository control class
-    '''
+    """
 
     def packages(self) -> List[Package]:
-        '''
+        """
         generate list of repository packages
         :return: list of packages properties
-        '''
+        """
         result: Dict[str, Package] = {}
-        for fn in os.listdir(self.paths.repository):
-            if not package_like(fn):
-                continue
-            full_path = os.path.join(self.paths.repository, fn)
+        for full_path in filter(package_like, self.paths.repository.iterdir()):
             try:
                 local = Package.load(full_path, self.pacman, self.aur_url)
                 result.setdefault(local.base, local).packages.update(local.packages)
             except Exception:
-                self.logger.exception(f'could not load package from {fn}', exc_info=True)
+                self.logger.exception("could not load package from %s", full_path)
                 continue
         return list(result.values())
 
-    def packages_built(self) -> List[str]:
-        '''
+    def packages_built(self) -> List[Path]:
+        """
         get list of files in built packages directory
         :return: list of filenames from the directory
-        '''
-        return [
-            os.path.join(self.paths.packages, fn)
-            for fn in os.listdir(self.paths.packages)
-        ]
+        """
+        return list(filter(package_like, self.paths.packages.iterdir()))

src/ahriman/core/repository/update_handler.py

@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2021 Evgenii Alekseev.
+# Copyright (c) 2021 ahriman team.
 #
 # This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).
@@ -17,8 +17,6 @@
 # You should have received a copy of the GNU General Public License
 # along with this program. If not, see <http://www.gnu.org/licenses/>.
 #
-import os
-
 from typing import Iterable, List
 
 from ahriman.core.repository.cleaner import Cleaner
@@ -26,31 +24,28 @@ from ahriman.models.package import Package
 
 
 class UpdateHandler(Cleaner):
-    '''
+    """
     trait to get package update list
-    '''
+    """
 
     def packages(self) -> List[Package]:
-        '''
+        """
         generate list of repository packages
         :return: list of packages properties
-        '''
+        """
         raise NotImplementedError
 
     def updates_aur(self, filter_packages: Iterable[str], no_vcs: bool) -> List[Package]:
-        '''
+        """
         check AUR for updates
         :param filter_packages: do not check every package just specified in the list
         :param no_vcs: do not check VCS packages
         :return: list of packages which are out-of-dated
-        '''
+        """
         result: List[Package] = []
 
-        build_section = self.config.get_section_name('build', self.architecture)
-        ignore_list = self.config.getlist(build_section, 'ignore_packages')
-
         for local in self.packages():
-            if local.base in ignore_list:
+            if local.base in self.ignore_list:
                 continue
             if local.is_vcs and no_vcs:
                 continue
@@ -64,29 +59,29 @@ class UpdateHandler(Cleaner):
                     result.append(remote)
             except Exception:
                 self.reporter.set_failed(local.base)
-                self.logger.exception(f'could not load remote package {local.base}', exc_info=True)
+                self.logger.exception("could not load remote package %s", local.base)
                 continue
 
         return result
 
     def updates_manual(self) -> List[Package]:
-        '''
+        """
         check for packages for which manual update has been requested
         :return: list of packages which are out-of-dated
-        '''
+        """
         result: List[Package] = []
         known_bases = {package.base for package in self.packages()}
 
-        for fn in os.listdir(self.paths.manual):
+        for fn in self.paths.manual.iterdir():
             try:
-                local = Package.load(os.path.join(self.paths.manual, fn), self.pacman, self.aur_url)
+                local = Package.load(fn, self.pacman, self.aur_url)
                 result.append(local)
                 if local.base not in known_bases:
                     self.reporter.set_unknown(local)
                 else:
                     self.reporter.set_pending(local.base)
             except Exception:
-                self.logger.exception(f'could not add package from {fn}', exc_info=True)
+                self.logger.exception("could not add package from %s", fn)
         self.clear_manual()
 
         return result

src/ahriman/core/sign/__init__.py

@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2021 Evgenii Alekseev.
+# Copyright (c) 2021 ahriman team.
 #
 # This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).

src/ahriman/core/sign/gpg.py

@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2021 Evgenii Alekseev.
+# Copyright (c) 2021 ahriman team.
 #
 # This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).
@@ -18,90 +18,144 @@
 # along with this program. If not, see <http://www.gnu.org/licenses/>.
 #
 import logging
-import os
+import requests
 
-from typing import List
+from pathlib import Path
+from typing import List, Optional, Set, Tuple
 
 from ahriman.core.configuration import Configuration
 from ahriman.core.exceptions import BuildFailed
-from ahriman.core.util import check_output
+from ahriman.core.util import check_output, exception_response_text
 from ahriman.models.sign_settings import SignSettings
 
 
 class GPG:
-    '''
+    """
     gnupg wrapper
     :ivar architecture: repository architecture
-    :ivar config: configuration instance
+    :ivar configuration: configuration instance
     :ivar default_key: default PGP key ID to use
     :ivar logger: class logger
-    :ivar target: list of targets to sign (repository, package etc)
-    '''
+    :ivar targets: list of targets to sign (repository, package etc)
+    """
 
-    def __init__(self, architecture: str, config: Configuration) -> None:
-        '''
+    _check_output = check_output
+
+    def __init__(self, architecture: str, configuration: Configuration) -> None:
+        """
         default constructor
         :param architecture: repository architecture
-        :param config: configuration instance
-        '''
-        self.logger = logging.getLogger('build_details')
-        self.config = config
-        self.section = config.get_section_name('sign', architecture)
-        self.target = [SignSettings.from_option(opt) for opt in config.getlist(self.section, 'target')]
-        self.default_key = config.get(self.section, 'key') if self.target else ''
+        :param configuration: configuration instance
+        """
+        self.logger = logging.getLogger("build_details")
+        self.architecture = architecture
+        self.configuration = configuration
+        self.targets, self.default_key = self.sign_options(configuration)
 
     @property
     def repository_sign_args(self) -> List[str]:
-        '''
+        """
         :return: command line arguments for repo-add command to sign database
-        '''
-        if SignSettings.SignRepository not in self.target:
+        """
+        if SignSettings.Repository not in self.targets:
             return []
-        return ['--sign', '--key', self.default_key]
+        if self.default_key is None:
+            self.logger.error("no default key set, skip repository sign")
+            return []
+        return ["--sign", "--key", self.default_key]
 
     @staticmethod
-    def sign_cmd(path: str, key: str) -> List[str]:
-        '''
+    def sign_command(path: Path, key: str) -> List[str]:
+        """
         gpg command to run
         :param path: path to file to sign
         :param key: PGP key ID
         :return: gpg command with all required arguments
-        '''
-        return ['gpg', '-u', key, '-b', path]
+        """
+        return ["gpg", "-u", key, "-b", str(path)]
 
-    def process(self, path: str, key: str) -> List[str]:
-        '''
+    @staticmethod
+    def sign_options(configuration: Configuration) -> Tuple[Set[SignSettings], Optional[str]]:
+        """
+        extract default sign options from configuration
+        :param configuration: configuration instance
+        :return: tuple of sign targets and default PGP key
+        """
+        targets = {
+            SignSettings.from_option(option)
+            for option in configuration.getlist("sign", "target")
+        }
+        default_key = configuration.get("sign", "key") if targets else None
+        return targets, default_key
+
+    def download_key(self, server: str, key: str) -> str:
+        """
+        download key from public PGP server
+        :param server: public PGP server which will be used to download the key
+        :param key: key ID to download
+        :return: key as plain text
+        """
+        key = key if key.startswith("0x") else f"0x{key}"
+        try:
+            response = requests.get(f"http://{server}/pks/lookup", params={
+                "op": "get",
+                "options": "mr",
+                "search": key
+            })
+            response.raise_for_status()
+        except requests.exceptions.HTTPError as e:
+            self.logger.exception("could not download key %s from %s: %s", key, server, exception_response_text(e))
+            raise
+        return response.text
+
+    def import_key(self, server: str, key: str) -> None:
+        """
+        import key to current user and sign it locally
+        :param server: public PGP server which will be used to download the key
+        :param key: key ID to import
+        """
+        key_body = self.download_key(server, key)
+        GPG._check_output("gpg", "--import", input_data=key_body, exception=None, logger=self.logger)
+        GPG._check_output("gpg", "--quick-lsign-key", key, exception=None, logger=self.logger)
+
+    def process(self, path: Path, key: str) -> List[Path]:
+        """
         gpg command wrapper
         :param path: path to file to sign
         :param key: PGP key ID
         :return: list of generated files including original file
-        '''
-        check_output(
-            *GPG.sign_cmd(path, key),
-            exception=BuildFailed(path),
-            cwd=os.path.dirname(path),
+        """
+        GPG._check_output(
+            *GPG.sign_command(path, key),
+            exception=BuildFailed(path.name),
             logger=self.logger)
-        return [path, f'{path}.sig']
+        return [path, path.parent / f"{path.name}.sig"]
 
-    def sign_package(self, path: str, base: str) -> List[str]:
-        '''
+    def sign_package(self, path: Path, base: str) -> List[Path]:
+        """
         sign package if required by configuration
         :param path: path to file to sign
         :param base: package base required to check for key overrides
         :return: list of generated files including original file
-        '''
-        if SignSettings.SignPackages not in self.target:
+        """
+        if SignSettings.Packages not in self.targets:
+            return [path]
+        key = self.configuration.get("sign", f"key_{base}", fallback=self.default_key)
+        if key is None:
+            self.logger.error("no default key set, skip package %s sign", path)
             return [path]
-        key = self.config.get(self.section, f'key_{base}', fallback=self.default_key)
         return self.process(path, key)
 
-    def sign_repository(self, path: str) -> List[str]:
-        '''
+    def sign_repository(self, path: Path) -> List[Path]:
+        """
         sign repository if required by configuration
         :note: more likely you just want to pass `repository_sign_args` to repo wrapper
         :param path: path to repository database
         :return: list of generated files including original file
-        '''
-        if SignSettings.SignRepository not in self.target:
+        """
+        if SignSettings.Repository not in self.targets:
+            return [path]
+        if self.default_key is None:
+            self.logger.error("no default key set, skip repository sign")
             return [path]
         return self.process(path, self.default_key)

src/ahriman/core/spawn.py

@@ -0,0 +1,140 @@
+#
+# Copyright (c) 2021 ahriman team.
+#
+# This file is part of ahriman
+# (see https://github.com/arcan1s/ahriman).
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+from __future__ import annotations
+
+import argparse
+import logging
+import uuid
+
+from multiprocessing import Process, Queue
+from threading import Lock, Thread
+from typing import Callable, Dict, Iterable, Tuple
+
+from ahriman.core.configuration import Configuration
+from ahriman.models.package_source import PackageSource
+
+
+class Spawn(Thread):
+    """
+    helper to spawn external ahriman process
+    MUST NOT be used directly, the only one usage allowed is to spawn process from web services
+    :ivar active: map of active child processes required to avoid zombies
+    :ivar architecture: repository architecture
+    :ivar configuration: configuration instance
+    :ivar logger: spawner logger
+    :ivar queue: multiprocessing queue to read updates from processes
+    """
+
+    def __init__(self, args_parser: argparse.ArgumentParser, architecture: str, configuration: Configuration) -> None:
+        """
+        default constructor
+        :param args_parser: command line parser for the application
+        :param architecture: repository architecture
+        :param configuration: configuration instance
+        """
+        Thread.__init__(self, name="spawn")
+        self.architecture = architecture
+        self.args_parser = args_parser
+        self.configuration = configuration
+        self.logger = logging.getLogger("http")
+
+        self.lock = Lock()
+        self.active: Dict[str, Process] = {}
+        # stupid pylint does not know that it is possible
+        self.queue: Queue[Tuple[str, bool]] = Queue()  # pylint: disable=unsubscriptable-object
+
+    @staticmethod
+    def process(callback: Callable[[argparse.Namespace, str], bool], args: argparse.Namespace, architecture: str,
+                process_id: str, queue: Queue[Tuple[str, bool]]) -> None:  # pylint: disable=unsubscriptable-object
+        """
+        helper to run external process
+        :param callback: application run function (i.e. Handler.run method)
+        :param args: command line arguments
+        :param architecture: repository architecture
+        :param process_id: process unique identifier
+        :param queue: output queue
+        """
+        result = callback(args, architecture)
+        queue.put((process_id, result))
+
+    def packages_add(self, packages: Iterable[str], now: bool) -> None:
+        """
+        add packages
+        :param packages: packages list to add
+        :param now: build packages now
+        """
+        kwargs = {"source": PackageSource.AUR.value}  # avoid abusing by building non-aur packages
+        if now:
+            kwargs["now"] = ""
+        self.spawn_process("add", *packages, **kwargs)
+
+    def packages_remove(self, packages: Iterable[str]) -> None:
+        """
+        remove packages
+        :param packages: packages list to remove
+        """
+        self.spawn_process("remove", *packages)
+
+    def spawn_process(self, command: str, *args: str, **kwargs: str) -> None:
+        """
+        spawn external ahriman process with supplied arguments
+        :param command: subcommand to run
+        :param args: positional command arguments
+        :param kwargs: named command arguments
+        """
+        # default arguments
+        arguments = ["--architecture", self.architecture]
+        if self.configuration.path is not None:
+            arguments.extend(["--configuration", str(self.configuration.path)])
+        # positional command arguments
+        arguments.append(command)
+        arguments.extend(args)
+        # named command arguments
+        for argument, value in kwargs.items():
+            arguments.append(f"--{argument}")
+            if value:
+                arguments.append(value)
+
+        process_id = str(uuid.uuid4())
+        self.logger.info("full command line arguments of %s are %s", process_id, arguments)
+        parsed = self.args_parser.parse_args(arguments)
+
+        callback = parsed.handler.call
+        process = Process(target=self.process,
+                          args=(callback, parsed, self.architecture, process_id, self.queue),
+                          daemon=True)
+        process.start()
+
+        with self.lock:
+            self.active[process_id] = process
+
+    def run(self) -> None:
+        """
+        thread run method
+        """
+        for process_id, status in iter(self.queue.get, None):
+            self.logger.info("process %s has been terminated with status %s", process_id, status)
+
+            with self.lock:
+                process = self.active.pop(process_id, None)
+
+            if process is not None:
+                process.terminate()  # make sure lol
+                process.join()

src/ahriman/core/status/__init__.py

@@ -0,0 +1,19 @@
+#
+# Copyright (c) 2021 ahriman team.
+#
+# This file is part of ahriman
+# (see https://github.com/arcan1s/ahriman).
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#

src/ahriman/core/status/client.py

@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2021 Evgenii Alekseev.
+# Copyright (c) 2021 ahriman team.
 #
 # This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).
@@ -19,110 +19,119 @@
 #
 from __future__ import annotations
 
-from typing import List, Optional, Tuple
+from typing import List, Optional, Tuple, Type
 
 from ahriman.core.configuration import Configuration
 from ahriman.models.build_status import BuildStatus, BuildStatusEnum
+from ahriman.models.internal_status import InternalStatus
 from ahriman.models.package import Package
 
 
 class Client:
-    '''
+    """
     base build status reporter client
-    '''
+    """
+
+    @classmethod
+    def load(cls: Type[Client], configuration: Configuration) -> Client:
+        """
+        load client from settings
+        :param configuration: configuration instance
+        :return: client according to current settings
+        """
+        address = configuration.get("web", "address", fallback=None)
+        host = configuration.get("web", "host", fallback=None)
+        port = configuration.getint("web", "port", fallback=None)
+        if address or (host and port):
+            from ahriman.core.status.web_client import WebClient
+            return WebClient(configuration)
+        return cls()
 
     def add(self, package: Package, status: BuildStatusEnum) -> None:
-        '''
+        """
         add new package with status
         :param package: package properties
         :param status: current package build status
-        '''
+        """
 
-    # pylint: disable=R0201
-    def get(self, base: Optional[str]) -> List[Tuple[Package, BuildStatus]]:
-        '''
+    def get(self, base: Optional[str]) -> List[Tuple[Package, BuildStatus]]:  # pylint: disable=no-self-use
+        """
         get package status
         :param base: package base to get
         :return: list of current package description and status if it has been found
-        '''
+        """
         del base
         return []
 
-    # pylint: disable=R0201
-    def get_self(self) -> BuildStatus:
-        '''
+    def get_internal(self) -> InternalStatus:  # pylint: disable=no-self-use
+        """
+        get internal service status
+        :return: current internal (web) service status
+        """
+        return InternalStatus()
+
+    def get_self(self) -> BuildStatus:  # pylint: disable=no-self-use
+        """
         get ahriman status itself
         :return: current ahriman status
-        '''
+        """
         return BuildStatus()
 
+    def reload_auth(self) -> None:
+        """
+        reload authentication module call
+        """
+
     def remove(self, base: str) -> None:
-        '''
+        """
         remove packages from watcher
         :param base: package base to remove
-        '''
+        """
 
     def update(self, base: str, status: BuildStatusEnum) -> None:
-        '''
+        """
         update package build status. Unlike `add` it does not update package properties
         :param base: package base to update
         :param status: current package build status
-        '''
+        """
 
     def update_self(self, status: BuildStatusEnum) -> None:
-        '''
+        """
         update ahriman status itself
         :param status: current ahriman status
-        '''
+        """
 
     def set_building(self, base: str) -> None:
-        '''
+        """
         set package status to building
         :param base: package base to update
-        '''
+        """
         return self.update(base, BuildStatusEnum.Building)
 
     def set_failed(self, base: str) -> None:
-        '''
+        """
         set package status to failed
         :param base: package base to update
-        '''
+        """
         return self.update(base, BuildStatusEnum.Failed)
 
     def set_pending(self, base: str) -> None:
-        '''
+        """
         set package status to pending
         :param base: package base to update
-        '''
+        """
         return self.update(base, BuildStatusEnum.Pending)
 
     def set_success(self, package: Package) -> None:
-        '''
+        """
         set package status to success
         :param package: current package properties
-        '''
+        """
         return self.add(package, BuildStatusEnum.Success)
 
     def set_unknown(self, package: Package) -> None:
-        '''
+        """
         set package status to unknown
         :param package: current package properties
-        '''
+        """
         return self.add(package, BuildStatusEnum.Unknown)
-
-    @staticmethod
-    def load(architecture: str, config: Configuration) -> Client:
-        '''
-        load client from settings
-        :param architecture: repository architecture
-        :param config: configuration instance
-        :return: client according to current settings
-        '''
-        section = config.get_section_name('web', architecture)
-        host = config.get(section, 'host', fallback=None)
-        port = config.getint(section, 'port', fallback=None)
-        if host is None or port is None:
-            return Client()
-
-        from ahriman.core.watcher.web_client import WebClient
-        return WebClient(host, port)

src/ahriman/core/status/watcher.py

@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2021 Evgenii Alekseev.
+# Copyright (c) 2021 ahriman team.
 #
 # This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).
@@ -19,103 +19,111 @@
 #
 import json
 import logging
-import os
 
+from pathlib import Path
 from typing import Any, Dict, List, Optional, Tuple
 
 from ahriman.core.configuration import Configuration
+from ahriman.core.exceptions import UnknownPackage
 from ahriman.core.repository.repository import Repository
 from ahriman.models.build_status import BuildStatus, BuildStatusEnum
 from ahriman.models.package import Package
 
 
 class Watcher:
-    '''
+    """
     package status watcher
     :ivar architecture: repository architecture
     :ivar known: list of known packages. For the most cases `packages` should be used instead
     :ivar logger: class logger
     :ivar repository: repository object
     :ivar status: daemon status
-    '''
+    """
 
-    def __init__(self, architecture: str, config: Configuration) -> None:
-        '''
+    def __init__(self, architecture: str, configuration: Configuration) -> None:
+        """
         default constructor
         :param architecture: repository architecture
-        :param config: configuration instance
-        '''
-        self.logger = logging.getLogger('http')
+        :param configuration: configuration instance
+        """
+        self.logger = logging.getLogger("http")
 
         self.architecture = architecture
-        self.repository = Repository(architecture, config)
+        self.repository = Repository(architecture, configuration, no_report=True)
 
         self.known: Dict[str, Tuple[Package, BuildStatus]] = {}
         self.status = BuildStatus()
 
     @property
-    def cache_path(self) -> str:
-        '''
+    def cache_path(self) -> Path:
+        """
         :return: path to dump with json cache
-        '''
-        return os.path.join(self.repository.paths.root, 'status_cache.json')
+        """
+        return self.repository.paths.root / "status_cache.json"
 
     @property
     def packages(self) -> List[Tuple[Package, BuildStatus]]:
-        '''
+        """
         :return: list of packages together with their statuses
-        '''
+        """
         return list(self.known.values())
 
     def _cache_load(self) -> None:
-        '''
+        """
         update current state from cache
-        '''
+        """
         def parse_single(properties: Dict[str, Any]) -> None:
-            package = Package.from_json(properties['package'])
-            status = BuildStatus.from_json(properties['status'])
+            package = Package.from_json(properties["package"])
+            status = BuildStatus.from_json(properties["status"])
             if package.base in self.known:
                 self.known[package.base] = (package, status)
 
-        if not os.path.isfile(self.cache_path):
+        if not self.cache_path.is_file():
             return
-        with open(self.cache_path) as cache:
-            dump = json.load(cache)
-        for item in dump['packages']:
+        with self.cache_path.open() as cache:
+            try:
+                dump = json.load(cache)
+            except Exception:
+                self.logger.exception("cannot parse json from file")
+                dump = {}
+        for item in dump.get("packages", []):
             try:
                 parse_single(item)
             except Exception:
-                self.logger.exception(f'cannot parse item f{item} to package', exc_info=True)
+                self.logger.exception("cannot parse item %s to package", item)
 
     def _cache_save(self) -> None:
-        '''
+        """
         dump current cache to filesystem
-        '''
+        """
         dump = {
-            'packages': [
+            "packages": [
                 {
-                    'package': package.view(),
-                    'status': status.view()
+                    "package": package.view(),
+                    "status": status.view()
                 } for package, status in self.packages
             ]
         }
         try:
-            with open(self.cache_path, 'w') as cache:
+            with self.cache_path.open("w") as cache:
                 json.dump(dump, cache)
         except Exception:
-            self.logger.exception('cannot dump cache', exc_info=True)
+            self.logger.exception("cannot dump cache")
 
     def get(self, base: str) -> Tuple[Package, BuildStatus]:
-        '''
+        """
         get current package base build status
         :return: package and its status
-        '''
-        return self.known[base]
+        """
+        try:
+            return self.known[base]
+        except KeyError:
+            raise UnknownPackage(base)
 
     def load(self) -> None:
-        '''
+        """
         load packages from local repository. In case if last status is known, it will use it
-        '''
+        """
         for package in self.repository.packages():
             # get status of build or assign unknown
             current = self.known.get(package.base)
@@ -127,29 +135,32 @@ class Watcher:
         self._cache_load()
 
     def remove(self, base: str) -> None:
-        '''
+        """
         remove package base from known list if any
         :param base: package base
-        '''
+        """
         self.known.pop(base, None)
         self._cache_save()
 
     def update(self, base: str, status: BuildStatusEnum, package: Optional[Package]) -> None:
-        '''
+        """
         update package status and description
         :param base: package base to update
         :param status: new build status
         :param package: optional new package description. In case if not set current properties will be used
-        '''
+        """
         if package is None:
-            package, _ = self.known[base]
+            try:
+                package, _ = self.known[base]
+            except KeyError:
+                raise UnknownPackage(base)
         full_status = BuildStatus(status)
         self.known[base] = (package, full_status)
         self._cache_save()
 
     def update_self(self, status: BuildStatusEnum) -> None:
-        '''
+        """
         update service status
         :param status: new service status
-        '''
+        """
         self.status = BuildStatus(status)

src/ahriman/core/status/web_client.py

@@ -0,0 +1,255 @@
+#
+# Copyright (c) 2021 ahriman team.
+#
+# This file is part of ahriman
+# (see https://github.com/arcan1s/ahriman).
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+import logging
+import requests
+
+from typing import List, Optional, Tuple
+
+from ahriman.core.configuration import Configuration
+from ahriman.core.status.client import Client
+from ahriman.core.util import exception_response_text
+from ahriman.models.build_status import BuildStatusEnum, BuildStatus
+from ahriman.models.internal_status import InternalStatus
+from ahriman.models.package import Package
+from ahriman.models.user import User
+
+
+class WebClient(Client):
+    """
+    build status reporter web client
+    :ivar address: address of the web service
+    :ivar logger: class logger
+    :ivar user: web service user descriptor
+    """
+
+    def __init__(self, configuration: Configuration) -> None:
+        """
+        default constructor
+        :param configuration: configuration instance
+        """
+        self.logger = logging.getLogger("http")
+        self.address = self.parse_address(configuration)
+        self.user = User.from_option(
+            configuration.get("web", "username", fallback=None),
+            configuration.get("web", "password", fallback=None))
+
+        self.__session = requests.session()
+        self._login()
+
+    @property
+    def _ahriman_url(self) -> str:
+        """
+        :return: full url for web service for ahriman service itself
+        """
+        return f"{self.address}/status-api/v1/ahriman"
+
+    @property
+    def _login_url(self) -> str:
+        """
+        :return: full url for web service to login
+        """
+        return f"{self.address}/user-api/v1/login"
+
+    @property
+    def _reload_auth_url(self) -> str:
+        """
+        :return: full url for web service to reload authentication module
+        """
+        return f"{self.address}/service-api/v1/reload-auth"
+
+    @property
+    def _status_url(self) -> str:
+        """
+        :return: full url for web service for status
+        """
+        return f"{self.address}/status-api/v1/status"
+
+    @staticmethod
+    def parse_address(configuration: Configuration) -> str:
+        """
+        parse address from configuration
+        :param configuration: configuration instance
+        :return: valid http address
+        """
+        address = configuration.get("web", "address", fallback=None)
+        if not address:
+            # build address from host and port directly
+            host = configuration.get("web", "host")
+            port = configuration.getint("web", "port")
+            address = f"http://{host}:{port}"
+        return address
+
+    def _login(self) -> None:
+        """
+        process login to the service
+        """
+        if self.user is None:
+            return  # no auth configured
+
+        payload = {
+            "username": self.user.username,
+            "password": self.user.password
+        }
+
+        try:
+            response = self.__session.post(self._login_url, json=payload)
+            response.raise_for_status()
+        except requests.exceptions.HTTPError as e:
+            self.logger.exception("could not login as %s: %s", self.user, exception_response_text(e))
+        except Exception:
+            self.logger.exception("could not login as %s", self.user)
+
+    def _package_url(self, base: str = "") -> str:
+        """
+        url generator
+        :param base: package base to generate url
+        :return: full url of web service for specific package base
+        """
+        return f"{self.address}/status-api/v1/packages/{base}"
+
+    def add(self, package: Package, status: BuildStatusEnum) -> None:
+        """
+        add new package with status
+        :param package: package properties
+        :param status: current package build status
+        """
+        payload = {
+            "status": status.value,
+            "package": package.view()
+        }
+
+        try:
+            response = self.__session.post(self._package_url(package.base), json=payload)
+            response.raise_for_status()
+        except requests.exceptions.HTTPError as e:
+            self.logger.exception("could not add %s: %s", package.base, exception_response_text(e))
+        except Exception:
+            self.logger.exception("could not add %s", package.base)
+
+    def get(self, base: Optional[str]) -> List[Tuple[Package, BuildStatus]]:
+        """
+        get package status
+        :param base: package base to get
+        :return: list of current package description and status if it has been found
+        """
+        try:
+            response = self.__session.get(self._package_url(base or ""))
+            response.raise_for_status()
+
+            status_json = response.json()
+            return [
+                (Package.from_json(package["package"]), BuildStatus.from_json(package["status"]))
+                for package in status_json
+            ]
+        except requests.exceptions.HTTPError as e:
+            self.logger.exception("could not get %s: %s", base, exception_response_text(e))
+        except Exception:
+            self.logger.exception("could not get %s", base)
+        return []
+
+    def get_internal(self) -> InternalStatus:
+        """
+        get internal service status
+        :return: current internal (web) service status
+        """
+        try:
+            response = self.__session.get(self._status_url)
+            response.raise_for_status()
+
+            status_json = response.json()
+            return InternalStatus.from_json(status_json)
+        except requests.exceptions.HTTPError as e:
+            self.logger.exception("could not get web service status: %s", exception_response_text(e))
+        except Exception:
+            self.logger.exception("could not get web service status")
+        return InternalStatus()
+
+    def get_self(self) -> BuildStatus:
+        """
+        get ahriman status itself
+        :return: current ahriman status
+        """
+        try:
+            response = self.__session.get(self._ahriman_url)
+            response.raise_for_status()
+
+            status_json = response.json()
+            return BuildStatus.from_json(status_json)
+        except requests.exceptions.HTTPError as e:
+            self.logger.exception("could not get service status: %s", exception_response_text(e))
+        except Exception:
+            self.logger.exception("could not get service status")
+        return BuildStatus()
+
+    def reload_auth(self) -> None:
+        """
+        reload authentication module call
+        """
+        try:
+            response = self.__session.post(self._reload_auth_url)
+            response.raise_for_status()
+        except requests.exceptions.HTTPError as e:
+            self.logger.exception("could not reload auth module: %s", exception_response_text(e))
+        except Exception:
+            self.logger.exception("could not reload auth module")
+
+    def remove(self, base: str) -> None:
+        """
+        remove packages from watcher
+        :param base: basename to remove
+        """
+        try:
+            response = self.__session.delete(self._package_url(base))
+            response.raise_for_status()
+        except requests.exceptions.HTTPError as e:
+            self.logger.exception("could not delete %s: %s", base, exception_response_text(e))
+        except Exception:
+            self.logger.exception("could not delete %s", base)
+
+    def update(self, base: str, status: BuildStatusEnum) -> None:
+        """
+        update package build status. Unlike `add` it does not update package properties
+        :param base: package base to update
+        :param status: current package build status
+        """
+        payload = {"status": status.value}
+
+        try:
+            response = self.__session.post(self._package_url(base), json=payload)
+            response.raise_for_status()
+        except requests.exceptions.HTTPError as e:
+            self.logger.exception("could not update %s: %s", base, exception_response_text(e))
+        except Exception:
+            self.logger.exception("could not update %s", base)
+
+    def update_self(self, status: BuildStatusEnum) -> None:
+        """
+        update ahriman status itself
+        :param status: current ahriman status
+        """
+        payload = {"status": status.value}
+
+        try:
+            response = self.__session.post(self._ahriman_url, json=payload)
+            response.raise_for_status()
+        except requests.exceptions.HTTPError as e:
+            self.logger.exception("could not update service status: %s", exception_response_text(e))
+        except Exception:
+            self.logger.exception("could not update service status")

src/ahriman/core/tree.py

@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2021 Evgenii Alekseev.
+# Copyright (c) 2021 ahriman team.
 #
 # This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).
@@ -22,74 +22,90 @@ from __future__ import annotations
 import shutil
 import tempfile
 
-from typing import Iterable, List, Set
+from pathlib import Path
+from typing import Iterable, List, Set, Type
 
 from ahriman.core.build_tools.task import Task
 from ahriman.models.package import Package
 
 
 class Leaf:
-    '''
+    """
     tree leaf implementation
     :ivar dependencies: list of package dependencies
     :ivar package: leaf package properties
-    '''
+    """
 
-    def __init__(self, package: Package) -> None:
-        '''
+    def __init__(self, package: Package, dependencies: Set[str]) -> None:
+        """
         default constructor
         :param package: package properties
-        '''
+        :param dependencies: package dependencies
+        """
         self.package = package
-        self.dependencies: Set[str] = set()
+        self.dependencies = dependencies
 
     @property
     def items(self) -> Iterable[str]:
-        '''
+        """
         :return: packages containing in this leaf
-        '''
+        """
         return self.package.packages.keys()
 
+    @classmethod
+    def load(cls: Type[Leaf], package: Package) -> Leaf:
+        """
+        load leaf from package with dependencies
+        :param package: package properties
+        :return: loaded class
+        """
+        clone_dir = Path(tempfile.mkdtemp())
+        try:
+            Task.fetch(clone_dir, package.git_url)
+            dependencies = Package.dependencies(clone_dir)
+        finally:
+            shutil.rmtree(clone_dir, ignore_errors=True)
+        return cls(package, dependencies)
+
     def is_root(self, packages: Iterable[Leaf]) -> bool:
-        '''
+        """
         check if package depends on any other package from list of not
         :param packages: list of known leaves
         :return: True if any of packages is dependency of the leaf, False otherwise
-        '''
+        """
         for leaf in packages:
             if self.dependencies.intersection(leaf.items):
                 return False
         return True
 
-    def load_dependencies(self) -> None:
-        '''
-        load dependencies for the leaf
-        '''
-        clone_dir = tempfile.mkdtemp()
-        try:
-            Task.fetch(clone_dir, self.package.git_url)
-            self.dependencies = Package.dependencies(clone_dir)
-        finally:
-            shutil.rmtree(clone_dir, ignore_errors=True)
-
 
 class Tree:
-    '''
+    """
     dependency tree implementation
     :ivar leaves: list of tree leaves
-    '''
+    """
 
-    def __init__(self) -> None:
-        '''
+    def __init__(self, leaves: List[Leaf]) -> None:
+        """
         default constructor
-        '''
-        self.leaves: List[Leaf] = []
+        :param leaves: leaves to build the tree
+        """
+        self.leaves = leaves
+
+    @classmethod
+    def load(cls: Type[Tree], packages: Iterable[Package]) -> Tree:
+        """
+        load tree from packages
+        :param packages: packages list
+        :return: loaded class
+        """
+        return cls([Leaf.load(package) for package in packages])
 
     def levels(self) -> List[List[Package]]:
-        '''
+        """
         get build levels starting from the packages which do not require any other package to build
         :return: list of packages lists
-        '''
+        """
         result: List[List[Package]] = []
 
         unprocessed = self.leaves[:]
@@ -98,13 +114,3 @@ class Tree:
             unprocessed = [leaf for leaf in unprocessed if not leaf.is_root(unprocessed)]
 
         return result
-
-    def load(self, packages: Iterable[Package]) -> None:
-        '''
-        load tree from packages
-        :param packages: packages list
-        '''
-        for package in packages:
-            leaf = Leaf(package)
-            leaf.load_dependencies()
-            self.leaves.append(leaf)

src/ahriman/core/upload/__init__.py

@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2021 Evgenii Alekseev.
+# Copyright (c) 2021 ahriman team.
 #
 # This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).