Release 0.13.0

* multi-sign and multi-web configuration
* change default configuration to do not use architecture
* change units to be templated
* some refactoring

CONFIGURING.md

@@ -1,19 +1,22 @@
 # ahriman configuration
 
-Some groups can be specified for each architecture separately with default values. E.g. if there are `build` and `build_x86_64` groups it will use the `build_x86_64` for the `x86_64` architecture and `build` for any other.
+Some groups can be specified for each architecture separately. E.g. if there are `build` and `build_x86_64` groups it will use the `build_x86_64` for the `x86_64` architecture and `build` for any other (architecture specific group has higher priority).
 
 ## `settings` group
 
-Base configuration settings:
+Base configuration settings.
 
 * `include` - path to directory with configuration files overrides, string, required.
 * `logging` - path to logging configuration, string, required. Check `logging.ini` for reference.
 
-## `aur` group
+## `alpm` group
 
-AUR related configuration:
+libalpm and AUR related configuration.
 
-* `url` - base url for AUR, string, required.
+* `aur_url` - base url for AUR, string, required.
+* `database` - path to pacman local database cache, string, required.
+* `repositories` - list of pacman repositories, space separated list of strings, required.
+* `root` - root for alpm library, string, required.
 
 ## `build_*` groups
 
@@ -27,25 +30,26 @@ Build related configuration. Group name must refer to architecture, e.g. it shou
 
 ## `repository` group
 
-Base repository settings:
+Base repository settings.
 
 * `name` - repository name, string, required.
 * `root` - root path for application, string, required.
 
-## `sign` group
+## `sign_*` groups
 
-Settings for signing packages or repository:
+Settings for signing packages or repository. Group name must refer to architecture, e.g. it should be `sign_x86_64` for x86_64 architecture.
 
-* `enabled` - configuration flag to enable signing, string, required. Allowed values are `disabled`, `package` (sign each package separately), `repository` (sign repository database file).
+* `target` - configuration flag to enable signing, space separated list of strings, required. Allowed values are `package` (sign each package separately), `repository` (sign repository database file).
-* `key` - PGP key, string, optional.
+* `key` - default PGP key, string, required. This key will also be used for database signing if enabled.
+* `key_*` settings - PGP key which will be used for specific packages, string, optional. For example, if there is `key_yay` option the specified key will be used for yay package and default key for others.
 
 ## `report` group
 
-Report generation settings:
+Report generation settings.
 
 * `target` - list of reports to be generated, space separated list of strings, optional. Allowed values are `html`.
 
-### `html_*` group
+### `html_*` groups
 
 Group name must refer to architecture, e.g. it should be `html_x86_64` for x86_64 architecture.
 
@@ -56,18 +60,26 @@ Group name must refer to architecture, e.g. it should be `html_x86_64` for x86_6
 
 ## `upload` group
 
-Remote synchronization settings:
+Remote synchronization settings.
 
 * `target` - list of synchronizations to be used, space separated list of strings, optional. Allowed values are `rsync`, `s3`.
 
-### `s3_*` group
+### `rsync_*` groups
+
+Group name must refer to architecture, e.g. it should be `rsync_x86_64` for x86_64 architecture. Requires `rsync` package to be installed. Do not forget to configure ssh for user `ahriman`.
+
+* `remote` - remote server to rsync (e.g. `1.2.3.4:5678:path/to/sync`), string, required.
+
+### `s3_*` groups
 
 Group name must refer to architecture, e.g. it should be `s3_x86_64` for x86_64 architecture. Requires `aws-cli` package to be installed. Do not forget to configure it for user `ahriman`.
 
 * `bucket` - bucket name (e.g. `s3://bucket/path`), string, required.
 
-### `rsync_*` group
+## `web_*` groups
 
-Group name must refer to architecture, e.g. it should be `rsync_x86_64` for x86_64 architecture. Requires `rsync` package to be installed. Do not forget to configure ssh for user `ahriman`.
+Web server settings. If any of `host`/`port` is not set, web integration will be disabled. Group name must refer to architecture, e.g. it should be `web_x86_64` for x86_64 architecture.
 
-* `remote` - remote server to rsync (e.g. `1.2.3.4:5678:path/to/sync`), string, required.
+* `host` - host to bind, string, optional.
+* `port` - port to bind, int, optional.
+* `templates` - path to templates directory, string, required.

README.md

@@ -2,20 +2,26 @@
 
 Wrapper for managing custom repository inspired by [repo-scripts](https://github.com/arcan1s/repo-scripts).
 
+## Features
+
+* Install-configure-forget manager for own repository
+* Multi-architecture support
+* VCS packages support
+* Sign support with gpg (repository, package, per package settings)
+* Synchronization to remote services (rsync, s3) and report generation (html)
+* Dependency manager
+* Repository status interface
+
 ## Installation and run
 
 * Install package as usual.
 * Change settings if required, see `CONFIGURING.md` for more details.
 * Create `/var/lib/ahriman/.makepkg.conf` with `makepkg.conf` overrides if required (at least you might want to set `PACKAGER`).
 * Configure build tools (it might be required if your package will use any custom repositories):
-    * create build command if required, e.g. `ln -s /usr/bin/archbuild /usr/local/bin/custom-x86_64-build` (you can choose any name for command);
+    * create build command, e.g. `ln -s /usr/bin/archbuild /usr/local/bin/custom-x86_64-build` (you can choose any name for command);
     * create configuration file, e.g. `cp /usr/share/devtools/pacman-{extra,custom}.conf`;
-    * change configuration file: add your own repository, add multilib repository;
+    * change configuration file, add your own repository, add multilib repository etc;
-    * set `build.build_command` to point to your command;
+    * set `build.build_command` setting to point to your command;
-    * configure `/etc/sudoers.d/ahriman` to allow to run command without password.
+    * configure `/etc/sudoers.d/ahriman` to allow running command without password.
 * Start and enable `ahriman.timer` via `systemctl`.
 * Add packages by using `ahriman add {package}` command.
-
-## Limitations
-
-* It does not manage dependencies, so you have to add them before main package.

package/archlinux/PKGBUILD

@@ -1,28 +1,29 @@
 # Maintainer: Evgeniy Alekseev
 
 pkgname='ahriman'
-pkgver=0.9.1
+pkgver=0.13.0
 pkgrel=1
 pkgdesc="ArcHlinux ReposItory MANager"
 arch=('any')
 url="https://github.com/arcan1s/ahriman"
 license=('GPL3')
-depends=('devtools' 'expac' 'git' 'python-aur' 'python-srcinfo')
+depends=('devtools' 'git' 'pyalpm' 'python-aur' 'python-srcinfo')
 makedepends=('python-pip')
 optdepends=('aws-cli: sync to s3'
             'breezy: -bzr packages support'
             'darcs: -darcs packages support'
             'gnupg: package and repository sign'
             'mercurial: -hg packages support'
+            'python-aiohttp: web server'
+            'python-aiohttp-jinja2: web server'
             'python-jinja: html report generation'
+            'python-requests: web server'
             'rsync: sync by using rsync'
             'subversion: -svn packages support')
 source=("https://github.com/arcan1s/ahriman/releases/download/$pkgver/$pkgname-$pkgver-src.tar.xz"
-        'ahriman.sudoers'
         'ahriman.sysusers'
         'ahriman.tmpfiles')
-sha512sums=('82a8208554956f009db0334b0cb20891889d96617b4c9b9d2af7a007d668f6cbc6d46d0be8b5ee11ffb2b69d124b5d5d6db1a6f7d5a0a2c719c0d8e07dca24d8'
+sha512sums=('b835d745fb77e400ca31ba4d93547b7db8e9dfe5d6c04b60e3953efeeaa7f561a1c60b2ade2684d3c7ba9a87e470c65610f33340315f192661c1676746b91298'
-            '8c9b5b63ac3f7b4d9debaf801a1e9c060877c33d3ecafe18010fcca778e5fa2f2e46909d3d0ff1b229ff8aa978445d8243fd36e1fc104117ed678d5e21901167'
             '13718afec2c6786a18f0b223ef8e58dccf0688bca4cdbe203f14071f5031ed20120eb0ce38b52c76cfd6e8b6581a9c9eaa2743eb11abbaca637451a84c33f075'
             '55b20f6da3d66e7bbf2add5d95a3b60632df121717d25a993e56e737d14f51fe063eb6f1b38bd81cc32e05db01c0c1d80aaa720c45cde87f238d8b46cdb8cbc4')
 backup=('etc/ahriman.ini'
@@ -39,7 +40,6 @@ package() {
 
   python setup.py install --root="$pkgdir"
 
-  install -Dm400 "$srcdir/$pkgname.sudoers" "$pkgdir/etc/sudoers.d/$pkgname"
   install -Dm644 "$srcdir/$pkgname.sysusers" "$pkgdir/usr/lib/sysusers.d/$pkgname.conf"
   install -Dm644 "$srcdir/$pkgname.tmpfiles" "$pkgdir/usr/lib/tmpfiles.d/$pkgname.conf"
 }

package/archlinux/ahriman.sudoers

@@ -1,4 +0,0 @@
-# Used by ArcHlinux ReposItory MANager with default settings
-
-Cmnd_Alias ARCHBUILD_CMD = /usr/bin/extra-x86_64-build *, /usr/bin/multilib-build *
-ahriman ALL=(ALL) NOPASSWD: ARCHBUILD_CMD

package/etc/ahriman.ini

@@ -2,10 +2,13 @@
 include = /etc/ahriman.ini.d
 logging = /etc/ahriman.ini.d/logging.ini
 
-[aur]
+[alpm]
-url = https://aur.archlinux.org
+aur_url = https://aur.archlinux.org
+database = /var/lib/pacman
+repositories = core extra community multilib
+root = /
 
-[build_x86_64]
+[build]
 archbuild_flags =
 build_command = extra-x86_64-build
 ignore_packages =
@@ -17,23 +20,28 @@ name = aur-clone
 root = /var/lib/ahriman
 
 [sign]
-enabled = disabled
+target =
 key =
 
 [report]
 target =
 
-[html_x86_64]
+[html]
 path =
 homepage =
 link_path =
-template_path = /usr/share/ahriman/index.jinja2
+template_path = /usr/share/ahriman/repo-index.jinja2
 
 [upload]
 target =
 
-[s3_x86_64]
+[rsync]
+remote =
+
+[s3]
 bucket =
 
-[rsync_x86_64]
+[web]
-remote =
+host =
+port =
+templates = /usr/share/ahriman

package/etc/ahriman.ini.d/logging.ini

@@ -1,8 +1,8 @@
 [loggers]
-keys = root,builder,build_details
+keys = root,builder,build_details,http
 
 [handlers]
-keys = console_handler,build_file_handler,file_handler
+keys = console_handler,build_file_handler,file_handler,http_handler
 
 [formatters]
 keys = generic_format
@@ -25,6 +25,12 @@ level = DEBUG
 formatter = generic_format
 args = ('/var/log/ahriman/build.log', 'a', 20971520, 20)
 
+[handler_http_handler]
+class = logging.handlers.RotatingFileHandler
+level = DEBUG
+formatter = generic_format
+args = ('/var/log/ahriman/http.log', 'a', 20971520, 20)
+
 [formatter_generic_format]
 format = %(asctime)s : %(levelname)s : %(funcName)s : %(message)s
 datefmt =
@@ -45,3 +51,9 @@ level = DEBUG
 handlers = build_file_handler
 qualname = build_details
 propagate = 0
+
+[logger_http]
+level = DEBUG
+handlers = http_handler
+qualname = http
+propagate = 0

package/lib/systemd/system/ahriman-web@.service

@@ -0,0 +1,15 @@
+[Unit]
+Description=ArcHlinux ReposItory MANager web server (%I architecture)
+After=network.target
+
+[Service]
+Type=simple
+ExecStart=/usr/bin/ahriman --architecture %i web
+User=ahriman
+Group=ahriman
+
+KillSignal=SIGQUIT
+SuccessExitStatus=SIGQUIT
+
+[Install]
+WantedBy=multi-user.target

package/lib/systemd/system/ahriman.service

@@ -1,7 +0,0 @@
-[Unit]
-Description=ArcHlinux ReposItory MANager
-
-[Service]
-ExecStart=/usr/bin/ahriman --architecture x86_64 update
-User=ahriman
-Group=ahriman

package/lib/systemd/system/ahriman@.service

@@ -0,0 +1,7 @@
+[Unit]
+Description=ArcHlinux ReposItory MANager (%I architecture)
+
+[Service]
+ExecStart=/usr/bin/ahriman --architecture %i update
+User=ahriman
+Group=ahriman

package/lib/systemd/system/ahriman@.timer

@@ -1,5 +1,5 @@
 [Unit]
-Description=ArcHlinux ReposItory MANager timer
+Description=ArcHlinux ReposItory MANager timer (%I architecture)
 
 [Timer]
 OnCalendar=daily

package/share/ahriman/build-status.jinja2

@@ -0,0 +1,54 @@
+<!doctype html>
+<html lang="en">
+    <head>
+        <title>{{ repository|e }}</title>
+
+        {% include "style.jinja2" %}
+
+        {% include "sorttable.jinja2" %}
+        {% include "search.jinja2" %}
+    </head>
+
+    <body>
+        <div class="root">
+            <h1>ahriman
+                <img src="https://img.shields.io/badge/version-{{ version|e }}-informational" alt="{{ version|e }}">
+                <img src="https://img.shields.io/badge/architecture-{{ architecture|e }}-informational" alt="{{ architecture|e }}">
+                <img src="https://img.shields.io/badge/service%20status-{{ service.status|e }}-{{ service.status_color|e }}" alt="{{ service.status|e }}" title="{{ service.timestamp|e }}">
+            </h1>
+
+            {% include "search-line.jinja2" %}
+
+            <section class="element">
+                <table class="sortable search-table">
+                    <tr class="header">
+                        <th>package base</th>
+                        <th>packages</th>
+                        <th>version</th>
+                        <th>last update</th>
+                        <th>status</th>
+                    </tr>
+
+                    {% for package in packages %}
+                        <tr class="package">
+                            <td class="include-search"><a href="{{ package.web_url|e }}" title="{{ package.base|e }}">{{ package.base|e }}</a></td>
+                            <td class="include-search">{{ package.packages|join("<br>"|safe) }}</td>
+                            <td>{{ package.version|e }}</td>
+                            <td>{{ package.timestamp|e }}</td>
+                            <td class="status package-{{ package.status|e }}">{{ package.status|e }}</td>
+                        </tr>
+                    {% endfor %}
+                </table>
+            </section>
+
+            <footer>
+                <ul class="navigation">
+                    <li><a href="https://github.com/arcan1s/ahriman" title="sources">ahriman</a></li>
+                    <li><a href="https://github.com/arcan1s/ahriman/releases" title="releases list">releases</a></li>
+                    <li><a href="https://github.com/arcan1s/ahriman/issues" title="issues tracker">report a bug</a></li>
+                </ul>
+            </footer>
+        </div>
+    </body>
+
+</html>

package/share/ahriman/index.jinja2

@@ -1,31 +0,0 @@
-<!doctype html>
-<html lang="en">
-    <head>
-        <title>{{ repository|e }}</title>
-    </head>
-
-    <body>
-        <h1>{{ repository|e }} ArchLinux custom repository</h1>
-
-        {% if pgp_key is not none %}
-            <p>All packages are signed with <a href="http://keys.gnupg.net/pks/lookup?search=0x{{ pgp_key|e }}" title="key search">{{ pgp_key|e }}</a>.</p>
-        {% endif %}
-
-        <code>
-            $ cat /etc/pacman.conf<br>
-            [{{ repository|e }}]<br>
-            Server = {{ link_path|e }}
-        </code>
-
-        <p>Packages:</p>
-        <ul>
-            {% for package, package_url in packages.items() %}
-                <li><a href="{{ package_url|e }}" title="{{ package|e }}">{{ package|e }}</a></li>
-            {% endfor %}
-        </ul>
-
-        {% if homepage is not none %}
-            <footer><a href="{{ homepage|e }}" title="homepage">Homepage</a></footer>
-        {% endif %}
-    </body>
-</html>

package/share/ahriman/repo-index.jinja2

@@ -0,0 +1,62 @@
+<!doctype html>
+<html lang="en">
+    <head>
+        <title>{{ repository|e }}</title>
+
+        {% include "style.jinja2" %}
+
+        {% include "sorttable.jinja2" %}
+        {% include "search.jinja2" %}
+    </head>
+
+    <body>
+        <div class="root">
+            <h1>Archlinux user repository</h1>
+
+            <section class="element">
+                {% if pgp_key is not none %}
+                    <p>This repository is signed with <a href="http://keys.gnupg.net/pks/lookup?search=0x{{ pgp_key|e }}&fingerprint=on&op=index" title="key search">{{ pgp_key|e }}</a> by default.</p>
+                {% endif %}
+
+                <code>
+                    $ cat /etc/pacman.conf<br>
+                    [{{ repository|e }}]<br>
+                    Server = {{ link_path|e }}<br>
+                    SigLevel = Database{% if has_repo_signed %}Required{% else %}Never{% endif %} Package{% if has_package_signed %}Required{% else %}Never{% endif %} TrustedOnly
+                </code>
+            </section>
+
+            {% include "search-line.jinja2" %}
+
+            <section class="element">
+                <table class="sortable search-table">
+                    <tr class="header">
+                        <th>package</th>
+                        <th>version</th>
+                        <th>archive size</th>
+                        <th>installed size</th>
+                        <th>build date</th>
+                    </tr>
+
+                    {% for package in packages %}
+                        <tr class="package">
+                            <td class="include-search"><a href="{{ link_path|e }}/{{ package.filename|e }}" title="{{ package.name|e }}">{{ package.name|e }}</a></td>
+                            <td>{{ package.version|e }}</td>
+                            <td>{{ package.archive_size|e }}</td>
+                            <td>{{ package.installed_size|e }}</td>
+                            <td>{{ package.build_date|e }}</td>
+                        </tr>
+                    {% endfor %}
+                </table>
+            </section>
+
+            <footer>
+                <ul class="navigation">
+                    {% if homepage is not none %}
+                        <li><a href="{{ homepage|e }}" title="homepage">Homepage</a></li>
+                    {% endif %}
+                </ul>
+            </footer>
+        </div>
+    </body>
+</html>

package/share/ahriman/search-line.jinja2

@@ -0,0 +1,3 @@
+<section class="element">
+    <input type="search" id="search" onkeyup="searchInTable()" placeholder="search for package" title="search for package"/>
+</section>

package/share/ahriman/search.jinja2

@@ -0,0 +1,25 @@
+<script type="text/javascript">
+    function searchInTable() {
+        const input = document.getElementById("search");
+        const filter = input.value.toLowerCase();
+        const tables = document.getElementsByClassName("search-table");
+
+        for (let i = 0; i < tables.length; i++) {
+            const tr = tables[i].getElementsByTagName("tr");
+            // from 1 coz of header
+            for (let i = 1; i < tr.length; i++) {
+                let td = tr[i].getElementsByClassName("include-search");
+                let display = "none";
+                for (let j = 0; j < td.length; j++) {
+                    if (td[j].tagName.toLowerCase() === "td") {
+                        if (td[j].innerHTML.toLowerCase().indexOf(filter) > -1) {
+                            display = "";
+                            break;
+                        }
+                    }
+                }
+                tr[i].style.display = display;
+            }
+        }
+    }
+</script>

package/share/ahriman/sorttable.jinja2

@@ -0,0 +1 @@
+<script src="https://www.kryogenix.org/code/browser/sorttable/sorttable.js"></script>

package/share/ahriman/style.jinja2

@@ -0,0 +1,136 @@
+<style>
+    :root {
+        --color-building: 255, 255, 146;
+        --color-failed: 255, 94, 94;
+        --color-pending: 255, 255, 146;
+        --color-success: 94, 255, 94;
+        --color-unknown: 225, 225, 225;
+
+        --color-header: 200, 200, 255;
+        --color-hover: 255, 255, 225;
+        --color-line-blue: 235, 235, 255;
+        --color-line-white: 255, 255, 255;
+    }
+
+    @keyframes blink-building {
+        0% { background-color: rgba(var(--color-building), 1.0); }
+        10% { background-color: rgba(var(--color-building), 0.9); }
+        20% { background-color: rgba(var(--color-building), 0.8); }
+        30% { background-color: rgba(var(--color-building), 0.7); }
+        40% { background-color: rgba(var(--color-building), 0.6); }
+        50% { background-color: rgba(var(--color-building), 0.5); }
+        60% { background-color: rgba(var(--color-building), 0.4); }
+        70% { background-color: rgba(var(--color-building), 0.3); }
+        80% { background-color: rgba(var(--color-building), 0.2); }
+        90% { background-color: rgba(var(--color-building), 0.1); }
+        100% { background-color: rgba(var(--color-building), 0.0); }
+    }
+
+    div.root {
+        width: 70%;
+        padding: 15px 15% 0;
+    }
+
+    section.element, footer {
+        width: 100%;
+        padding: 10px 0;
+    }
+
+    code, input, table {
+        width: inherit;
+    }
+
+    th, td {
+        padding: 5px;
+    }
+
+    tr.package:nth-child(odd) {
+        background-color: rgba(var(--color-line-white), 1.0);
+    }
+
+    tr.package:nth-child(even) {
+        background-color: rgba(var(--color-line-blue), 1.0);
+    }
+
+    tr.package:hover {
+        background-color: rgba(var(--color-hover), 1.0);
+    }
+
+    tr.header{
+        background-color: rgba(var(--color-header), 1.0);
+    }
+
+    td.status {
+        text-align: center;
+    }
+
+    td.package-unknown {
+        background-color: rgba(var(--color-unknown), 1.0);
+    }
+    td.package-pending {
+        background-color: rgba(var(--color-pending), 1.0);
+    }
+    td.package-building {
+        background-color: rgba(var(--color-building), 1.0);
+        animation-name: blink-building;
+        animation-duration: 1s;
+        animation-timing-function: linear;
+        animation-iteration-count: infinite;
+        animation-direction: alternate;
+    }
+    td.package-failed {
+        background-color: rgba(var(--color-failed), 1.0);
+    }
+    td.package-success {
+        background-color: rgba(var(--color-success), 1.0);
+    }
+
+    li.service-unknown {
+        background-color: rgba(var(--color-unknown), 1.0);
+    }
+    li.service-building {
+        background-color: rgba(var(--color-building), 1.0);
+        animation-name: blink-building;
+        animation-duration: 1s;
+        animation-timing-function: linear;
+        animation-iteration-count: infinite;
+        animation-direction: alternate;
+    }
+    li.service-failed {
+        background-color: rgba(var(--color-failed), 1.0);
+    }
+    li.service-success {
+        background-color: rgba(var(--color-success), 1.0);
+    }
+
+    ul.navigation {
+        list-style-type: none;
+        margin: 0;
+        padding: 0;
+        overflow: hidden;
+        background-color: rgba(var(--color-header), 1.0);
+    }
+
+    ul.navigation li {
+        float: left;
+    }
+
+    ul.navigation li.status {
+        display: block;
+        text-align: center;
+        text-decoration: none;
+        padding: 14px 16px;
+    }
+
+    ul.navigation li a {
+        display: block;
+        color: black;
+        text-align: center;
+        text-decoration: none;
+        padding: 14px 16px;
+    }
+
+    ul.navigation li a:hover {
+        background-color: rgba(var(--color-hover), 1.0);
+    }
+</style>

setup.py

@@ -28,6 +28,7 @@ setup(
     ],
     install_requires=[
         'aur',
+        'pyalpm',
         'srcinfo',
     ],
     setup_requires=[
@@ -42,17 +43,30 @@ setup(
         'package/bin/ahriman',
     ],
     data_files=[
-        ('/etc', ['package/etc/ahriman.ini']),
+        ('/etc', [
-        ('/etc/ahriman.ini.d', ['package/etc/ahriman.ini.d/logging.ini']),
+            'package/etc/ahriman.ini',
-        ('lib/systemd/system', [
+        ]),
-            'package/lib/systemd/system/ahriman.service',
+        ('/etc/ahriman.ini.d', [
-            'package/lib/systemd/system/ahriman.timer'
+            'package/etc/ahriman.ini.d/logging.ini',
+        ]),
+        ('lib/systemd/system', [
+            'package/lib/systemd/system/ahriman@.service',
+            'package/lib/systemd/system/ahriman@.timer',
+            'package/lib/systemd/system/ahriman-web@.service',
+        ]),
+        ('share/ahriman', [
+            'package/share/ahriman/build-status.jinja2',
+            'package/share/ahriman/repo-index.jinja2',
+            'package/share/ahriman/search.jinja2',
+            'package/share/ahriman/search-line.jinja2',
+            'package/share/ahriman/sorttable.jinja2',
+            'package/share/ahriman/style.jinja2',
         ]),
-        ('share/ahriman', ['package/share/ahriman/index.jinja2']),
     ],
 
     extras_require={
         'html-templates': ['Jinja2'],
         'test': ['coverage', 'pytest'],
+        'web': ['Jinja2', 'aiohttp', 'aiohttp_jinja2', 'requests'],
     },
 )

src/ahriman/__init__.py

@@ -1,7 +1,7 @@
 #
 # Copyright (c) 2021 Evgenii Alekseev.
 #
-# This file is part of ahriman 
+# This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).
 #
 # This program is free software: you can redistribute it and/or modify

src/ahriman/application/__init__.py

@@ -1,7 +1,7 @@
 #
 # Copyright (c) 2021 Evgenii Alekseev.
 #
-# This file is part of ahriman 
+# This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).
 #
 # This program is free software: you can redistribute it and/or modify

src/ahriman/application/ahriman.py

@@ -1,7 +1,7 @@
 #
 # Copyright (c) 2021 Evgenii Alekseev.
 #
-# This file is part of ahriman 
+# This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).
 #
 # This program is free software: you can redistribute it and/or modify
@@ -18,65 +18,123 @@
 # along with this program. If not, see <http://www.gnu.org/licenses/>.
 #
 import argparse
-import os
+
+from multiprocessing import Pool
 
 import ahriman.version as version
 
 from ahriman.application.application import Application
+from ahriman.application.lock import Lock
 from ahriman.core.configuration import Configuration
 
 
-def _get_app(args: argparse.Namespace) -> Application:
+def _call(args: argparse.Namespace, architecture: str, config: Configuration) -> None:
-    config = _get_config(args.config)
+    '''
-    return Application(args.architecture, config)
+    additional function to wrap all calls for multiprocessing library
+    :param args: command line args
+    :param architecture: repository architecture
+    :param config: configuration instance
+    '''
+    with Lock(args.lock, architecture, args.force, config):
+        args.fn(args, architecture, config)
 
 
-def _get_config(config_path: str) -> Configuration:
+def add(args: argparse.Namespace, architecture: str, config: Configuration) -> None:
-    config = Configuration()
+    '''
-    config.load(config_path)
+    add packages callback
-    config.load_logging()
+    :param args: command line args
-    return config
+    :param architecture: repository architecture
+    :param config: configuration instance
+    '''
+    Application(architecture, config).add(args.package, args.without_dependencies)
 
 
-def _remove_lock(path: str) -> None:
+def clean(args: argparse.Namespace, architecture: str, config: Configuration) -> None:
-    if os.path.exists(path):
+    '''
-        os.remove(path)
+    clean caches callback
+    :param args: command line args
+    :param architecture: repository architecture
+    :param config: configuration instance
+    '''
+    Application(architecture, config).clean()
 
 
-def add(args: argparse.Namespace) -> None:
+def rebuild(args: argparse.Namespace, architecture: str, config: Configuration) -> None:
-    _get_app(args).add(args.package)
+    '''
-
+    world rebuild callback
-
+    :param args: command line args
-def rebuild(args: argparse.Namespace) -> None:
+    :param architecture: repository architecture
-    app = _get_app(args)
+    :param config: configuration instance
+    '''
+    app = Application(architecture, config)
     packages = app.repository.packages()
     app.update(packages)
 
 
-def remove(args: argparse.Namespace) -> None:
+def remove(args: argparse.Namespace, architecture: str, config: Configuration) -> None:
-    _get_app(args).remove(args.package)
+    '''
+    remove packages callback
+    :param args: command line args
+    :param architecture: repository architecture
+    :param config: configuration instance
+    '''
+    Application(architecture, config).remove(args.package)
 
 
-def report(args: argparse.Namespace) -> None:
+def report(args: argparse.Namespace, architecture: str, config: Configuration) -> None:
-    _get_app(args).report(args.target)
+    '''
+    generate report callback
+    :param args: command line args
+    :param architecture: repository architecture
+    :param config: configuration instance
+    '''
+    Application(architecture, config).report(args.target)
 
 
-def sync(args: argparse.Namespace) -> None:
+def sync(args: argparse.Namespace, architecture: str, config: Configuration) -> None:
-    _get_app(args).sync(args.target)
+    '''
+    sync to remote server callback
+    :param args: command line args
+    :param architecture: repository architecture
+    :param config: configuration instance
+    '''
+    Application(architecture, config).sync(args.target)
 
 
-def update(args: argparse.Namespace) -> None:
+def update(args: argparse.Namespace, architecture: str, config: Configuration) -> None:
-    app = _get_app(args)
+    '''
-    log_fn = lambda line: print(line) if args.dry_run else app.logger.info(line)
+    update packages callback
-    packages = app.get_updates(args.no_aur, args.no_manual, args.no_vcs, log_fn)
+    :param args: command line args
+    :param architecture: repository architecture
+    :param config: configuration instance
+    '''
+    # typing workaround
+    def log_fn(line: str) -> None:
+        return print(line) if args.dry_run else app.logger.info(line)
+
+    app = Application(architecture, config)
+    packages = app.get_updates(args.package, args.no_aur, args.no_manual, args.no_vcs, log_fn)
     if args.dry_run:
         return
+
     app.update(packages)
 
 
+def web(args: argparse.Namespace, architecture: str, config: Configuration) -> None:
+    '''
+    web server callback
+    :param args: command line args
+    :param architecture: repository architecture
+    :param config: configuration instance
+    '''
+    from ahriman.web.web import run_server, setup_service
+    app = setup_service(architecture, config)
+    run_server(app, architecture)
+
+
 if __name__ == '__main__':
     parser = argparse.ArgumentParser(prog='ahriman', description='ArcHlinux ReposItory MANager')
-    parser.add_argument('-a', '--architecture', help='target architecture', required=True)
+    parser.add_argument('-a', '--architecture', help='target architectures', action='append')
     parser.add_argument('-c', '--config', help='configuration path', default='/etc/ahriman.ini')
     parser.add_argument('--force', help='force run, remove file lock', action='store_true')
     parser.add_argument('--lock', help='lock file', default='/tmp/ahriman.lock')
@@ -85,10 +143,16 @@ if __name__ == '__main__':
 
     add_parser = subparsers.add_parser('add', description='add package')
     add_parser.add_argument('package', help='package name or archive path', nargs='+')
+    add_parser.add_argument('--without-dependencies', help='do not add dependencies', action='store_true')
     add_parser.set_defaults(fn=add)
 
     check_parser = subparsers.add_parser('check', description='check for updates')
-    check_parser.set_defaults(fn=update, no_aur=False, no_manual=True, no_vcs=False, dry_run=True)
+    check_parser.add_argument('package', help='filter check by packages', nargs='*')
+    check_parser.add_argument('--no-vcs', help='do not check VCS packages', action='store_true')
+    check_parser.set_defaults(fn=update, no_aur=False, no_manual=True, dry_run=True)
+
+    clean_parser = subparsers.add_parser('clean', description='clear all local caches')
+    clean_parser.set_defaults(fn=clean)
 
     rebuild_parser = subparsers.add_parser('rebuild', description='rebuild whole repository')
     rebuild_parser.set_defaults(fn=rebuild)
@@ -106,26 +170,22 @@ if __name__ == '__main__':
     sync_parser.set_defaults(fn=sync)
 
     update_parser = subparsers.add_parser('update', description='run updates')
-    update_parser.add_argument('--dry-run', help='just perform check for updates, same as check command', action='store_true')
+    update_parser.add_argument('package', help='filter check by packages', nargs='*')
+    update_parser.add_argument(
+        '--dry-run', help='just perform check for updates, same as check command', action='store_true')
     update_parser.add_argument('--no-aur', help='do not check for AUR updates', action='store_true')
     update_parser.add_argument('--no-manual', help='do not include manual updates', action='store_true')
     update_parser.add_argument('--no-vcs', help='do not check VCS packages', action='store_true')
     update_parser.set_defaults(fn=update)
 
+    web_parser = subparsers.add_parser('web', description='start web server')
+    web_parser.set_defaults(fn=web, lock=None)
+
     args = parser.parse_args()
-
-    if args.force:
-        _remove_lock(args.lock)
-    if os.path.exists(args.lock):
-        raise RuntimeError('Another application instance is run')
-
     if 'fn' not in args:
         parser.print_help()
         exit(1)
 
-    try:
+    config = Configuration.from_path(args.config)
-        open(args.lock, 'w').close()
+    with Pool(len(args.architecture)) as pool:
-        args.fn(args)
+        pool.starmap(_call, [(args, architecture, config) for architecture in args.architecture])
-    finally:
-        _remove_lock(args.lock)
-

src/ahriman/application/application.py

@@ -1,7 +1,7 @@
 #
 # Copyright (c) 2021 Evgenii Alekseev.
 #
-# This file is part of ahriman 
+# This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).
 #
 # This program is free software: you can redistribute it and/or modify
@@ -21,32 +21,69 @@ import logging
 import os
 import shutil
 
-from typing import Callable, List, Optional
+from typing import Callable, Iterable, List, Optional, Set
 
 from ahriman.core.build_tools.task import Task
 from ahriman.core.configuration import Configuration
 from ahriman.core.repository import Repository
+from ahriman.core.tree import Tree
 from ahriman.models.package import Package
 
 
 class Application:
+    '''
+    base application class
+    :ivar architecture: repository architecture
+    :ivar config: configuration instance
+    :ivar logger: application logger
+    :ivar repository: repository instance
+    '''
 
     def __init__(self, architecture: str, config: Configuration) -> None:
+        '''
+        default constructor
+        :param architecture: repository architecture
+        :param config: configuration instance
+        '''
         self.logger = logging.getLogger('root')
         self.config = config
         self.architecture = architecture
         self.repository = Repository(architecture, config)
 
+    def _known_packages(self) -> Set[str]:
+        '''
+        load packages from repository and pacman repositories
+        :return: list of known packages
+        '''
+        known_packages: Set[str] = set()
+        # local set
+        for package in self.repository.packages():
+            known_packages.update(package.packages.keys())
+        known_packages.update(self.repository.pacman.all_packages())
+        return known_packages
+
     def _finalize(self) -> None:
+        '''
+        generate report and sync to remote server
+        '''
         self.report()
         self.sync()
 
-    def get_updates(self, no_aur: bool, no_manual: bool, no_vcs: bool,
+    def get_updates(self, filter_packages: List[str], no_aur: bool, no_manual: bool, no_vcs: bool,
                     log_fn: Callable[[str], None]) -> List[Package]:
+        '''
+        get list of packages to run update process
+        :param filter_packages: do not check every package just specified in the list
+        :param no_aur: do not check for aur updates
+        :param no_manual: do not check for manual updates
+        :param no_vcs: do not check VCS packages
+        :param log_fn: logger function to log updates
+        :return: list of out-of-dated packages
+        '''
         updates = []
 
         if not no_aur:
-            updates.extend(self.repository.updates_aur(no_vcs))
+            updates.extend(self.repository.updates_aur(filter_packages, no_vcs))
         if not no_manual:
             updates.extend(self.repository.updates_manual())
 
@@ -55,35 +92,91 @@ class Application:
 
         return updates
 
-    def add(self, names: List[str]) -> None:
+    def add(self, names: Iterable[str], without_dependencies: bool) -> None:
-        def add_manual(name: str) -> None:
+        '''
-            package = Package.load(name, self.config.get('aur', 'url'))
+        add packages for the next build
-            Task.fetch(os.path.join(self.repository.paths.manual, package.base), package.url)
+        :param names: list of package bases to add
+        :param without_dependencies: if set, dependency check will be disabled
+        '''
+        known_packages = self._known_packages()
+
+        def add_manual(name: str) -> str:
+            package = Package.load(name, self.repository.pacman, self.config.get('alpm', 'aur_url'))
+            path = os.path.join(self.repository.paths.manual, package.base)
+            Task.fetch(path, package.git_url)
+            return path
 
         def add_archive(src: str) -> None:
             dst = os.path.join(self.repository.paths.packages, os.path.basename(src))
             shutil.move(src, dst)
 
-        for name in names:
+        def process_dependencies(path: str) -> None:
-            if os.path.isfile(name):
+            if without_dependencies:
-                add_archive(name)
+                return
-            else:
+            dependencies = Package.dependencies(path)
-                add_manual(name)
+            self.add(dependencies.difference(known_packages), without_dependencies)
 
-    def remove(self, names: List[str]) -> None:
+        def process_single(name: str) -> None:
+            if not os.path.isfile(name):
+                path = add_manual(name)
+                process_dependencies(path)
+            else:
+                add_archive(name)
+
+        for name in names:
+            process_single(name)
+
+    def clean(self) -> None:
+        '''
+        run all clean methods
+        '''
+        self.repository._clear_build()
+        self.repository._clear_cache()
+        self.repository._clear_chroot()
+        self.repository._clear_manual()
+        self.repository._clear_packages()
+
+    def remove(self, names: Iterable[str]) -> None:
+        '''
+        remove packages from repository
+        :param names: list of packages (either base or name) to remove
+        '''
         self.repository.process_remove(names)
         self._finalize()
 
-    def report(self, target: Optional[List[str]] = None) -> None:
+    def report(self, target: Optional[Iterable[str]] = None) -> None:
+        '''
+        generate report
+        :param target: list of targets to run (e.g. html)
+        '''
         targets = target or None
         self.repository.process_report(targets)
 
-    def sync(self, target: Optional[List[str]] = None) -> None:
+    def sync(self, target: Optional[Iterable[str]] = None) -> None:
+        '''
+        sync to remote server
+        :param target: list of targets to run (e.g. s3)
+        '''
         targets = target or None
         self.repository.process_sync(targets)
 
-    def update(self, updates: List[Package]) -> None:
+    def update(self, updates: Iterable[Package]) -> None:
-        packages = self.repository.process_build(updates)
+        '''
-        self.repository.process_update(packages)
+        run package updates
-        self._finalize()
+        :param updates: list of packages to update
+        '''
+        def process_update(paths: Iterable[str]) -> None:
+            self.repository.process_update(paths)
+            self._finalize()
 
+        # process built packages
+        packages = self.repository.packages_built()
+        process_update(packages)
+
+        # process manual packages
+        tree = Tree()
+        tree.load(updates)
+        for num, level in enumerate(tree.levels()):
+            self.logger.info(f'processing level #{num} {[package.base for package in level]}')
+            packages = self.repository.process_build(level)
+            process_update(packages)

src/ahriman/application/lock.py

@@ -0,0 +1,108 @@
+#
+# Copyright (c) 2021 Evgenii Alekseev.
+#
+# This file is part of ahriman
+# (see https://github.com/arcan1s/ahriman).
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+from __future__ import annotations
+
+import os
+
+from types import TracebackType
+from typing import Literal, Optional, Type
+
+from ahriman.core.configuration import Configuration
+from ahriman.core.exceptions import DuplicateRun
+from ahriman.core.watcher.client import Client
+from ahriman.models.build_status import BuildStatusEnum
+
+
+class Lock:
+    '''
+    wrapper for application lock file
+    :ivar force: remove lock file on start if any
+    :ivar path: path to lock file if any
+    :ivar reporter: build status reporter instance
+    '''
+
+    def __init__(self, path: Optional[str], architecture: str, force: bool, config: Configuration) -> None:
+        '''
+        default constructor
+        :param path: optional path to lock file, if empty no file lock will be used
+        :param architecture: repository architecture
+        :param force: remove lock file on start if any
+        :param config: configuration instance
+        '''
+        self.path = f'{path}_{architecture}' if path is not None else None
+        self.force = force
+
+        self.reporter = Client.load(architecture, config)
+
+    def __enter__(self) -> Lock:
+        '''
+        default workflow is the following:
+
+            remove lock file if force flag is set
+            check if there is lock file
+            create lock file
+            report to web if enabled
+        '''
+        if self.force:
+            self.remove()
+        self.check()
+        self.create()
+        self.reporter.update_self(BuildStatusEnum.Building)
+        return self
+
+    def __exit__(self, exc_type: Optional[Type[Exception]], exc_val: Optional[Exception],
+                 exc_tb: TracebackType) -> Literal[False]:
+        '''
+        remove lock file when done
+        :param exc_type: exception type name if any
+        :param exc_val: exception raised if any
+        :param exc_tb: exception traceback if any
+        :return: always False (do not suppress any exception)
+        '''
+        self.remove()
+        status = BuildStatusEnum.Success if exc_val is None else BuildStatusEnum.Failed
+        self.reporter.update_self(status)
+        return False
+
+    def check(self) -> None:
+        '''
+        check if lock file exists, raise exception if it does
+        '''
+        if self.path is None:
+            return
+        if os.path.exists(self.path):
+            raise DuplicateRun()
+
+    def create(self) -> None:
+        '''
+        create lock file
+        '''
+        if self.path is None:
+            return
+        open(self.path, 'w').close()
+
+    def remove(self) -> None:
+        '''
+        remove lock file
+        '''
+        if self.path is None:
+            return
+        if os.path.exists(self.path):
+            os.remove(self.path)

src/ahriman/core/__init__.py

@@ -1,7 +1,7 @@
 #
 # Copyright (c) 2021 Evgenii Alekseev.
 #
-# This file is part of ahriman 
+# This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).
 #
 # This program is free software: you can redistribute it and/or modify

src/ahriman/core/alpm/__init__.py

@@ -1,7 +1,7 @@
 #
 # Copyright (c) 2021 Evgenii Alekseev.
 #
-# This file is part of ahriman 
+# This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).
 #
 # This program is free software: you can redistribute it and/or modify

src/ahriman/core/alpm/pacman.py

@@ -0,0 +1,52 @@
+#
+# Copyright (c) 2021 Evgenii Alekseev.
+#
+# This file is part of ahriman
+# (see https://github.com/arcan1s/ahriman).
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+from pyalpm import Handle  # type: ignore
+from typing import List, Set
+
+from ahriman.core.configuration import Configuration
+
+
+class Pacman:
+    '''
+    alpm wrapper
+    :ivar handle: pyalpm root `Handle`
+    '''
+
+    def __init__(self, config: Configuration) -> None:
+        '''
+        default constructor
+        :param config: configuration instance
+        '''
+        root = config.get('alpm', 'root')
+        pacman_root = config.get('alpm', 'database')
+        self.handle = Handle(root, pacman_root)
+        for repository in config.getlist('alpm', 'repositories'):
+            self.handle.register_syncdb(repository, 0)  # 0 is pgp_level
+
+    def all_packages(self) -> List[str]:
+        '''
+        get list of packages known for alpm
+        :return: list of package names
+        '''
+        result: Set[str] = set()
+        for database in self.handle.get_syncdbs():
+            result.update({package.name for package in database.pkgcache})
+
+        return list(result)

src/ahriman/core/alpm/repo.py

@@ -1,7 +1,7 @@
 #
 # Copyright (c) 2021 Evgenii Alekseev.
 #
-# This file is part of ahriman 
+# This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).
 #
 # This program is free software: you can redistribute it and/or modify
@@ -27,9 +27,22 @@ from ahriman.core.util import check_output
 from ahriman.models.repository_paths import RepositoryPaths
 
 
-class RepoWrapper:
+class Repo:
+    '''
+    repo-add and repo-remove wrapper
+    :ivar logger: class logger
+    :ivar name: repository name
+    :ivar paths: repository paths instance
+    :ivar sign_args: additional args which have to be used to sign repository archive
+    '''
 
     def __init__(self, name: str, paths: RepositoryPaths, sign_args: List[str]) -> None:
+        '''
+        default constructor
+        :param name: repository name
+        :param paths: repository paths instance
+        :param sign_args: additional args which have to be used to sign repository archive
+        '''
         self.logger = logging.getLogger('build_details')
         self.name = name
         self.paths = paths
@@ -37,19 +50,32 @@ class RepoWrapper:
 
     @property
     def repo_path(self) -> str:
+        '''
+        :return: path to repository database
+        '''
         return os.path.join(self.paths.repository, f'{self.name}.db.tar.gz')
 
     def add(self, path: str) -> None:
+        '''
+        add new package to repository
+        :param path: path to archive to add
+        '''
         check_output(
             'repo-add', *self.sign_args, '-R', self.repo_path, path,
             exception=BuildFailed(path),
             cwd=self.paths.repository,
             logger=self.logger)
 
-    def remove(self, prefix: str, package: str) -> None:
+    def remove(self, package: str) -> None:
-        for fn in filter(lambda f: f.startswith(prefix), os.listdir(self.paths.repository)):
+        '''
+        remove package from repository
+        :param package: package name to remove
+        '''
+        # remove package and signature (if any) from filesystem
+        for fn in filter(lambda f: f.startswith(package), os.listdir(self.paths.repository)):
             full_path = os.path.join(self.paths.repository, fn)
             os.remove(full_path)
+        # remove package from registry
         check_output(
             'repo-remove', *self.sign_args, self.repo_path, package,
             exception=BuildFailed(package),

src/ahriman/core/build_tools/__init__.py

@@ -1,7 +1,7 @@
 #
 # Copyright (c) 2021 Evgenii Alekseev.
 #
-# This file is part of ahriman 
+# This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).
 #
 # This program is free software: you can redistribute it and/or modify

src/ahriman/core/build_tools/task.py

@@ -1,7 +1,7 @@
 #
 # Copyright (c) 2021 Evgenii Alekseev.
 #
-# This file is part of ahriman 
+# This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).
 #
 # This program is free software: you can redistribute it and/or modify
@@ -31,29 +31,67 @@ from ahriman.models.repository_paths import RepositoryPaths
 
 
 class Task:
+    '''
+    base package build task
+    :ivar build_logger: logger for build process
+    :ivar logger: class logger
+    :ivar package: package definitions
+    :ivar paths: repository paths instance
+    '''
 
     def __init__(self, package: Package, architecture: str, config: Configuration, paths: RepositoryPaths) -> None:
+        '''
+        default constructor
+        :param package: package definitions
+        :param architecture: repository architecture
+        :param config: configuration instance
+        :param paths: repository paths instance
+        '''
         self.logger = logging.getLogger('builder')
         self.build_logger = logging.getLogger('build_details')
         self.package = package
         self.paths = paths
 
         section = config.get_section_name('build', architecture)
-        self.archbuild_flags = config.get_list(section, 'archbuild_flags')
+        self.archbuild_flags = config.getlist(section, 'archbuild_flags')
         self.build_command = config.get(section, 'build_command')
-        self.makepkg_flags = config.get_list(section, 'makepkg_flags')
+        self.makepkg_flags = config.getlist(section, 'makepkg_flags')
-        self.makechrootpkg_flags = config.get_list(section, 'makechrootpkg_flags')
+        self.makechrootpkg_flags = config.getlist(section, 'makechrootpkg_flags')
+
+    @property
+    def cache_path(self) -> str:
+        '''
+        :return: path to cached packages
+        '''
+        return os.path.join(self.paths.cache, self.package.base)
 
     @property
     def git_path(self) -> str:
+        '''
+        :return: path to clone package from git
+        '''
         return os.path.join(self.paths.sources, self.package.base)
 
     @staticmethod
-    def fetch(local: str, remote: str) -> None:
+    def fetch(local: str, remote: str, branch: str = 'master') -> None:
-        shutil.rmtree(local, ignore_errors=True)  # remove in case if file exists
+        '''
-        check_output('git', 'clone', remote, local, exception=None)
+        either clone repository or update it to origin/`branch`
+        :param local: local path to fetch
+        :param remote: remote target (from where to fetch)
+        :param branch: branch name to checkout, master by default
+        '''
+        if os.path.isdir(local):
+            check_output('git', 'fetch', 'origin', branch, cwd=local, exception=None)
+        else:
+            check_output('git', 'clone', remote, local, exception=None)
+        # and now force reset to our branch
+        check_output('git', 'reset', '--hard', f'origin/{branch}', cwd=local, exception=None)
 
     def build(self) -> List[str]:
+        '''
+        run package build
+        :return: paths of produced packages
+        '''
         cmd = [self.build_command, '-r', self.paths.chroot]
         cmd.extend(self.archbuild_flags)
         cmd.extend(['--'] + self.makechrootpkg_flags)
@@ -71,6 +109,13 @@ class Task:
                             exception=BuildFailed(self.package.base),
                             cwd=self.git_path).splitlines()
 
-    def clone(self, path: Optional[str] = None) -> None:
+    def init(self, path: Optional[str] = None) -> None:
+        '''
+        fetch package from git
+        :param path: optional local path to fetch. If not set default path will be used
+        '''
         git_path = path or self.git_path
-        return Task.fetch(git_path, self.package.url)
+        if os.path.isdir(self.cache_path):
+            # no need to clone whole repository, just copy from cache first
+            shutil.copytree(self.cache_path, git_path)
+        return Task.fetch(git_path, self.package.git_url)

src/ahriman/core/configuration.py

@@ -1,7 +1,7 @@
 #
 # Copyright (c) 2021 Evgenii Alekseev.
 #
-# This file is part of ahriman 
+# This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).
 #
 # This program is free software: you can redistribute it and/or modify
@@ -17,46 +17,90 @@
 # You should have received a copy of the GNU General Public License
 # along with this program. If not, see <http://www.gnu.org/licenses/>.
 #
+from __future__ import annotations
+
 import configparser
 import os
 
 from logging.config import fileConfig
-from typing import List, Optional
+from typing import List, Optional, Type
 
 
-# built-in configparser extension
 class Configuration(configparser.RawConfigParser):
+    '''
+    extension for built-in configuration parser
+    :ivar path: path to root configuration file
+    '''
 
     def __init__(self) -> None:
+        '''
+        default constructor
+        '''
         configparser.RawConfigParser.__init__(self, allow_no_value=True)
         self.path: Optional[str] = None
 
     @property
     def include(self) -> str:
+        '''
+        :return: path to directory with configuration includes
+        '''
         return self.get('settings', 'include')
 
-    def get_list(self, section: str, key: str) -> List[str]:
+    @classmethod
+    def from_path(cls: Type[Configuration], path: str) -> Configuration:
+        '''
+        constructor with full object initialization
+        :param path: path to root configuration file
+        :return: configuration instance
+        '''
+        config = cls()
+        config.load(path)
+        config.load_logging()
+        return config
+
+    def getlist(self, section: str, key: str) -> List[str]:
+        '''
+        get space separated string list option
+        :param section: section name
+        :param key: key name
+        :return: list of string if option is set, empty list otherwise
+        '''
         raw = self.get(section, key, fallback=None)
         if not raw:  # empty string or none
             return []
         return raw.split()
 
     def get_section_name(self, prefix: str, suffix: str) -> str:
+        '''
+        check if there is `prefix`_`suffix` section and return it on success. Return `prefix` otherwise
+        :param prefix: section name prefix
+        :param suffix: section name suffix (e.g. architecture name)
+        :return: found section name
+        '''
         probe = f'{prefix}_{suffix}'
         return probe if self.has_section(probe) else prefix
 
     def load(self, path: str) -> None:
+        '''
+        fully load configuration
+        :param path: path to root configuration file
+        '''
         self.path = path
         self.read(self.path)
         self.load_includes()
 
     def load_includes(self) -> None:
+        '''
+        load configuration includes
+        '''
         try:
-            include_dir = self.include
+            for conf in filter(lambda p: p.endswith('.ini'), sorted(os.listdir(self.include))):
-            for conf in filter(lambda p: p.endswith('.ini'), sorted(os.listdir(include_dir))):
                 self.read(os.path.join(self.include, conf))
         except (FileNotFoundError, configparser.NoOptionError):
             pass
 
     def load_logging(self) -> None:
+        '''
+        setup logging settings from configuration
+        '''
         fileConfig(self.get('settings', 'logging'))

src/ahriman/core/exceptions.py

@@ -1,7 +1,7 @@
 #
 # Copyright (c) 2021 Evgenii Alekseev.
 #
-# This file is part of ahriman 
+# This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).
 #
 # This program is free software: you can redistribute it and/or modify
@@ -21,30 +21,87 @@ from typing import Any
 
 
 class BuildFailed(Exception):
+    '''
+    base exception for failed builds
+    '''
+
     def __init__(self, package: str) -> None:
+        '''
+        default constructor
+        :param package: package base raised exception
+        '''
         Exception.__init__(self, f'Package {package} build failed, check logs for details')
 
 
-class InvalidOptionException(Exception):
+class DuplicateRun(Exception):
+    '''
+    exception which will be raised if there is another application instance
+    '''
+
+    def __init__(self) -> None:
+        '''
+        default constructor
+        '''
+        Exception.__init__(self, 'Another application instance is run')
+
+
+class InitializeException(Exception):
+    '''
+    base service initialization exception
+    '''
+
+    def __init__(self) -> None:
+        '''
+        default constructor
+        '''
+        Exception.__init__(self, 'Could not load service')
+
+
+class InvalidOption(Exception):
+    '''
+    exception which will be raised on configuration errors
+    '''
+
     def __init__(self, value: Any) -> None:
+        '''
+        default constructor
+        :param value: option value
+        '''
         Exception.__init__(self, f'Invalid or unknown option value `{value}`')
 
 
 class InvalidPackageInfo(Exception):
+    '''
+    exception which will be raised on package load errors
+    '''
+
     def __init__(self, details: Any) -> None:
+        '''
+        default constructor
+        :param details: error details
+        '''
         Exception.__init__(self, f'There are errors during reading package information: `{details}`')
 
 
-class MissingConfiguration(Exception):
-    def __init__(self, name: str) -> None:
-        Exception.__init__(self, f'No section `{name}` found')
-
-
 class ReportFailed(Exception):
-    def __init__(self, cause: Exception) -> None:
+    '''
-        Exception.__init__(self, f'Report failed with reason {cause}')
+    report generation exception
+    '''
+
+    def __init__(self) -> None:
+        '''
+        default constructor
+        '''
+        Exception.__init__(self, 'Report failed')
 
 
 class SyncFailed(Exception):
-    def __init__(self, cause: Exception) -> None:
+    '''
-        Exception.__init__(self, f'Sync failed with reason {cause}')
+    remote synchronization exception
+    '''
+
+    def __init__(self) -> None:
+        '''
+        default constructor
+        '''
+        Exception.__init__(self, 'Sync failed')

src/ahriman/core/report/__init__.py

@@ -1,7 +1,7 @@
 #
 # Copyright (c) 2021 Evgenii Alekseev.
 #
-# This file is part of ahriman 
+# This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).
 #
 # This program is free software: you can redistribute it and/or modify

src/ahriman/core/report/html.py

@@ -20,51 +20,89 @@
 import jinja2
 import os
 
-from typing import Dict
+from typing import Callable, Dict, Iterable
 
 from ahriman.core.configuration import Configuration
 from ahriman.core.report.report import Report
-from ahriman.core.util import package_like
+from ahriman.core.util import pretty_size, pretty_datetime
+from ahriman.models.package import Package
 from ahriman.models.sign_settings import SignSettings
 
 
 class HTML(Report):
+    '''
+    html report generator
+
+    It uses jinja2 templates for report generation, the following variables are allowed:
+
+        homepage - link to homepage, string, optional
+        link_path - prefix fo packages to download, string, required
+        has_package_signed - True in case if package sign enabled, False otherwise, required
+        has_repo_signed - True in case if repository database sign enabled, False otherwise, required
+        packages - sorted list of packages properties: archive_size, build_date, filename, installed_size, name, version. Required
+        pgp_key - default PGP key ID, string, optional
+        repository - repository name, string, required
+
+    :ivar homepage: homepage link if any (for footer)
+    :ivar link_path: prefix fo packages to download
+    :ivar name: repository name
+    :ivar pgp_key: default PGP key
+    :ivar report_path: output path to html report
+    :ivar sign_targets: targets to sign enabled in configuration
+    :ivar tempate_path: path to directory with jinja templates
+    '''
 
     def __init__(self, architecture: str, config: Configuration) -> None:
+        '''
+        default constructor
+        :param architecture: repository architecture
+        :param config: configuration instance
+        '''
         Report.__init__(self, architecture, config)
-        section = self.config.get_section_name('html', self.architecture)
+        section = config.get_section_name('html', architecture)
         self.report_path = config.get(section, 'path')
-
         self.link_path = config.get(section, 'link_path')
         self.template_path = config.get(section, 'template_path')
 
         # base template vars
-        if SignSettings.from_option(config.get('sign', 'enabled')) != SignSettings.Disabled:
-            self.pgp_key = config.get('sign', 'key', fallback=None)
-        else:
-            self.pgp_key = None
         self.homepage = config.get(section, 'homepage', fallback=None)
-        self.repository = config.get('repository', 'name')
+        self.name = config.get('repository', 'name')
 
-    def generate(self, path: str) -> None:
+        sign_section = config.get_section_name('sign', architecture)
+        self.sign_targets = [SignSettings.from_option(opt) for opt in config.getlist(sign_section, 'target')]
+        self.pgp_key = config.get(sign_section, 'key') if self.sign_targets else None
+
+    def generate(self, packages: Iterable[Package]) -> None:
+        '''
+        generate report for the specified packages
+        :param packages: list of packages to generate report
+        '''
         # idea comes from https://stackoverflow.com/a/38642558
         templates_dir, template_name = os.path.split(self.template_path)
         loader = jinja2.FileSystemLoader(searchpath=templates_dir)
         environment = jinja2.Environment(loader=loader)
         template = environment.get_template(template_name)
 
-        packages: Dict[str, str] = {}
+        content = [
-        for fn in sorted(os.listdir(path)):
+            {
-            if not package_like(fn):
+                'archive_size': pretty_size(properties.archive_size),
-                continue
+                'build_date': pretty_datetime(properties.build_date),
-            packages[fn] = f'{self.link_path}/{fn}'
+                'filename': properties.filename,
+                'installed_size': pretty_size(properties.installed_size),
+                'name': package,
+                'version': base.version
+            } for base in packages for package, properties in base.packages.items()
+        ]
+        comparator: Callable[[Dict[str, str]], str] = lambda item: item['filename']
 
         html = template.render(
             homepage=self.homepage,
             link_path=self.link_path,
-            packages=packages,
+            has_package_signed=SignSettings.SignPackages in self.sign_targets,
+            has_repo_signed=SignSettings.SignRepository in self.sign_targets,
+            packages=sorted(content, key=comparator),
             pgp_key=self.pgp_key,
-            repository=self.repository)
+            repository=self.name)
 
         with open(self.report_path, 'w') as out:
             out.write(html)

src/ahriman/core/report/report.py

@@ -19,32 +19,57 @@
 #
 import logging
 
+from typing import Iterable
+
 from ahriman.core.configuration import Configuration
 from ahriman.core.exceptions import ReportFailed
+from ahriman.models.package import Package
 from ahriman.models.report_settings import ReportSettings
 
 
 class Report:
+    '''
+    base report generator
+    :ivar architecture: repository architecture
+    :ivar config: configuration instance
+    :ivar logger: class logger
+    '''
 
     def __init__(self, architecture: str, config: Configuration) -> None:
+        '''
+        default constructor
+        :param architecture: repository architecture
+        :param config: configuration instance
+        '''
+        self.logger = logging.getLogger('builder')
         self.architecture = architecture
         self.config = config
-        self.logger = logging.getLogger('builder')
 
     @staticmethod
-    def run(architecture: str, config: Configuration, target: str, path: str) -> None:
+    def run(architecture: str, config: Configuration, target: str, packages: Iterable[Package]) -> None:
+        '''
+        run report generation
+        :param architecture: repository architecture
+        :param config: configuration instance
+        :param target: target to generate report (e.g. html)
+        :param packages: list of packages to generate report
+        '''
         provider = ReportSettings.from_option(target)
         if provider == ReportSettings.HTML:
             from ahriman.core.report.html import HTML
             report: Report = HTML(architecture, config)
         else:
-            from ahriman.core.report.dummy import Dummy
+            report = Report(architecture, config)
-            report = Dummy(architecture, config)
 
         try:
-            report.generate(path)
+            report.generate(packages)
-        except Exception as e:
+        except Exception:
-            raise ReportFailed(e) from e
+            report.logger.exception('report generation failed', exc_info=True)
+            raise ReportFailed()
 
-    def generate(self, path: str) -> None:
+    def generate(self, packages: Iterable[Package]) -> None:
-        raise NotImplementedError
+        '''
+        generate report for the specified packages
+        :param packages: list of packages to generate report
+        '''
+        pass

src/ahriman/core/repository.py

@@ -1,7 +1,7 @@
 #
 # Copyright (c) 2021 Evgenii Alekseev.
 #
-# This file is part of ahriman 
+# This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).
 #
 # This program is free software: you can redistribute it and/or modify
@@ -21,65 +21,130 @@ import logging
 import os
 import shutil
 
-from typing import Dict, List, Optional
+from typing import Dict, Iterable, List, Optional
 
+from ahriman.core.alpm.pacman import Pacman
+from ahriman.core.alpm.repo import Repo
 from ahriman.core.build_tools.task import Task
 from ahriman.core.configuration import Configuration
-from ahriman.core.repo.repo_wrapper import RepoWrapper
 from ahriman.core.report.report import Report
-from ahriman.core.sign.gpg_wrapper import GPGWrapper
+from ahriman.core.sign.gpg import GPG
 from ahriman.core.upload.uploader import Uploader
 from ahriman.core.util import package_like
+from ahriman.core.watcher.client import Client
 from ahriman.models.package import Package
 from ahriman.models.repository_paths import RepositoryPaths
 
 
 class Repository:
+    '''
+    base repository control class
+    :ivar architecture: repository architecture
+    :ivar aur_url: base AUR url
+    :ivar config: configuration instance
+    :ivar logger: class logger
+    :ivar name: repository name
+    :ivar pacman: alpm wrapper instance
+    :ivar paths: repository paths instance
+    :ivar repo: repo commands wrapper instance
+    :ivar reporter: build status reporter instance
+    :ivar sign: GPG wrapper instance
+    '''
 
     def __init__(self, architecture: str, config: Configuration) -> None:
+        '''
+        default constructor
+        :param architecture: repository architecture
+        :param config: configuration instance
+        '''
         self.logger = logging.getLogger('builder')
         self.architecture = architecture
         self.config = config
 
-        self.aur_url = config.get('aur', 'url')
+        self.aur_url = config.get('alpm', 'aur_url')
         self.name = config.get('repository', 'name')
 
-        self.paths = RepositoryPaths(config.get('repository', 'root'), self.architecture)
+        self.paths = RepositoryPaths(config.get('repository', 'root'), architecture)
         self.paths.create_tree()
 
-        self.sign = GPGWrapper(config)
+        self.pacman = Pacman(config)
-        self.wrapper = RepoWrapper(self.name, self.paths, self.sign.repository_sign_args)
+        self.sign = GPG(architecture, config)
+        self.repo = Repo(self.name, self.paths, self.sign.repository_sign_args)
+        self.reporter = Client.load(architecture, config)
 
     def _clear_build(self) -> None:
+        '''
+        clear sources directory
+        '''
         for package in os.listdir(self.paths.sources):
             shutil.rmtree(os.path.join(self.paths.sources, package))
 
+    def _clear_cache(self) -> None:
+        '''
+        clear cache directory
+        '''
+        for package in os.listdir(self.paths.cache):
+            shutil.rmtree(os.path.join(self.paths.cache, package))
+
+    def _clear_chroot(self) -> None:
+        '''
+        clear cache directory. Warning: this method is architecture independent and will clear every chroot
+        '''
+        for chroot in os.listdir(self.paths.chroot):
+            shutil.rmtree(os.path.join(self.paths.chroot, chroot))
+
     def _clear_manual(self) -> None:
+        '''
+        clear directory with manual package updates
+        '''
         for package in os.listdir(self.paths.manual):
             shutil.rmtree(os.path.join(self.paths.manual, package))
 
     def _clear_packages(self) -> None:
-        for package in os.listdir(self.paths.packages):
+        '''
-            os.remove(os.path.join(self.paths.packages, package))
+        clear directory with built packages (NOT repository itself)
+        '''
+        for package in self.packages_built():
+            os.remove(package)
 
     def packages(self) -> List[Package]:
+        '''
+        generate list of repository packages
+        :return: list of packages properties
+        '''
         result: Dict[str, Package] = {}
         for fn in os.listdir(self.paths.repository):
             if not package_like(fn):
                 continue
             full_path = os.path.join(self.paths.repository, fn)
             try:
-                local = Package.load(full_path, self.aur_url)
+                local = Package.load(full_path, self.pacman, self.aur_url)
                 result.setdefault(local.base, local).packages.update(local.packages)
             except Exception:
                 self.logger.exception(f'could not load package from {fn}', exc_info=True)
                 continue
         return list(result.values())
 
-    def process_build(self, updates: List[Package]) -> List[str]:
+    def packages_built(self) -> List[str]:
+        '''
+        get list of files in built packages directory
+        :return: list of filenames from the directory
+        '''
+        return [
+            os.path.join(self.paths.packages, fn)
+            for fn in os.listdir(self.paths.packages)
+        ]
+
+    def process_build(self, updates: Iterable[Package]) -> List[str]:
+        '''
+        build packages
+        :param updates: list of packages properties to build
+        :return: `packages_built`
+        '''
         def build_single(package: Package) -> None:
+            self.reporter.set_building(package.base)
             task = Task(package, self.architecture, self.config, self.paths)
-            task.clone()
+            task.init()
             built = task.build()
             for src in built:
                 dst = os.path.join(self.paths.packages, os.path.basename(src))
@@ -89,85 +154,128 @@ class Repository:
             try:
                 build_single(package)
             except Exception:
+                self.reporter.set_failed(package.base)
                 self.logger.exception(f'{package.base} ({self.architecture}) build exception', exc_info=True)
                 continue
         self._clear_build()
 
-        return [
+        return self.packages_built()
-            os.path.join(self.paths.packages, fn)
-            for fn in os.listdir(self.paths.packages)
-        ]
 
-    def process_remove(self, packages: List[str]) -> str:
+    def process_remove(self, packages: Iterable[str]) -> str:
+        '''
+        remove packages from list
+        :param packages: list of package names or bases to rmeove
+        :return: path to repository database
+        '''
         def remove_single(package: str) -> None:
             try:
-                self.wrapper.remove(package, package)
+                self.repo.remove(package)
             except Exception:
                 self.logger.exception(f'could not remove {package}', exc_info=True)
 
+        requested = set(packages)
         for local in self.packages():
             if local.base in packages:
-                to_remove = local.packages
+                to_remove = set(local.packages.keys())
-            elif local.packages.intersection(packages):
+                self.reporter.remove(local.base)  # we only update status page in case of base removal
-                to_remove = local.packages.intersection(packages)
+            elif requested.intersection(local.packages.keys()):
+                to_remove = requested.intersection(local.packages.keys())
             else:
                 to_remove = set()
             for package in to_remove:
                 remove_single(package)
 
-        return self.wrapper.repo_path
+        return self.repo.repo_path
 
-    def process_report(self, targets: Optional[List[str]]) -> None:
+    def process_report(self, targets: Optional[Iterable[str]]) -> None:
+        '''
+        generate reports
+        :param targets: list of targets to generate reports. Configuration option will be used if it is not set
+        '''
         if targets is None:
-            targets = self.config.get_list('report', 'target')
+            targets = self.config.getlist('report', 'target')
         for target in targets:
-            Report.run(self.architecture, self.config, target, self.paths.repository)
+            Report.run(self.architecture, self.config, target, self.packages())
 
-    def process_sync(self, targets: Optional[List[str]]) -> None:
+    def process_sync(self, targets: Optional[Iterable[str]]) -> None:
+        '''
+        process synchronization to remote servers
+        :param targets: list of targets to sync. Configuration option will be used if it is not set
+        '''
         if targets is None:
-            targets = self.config.get_list('upload', 'target')
+            targets = self.config.getlist('upload', 'target')
         for target in targets:
             Uploader.run(self.architecture, self.config, target, self.paths.repository)
 
-    def process_update(self, packages: List[str]) -> str:
+    def process_update(self, packages: Iterable[str]) -> str:
+        '''
+        sign packages, add them to repository and update repository database
+        :param packages: list of filenames to run
+        :return: path to repository database
+        '''
         for package in packages:
-            files = self.sign.sign_package(package)
+            local = Package.load(package, self.pacman, self.aur_url)  # we will use it for status reports
-            for src in files:
+            try:
-                dst = os.path.join(self.paths.repository, os.path.basename(src))
+                files = self.sign.sign_package(package, local.base)
-                shutil.move(src, dst)
+                for src in files:
-            package_fn = os.path.join(self.paths.repository, os.path.basename(package))
+                    dst = os.path.join(self.paths.repository, os.path.basename(src))
-            self.wrapper.add(package_fn)
+                    shutil.move(src, dst)
+                package_fn = os.path.join(self.paths.repository, os.path.basename(package))
+                self.repo.add(package_fn)
+                self.reporter.set_success(local)
+            except Exception:
+                self.logger.exception(f'could not process {package}', exc_info=True)
+                self.reporter.set_failed(local.base)
         self._clear_packages()
 
-        return self.wrapper.repo_path
+        return self.repo.repo_path
 
-    def updates_aur(self, no_vcs: bool) -> List[Package]:
+    def updates_aur(self, filter_packages: Iterable[str], no_vcs: bool) -> List[Package]:
+        '''
+        check AUR for updates
+        :param filter_packages: do not check every package just specified in the list
+        :param no_vcs: do not check VCS packages
+        :return: list of packages which are out-of-dated
+        '''
         result: List[Package] = []
-        ignore_list = self.config.get_list(
+
-            self.config.get_section_name('build', self.architecture), 'ignore_packages')
+        build_section = self.config.get_section_name('build', self.architecture)
+        ignore_list = self.config.getlist(build_section, 'ignore_packages')
 
         for local in self.packages():
             if local.base in ignore_list:
                 continue
             if local.is_vcs and no_vcs:
                 continue
+            if filter_packages and local.base not in filter_packages:
+                continue
 
             try:
-                remote = Package.load(local.base, self.aur_url)
+                remote = Package.load(local.base, self.pacman, self.aur_url)
-                if local.is_outdated(remote):
+                if local.is_outdated(remote, self.paths):
                     result.append(remote)
+                    self.reporter.set_pending(local.base)
             except Exception:
+                self.reporter.set_failed(local.base)
                 self.logger.exception(f'could not load remote package {local.base}', exc_info=True)
                 continue
 
         return result
 
     def updates_manual(self) -> List[Package]:
+        '''
+        check for packages for which manual update has been requested
+        :return: list of packages which are out-of-dated
+        '''
         result: List[Package] = []
 
         for fn in os.listdir(self.paths.manual):
-            local = Package.load(os.path.join(self.paths.manual, fn), self.aur_url)
+            try:
-            result.append(local)
+                local = Package.load(os.path.join(self.paths.manual, fn), self.pacman, self.aur_url)
+                result.append(local)
+                self.reporter.set_unknown(local)
+            except Exception:
+                self.logger.exception(f'could not add package from {fn}', exc_info=True)
         self._clear_manual()
 
-        return result
+        return result

src/ahriman/core/sign/__init__.py

@@ -1,7 +1,7 @@
 #
 # Copyright (c) 2021 Evgenii Alekseev.
 #
-# This file is part of ahriman 
+# This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).
 #
 # This program is free software: you can redistribute it and/or modify

src/ahriman/core/sign/gpg.py

@@ -0,0 +1,106 @@
+#
+# Copyright (c) 2021 Evgenii Alekseev.
+#
+# This file is part of ahriman
+# (see https://github.com/arcan1s/ahriman).
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+import logging
+import os
+
+from typing import List
+
+from ahriman.core.configuration import Configuration
+from ahriman.core.exceptions import BuildFailed
+from ahriman.core.util import check_output
+from ahriman.models.sign_settings import SignSettings
+
+
+class GPG:
+    '''
+    gnupg wrapper
+    :ivar architecture: repository architecture
+    :ivar config: configuration instance
+    :ivar default_key: default PGP key ID to use
+    :ivar logger: class logger
+    :ivar target: list of targets to sign (repository, package etc)
+    '''
+
+    def __init__(self, architecture: str, config: Configuration) -> None:
+        '''
+        default constructor
+        :param architecture: repository architecture
+        :param config: configuration instance
+        '''
+        self.logger = logging.getLogger('build_details')
+        self.config = config
+        self.section = config.get_section_name('sign', architecture)
+        self.target = [SignSettings.from_option(opt) for opt in config.getlist(self.section, 'target')]
+        self.default_key = config.get(self.section, 'key') if self.target else ''
+
+    @property
+    def repository_sign_args(self) -> List[str]:
+        '''
+        :return: command line arguments for repo-add command to sign database
+        '''
+        if SignSettings.SignRepository not in self.target:
+            return []
+        return ['--sign', '--key', self.default_key]
+
+    def process(self, path: str, key: str) -> List[str]:
+        '''
+        gpg command wrapper
+        :param path: path to file to sign
+        :param key: PGP key ID
+        :return: list of generated files including original file
+        '''
+        check_output(
+            *self.sign_cmd(path, key),
+            exception=BuildFailed(path),
+            cwd=os.path.dirname(path),
+            logger=self.logger)
+        return [path, f'{path}.sig']
+
+    def sign_cmd(self, path: str, key: str) -> List[str]:
+        '''
+        gpg command to run
+        :param path: path to file to sign
+        :param key: PGP key ID
+        :return: gpg command with all required arguments
+        '''
+        return ['gpg', '-u', key, '-b', path]
+
+    def sign_package(self, path: str, base: str) -> List[str]:
+        '''
+        sign package if required by configuration
+        :param path: path to file to sign
+        :param base: package base required to check for key overrides
+        :return: list of generated files including original file
+        '''
+        if SignSettings.SignPackages not in self.target:
+            return [path]
+        key = self.config.get(self.section, f'key_{base}', fallback=self.default_key)
+        return self.process(path, key)
+
+    def sign_repository(self, path: str) -> List[str]:
+        '''
+        sign repository if required by configuration
+        :note: more likely you just want to pass `repository_sign_args` to repo wrapper
+        :param path: path to repository database
+        :return: list of generated files including original file
+        '''
+        if SignSettings.SignRepository not in self.target:
+            return [path]
+        return self.process(path, self.default_key)

src/ahriman/core/sign/gpg_wrapper.py

@@ -1,68 +0,0 @@
-#
-# Copyright (c) 2021 Evgenii Alekseev.
-#
-# This file is part of ahriman 
-# (see https://github.com/arcan1s/ahriman).
-#
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program. If not, see <http://www.gnu.org/licenses/>.
-#
-import logging
-import os
-
-from typing import List
-
-from ahriman.core.configuration import Configuration
-from ahriman.core.exceptions import BuildFailed
-from ahriman.core.util import check_output
-from ahriman.models.sign_settings import SignSettings
-
-
-class GPGWrapper:
-
-    def __init__(self, config: Configuration) -> None:
-        self.logger = logging.getLogger('build_details')
-
-        self.key = config.get('sign', 'key', fallback=None)
-        self.sign = SignSettings.from_option(config.get('sign', 'enabled'))
-
-    @property
-    def repository_sign_args(self) -> List[str]:
-        if self.sign != SignSettings.SignRepository:
-            return []
-        return ['--sign', '--key', self.key] if self.key else ['--sign']
-
-    def process(self, path: str) -> List[str]:
-        check_output(
-            *self.sign_cmd(path),
-            exception=BuildFailed(path),
-            cwd=os.path.dirname(path),
-            logger=self.logger)
-        return [path, f'{path}.sig']
-
-    def sign_cmd(self, path: str) -> List[str]:
-        cmd = ['gpg']
-        if self.key is not None:
-            cmd.extend(['-u', self.key])
-        cmd.extend(['-b', path])
-        return cmd
-
-    def sign_package(self, path: str) -> List[str]:
-        if self.sign != SignSettings.SignPackages:
-            return [path]
-        return self.process(path)
-
-    def sign_repository(self, path: str) -> List[str]:
-        if self.sign != SignSettings.SignRepository:
-            return [path]
-        return self.process(path)

src/ahriman/core/tree.py

@@ -0,0 +1,110 @@
+#
+# Copyright (c) 2021 Evgenii Alekseev.
+#
+# This file is part of ahriman
+# (see https://github.com/arcan1s/ahriman).
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+from __future__ import annotations
+
+import shutil
+import tempfile
+
+from typing import Iterable, List, Set
+
+from ahriman.core.build_tools.task import Task
+from ahriman.models.package import Package
+
+
+class Leaf:
+    '''
+    tree leaf implementation
+    :ivar dependencies: list of package dependencies
+    :ivar package: leaf package properties
+    '''
+
+    def __init__(self, package: Package) -> None:
+        '''
+        default constructor
+        :param package: package properties
+        '''
+        self.package = package
+        self.dependencies: Set[str] = set()
+
+    @property
+    def items(self) -> Iterable[str]:
+        '''
+        :return: packages containing in this leaf
+        '''
+        return self.package.packages.keys()
+
+    def is_root(self, packages: Iterable[Leaf]) -> bool:
+        '''
+        check if package depends on any other package from list of not
+        :param packages: list of known leaves
+        :return: True if any of packages is dependency of the leaf, False otherwise
+        '''
+        for leaf in packages:
+            if self.dependencies.intersection(leaf.items):
+                return False
+        return True
+
+    def load_dependencies(self) -> None:
+        '''
+        load dependencies for the leaf
+        '''
+        clone_dir = tempfile.mkdtemp()
+        try:
+            Task.fetch(clone_dir, self.package.git_url)
+            self.dependencies = Package.dependencies(clone_dir)
+        finally:
+            shutil.rmtree(clone_dir, ignore_errors=True)
+
+
+class Tree:
+    '''
+    dependency tree implementation
+    :ivar leaves: list of tree leaves
+    '''
+
+    def __init__(self) -> None:
+        '''
+        default constructor
+        '''
+        self.leaves: List[Leaf] = []
+
+    def levels(self) -> List[List[Package]]:
+        '''
+        get build levels starting from the packages which do not require any other package to build
+        :return: list of packages lists
+        '''
+        result: List[List[Package]] = []
+
+        unprocessed = [leaf for leaf in self.leaves]
+        while unprocessed:
+            result.append([leaf.package for leaf in unprocessed if leaf.is_root(unprocessed)])
+            unprocessed = [leaf for leaf in unprocessed if not leaf.is_root(unprocessed)]
+
+        return result
+
+    def load(self, packages: Iterable[Package]) -> None:
+        '''
+        load tree from packages
+        :param packages: packages list
+        '''
+        for package in packages:
+            leaf = Leaf(package)
+            leaf.load_dependencies()
+            self.leaves.append(leaf)

src/ahriman/core/upload/__init__.py

@@ -1,7 +1,7 @@
 #
 # Copyright (c) 2021 Evgenii Alekseev.
 #
-# This file is part of ahriman 
+# This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).
 #
 # This program is free software: you can redistribute it and/or modify

src/ahriman/core/upload/rsync.py

@@ -23,13 +23,26 @@ from ahriman.core.util import check_output
 
 
 class Rsync(Uploader):
+    '''
+    rsync wrapper
+    :ivar remote: remote address to sync
+    '''
 
     def __init__(self, architecture: str, config: Configuration) -> None:
+        '''
+        default constructor
+        :param architecture: repository architecture
+        :param config: configuration instance
+        '''
         Uploader.__init__(self, architecture, config)
-        section = self.config.get_section_name('rsync', self.architecture)
+        section = config.get_section_name('rsync', architecture)
-        self.remote = self.config.get(section, 'remote')
+        self.remote = config.get(section, 'remote')
 
     def sync(self, path: str) -> None:
+        '''
+        sync data to remote server
+        :param path: local path to sync
+        '''
         check_output('rsync', '--archive', '--verbose', '--compress', '--partial', '--progress', '--delete', path, self.remote,
                      exception=None,
                      logger=self.logger)

src/ahriman/core/upload/s3.py

@@ -23,13 +23,26 @@ from ahriman.core.util import check_output
 
 
 class S3(Uploader):
+    '''
+    aws-cli wrapper
+    :ivar bucket: full bucket name
+    '''
 
     def __init__(self, architecture: str, config: Configuration) -> None:
+        '''
+        default constructor
+        :param architecture: repository architecture
+        :param config: configuration instance
+        '''
         Uploader.__init__(self, architecture, config)
-        section = self.config.get_section_name('s3', self.architecture)
+        section = config.get_section_name('s3', architecture)
-        self.bucket = self.config.get(section, 'bucket')
+        self.bucket = config.get(section, 'bucket')
 
     def sync(self, path: str) -> None:
+        '''
+        sync data to remote server
+        :param path: local path to sync
+        '''
         # TODO rewrite to boto, but it is bullshit
         check_output('aws', 's3', 'sync', '--delete', path, self.bucket,
                      exception=None,

src/ahriman/core/upload/uploader.py

@@ -25,14 +25,32 @@ from ahriman.models.upload_settings import UploadSettings
 
 
 class Uploader:
+    '''
+    base remote sync class
+    :ivar architecture: repository architecture
+    :ivar config: configuration instance
+    :ivar logger: application logger
+    '''
 
     def __init__(self, architecture: str, config: Configuration) -> None:
+        '''
+        default constructor
+        :param architecture: repository architecture
+        :param config: configuration instance
+        '''
+        self.logger = logging.getLogger('builder')
         self.architecture = architecture
         self.config = config
-        self.logger = logging.getLogger('builder')
 
     @staticmethod
     def run(architecture: str, config: Configuration, target: str, path: str) -> None:
+        '''
+        run remote sync
+        :param architecture: repository architecture
+        :param config: configuration instance
+        :param target: target to run sync (e.g. s3)
+        :param path: local path to sync
+        '''
         provider = UploadSettings.from_option(target)
         if provider == UploadSettings.Rsync:
             from ahriman.core.upload.rsync import Rsync
@@ -41,13 +59,17 @@ class Uploader:
             from ahriman.core.upload.s3 import S3
             uploader = S3(architecture, config)
         else:
-            from ahriman.core.upload.dummy import Dummy
+            uploader = Uploader(architecture, config)
-            uploader = Dummy(architecture, config)
 
         try:
             uploader.sync(path)
-        except Exception as e:
+        except Exception:
-            raise SyncFailed(e) from e
+            uploader.logger.exception('remote sync failed', exc_info=True)
+            raise SyncFailed()
 
     def sync(self, path: str) -> None:
-        raise NotImplementedError
+        '''
+        sync data to remote server
+        :param path: local path to sync
+        '''
+        pass

src/ahriman/core/util.py

@@ -1,7 +1,7 @@
 #
 # Copyright (c) 2021 Evgenii Alekseev.
 #
-# This file is part of ahriman 
+# This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).
 #
 # This program is free software: you can redistribute it and/or modify
@@ -17,15 +17,27 @@
 # You should have received a copy of the GNU General Public License
 # along with this program. If not, see <http://www.gnu.org/licenses/>.
 #
+import datetime
 import subprocess
 
 from logging import Logger
 from typing import Optional
 
+from ahriman.core.exceptions import InvalidOption
+
 
 def check_output(*args: str, exception: Optional[Exception],
-                 cwd = None, stderr: int = subprocess.STDOUT,
+                 cwd: Optional[str] = None, stderr: int = subprocess.STDOUT,
                  logger: Optional[Logger] = None) -> str:
+    '''
+    subprocess wrapper
+    :param args: command line arguments
+    :param exception: exception which has to be reraised instead of default subprocess exception
+    :param cwd: current working directory
+    :param stderr: standard error output mode
+    :param logger: logger to log command result if required
+    :return: command output
+    '''
     try:
         result = subprocess.check_output(args, cwd=cwd, stderr=stderr).decode('utf8').strip()
         if logger is not None:
@@ -40,4 +52,43 @@ def check_output(*args: str, exception: Optional[Exception],
 
 
 def package_like(filename: str) -> bool:
+    '''
+    check if file looks like package
+    :param filename: name of file to check
+    :return: True in case if name contains `.pkg.` and not signature, False otherwise
+    '''
     return '.pkg.' in filename and not filename.endswith('.sig')
+
+
+def pretty_datetime(timestamp: Optional[datetime.datetime]) -> str:
+    '''
+    convert datetime object to string
+    :param timestamp: datetime to convert
+    :return: pretty printable datetime as string
+    '''
+    return '' if timestamp is None else timestamp.strftime('%Y-%m-%d %H:%M:%S')
+
+
+def pretty_size(size: Optional[float], level: int = 0) -> str:
+    '''
+    convert size to string
+    :param size: size to convert
+    :param level: represents current units, 0 is B, 1 is KiB etc
+    :return: pretty printable size as string
+    '''
+    def str_level() -> str:
+        if level == 0:
+            return 'B'
+        elif level == 1:
+            return 'KiB'
+        elif level == 2:
+            return 'MiB'
+        elif level == 3:
+            return 'GiB'
+        raise InvalidOption(level)  # I hope it will not be more than 1024 GiB
+
+    if size is None:
+        return ''
+    elif size < 1024:
+        return f'{round(size, 2)} {str_level()}'
+    return pretty_size(size / 1024, level + 1)

src/ahriman/core/watcher/__init__.py

@@ -17,10 +17,3 @@
 # You should have received a copy of the GNU General Public License
 # along with this program. If not, see <http://www.gnu.org/licenses/>.
 #
-from ahriman.core.report.report import Report
-
-
-class Dummy(Report):
-
-    def generate(self, path: str) -> None:
-        pass

src/ahriman/core/watcher/client.py

@@ -0,0 +1,112 @@
+#
+# Copyright (c) 2021 Evgenii Alekseev.
+#
+# This file is part of ahriman
+# (see https://github.com/arcan1s/ahriman).
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+from __future__ import annotations
+
+from ahriman.core.configuration import Configuration
+from ahriman.models.build_status import BuildStatusEnum
+from ahriman.models.package import Package
+
+
+class Client:
+    '''
+    base build status reporter client
+    '''
+
+    def add(self, package: Package, status: BuildStatusEnum) -> None:
+        '''
+        add new package with status
+        :param package: package properties
+        :param status: current package build status
+        '''
+        pass
+
+    def remove(self, base: str) -> None:
+        '''
+        remove packages from watcher
+        :param base: basename to remove
+        '''
+        pass
+
+    def update(self, base: str, status: BuildStatusEnum) -> None:
+        '''
+        update package build status. Unlike `add` it does not update package properties
+        :param base: package base to update
+        :param status: current package build status
+        '''
+        pass
+
+    def update_self(self, status: BuildStatusEnum) -> None:
+        '''
+        update ahriman status itself
+        :param status: current ahriman status
+        '''
+        pass
+
+    def set_building(self, base: str) -> None:
+        '''
+        set package status to building
+        :param base: package base to update
+        '''
+        return self.update(base, BuildStatusEnum.Building)
+
+    def set_failed(self, base: str) -> None:
+        '''
+        set package status to failed
+        :param base: package base to update
+        '''
+        return self.update(base, BuildStatusEnum.Failed)
+
+    def set_pending(self, base: str) -> None:
+        '''
+        set package status to pending
+        :param base: package base to update
+        '''
+        return self.update(base, BuildStatusEnum.Pending)
+
+    def set_success(self, package: Package) -> None:
+        '''
+        set package status to success
+        :param package: current package properties
+        '''
+        return self.add(package, BuildStatusEnum.Success)
+
+    def set_unknown(self, package: Package) -> None:
+        '''
+        set package status to unknown
+        :param package: current package properties
+        '''
+        return self.add(package, BuildStatusEnum.Unknown)
+
+    @staticmethod
+    def load(architecture: str, config: Configuration) -> Client:
+        '''
+        load client from settings
+        :param architecture: repository architecture
+        :param config: configuration instance
+        :return: client according to current settings
+        '''
+        section = config.get_section_name('web', architecture)
+        host = config.get(section, 'host', fallback=None)
+        port = config.getint(section, 'port', fallback=None)
+        if host is None or port is None:
+            return Client()
+
+        from ahriman.core.watcher.web_client import WebClient
+        return WebClient(host, port)

src/ahriman/core/watcher/watcher.py

@@ -0,0 +1,92 @@
+#
+# Copyright (c) 2021 Evgenii Alekseev.
+#
+# This file is part of ahriman
+# (see https://github.com/arcan1s/ahriman).
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+from typing import Dict, List, Optional, Tuple
+
+from ahriman.core.configuration import Configuration
+from ahriman.core.repository import Repository
+from ahriman.models.build_status import BuildStatus, BuildStatusEnum
+from ahriman.models.package import Package
+
+
+class Watcher:
+    '''
+    package status watcher
+    :ivar architecture: repository architecture
+    :ivar known: list of known packages. For the most cases `packages` should be used instead
+    :ivar repository: repository object
+    '''
+
+    def __init__(self, architecture: str, config: Configuration) -> None:
+        '''
+        default constructor
+        :param architecture: repository architecture
+        :param config: configuration instance
+        '''
+        self.architecture = architecture
+        self.repository = Repository(architecture, config)
+
+        self.known: Dict[str, Tuple[Package, BuildStatus]] = {}
+        self.status = BuildStatus()
+
+    @property
+    def packages(self) -> List[Tuple[Package, BuildStatus]]:
+        '''
+        :return: list of packages together with their statuses
+        '''
+        return [pair for pair in self.known.values()]
+
+    def load(self) -> None:
+        '''
+        load packages from local repository. In case if last status is known, it will use it
+        '''
+        for package in self.repository.packages():
+            # get status of build or assign unknown
+            current = self.known.get(package.base)
+            if current is None:
+                status = BuildStatus()
+            else:
+                _, status = current
+            self.known[package.base] = (package, status)
+
+    def remove(self, base: str) -> None:
+        '''
+        remove package base from known list if any
+        :param base: package base
+        '''
+        self.known.pop(base, None)
+
+    def update(self, base: str, status: BuildStatusEnum, package: Optional[Package]) -> None:
+        '''
+        update package status and description
+        :param base: package base to update
+        :param status: new build status
+        :param package: optional new package description. In case if not set current properties will be used
+        '''
+        if package is None:
+            package, _ = self.known[base]
+        full_status = BuildStatus(status)
+        self.known[base] = (package, full_status)
+
+    def update_self(self, status: BuildStatusEnum) -> None:
+        '''
+        update service status
+        :param status: new service status
+        '''
+        self.status = BuildStatus(status)

src/ahriman/core/watcher/web_client.py

@@ -0,0 +1,117 @@
+#
+# Copyright (c) 2021 Evgenii Alekseev.
+#
+# This file is part of ahriman
+# (see https://github.com/arcan1s/ahriman).
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+import logging
+import requests
+
+from dataclasses import asdict
+from typing import Any, Dict
+
+from ahriman.core.watcher.client import Client
+from ahriman.models.build_status import BuildStatusEnum
+from ahriman.models.package import Package
+
+
+class WebClient(Client):
+    '''
+    build status reporter web client
+    :ivar host: host of web service
+    :ivar logger: class logger
+    :ivar port: port of web service
+    '''
+
+    def __init__(self, host: str, port: int) -> None:
+        '''
+        default constructor
+        :param host: host of web service
+        :param port: port of web service
+        '''
+        self.logger = logging.getLogger('http')
+        self.host = host
+        self.port = port
+
+    def _ahriman_url(self) -> str:
+        '''
+        url generator
+        :return: full url for web service for ahriman service itself
+        '''
+        return f'http://{self.host}:{self.port}/api/v1/ahriman'
+
+    def _package_url(self, base: str) -> str:
+        '''
+        url generator
+        :param base: package base to generate url
+        :return: full url of web service for specific package base
+        '''
+        return f'http://{self.host}:{self.port}/api/v1/packages/{base}'
+
+    def add(self, package: Package, status: BuildStatusEnum) -> None:
+        '''
+        add new package with status
+        :param package: package properties
+        :param status: current package build status
+        '''
+        payload: Dict[str, Any] = {
+            'status': status.value,
+            'package': asdict(package)
+        }
+
+        try:
+            response = requests.post(self._package_url(package.base), json=payload)
+            response.raise_for_status()
+        except Exception:
+            self.logger.exception(f'could not add {package.base}', exc_info=True)
+
+    def remove(self, base: str) -> None:
+        '''
+        remove packages from watcher
+        :param base: basename to remove
+        '''
+        try:
+            response = requests.delete(self._package_url(base))
+            response.raise_for_status()
+        except Exception:
+            self.logger.exception(f'could not delete {base}', exc_info=True)
+
+    def update(self, base: str, status: BuildStatusEnum) -> None:
+        '''
+        update package build status. Unlike `add` it does not update package properties
+        :param base: package base to update
+        :param status: current package build status
+        '''
+        payload: Dict[str, Any] = {'status': status.value}
+
+        try:
+            response = requests.post(self._package_url(base), json=payload)
+            response.raise_for_status()
+        except Exception:
+            self.logger.exception(f'could not update {base}', exc_info=True)
+
+    def update_self(self, status: BuildStatusEnum) -> None:
+        '''
+        update ahriman status itself
+        :param status: current ahriman status
+        '''
+        payload: Dict[str, Any] = {'status': status.value}
+
+        try:
+            response = requests.post(self._ahriman_url(), json=payload)
+            response.raise_for_status()
+        except Exception:
+            self.logger.exception(f'could not update service status', exc_info=True)

src/ahriman/models/__init__.py

@@ -1,7 +1,7 @@
 #
 # Copyright (c) 2021 Evgenii Alekseev.
 #
-# This file is part of ahriman 
+# This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).
 #
 # This program is free software: you can redistribute it and/or modify

src/ahriman/models/build_status.py

@@ -0,0 +1,73 @@
+#
+# Copyright (c) 2021 Evgenii Alekseev.
+#
+# This file is part of ahriman
+# (see https://github.com/arcan1s/ahriman).
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+import datetime
+
+from enum import Enum
+from typing import Optional, Union
+
+
+class BuildStatusEnum(Enum):
+    '''
+    build status enumeration
+    :cvar Unknown: build status is unknown
+    :cvar Pending: package is out-of-dated and will be built soon
+    :cvar Building: package is building right now
+    :cvar Failed: package build failed
+    :cvar Success: package has been built without errors
+    '''
+
+    Unknown = 'unknown'
+    Pending = 'pending'
+    Building = 'building'
+    Failed = 'failed'
+    Success = 'success'
+
+    def badges_color(self) -> str:
+        '''
+        convert itself to shield.io badges color
+        :return: shields.io color
+        '''
+        if self == BuildStatusEnum.Pending:
+            return 'yellow'
+        elif self == BuildStatusEnum.Building:
+            return 'yellow'
+        elif self == BuildStatusEnum.Failed:
+            return 'critical'
+        elif self == BuildStatusEnum.Success:
+            return 'success'
+        return 'inactive'
+
+
+class BuildStatus:
+    '''
+    build status holder
+    :ivar status: build status
+    :ivar _timestamp: build status update time
+    '''
+
+    def __init__(self, status: Union[BuildStatusEnum, str, None] = None,
+                 timestamp: Optional[datetime.datetime] = None) -> None:
+        '''
+        default constructor
+        :param status: current build status if known. `BuildStatusEnum.Unknown` will be used if not set
+        :param timestamp: build status timestamp. Current timestamp will be used if not set
+        '''
+        self.status = BuildStatusEnum(status) if status else BuildStatusEnum.Unknown
+        self.timestamp = timestamp or datetime.datetime.utcnow()

src/ahriman/models/package.py

@@ -1,7 +1,7 @@
 #
 # Copyright (c) 2021 Evgenii Alekseev.
 #
-# This file is part of ahriman 
+# This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).
 #
 # This program is free software: you can redistribute it and/or modify
@@ -19,100 +19,189 @@
 #
 from __future__ import annotations
 
-import shutil
+import aur  # type: ignore
-
+import datetime
-import aur
 import os
-import tempfile
 
-from configparser import RawConfigParser
+from dataclasses import dataclass
-from dataclasses import dataclass, field
+from pyalpm import vercmp  # type: ignore
-from srcinfo.parse import parse_srcinfo
+from srcinfo.parse import parse_srcinfo  # type: ignore
-from typing import Set, Type
+from typing import Dict, List, Optional, Set, Type
 
+from ahriman.core.alpm.pacman import Pacman
 from ahriman.core.exceptions import InvalidPackageInfo
 from ahriman.core.util import check_output
+from ahriman.models.package_desciption import PackageDescription
+from ahriman.models.repository_paths import RepositoryPaths
 
 
 @dataclass
 class Package:
+    '''
+    package properties representation
+    :ivar aurl_url: AUR root url
+    :ivar base: package base name
+    :ivar packages: map of package names to their properties. Filled only on load from archive
+    :ivar version: package full version
+    '''
+
     base: str
     version: str
-    url: str
+    aur_url: str
-    packages: Set[str] = field(default_factory=set)
+    packages: Dict[str, PackageDescription]
+
+    @property
+    def git_url(self) -> str:
+        '''
+        :return: package git url to clone
+        '''
+        return f'{self.aur_url}/{self.base}.git'
 
     @property
     def is_vcs(self) -> bool:
+        '''
+        :return: True in case if package base looks like VCS package and false otherwise
+        '''
         return self.base.endswith('-bzr') \
-               or self.base.endswith('-csv')\
+            or self.base.endswith('-csv')\
-               or self.base.endswith('-darcs')\
+            or self.base.endswith('-darcs')\
-               or self.base.endswith('-git')\
+            or self.base.endswith('-git')\
-               or self.base.endswith('-hg')\
+            or self.base.endswith('-hg')\
-               or self.base.endswith('-svn')
+            or self.base.endswith('-svn')
 
-    # additional method to handle vcs versions
+    @property
-    def actual_version(self) -> str:
+    def web_url(self) -> str:
+        '''
+        :return: package AUR url
+        '''
+        return f'{self.aur_url}/packages/{self.base}'
+
+    def actual_version(self, paths: RepositoryPaths) -> str:
+        '''
+        additional method to handle VCS package versions
+        :param paths: repository paths instance
+        :return: package version if package is not VCS and current version according to VCS otherwise
+        '''
         if not self.is_vcs:
             return self.version
 
         from ahriman.core.build_tools.task import Task
-        clone_dir = tempfile.mkdtemp()
+        clone_dir = os.path.join(paths.cache, self.base)
-        try:
+
-            Task.fetch(clone_dir, self.url)
+        Task.fetch(clone_dir, self.git_url)
-            # update pkgver first
+        # update pkgver first
-            check_output('makepkg', '--nodeps', '--noprepare', '--nobuild',
+        check_output('makepkg', '--nodeps', '--nobuild', exception=None, cwd=clone_dir)
-                         exception=None, cwd=clone_dir)
+        # generate new .SRCINFO and put it to parser
-            # generate new .SRCINFO and put it to parser
+        src_info_source = check_output('makepkg', '--printsrcinfo', exception=None, cwd=clone_dir)
-            src_info_source = check_output('makepkg', '--printsrcinfo',
+        src_info, errors = parse_srcinfo(src_info_source)
-                                           exception=None, cwd=clone_dir)
+        if errors:
-            src_info, errors = parse_srcinfo(src_info_source)
+            raise InvalidPackageInfo(errors)
-            if errors:
+
-                raise InvalidPackageInfo(errors)
+        return self.full_version(src_info.get('epoch'), src_info['pkgver'], src_info['pkgrel'])
-            return f'{src_info["pkgver"]}-{src_info["pkgrel"]}'
-        finally:
-            shutil.rmtree(clone_dir, ignore_errors=True)
 
     @classmethod
-    def from_archive(cls: Type[Package], path: str, aur_url: str) -> Package:
+    def from_archive(cls: Type[Package], path: str, pacman: Pacman, aur_url: str) -> Package:
-        package, base, version = check_output('expac', '-p', '%n %e %v', path, exception=None).split()
+        '''
-        return cls(base, version, f'{aur_url}/{base}.git', packages={package})
+        construct package properties from package archive
+        :param path: path to package archive
+        :param pacman: alpm wrapper instance
+        :param aur_url: AUR root url
+        :return: package properties
+        '''
+        package = pacman.handle.load_pkg(path)
+        build_date = datetime.datetime.fromtimestamp(package.builddate)
+        properties = PackageDescription(package.size, build_date, os.path.basename(path), package.isize)
+        return cls(package.base, package.version, aur_url, {package.name: properties})
 
     @classmethod
-    def from_aur(cls: Type[Package], name: str, aur_url: str)-> Package:
+    def from_aur(cls: Type[Package], name: str, aur_url: str) -> Package:
+        '''
+        construct package properties from AUR page
+        :param name: package name (either base or normal name)
+        :param aur_url: AUR root url
+        :return: package properties
+        '''
         package = aur.info(name)
-        return cls(package.package_base, package.version, f'{aur_url}/{package.package_base}.git',
+        return cls(package.package_base, package.version, aur_url, {package.name: PackageDescription()})
-                   packages={package.name})
 
     @classmethod
-    def from_build(cls: Type[Package], path: str) -> Package:
+    def from_build(cls: Type[Package], path: str, aur_url: str) -> Package:
-        git_config = RawConfigParser()
+        '''
-        git_config.read(os.path.join(path, '.git', 'config'))
+        construct package properties from sources directory
-
+        :param path: path to package sources directory
+        :param aur_url: AUR root url
+        :return: package properties
+        '''
         with open(os.path.join(path, '.SRCINFO')) as fn:
             src_info, errors = parse_srcinfo(fn.read())
         if errors:
             raise InvalidPackageInfo(errors)
+        packages = {key: PackageDescription() for key in src_info['packages'].keys()}
+        version = cls.full_version(src_info.get('epoch'), src_info['pkgver'], src_info['pkgrel'])
+
+        return cls(src_info['pkgbase'], version, aur_url, packages)
+
+    @staticmethod
+    def dependencies(path: str) -> Set[str]:
+        '''
+        load dependencies from package sources
+        :param path: path to package sources directory
+        :return: list of package dependencies including makedepends array, but excluding packages from this base
+        '''
+        with open(os.path.join(path, '.SRCINFO')) as fn:
+            src_info, errors = parse_srcinfo(fn.read())
+        if errors:
+            raise InvalidPackageInfo(errors)
+        makedepends = src_info.get('makedepends', [])
+        # sum over each package
+        depends: List[str] = src_info.get('depends', [])
+        for package in src_info['packages'].values():
+            depends.extend(package.get('depends', []))
+        # we are not interested in dependencies inside pkgbase
         packages = set(src_info['packages'].keys())
+        return set(depends + makedepends) - packages
 
-        return cls(src_info['pkgbase'], f'{src_info["pkgver"]}-{src_info["pkgrel"]}',
+    @staticmethod
-                   git_config.get('remote "origin"', 'url'), packages-packages)
+    def full_version(epoch: Optional[str], pkgver: str, pkgrel: str) -> str:
+        '''
+        generate full version from components
+        :param epoch: package epoch if any
+        :param pkgver: package version
+        :param pkgrel: package release version (archlinux specific)
+        :return: generated version
+        '''
+        prefix = f'{epoch}:' if epoch else ''
+        return f'{prefix}{pkgver}-{pkgrel}'
 
-    @classmethod
+    @staticmethod
-    def load(cls: Type[Package], path: str, aur_url: str) -> Package:
+    def load(path: str, pacman: Pacman, aur_url: str) -> Package:
+        '''
+        package constructor from available sources
+        :param path: one of path to sources directory, path to archive or package name/base
+        :param pacman: alpm wrapper instance (required to load from archive)
+        :param aur_url: AUR root url
+        :return: package properties
+        '''
         try:
             if os.path.isdir(path):
-                package: Package = cls.from_build(path)
+                package: Package = Package.from_build(path, aur_url)
             elif os.path.exists(path):
-                package = cls.from_archive(path, aur_url)
+                package = Package.from_archive(path, pacman, aur_url)
             else:
-                package = cls.from_aur(path, aur_url)
+                package = Package.from_aur(path, aur_url)
             return package
         except InvalidPackageInfo:
             raise
         except Exception as e:
             raise InvalidPackageInfo(str(e))
 
-    def is_outdated(self, remote: Package) -> bool:
+    def is_outdated(self, remote: Package, paths: RepositoryPaths) -> bool:
-        remote_version = remote.actual_version()  # either normal version or updated VCS
+        '''
-        result = check_output('vercmp', self.version, remote_version, exception=None)
+        check if package is out-of-dated
-        return True if int(result) < 0 else False
+        :param remote: package properties from remote source
+        :param paths: repository paths instance. Required for VCS packages cache
+        :return: True if the package is out-of-dated and False otherwise
+        '''
+        remote_version = remote.actual_version(paths)  # either normal version or updated VCS
+        result: int = vercmp(self.version, remote_version)
+        return result < 0

src/ahriman/models/package_desciption.py

@@ -0,0 +1,39 @@
+#
+# Copyright (c) 2021 Evgenii Alekseev.
+#
+# This file is part of ahriman
+# (see https://github.com/arcan1s/ahriman).
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+import datetime
+
+from dataclasses import dataclass
+from typing import Optional
+
+
+@dataclass
+class PackageDescription:
+    '''
+    package specific properties
+    :ivar archive_size: package archive size
+    :ivar build_date: package build date
+    :ivar filename: package archive name
+    :ivar installed_size: package installed size
+    '''
+
+    archive_size: Optional[int] = None
+    build_date: Optional[datetime.datetime] = None
+    filename: Optional[str] = None
+    installed_size: Optional[int] = None

src/ahriman/models/report_settings.py

@@ -1,7 +1,7 @@
 #
 # Copyright (c) 2021 Evgenii Alekseev.
 #
-# This file is part of ahriman 
+# This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).
 #
 # This program is free software: you can redistribute it and/or modify
@@ -20,16 +20,25 @@
 from __future__ import annotations
 
 from enum import Enum, auto
-from typing import Type
 
-from ahriman.core.exceptions import InvalidOptionException
+from ahriman.core.exceptions import InvalidOption
 
 
 class ReportSettings(Enum):
+    '''
+    report targets enumeration
+    :cvar HTML: html report generation
+    '''
+
     HTML = auto()
 
-    @classmethod
+    @staticmethod
-    def from_option(cls: Type[ReportSettings], value: str) -> ReportSettings:
+    def from_option(value: str) -> ReportSettings:
+        '''
+        construct value from configuration
+        :param value: configuration value
+        :return: parsed value
+        '''
         if value.lower() in ('html',):
-            return cls.HTML
+            return ReportSettings.HTML
-        raise InvalidOptionException(value)
+        raise InvalidOption(value)

src/ahriman/models/repository_paths.py

@@ -1,7 +1,7 @@
 #
 # Copyright (c) 2021 Evgenii Alekseev.
 #
-# This file is part of ahriman 
+# This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).
 #
 # This program is free software: you can redistribute it and/or modify
@@ -24,32 +24,65 @@ from dataclasses import dataclass
 
 @dataclass
 class RepositoryPaths:
+    '''
+    repository paths holder. For the most operations with paths you want to use this object
+    :ivar root: repository root (i.e. ahriman home)
+    :ivar architecture: repository architecture
+    '''
+
     root: str
     architecture: str
 
+    @property
+    def cache(self) -> str:
+        '''
+        :return: directory for packages cache (mainly used for VCS packages)
+        '''
+        return os.path.join(self.root, 'cache')
+
     @property
     def chroot(self) -> str:
+        '''
+        :return: directory for devtools chroot
+        '''
+        # for the chroot directory devtools will create own tree and we don't have to specify architecture here
         return os.path.join(self.root, 'chroot')
 
     @property
     def manual(self) -> str:
-        return os.path.join(self.root, 'manual')
+        '''
+        :return: directory for manual updates (i.e. from add command)
+        '''
+        return os.path.join(self.root, 'manual', self.architecture)
 
     @property
     def packages(self) -> str:
-        return os.path.join(self.root, 'packages')
+        '''
+        :return: directory for built packages
+        '''
+        return os.path.join(self.root, 'packages', self.architecture)
 
     @property
     def repository(self) -> str:
+        '''
+        :return: repository directory
+        '''
         return os.path.join(self.root, 'repository', self.architecture)
 
     @property
     def sources(self) -> str:
-        return os.path.join(self.root, 'sources')
+        '''
+        :return: directory for downloaded PKGBUILDs for current build
+        '''
+        return os.path.join(self.root, 'sources', self.architecture)
 
     def create_tree(self) -> None:
+        '''
+        create ahriman working tree
+        '''
+        os.makedirs(self.cache, mode=0o755, exist_ok=True)
         os.makedirs(self.chroot, mode=0o755, exist_ok=True)
         os.makedirs(self.manual, mode=0o755, exist_ok=True)
         os.makedirs(self.packages, mode=0o755, exist_ok=True)
         os.makedirs(self.repository, mode=0o755, exist_ok=True)
-        os.makedirs(self.sources, mode=0o755, exist_ok=True)
+        os.makedirs(self.sources, mode=0o755, exist_ok=True)

src/ahriman/models/sign_settings.py

@@ -1,7 +1,7 @@
 #
 # Copyright (c) 2021 Evgenii Alekseev.
 #
-# This file is part of ahriman 
+# This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).
 #
 # This program is free software: you can redistribute it and/or modify
@@ -20,22 +20,29 @@
 from __future__ import annotations
 
 from enum import Enum, auto
-from typing import Type
 
-from ahriman.core.exceptions import InvalidOptionException
+from ahriman.core.exceptions import InvalidOption
 
 
 class SignSettings(Enum):
-    Disabled = auto()
+    '''
+    sign targets enumeration
+    :cvar SignPackages: sign each package
+    :cvar SignRepository: sign repository database file
+    '''
+
     SignPackages = auto()
     SignRepository = auto()
 
-    @classmethod
+    @staticmethod
-    def from_option(cls: Type[SignSettings], value: str) -> SignSettings:
+    def from_option(value: str) -> SignSettings:
-        if value.lower() in ('no', 'disabled'):
+        '''
-            return cls.Disabled
+        construct value from configuration
-        elif value.lower() in ('package', 'packages', 'sign-package'):
+        :param value: configuration value
-            return cls.SignPackages
+        :return: parsed value
+        '''
+        if value.lower() in ('package', 'packages', 'sign-package'):
+            return SignSettings.SignPackages
         elif value.lower() in ('repository', 'sign-repository'):
-            return cls.SignRepository
+            return SignSettings.SignRepository
-        raise InvalidOptionException(value)
+        raise InvalidOption(value)

src/ahriman/models/upload_settings.py

@@ -1,7 +1,7 @@
 #
 # Copyright (c) 2021 Evgenii Alekseev.
 #
-# This file is part of ahriman 
+# This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).
 #
 # This program is free software: you can redistribute it and/or modify
@@ -20,19 +20,29 @@
 from __future__ import annotations
 
 from enum import Enum, auto
-from typing import Type
 
-from ahriman.core.exceptions import InvalidOptionException
+from ahriman.core.exceptions import InvalidOption
 
 
 class UploadSettings(Enum):
+    '''
+    remote synchronization targets enumeration
+    :cvar Rsync: sync via rsync
+    :cvar S3: sync to Amazon S3
+    '''
+
     Rsync = auto()
     S3 = auto()
 
-    @classmethod
+    @staticmethod
-    def from_option(cls: Type[UploadSettings], value: str) -> UploadSettings:
+    def from_option(value: str) -> UploadSettings:
+        '''
+        construct value from configuration
+        :param value: configuration value
+        :return: parsed value
+        '''
         if value.lower() in ('rsync',):
-            return cls.Rsync
+            return UploadSettings.Rsync
         elif value.lower() in ('s3',):
-            return cls.S3
+            return UploadSettings.S3
-        raise InvalidOptionException(value)
+        raise InvalidOption(value)

src/ahriman/version.py

@@ -1,7 +1,7 @@
 #
 # Copyright (c) 2021 Evgenii Alekseev.
 #
-# This file is part of ahriman 
+# This file is part of ahriman
 # (see https://github.com/arcan1s/ahriman).
 #
 # This program is free software: you can redistribute it and/or modify
@@ -17,4 +17,4 @@
 # You should have received a copy of the GNU General Public License
 # along with this program. If not, see <http://www.gnu.org/licenses/>.
 #
-__version__ = '0.9.1'
+__version__ = '0.13.0'

src/ahriman/web/__init__.py

@@ -17,10 +17,3 @@
 # You should have received a copy of the GNU General Public License
 # along with this program. If not, see <http://www.gnu.org/licenses/>.
 #
-from ahriman.core.upload.uploader import Uploader
-
-
-class Dummy(Uploader):
-
-    def sync(self, path: str) -> None:
-        pass

src/ahriman/web/middlewares/__init__.py

@@ -0,0 +1,19 @@
+#
+# Copyright (c) 2021 Evgenii Alekseev.
+#
+# This file is part of ahriman
+# (see https://github.com/arcan1s/ahriman).
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#

src/ahriman/web/middlewares/exception_handler.py

@@ -0,0 +1,46 @@
+#
+# Copyright (c) 2021 Evgenii Alekseev.
+#
+# This file is part of ahriman
+# (see https://github.com/arcan1s/ahriman).
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+from aiohttp.web import middleware, Request
+from aiohttp.web_exceptions import HTTPClientError
+from aiohttp.web_response import StreamResponse
+from logging import Logger
+from typing import Awaitable, Callable
+
+
+HandlerType = Callable[[Request], Awaitable[StreamResponse]]
+
+
+def exception_handler(logger: Logger) -> Callable[[Request, HandlerType], Awaitable[StreamResponse]]:
+    '''
+    exception handler middleware. Just log any exception (except for client ones)
+    :param logger: class logger
+    :return: built middleware
+    '''
+    @middleware
+    async def handle(request: Request, handler: HandlerType) -> StreamResponse:
+        try:
+            return await handler(request)
+        except HTTPClientError:
+            raise
+        except Exception:
+            logger.exception(f'exception during performing request to {request.path}', exc_info=True)
+            raise
+
+    return handle

src/ahriman/web/routes.py

@@ -0,0 +1,54 @@
+#
+# Copyright (c) 2021 Evgenii Alekseev.
+#
+# This file is part of ahriman
+# (see https://github.com/arcan1s/ahriman).
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+from aiohttp.web import Application
+
+from ahriman.web.views.ahriman import AhrimanView
+from ahriman.web.views.index import IndexView
+from ahriman.web.views.package import PackageView
+from ahriman.web.views.packages import PackagesView
+
+
+def setup_routes(application: Application) -> None:
+    '''
+    setup all defined routes
+
+    Available routes are:
+
+        GET /                           get build status page
+        GET /index.html                 same as above
+
+        POST /api/v1/ahriman            update service status
+
+        POST /api/v1/packages           force update every package from repository
+
+        POST /api/v1/package/:base      update package base status
+        DELETE /api/v1/package/:base    delete package base from status page
+
+    :param application: web application instance
+    '''
+    application.router.add_get('/', IndexView)
+    application.router.add_get('/index.html', IndexView)
+
+    application.router.add_post('/api/v1/ahriman', AhrimanView)
+
+    application.router.add_post('/api/v1/packages', PackagesView)
+
+    application.router.add_delete('/api/v1/packages/{package}', PackageView)
+    application.router.add_post('/api/v1/packages/{package}', PackageView)

src/ahriman/web/views/__init__.py

@@ -0,0 +1,19 @@
+#
+# Copyright (c) 2021 Evgenii Alekseev.
+#
+# This file is part of ahriman
+# (see https://github.com/arcan1s/ahriman).
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#

src/ahriman/web/views/ahriman.py

@@ -0,0 +1,51 @@
+#
+# Copyright (c) 2021 Evgenii Alekseev.
+#
+# This file is part of ahriman
+# (see https://github.com/arcan1s/ahriman).
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+from aiohttp.web import HTTPBadRequest, HTTPOk, Response
+
+from ahriman.models.build_status import BuildStatusEnum
+from ahriman.web.views.base import BaseView
+
+
+class AhrimanView(BaseView):
+    '''
+    service status web view
+    '''
+
+    async def post(self) -> Response:
+        '''
+        update service status
+
+        JSON body must be supplied, the following model is used:
+        {
+            "status": "unknown",   # service status string, must be valid `BuildStatusEnum`
+        }
+
+        :return: 200 on success
+        '''
+        data = await self.request.json()
+
+        try:
+            status = BuildStatusEnum(data['status'])
+        except Exception as e:
+            raise HTTPBadRequest(text=str(e))
+
+        self.service.update_self(status)
+
+        return HTTPOk()

src/ahriman/web/views/base.py

@@ -0,0 +1,36 @@
+#
+# Copyright (c) 2021 Evgenii Alekseev.
+#
+# This file is part of ahriman
+# (see https://github.com/arcan1s/ahriman).
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+from aiohttp.web import View
+
+from ahriman.core.watcher.watcher import Watcher
+
+
+class BaseView(View):
+    '''
+    base web view to make things typed
+    '''
+
+    @property
+    def service(self) -> Watcher:
+        '''
+        :return: build status watcher instance
+        '''
+        watcher: Watcher = self.request.app['watcher']
+        return watcher

src/ahriman/web/views/index.py

@@ -0,0 +1,73 @@
+#
+# Copyright (c) 2021 Evgenii Alekseev.
+#
+# This file is part of ahriman
+# (see https://github.com/arcan1s/ahriman).
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+import aiohttp_jinja2  # type: ignore
+
+from typing import Any, Dict
+
+import ahriman.version as version
+
+from ahriman.core.util import pretty_datetime
+from ahriman.web.views.base import BaseView
+
+
+class IndexView(BaseView):
+    '''
+    root view
+
+    It uses jinja2 templates for report generation, the following variables are allowed:
+
+        architecture - repository architecture, string, required
+        packages - sorted list of packages properties: base, packages (sorted list), status,
+                   timestamp, version, web_url. Required
+        repository - repository name, string, required
+        service - service status properties: status, status_color, timestamp. Required
+        version - ahriman version, string, required
+    '''
+
+    @aiohttp_jinja2.template("build-status.jinja2")  # type: ignore
+    async def get(self) -> Dict[str, Any]:
+        '''
+        process get request. No parameters supported here
+        :return: parameters for jinja template
+        '''
+        # some magic to make it jinja-friendly
+        packages = [
+            {
+                'base': package.base,
+                'packages': [p for p in sorted(package.packages)],
+                'status': status.status.value,
+                'timestamp': pretty_datetime(status.timestamp),
+                'version': package.version,
+                'web_url': package.web_url
+            } for package, status in sorted(self.service.packages, key=lambda item: item[0].base)
+        ]
+        service = {
+            'status': self.service.status.status.value,
+            'status_color': self.service.status.status.badges_color(),
+            'timestamp': pretty_datetime(self.service.status.timestamp)
+        }
+
+        return {
+            'architecture': self.service.architecture,
+            'packages': packages,
+            'repository': self.service.repository.name,
+            'service': service,
+            'version': version.__version__,
+        }

src/ahriman/web/views/package.py

@@ -0,0 +1,69 @@
+#
+# Copyright (c) 2021 Evgenii Alekseev.
+#
+# This file is part of ahriman
+# (see https://github.com/arcan1s/ahriman).
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+from aiohttp.web import HTTPBadRequest, HTTPOk, Response
+
+from ahriman.models.build_status import BuildStatusEnum
+from ahriman.models.package import Package
+from ahriman.web.views.base import BaseView
+
+
+class PackageView(BaseView):
+    '''
+    package base specific web view
+    '''
+
+    async def delete(self) -> Response:
+        '''
+        delete package base from status page
+        :return: 200 on success
+        '''
+        base = self.request.match_info['package']
+        self.service.remove(base)
+
+        return HTTPOk()
+
+    async def post(self) -> Response:
+        '''
+        update package build status
+
+        JSON body must be supplied, the following model is used:
+        {
+            "status": "unknown",   # package build status string, must be valid `BuildStatusEnum`
+            "package": {}  # package body (use `dataclasses.asdict` to generate one), optional.
+                           # Must be supplied in case if package base is unknown
+        }
+
+        :return: 200 on success
+        '''
+        base = self.request.match_info['package']
+        data = await self.request.json()
+
+        try:
+            package = Package(**data['package']) if 'package' in data else None
+            status = BuildStatusEnum(data['status'])
+        except Exception as e:
+            raise HTTPBadRequest(text=str(e))
+
+        try:
+            self.service.update(base, status, package)
+        except KeyError:
+            raise HTTPBadRequest(text=f'Package {base} is unknown, but no package body set')
+
+        return HTTPOk()

src/ahriman/web/views/packages.py

@@ -0,0 +1,37 @@
+#
+# Copyright (c) 2021 Evgenii Alekseev.
+#
+# This file is part of ahriman
+# (see https://github.com/arcan1s/ahriman).
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+from aiohttp.web import HTTPOk, Response
+
+from ahriman.web.views.base import BaseView
+
+
+class PackagesView(BaseView):
+    '''
+    global watcher view
+    '''
+
+    async def post(self) -> Response:
+        '''
+        reload all packages from repository. No parameters supported here
+        :return: 200 on success
+        '''
+        self.service.load()
+
+        return HTTPOk()

src/ahriman/web/web.py

@@ -0,0 +1,94 @@
+#
+# Copyright (c) 2021 Evgenii Alekseev.
+#
+# This file is part of ahriman
+# (see https://github.com/arcan1s/ahriman).
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+import aiohttp_jinja2  # type: ignore
+import jinja2
+import logging
+
+from aiohttp import web
+
+from ahriman.core.configuration import Configuration
+from ahriman.core.exceptions import InitializeException
+from ahriman.core.watcher.watcher import Watcher
+from ahriman.web.middlewares.exception_handler import exception_handler
+from ahriman.web.routes import setup_routes
+
+
+async def on_shutdown(application: web.Application) -> None:
+    '''
+    web application shutdown handler
+    :param application: web application instance
+    '''
+    application.logger.warning('server terminated')
+
+
+async def on_startup(application: web.Application) -> None:
+    '''
+    web application start handler
+    :param application: web application instance
+    '''
+    application.logger.info('server started')
+    try:
+        application['watcher'].load()
+    except Exception:
+        application.logger.exception('could not load packages', exc_info=True)
+        raise InitializeException()
+
+
+def run_server(application: web.Application, architecture: str) -> None:
+    '''
+    run web application
+    :param application: web application instance
+    :param architecture: repository architecture
+    '''
+    application.logger.info('start server')
+
+    section = application['config'].get_section_name('web', architecture)
+    host = application['config'].get(section, 'host')
+    port = application['config'].getint(section, 'port')
+
+    web.run_app(application, host=host, port=port, handle_signals=False)
+
+
+def setup_service(architecture: str, config: Configuration) -> web.Application:
+    '''
+    create web application
+    :param architecture: repository architecture
+    :param config: configuration instance
+    :return: web application instance
+    '''
+    app = web.Application(logger=logging.getLogger('http'))
+    app.on_shutdown.append(on_shutdown)
+    app.on_startup.append(on_startup)
+
+    app.middlewares.append(web.normalize_path_middleware(append_slash=False, remove_slash=True))
+    app.middlewares.append(exception_handler(app.logger))
+
+    app.logger.info('setup routes')
+    setup_routes(app)
+    app.logger.info('setup templates')
+    aiohttp_jinja2.setup(app, loader=jinja2.FileSystemLoader(config.get('web', 'templates')))
+
+    app.logger.info('setup configuration')
+    app['config'] = config
+
+    app.logger.info('setup watcher')
+    app['watcher'] = Watcher(architecture, config)
+
+    return app