improvements

* multi-sign and multi-web configuration
* change default configuration to do not use architecture
* change units to be templated
* some refactoring
This commit is contained in:
Evgenii Alekseev 2021-03-11 03:55:17 +03:00
parent 30ededb2cd
commit 1770793e69
32 changed files with 235 additions and 173 deletions

View File

@ -1,17 +1,17 @@
# ahriman configuration # ahriman configuration
Some groups can be specified for each architecture separately with default values. E.g. if there are `build` and `build_x86_64` groups it will use the `build_x86_64` for the `x86_64` architecture and `build` for any other. Some groups can be specified for each architecture separately. E.g. if there are `build` and `build_x86_64` groups it will use the `build_x86_64` for the `x86_64` architecture and `build` for any other (architecture specific group has higher priority).
## `settings` group ## `settings` group
Base configuration settings: Base configuration settings.
* `include` - path to directory with configuration files overrides, string, required. * `include` - path to directory with configuration files overrides, string, required.
* `logging` - path to logging configuration, string, required. Check `logging.ini` for reference. * `logging` - path to logging configuration, string, required. Check `logging.ini` for reference.
## `aur` group ## `aur` group
AUR related configuration: AUR related configuration.
* `url` - base url for AUR, string, required. * `url` - base url for AUR, string, required.
@ -27,25 +27,25 @@ Build related configuration. Group name must refer to architecture, e.g. it shou
## `repository` group ## `repository` group
Base repository settings: Base repository settings.
* `name` - repository name, string, required. * `name` - repository name, string, required.
* `root` - root path for application, string, required. * `root` - root path for application, string, required.
## `sign` group ## `sign_*` groups
Settings for signing packages or repository: Settings for signing packages or repository. Group name must refer to architecture, e.g. it should be `sign_x86_64` for x86_64 architecture.
* `target` - configuration flag to enable signing, space separated list of strings, required. Allowed values are `package` (sign each package separately), `repository` (sign repository database file). * `target` - configuration flag to enable signing, space separated list of strings, required. Allowed values are `package` (sign each package separately), `repository` (sign repository database file).
* `key` - PGP key, string, required. * `key` - PGP key, string, required.
## `report` group ## `report` group
Report generation settings: Report generation settings.
* `target` - list of reports to be generated, space separated list of strings, optional. Allowed values are `html`. * `target` - list of reports to be generated, space separated list of strings, optional. Allowed values are `html`.
### `html_*` group ### `html_*` groups
Group name must refer to architecture, e.g. it should be `html_x86_64` for x86_64 architecture. Group name must refer to architecture, e.g. it should be `html_x86_64` for x86_64 architecture.
@ -56,25 +56,25 @@ Group name must refer to architecture, e.g. it should be `html_x86_64` for x86_6
## `upload` group ## `upload` group
Remote synchronization settings: Remote synchronization settings.
* `target` - list of synchronizations to be used, space separated list of strings, optional. Allowed values are `rsync`, `s3`. * `target` - list of synchronizations to be used, space separated list of strings, optional. Allowed values are `rsync`, `s3`.
### `rsync_*` group ### `rsync_*` groups
Group name must refer to architecture, e.g. it should be `rsync_x86_64` for x86_64 architecture. Requires `rsync` package to be installed. Do not forget to configure ssh for user `ahriman`. Group name must refer to architecture, e.g. it should be `rsync_x86_64` for x86_64 architecture. Requires `rsync` package to be installed. Do not forget to configure ssh for user `ahriman`.
* `remote` - remote server to rsync (e.g. `1.2.3.4:5678:path/to/sync`), string, required. * `remote` - remote server to rsync (e.g. `1.2.3.4:5678:path/to/sync`), string, required.
### `s3_*` group ### `s3_*` groups
Group name must refer to architecture, e.g. it should be `s3_x86_64` for x86_64 architecture. Requires `aws-cli` package to be installed. Do not forget to configure it for user `ahriman`. Group name must refer to architecture, e.g. it should be `s3_x86_64` for x86_64 architecture. Requires `aws-cli` package to be installed. Do not forget to configure it for user `ahriman`.
* `bucket` - bucket name (e.g. `s3://bucket/path`), string, required. * `bucket` - bucket name (e.g. `s3://bucket/path`), string, required.
## `web` group ## `web_*` groups
Web server settings. If any of `host`/`port` is not set, web intergration will be disabled. Web server settings. If any of `host`/`port` is not set, web integration will be disabled. Group name must refer to architecture, e.g. it should be `web_x86_64` for x86_64 architecture.
* `host` - host to bind, string, optional. * `host` - host to bind, string, optional.
* `port` - port to bind, int, optional. * `port` - port to bind, int, optional.

View File

@ -5,9 +5,10 @@ Wrapper for managing custom repository inspired by [repo-scripts](https://github
## Features ## Features
* Install-configure-forget manager for own repository * Install-configure-forget manager for own repository
* Multi-architecture support
* VCS packages support * VCS packages support
* Sign support with gpg * Sign support with gpg
* Synchronization to remote services and report generation * Synchronization to remote services (rsync, s3) and report generation (html)
* Repository status interface * Repository status interface
## Installation and run ## Installation and run
@ -16,11 +17,11 @@ Wrapper for managing custom repository inspired by [repo-scripts](https://github
* Change settings if required, see `CONFIGURING.md` for more details. * Change settings if required, see `CONFIGURING.md` for more details.
* Create `/var/lib/ahriman/.makepkg.conf` with `makepkg.conf` overrides if required (at least you might want to set `PACKAGER`). * Create `/var/lib/ahriman/.makepkg.conf` with `makepkg.conf` overrides if required (at least you might want to set `PACKAGER`).
* Configure build tools (it might be required if your package will use any custom repositories): * Configure build tools (it might be required if your package will use any custom repositories):
* create build command if required, e.g. `ln -s /usr/bin/archbuild /usr/local/bin/custom-x86_64-build` (you can choose any name for command); * create build command, e.g. `ln -s /usr/bin/archbuild /usr/local/bin/custom-x86_64-build` (you can choose any name for command);
* create configuration file, e.g. `cp /usr/share/devtools/pacman-{extra,custom}.conf`; * create configuration file, e.g. `cp /usr/share/devtools/pacman-{extra,custom}.conf`;
* change configuration file: add your own repository, add multilib repository; * change configuration file, add your own repository, add multilib repository etc;
* set `build.build_command` to point to your command; * set `build.build_command` setting to point to your command;
* configure `/etc/sudoers.d/ahriman` to allow to run command without password. * configure `/etc/sudoers.d/ahriman` to allow running command without password.
* Start and enable `ahriman.timer` via `systemctl`. * Start and enable `ahriman.timer` via `systemctl`.
* Add packages by using `ahriman add {package}` command. * Add packages by using `ahriman add {package}` command.

View File

@ -21,11 +21,9 @@ optdepends=('aws-cli: sync to s3'
'rsync: sync by using rsync' 'rsync: sync by using rsync'
'subversion: -svn packages support') 'subversion: -svn packages support')
source=("https://github.com/arcan1s/ahriman/releases/download/$pkgver/$pkgname-$pkgver-src.tar.xz" source=("https://github.com/arcan1s/ahriman/releases/download/$pkgver/$pkgname-$pkgver-src.tar.xz"
'ahriman.sudoers'
'ahriman.sysusers' 'ahriman.sysusers'
'ahriman.tmpfiles') 'ahriman.tmpfiles')
sha512sums=('2c811060106aea6f8826cc6beac9f5733370386a43448b359051ea377e233218aae0c5ec3ef7b1ec399fa6a53c02059015b7398b0d88b5a2e7129f167d025539' sha512sums=('941821639fe4410152a21251d9b0fe5f96ee3a60b88e2067ea4a83ef04b5d1393828152ef4843575449bdef8d44ad6a69f9e41e82516d4d1850bd14f17822785'
'8c9b5b63ac3f7b4d9debaf801a1e9c060877c33d3ecafe18010fcca778e5fa2f2e46909d3d0ff1b229ff8aa978445d8243fd36e1fc104117ed678d5e21901167'
'13718afec2c6786a18f0b223ef8e58dccf0688bca4cdbe203f14071f5031ed20120eb0ce38b52c76cfd6e8b6581a9c9eaa2743eb11abbaca637451a84c33f075' '13718afec2c6786a18f0b223ef8e58dccf0688bca4cdbe203f14071f5031ed20120eb0ce38b52c76cfd6e8b6581a9c9eaa2743eb11abbaca637451a84c33f075'
'55b20f6da3d66e7bbf2add5d95a3b60632df121717d25a993e56e737d14f51fe063eb6f1b38bd81cc32e05db01c0c1d80aaa720c45cde87f238d8b46cdb8cbc4') '55b20f6da3d66e7bbf2add5d95a3b60632df121717d25a993e56e737d14f51fe063eb6f1b38bd81cc32e05db01c0c1d80aaa720c45cde87f238d8b46cdb8cbc4')
backup=('etc/ahriman.ini' backup=('etc/ahriman.ini'
@ -42,7 +40,6 @@ package() {
python setup.py install --root="$pkgdir" python setup.py install --root="$pkgdir"
install -Dm400 "$srcdir/$pkgname.sudoers" "$pkgdir/etc/sudoers.d/$pkgname"
install -Dm644 "$srcdir/$pkgname.sysusers" "$pkgdir/usr/lib/sysusers.d/$pkgname.conf" install -Dm644 "$srcdir/$pkgname.sysusers" "$pkgdir/usr/lib/sysusers.d/$pkgname.conf"
install -Dm644 "$srcdir/$pkgname.tmpfiles" "$pkgdir/usr/lib/tmpfiles.d/$pkgname.conf" install -Dm644 "$srcdir/$pkgname.tmpfiles" "$pkgdir/usr/lib/tmpfiles.d/$pkgname.conf"
} }

View File

@ -1,4 +0,0 @@
# Used by ArcHlinux ReposItory MANager with default settings
Cmnd_Alias ARCHBUILD_CMD = /usr/bin/extra-x86_64-build *, /usr/bin/multilib-build *
ahriman ALL=(ALL) NOPASSWD: ARCHBUILD_CMD

View File

@ -5,7 +5,7 @@ logging = /etc/ahriman.ini.d/logging.ini
[aur] [aur]
url = https://aur.archlinux.org url = https://aur.archlinux.org
[build_x86_64] [build]
archbuild_flags = archbuild_flags =
build_command = extra-x86_64-build build_command = extra-x86_64-build
ignore_packages = ignore_packages =
@ -23,7 +23,7 @@ key =
[report] [report]
target = target =
[html_x86_64] [html]
path = path =
homepage = homepage =
link_path = link_path =
@ -32,10 +32,10 @@ template_path = /usr/share/ahriman/repo-index.jinja2
[upload] [upload]
target = target =
[rsync_x86_64] [rsync]
remote = remote =
[s3_x86_64] [s3]
bucket = bucket =
[web] [web]

View File

@ -1,11 +0,0 @@
[Unit]
Description=ArcHlinux ReposItory MANager web server
[Service]
Type=simple
ExecStart=/usr/bin/ahriman --architecture x86_64 web
User=ahriman
Group=ahriman
[Install]
WantedBy=multi-user.target

View File

@ -0,0 +1,15 @@
[Unit]
Description=ArcHlinux ReposItory MANager web server (%I architecture)
After=network.target
[Service]
Type=simple
ExecStart=/usr/bin/ahriman --architecture %i web
User=ahriman
Group=ahriman
KillSignal=SIGQUIT
SuccessExitStatus=SIGQUIT
[Install]
WantedBy=multi-user.target

View File

@ -1,7 +0,0 @@
[Unit]
Description=ArcHlinux ReposItory MANager
[Service]
ExecStart=/usr/bin/ahriman --architecture x86_64 update
User=ahriman
Group=ahriman

View File

@ -0,0 +1,7 @@
[Unit]
Description=ArcHlinux ReposItory MANager (%I architecture)
[Service]
ExecStart=/usr/bin/ahriman --architecture %i update
User=ahriman
Group=ahriman

View File

@ -1,5 +1,5 @@
[Unit] [Unit]
Description=ArcHlinux ReposItory MANager timer Description=ArcHlinux ReposItory MANager timer (%I architecture)
[Timer] [Timer]
OnCalendar=daily OnCalendar=daily

View File

@ -49,12 +49,12 @@ setup(
'package/etc/ahriman.ini.d/logging.ini', 'package/etc/ahriman.ini.d/logging.ini',
]), ]),
('lib/systemd/system', [ ('lib/systemd/system', [
'package/lib/systemd/system/ahriman.service', 'package/lib/systemd/system/ahriman@.service',
'package/lib/systemd/system/ahriman.timer', 'package/lib/systemd/system/ahriman@.timer',
'package/lib/systemd/system/ahriman-web.service', 'package/lib/systemd/system/ahriman-web@.service',
]), ]),
('share/ahriman', [ ('share/ahriman', [
'package/share/ahriman/index.jinja2', 'package/share/ahriman/build-status.jinja2',
'package/share/ahriman/repo-index.jinja2', 'package/share/ahriman/repo-index.jinja2',
]), ]),
], ],

View File

@ -18,72 +18,38 @@
# along with this program. If not, see <http://www.gnu.org/licenses/>. # along with this program. If not, see <http://www.gnu.org/licenses/>.
# #
import argparse import argparse
import os
from typing import Optional
import ahriman.version as version import ahriman.version as version
from ahriman.application.application import Application from ahriman.application.application import Application
from ahriman.application.lock import Lock
from ahriman.core.configuration import Configuration from ahriman.core.configuration import Configuration
def _get_app(args: argparse.Namespace) -> Application:
config = _get_config(args.config)
return Application(args.architecture, config)
def _get_config(config_path: str) -> Configuration:
config = Configuration()
config.load(config_path)
config.load_logging()
return config
def _lock_check(path: Optional[str]) -> None:
if path is None:
return
if os.path.exists(args.lock):
raise RuntimeError('Another application instance is run')
def _lock_create(path: Optional[str]) -> None:
if path is None:
return
open(path, 'w').close()
def _lock_remove(path: Optional[str]) -> None:
if path is None:
return
if os.path.exists(path):
os.remove(path)
def add(args: argparse.Namespace) -> None: def add(args: argparse.Namespace) -> None:
_get_app(args).add(args.package) Application.from_args(args).add(args.package)
def rebuild(args: argparse.Namespace) -> None: def rebuild(args: argparse.Namespace) -> None:
app = _get_app(args) app = Application.from_args(args)
packages = app.repository.packages() packages = app.repository.packages()
app.update(packages) app.update(packages)
def remove(args: argparse.Namespace) -> None: def remove(args: argparse.Namespace) -> None:
_get_app(args).remove(args.package) Application.from_args(args).remove(args.package)
def report(args: argparse.Namespace) -> None: def report(args: argparse.Namespace) -> None:
_get_app(args).report(args.target) Application.from_args(args).report(args.target)
def sync(args: argparse.Namespace) -> None: def sync(args: argparse.Namespace) -> None:
_get_app(args).sync(args.target) Application.from_args(args).sync(args.target)
def update(args: argparse.Namespace) -> None: def update(args: argparse.Namespace) -> None:
app = _get_app(args) app = Application.from_args(args)
log_fn = lambda line: print(line) if args.dry_run else app.logger.info(line) log_fn = lambda line: print(line) if args.dry_run else app.logger.info(line)
packages = app.get_updates(args.no_aur, args.no_manual, args.no_vcs, log_fn) packages = app.get_updates(args.no_aur, args.no_manual, args.no_vcs, log_fn)
if args.dry_run: if args.dry_run:
@ -93,9 +59,9 @@ def update(args: argparse.Namespace) -> None:
def web(args: argparse.Namespace) -> None: def web(args: argparse.Namespace) -> None:
from ahriman.web.web import run_server, setup_service from ahriman.web.web import run_server, setup_service
config = _get_config(args.config) config = Configuration.from_path(args.config)
app = setup_service(args.architecture, config) app = setup_service(args.architecture, config)
run_server(app) run_server(app, args.architecture)
if __name__ == '__main__': if __name__ == '__main__':
@ -140,18 +106,9 @@ if __name__ == '__main__':
web_parser.set_defaults(fn=web, lock=None) web_parser.set_defaults(fn=web, lock=None)
args = parser.parse_args() args = parser.parse_args()
if args.force:
_lock_remove(args.lock)
_lock_check(args.lock)
if 'fn' not in args: if 'fn' not in args:
parser.print_help() parser.print_help()
exit(1) exit(1)
try: with Lock(args.lock, args.force):
_lock_create(args.lock)
args.fn(args) args.fn(args)
finally:
_lock_remove(args.lock)

View File

@ -17,11 +17,14 @@
# You should have received a copy of the GNU General Public License # You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>. # along with this program. If not, see <http://www.gnu.org/licenses/>.
# #
from __future__ import annotations
import argparse
import logging import logging
import os import os
import shutil import shutil
from typing import Callable, List, Optional from typing import Callable, List, Optional, Type
from ahriman.core.build_tools.task import Task from ahriman.core.build_tools.task import Task
from ahriman.core.configuration import Configuration from ahriman.core.configuration import Configuration
@ -37,6 +40,11 @@ class Application:
self.architecture = architecture self.architecture = architecture
self.repository = Repository(architecture, config) self.repository = Repository(architecture, config)
@classmethod
def from_args(cls: Type[Application], args: argparse.Namespace) -> Application:
config = Configuration.from_path(args.config)
return cls(args.architecture, config)
def _finalize(self) -> None: def _finalize(self) -> None:
self.report() self.report()
self.sync() self.sync()

View File

@ -0,0 +1,59 @@
#
# Copyright (c) 2021 Evgenii Alekseev.
#
# This file is part of ahriman
# (see https://github.com/arcan1s/ahriman).
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
import os
from typing import Optional
from ahriman.core.exceptions import DuplicateRun
class Lock:
def __init__(self, path: Optional[str], force: bool) -> None:
self.path = path
self.force = force
def __enter__(self):
if self.force:
self.remove()
self.check()
self.create()
return self
def __exit__(self, exc_type, exc_val, exc_tb):
self.remove()
return True
def check(self) -> None:
if self.path is None:
return
if os.path.exists(self.path):
raise DuplicateRun()
def create(self) -> None:
if self.path is None:
return
open(self.path, 'w').close()
def remove(self) -> None:
if self.path is None:
return
if os.path.exists(self.path):
os.remove(self.path)

View File

@ -39,10 +39,10 @@ class Task:
self.paths = paths self.paths = paths
section = config.get_section_name('build', architecture) section = config.get_section_name('build', architecture)
self.archbuild_flags = config.get_list(section, 'archbuild_flags') self.archbuild_flags = config.getlist(section, 'archbuild_flags')
self.build_command = config.get(section, 'build_command') self.build_command = config.get(section, 'build_command')
self.makepkg_flags = config.get_list(section, 'makepkg_flags') self.makepkg_flags = config.getlist(section, 'makepkg_flags')
self.makechrootpkg_flags = config.get_list(section, 'makechrootpkg_flags') self.makechrootpkg_flags = config.getlist(section, 'makechrootpkg_flags')
@property @property
def git_path(self) -> str: def git_path(self) -> str:

View File

@ -17,11 +17,13 @@
# You should have received a copy of the GNU General Public License # You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>. # along with this program. If not, see <http://www.gnu.org/licenses/>.
# #
from __future__ import annotations
import configparser import configparser
import os import os
from logging.config import fileConfig from logging.config import fileConfig
from typing import List, Optional, Set from typing import List, Optional, Type
# built-in configparser extension # built-in configparser extension
@ -35,7 +37,14 @@ class Configuration(configparser.RawConfigParser):
def include(self) -> str: def include(self) -> str:
return self.get('settings', 'include') return self.get('settings', 'include')
def get_list(self, section: str, key: str) -> List[str]: @classmethod
def from_path(cls: Type[Configuration], path: str) -> Configuration:
config = cls()
config.load(path)
config.load_logging()
return config
def getlist(self, section: str, key: str) -> List[str]:
raw = self.get(section, key, fallback=None) raw = self.get(section, key, fallback=None)
if not raw: # empty string or none if not raw: # empty string or none
return [] return []
@ -52,8 +61,7 @@ class Configuration(configparser.RawConfigParser):
def load_includes(self) -> None: def load_includes(self) -> None:
try: try:
include_dir = self.include for conf in filter(lambda p: p.endswith('.ini'), sorted(os.listdir(self.include))):
for conf in filter(lambda p: p.endswith('.ini'), sorted(os.listdir(include_dir))):
self.read(os.path.join(self.include, conf)) self.read(os.path.join(self.include, conf))
except (FileNotFoundError, configparser.NoOptionError): except (FileNotFoundError, configparser.NoOptionError):
pass pass

View File

@ -25,6 +25,11 @@ class BuildFailed(Exception):
Exception.__init__(self, f'Package {package} build failed, check logs for details') Exception.__init__(self, f'Package {package} build failed, check logs for details')
class DuplicateRun(Exception):
def __init__(self) -> None:
Exception.__init__(self, 'Another application instance is run')
class InitializeException(Exception): class InitializeException(Exception):
def __init__(self) -> None: def __init__(self) -> None:
Exception.__init__(self, 'Could not load service') Exception.__init__(self, 'Could not load service')
@ -40,16 +45,11 @@ class InvalidPackageInfo(Exception):
Exception.__init__(self, f'There are errors during reading package information: `{details}`') Exception.__init__(self, f'There are errors during reading package information: `{details}`')
class MissingConfiguration(Exception):
def __init__(self, name: str) -> None:
Exception.__init__(self, f'No section `{name}` found')
class ReportFailed(Exception): class ReportFailed(Exception):
def __init__(self, cause: Exception) -> None: def __init__(self) -> None:
Exception.__init__(self, f'Report failed with reason {cause}') Exception.__init__(self, 'Report failed')
class SyncFailed(Exception): class SyncFailed(Exception):
def __init__(self, cause: Exception) -> None: def __init__(self) -> None:
Exception.__init__(self, f'Sync failed with reason {cause}') Exception.__init__(self, 'Sync failed')

View File

@ -32,14 +32,14 @@ class HTML(Report):
def __init__(self, architecture: str, config: Configuration) -> None: def __init__(self, architecture: str, config: Configuration) -> None:
Report.__init__(self, architecture, config) Report.__init__(self, architecture, config)
section = self.config.get_section_name('html', self.architecture) section = config.get_section_name('html', architecture)
self.report_path = config.get(section, 'path') self.report_path = config.get(section, 'path')
self.link_path = config.get(section, 'link_path') self.link_path = config.get(section, 'link_path')
self.template_path = config.get(section, 'template_path') self.template_path = config.get(section, 'template_path')
# base template vars # base template vars
self.sign_targets = [SignSettings.from_option(opt) for opt in config.get_list('sign', 'target')] self.sign_targets = [SignSettings.from_option(opt) for opt in config.getlist('sign', 'target')]
self.pgp_key = config.get('sign', 'key', fallback=None) self.pgp_key = config.get('sign', 'key', fallback=None)
self.homepage = config.get(section, 'homepage', fallback=None) self.homepage = config.get(section, 'homepage', fallback=None)
self.repository = config.get('repository', 'name') self.repository = config.get('repository', 'name')

View File

@ -27,9 +27,9 @@ from ahriman.models.report_settings import ReportSettings
class Report: class Report:
def __init__(self, architecture: str, config: Configuration) -> None: def __init__(self, architecture: str, config: Configuration) -> None:
self.logger = logging.getLogger('builder')
self.architecture = architecture self.architecture = architecture
self.config = config self.config = config
self.logger = logging.getLogger('builder')
@staticmethod @staticmethod
def run(architecture: str, config: Configuration, target: str, path: str) -> None: def run(architecture: str, config: Configuration, target: str, path: str) -> None:
@ -42,8 +42,9 @@ class Report:
try: try:
report.generate(path) report.generate(path)
except Exception as e: except Exception:
raise ReportFailed(e) from e report.logger.exception('report generation failed', exc_info=True)
raise ReportFailed()
def generate(self, path: str) -> None: def generate(self, path: str) -> None:
pass pass

View File

@ -31,7 +31,6 @@ from ahriman.core.sign.gpg_wrapper import GPGWrapper
from ahriman.core.upload.uploader import Uploader from ahriman.core.upload.uploader import Uploader
from ahriman.core.util import package_like from ahriman.core.util import package_like
from ahriman.core.watcher.client import Client from ahriman.core.watcher.client import Client
from ahriman.models.build_status import BuildStatusEnum
from ahriman.models.package import Package from ahriman.models.package import Package
from ahriman.models.repository_paths import RepositoryPaths from ahriman.models.repository_paths import RepositoryPaths
@ -46,13 +45,12 @@ class Repository:
self.aur_url = config.get('aur', 'url') self.aur_url = config.get('aur', 'url')
self.name = config.get('repository', 'name') self.name = config.get('repository', 'name')
self.paths = RepositoryPaths(config.get('repository', 'root'), self.architecture) self.paths = RepositoryPaths(config.get('repository', 'root'), architecture)
self.paths.create_tree() self.paths.create_tree()
self.sign = GPGWrapper(config) self.sign = GPGWrapper(architecture, config)
self.wrapper = RepoWrapper(self.name, self.paths, self.sign.repository_sign_args) self.wrapper = RepoWrapper(self.name, self.paths, self.sign.repository_sign_args)
self.web_report = Client.load(architecture, config)
self.web_report = Client.load(config)
def _clear_build(self) -> None: def _clear_build(self) -> None:
for package in os.listdir(self.paths.sources): for package in os.listdir(self.paths.sources):
@ -82,7 +80,7 @@ class Repository:
def process_build(self, updates: List[Package]) -> List[str]: def process_build(self, updates: List[Package]) -> List[str]:
def build_single(package: Package) -> None: def build_single(package: Package) -> None:
self.web_report.update(package.base, BuildStatusEnum.Building) self.web_report.set_building(package.base)
task = Task(package, self.architecture, self.config, self.paths) task = Task(package, self.architecture, self.config, self.paths)
task.clone() task.clone()
built = task.build() built = task.build()
@ -94,7 +92,7 @@ class Repository:
try: try:
build_single(package) build_single(package)
except Exception: except Exception:
self.web_report.update(package.base, BuildStatusEnum.Failed) self.web_report.set_failed(package.base)
self.logger.exception(f'{package.base} ({self.architecture}) build exception', exc_info=True) self.logger.exception(f'{package.base} ({self.architecture}) build exception', exc_info=True)
continue continue
self._clear_build() self._clear_build()
@ -126,13 +124,13 @@ class Repository:
def process_report(self, targets: Optional[List[str]]) -> None: def process_report(self, targets: Optional[List[str]]) -> None:
if targets is None: if targets is None:
targets = self.config.get_list('report', 'target') targets = self.config.getlist('report', 'target')
for target in targets: for target in targets:
Report.run(self.architecture, self.config, target, self.paths.repository) Report.run(self.architecture, self.config, target, self.paths.repository)
def process_sync(self, targets: Optional[List[str]]) -> None: def process_sync(self, targets: Optional[List[str]]) -> None:
if targets is None: if targets is None:
targets = self.config.get_list('upload', 'target') targets = self.config.getlist('upload', 'target')
for target in targets: for target in targets:
Uploader.run(self.architecture, self.config, target, self.paths.repository) Uploader.run(self.architecture, self.config, target, self.paths.repository)
@ -146,18 +144,19 @@ class Repository:
shutil.move(src, dst) shutil.move(src, dst)
package_fn = os.path.join(self.paths.repository, os.path.basename(package)) package_fn = os.path.join(self.paths.repository, os.path.basename(package))
self.wrapper.add(package_fn) self.wrapper.add(package_fn)
self.web_report.add(local, BuildStatusEnum.Success) self.web_report.set_success(local)
except Exception: except Exception:
self.logger.exception(f'could not process {package}', exc_info=True) self.logger.exception(f'could not process {package}', exc_info=True)
self.web_report.update(local.base, BuildStatusEnum.Failed) self.web_report.set_failed(local.base)
self._clear_packages() self._clear_packages()
return self.wrapper.repo_path return self.wrapper.repo_path
def updates_aur(self, no_vcs: bool) -> List[Package]: def updates_aur(self, no_vcs: bool) -> List[Package]:
result: List[Package] = [] result: List[Package] = []
ignore_list = self.config.get_list(
self.config.get_section_name('build', self.architecture), 'ignore_packages') build_section = self.config.get_section_name('build', self.architecture)
ignore_list = self.config.getlist(build_section, 'ignore_packages')
for local in self.packages(): for local in self.packages():
if local.base in ignore_list: if local.base in ignore_list:
@ -169,9 +168,9 @@ class Repository:
remote = Package.load(local.base, self.aur_url) remote = Package.load(local.base, self.aur_url)
if local.is_outdated(remote): if local.is_outdated(remote):
result.append(remote) result.append(remote)
self.web_report.update(local.base, BuildStatusEnum.Pending) self.web_report.set_pending(local.base)
except Exception: except Exception:
self.web_report.update(local.base, BuildStatusEnum.Failed) self.web_report.set_failed(local.base)
self.logger.exception(f'could not load remote package {local.base}', exc_info=True) self.logger.exception(f'could not load remote package {local.base}', exc_info=True)
continue continue
@ -184,7 +183,7 @@ class Repository:
try: try:
local = Package.load(os.path.join(self.paths.manual, fn), self.aur_url) local = Package.load(os.path.join(self.paths.manual, fn), self.aur_url)
result.append(local) result.append(local)
self.web_report.add(local, BuildStatusEnum.Unknown) self.web_report.set_unknown(local)
except Exception: except Exception:
self.logger.exception(f'could not add package from {fn}', exc_info=True) self.logger.exception(f'could not add package from {fn}', exc_info=True)
self._clear_manual() self._clear_manual()

View File

@ -30,11 +30,11 @@ from ahriman.models.sign_settings import SignSettings
class GPGWrapper: class GPGWrapper:
def __init__(self, config: Configuration) -> None: def __init__(self, architecture: str, config: Configuration) -> None:
self.logger = logging.getLogger('build_details') self.logger = logging.getLogger('build_details')
section = config.get_section_name('sign', architecture)
self.target = [SignSettings.from_option(opt) for opt in config.get_list('sign', 'target')] self.target = [SignSettings.from_option(opt) for opt in config.getlist(section, 'target')]
self.key = config.get('sign', 'key') if self.target else None self.key = config.get(section, 'key') if self.target else None
@property @property
def repository_sign_args(self) -> List[str]: def repository_sign_args(self) -> List[str]:

View File

@ -26,8 +26,8 @@ class Rsync(Uploader):
def __init__(self, architecture: str, config: Configuration) -> None: def __init__(self, architecture: str, config: Configuration) -> None:
Uploader.__init__(self, architecture, config) Uploader.__init__(self, architecture, config)
section = self.config.get_section_name('rsync', self.architecture) section = config.get_section_name('rsync', architecture)
self.remote = self.config.get(section, 'remote') self.remote = config.get(section, 'remote')
def sync(self, path: str) -> None: def sync(self, path: str) -> None:
check_output('rsync', '--archive', '--verbose', '--compress', '--partial', '--progress', '--delete', path, self.remote, check_output('rsync', '--archive', '--verbose', '--compress', '--partial', '--progress', '--delete', path, self.remote,

View File

@ -26,8 +26,8 @@ class S3(Uploader):
def __init__(self, architecture: str, config: Configuration) -> None: def __init__(self, architecture: str, config: Configuration) -> None:
Uploader.__init__(self, architecture, config) Uploader.__init__(self, architecture, config)
section = self.config.get_section_name('s3', self.architecture) section = config.get_section_name('s3', architecture)
self.bucket = self.config.get(section, 'bucket') self.bucket = config.get(section, 'bucket')
def sync(self, path: str) -> None: def sync(self, path: str) -> None:
# TODO rewrite to boto, but it is bullshit # TODO rewrite to boto, but it is bullshit

View File

@ -27,9 +27,9 @@ from ahriman.models.upload_settings import UploadSettings
class Uploader: class Uploader:
def __init__(self, architecture: str, config: Configuration) -> None: def __init__(self, architecture: str, config: Configuration) -> None:
self.logger = logging.getLogger('builder')
self.architecture = architecture self.architecture = architecture
self.config = config self.config = config
self.logger = logging.getLogger('builder')
@staticmethod @staticmethod
def run(architecture: str, config: Configuration, target: str, path: str) -> None: def run(architecture: str, config: Configuration, target: str, path: str) -> None:
@ -45,8 +45,9 @@ class Uploader:
try: try:
uploader.sync(path) uploader.sync(path)
except Exception as e: except Exception:
raise SyncFailed(e) from e uploader.logger.exception('remote sync failed', exc_info=True)
raise SyncFailed()
def sync(self, path: str) -> None: def sync(self, path: str) -> None:
pass pass

View File

@ -39,10 +39,26 @@ class Client:
def update(self, base: str, status: BuildStatusEnum) -> None: def update(self, base: str, status: BuildStatusEnum) -> None:
pass pass
def set_building(self, base: str) -> None:
return self.update(base, BuildStatusEnum.Building)
def set_failed(self, base: str) -> None:
return self.update(base, BuildStatusEnum.Failed)
def set_pending(self, base: str) -> None:
return self.update(base, BuildStatusEnum.Pending)
def set_success(self, package: Package) -> None:
return self.add(package, BuildStatusEnum.Success)
def set_unknown(self, package: Package) -> None:
return self.add(package, BuildStatusEnum.Unknown)
@staticmethod @staticmethod
def load(config: Configuration) -> Client: def load(architecture: str, config: Configuration) -> Client:
host = config.get('web', 'host', fallback=None) section = config.get_section_name('web', architecture)
port = config.getint('web', 'port', fallback=None) host = config.get(section, 'host', fallback=None)
port = config.getint(section, 'port', fallback=None)
if host is None or port is None: if host is None or port is None:
return Client() return Client()
return WebClient(host, port) return WebClient(host, port)

View File

@ -19,10 +19,9 @@
# #
from __future__ import annotations from __future__ import annotations
import shutil
import aur import aur
import os import os
import shutil
import tempfile import tempfile
from dataclasses import dataclass, field from dataclasses import dataclass, field

View File

@ -29,22 +29,37 @@ class RepositoryPaths:
@property @property
def chroot(self) -> str: def chroot(self) -> str:
'''
:return: directory for devtools chroot
'''
return os.path.join(self.root, 'chroot') return os.path.join(self.root, 'chroot')
@property @property
def manual(self) -> str: def manual(self) -> str:
'''
:return: directory for manual updates (i.e. from add command)
'''
return os.path.join(self.root, 'manual') return os.path.join(self.root, 'manual')
@property @property
def packages(self) -> str: def packages(self) -> str:
'''
:return: directory for built packages
'''
return os.path.join(self.root, 'packages') return os.path.join(self.root, 'packages')
@property @property
def repository(self) -> str: def repository(self) -> str:
'''
:return: repository directory
'''
return os.path.join(self.root, 'repository', self.architecture) return os.path.join(self.root, 'repository', self.architecture)
@property @property
def sources(self) -> str: def sources(self) -> str:
'''
:return: directory for downloaded PKGBUILDs for current build
'''
return os.path.join(self.root, 'sources') return os.path.join(self.root, 'sources')
def create_tree(self) -> None: def create_tree(self) -> None:

View File

@ -18,11 +18,10 @@
# along with this program. If not, see <http://www.gnu.org/licenses/>. # along with this program. If not, see <http://www.gnu.org/licenses/>.
# #
from aiohttp.web import middleware, Request, Response from aiohttp.web import middleware, Request, Response
from aiohttp.web_exceptions import HTTPClientError
from logging import Logger from logging import Logger
from typing import Callable from typing import Callable
from aiohttp.web_exceptions import HTTPClientError
def exception_handler(logger: Logger) -> Callable: def exception_handler(logger: Logger) -> Callable:
@middleware @middleware

View File

@ -22,6 +22,7 @@ from aiohttp.web import View
from ahriman.core.watcher.watcher import Watcher from ahriman.core.watcher.watcher import Watcher
# special class to make it typed
class BaseView(View): class BaseView(View):
@property @property

View File

@ -17,18 +17,17 @@
# You should have received a copy of the GNU General Public License # You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>. # along with this program. If not, see <http://www.gnu.org/licenses/>.
# #
from typing import Any, Dict
from aiohttp_jinja2 import template from aiohttp_jinja2 import template
from typing import Any, Dict
from ahriman.web.views.base import BaseView from ahriman.web.views.base import BaseView
class IndexView(BaseView): class IndexView(BaseView):
@template("index.jinja2") @template("build-status.jinja2")
async def get(self) -> Dict[str, Any]: async def get(self) -> Dict[str, Any]:
# some magic to make it jinja-readable # some magic to make it jinja-friendly
packages = [ packages = [
{ {
'base': package.base, 'base': package.base,

View File

@ -38,17 +38,19 @@ async def on_startup(app: web.Application) -> None:
app.logger.info('server started') app.logger.info('server started')
try: try:
app['watcher'].load() app['watcher'].load()
except Exception as e: except Exception:
app.logger.exception('could not load packages', exc_info=True) app.logger.exception('could not load packages', exc_info=True)
raise InitializeException() from e raise InitializeException()
def run_server(app: web.Application) -> None: def run_server(app: web.Application, architecture: str) -> None:
app.logger.info('start server') app.logger.info('start server')
web.run_app(app,
host=app['config'].get('web', 'host'), section = app['config'].get_section_name('web', architecture)
port=app['config'].getint('web', 'port'), host = app['config'].get(section, 'host')
handle_signals=False) port = app['config'].getint(section, 'port')
web.run_app(app, host=host, port=port, handle_signals=False)
def setup_service(architecture: str, config: Configuration) -> web.Application: def setup_service(architecture: str, config: Configuration) -> web.Application: