Compare commits

...

22 Commits

Author SHA1 Message Date
67b97a64ea Release 0.11.7 2021-03-14 19:28:58 +03:00
7ace74af44 handle makedepends as optional 2021-03-14 19:28:29 +03:00
b7d481858d Release 0.11.6 2021-03-13 19:24:23 +03:00
f753563804 soft colours 2021-03-13 19:24:08 +03:00
4727894349 drop architecture coz it is always same 2021-03-13 17:12:36 +03:00
4b98b21a70 strict typing, change colors a bit, architecture depending lock 2021-03-13 16:57:58 +03:00
9410c521a1 Release 0.11.5 2021-03-13 05:18:44 +03:00
dd42cd0cd6 sort package list 2021-03-13 05:18:27 +03:00
50b409cd3e Release 0.11.4 2021-03-13 05:13:33 +03:00
356cd35c5f better templating 2021-03-13 05:12:53 +03:00
3405105dce pretty status html 2021-03-13 03:57:27 +03:00
4445c8c871 Release 0.11.3 2021-03-13 02:27:38 +03:00
a3a66c7c9a count epoch 2021-03-13 02:27:27 +03:00
45b762e3d9 Release 0.11.2 2021-03-13 01:57:26 +03:00
c5db7e64ca process prepare call for vcs packages 2021-03-13 01:57:10 +03:00
0dd4d098f6 Release 0.11.1 2021-03-12 00:24:49 +03:00
4866548224 handle built packages during update 2021-03-12 00:24:26 +03:00
5d526e1bd8 Release 0.11.0 2021-03-12 00:15:21 +03:00
c66325ff38 fix interaction with web 2021-03-12 00:14:31 +03:00
371019f899 add depdendency manager and switch to pyalpm instead of expac 2021-03-12 00:04:37 +03:00
2d351fa94f allow to specify key overrides for packages 2021-03-11 04:06:20 +03:00
1770793e69 improvements
* multi-sign and multi-web configuration
* change default configuration to do not use architecture
* change units to be templated
* some refactoring
2021-03-11 03:57:23 +03:00
48 changed files with 765 additions and 399 deletions

View File

@ -1,19 +1,22 @@
# ahriman configuration
Some groups can be specified for each architecture separately with default values. E.g. if there are `build` and `build_x86_64` groups it will use the `build_x86_64` for the `x86_64` architecture and `build` for any other.
Some groups can be specified for each architecture separately. E.g. if there are `build` and `build_x86_64` groups it will use the `build_x86_64` for the `x86_64` architecture and `build` for any other (architecture specific group has higher priority).
## `settings` group
Base configuration settings:
Base configuration settings.
* `include` - path to directory with configuration files overrides, string, required.
* `logging` - path to logging configuration, string, required. Check `logging.ini` for reference.
## `aur` group
## `alpm` group
AUR related configuration:
libalpm and AUR related configuration.
* `url` - base url for AUR, string, required.
* `aur_url` - base url for AUR, string, required.
* `database` - path to pacman local database cache, string, required.
* `repositories` - list of pacman repositories, space separated list of strings, required.
* `root` - root for alpm library, string, required.
## `build_*` groups
@ -27,25 +30,26 @@ Build related configuration. Group name must refer to architecture, e.g. it shou
## `repository` group
Base repository settings:
Base repository settings.
* `name` - repository name, string, required.
* `root` - root path for application, string, required.
## `sign` group
## `sign_*` groups
Settings for signing packages or repository:
Settings for signing packages or repository. Group name must refer to architecture, e.g. it should be `sign_x86_64` for x86_64 architecture.
* `target` - configuration flag to enable signing, space separated list of strings, required. Allowed values are `package` (sign each package separately), `repository` (sign repository database file).
* `key` - PGP key, string, required.
* `key` - default PGP key, string, required. This key will also be used for database signing if enabled.
* `key_*` settings - PGP key which will be used for specific packages, string, optional. For example, if there is `key_yay` option the specified key will be used for yay package and default key for others.
## `report` group
Report generation settings:
Report generation settings.
* `target` - list of reports to be generated, space separated list of strings, optional. Allowed values are `html`.
### `html_*` group
### `html_*` groups
Group name must refer to architecture, e.g. it should be `html_x86_64` for x86_64 architecture.
@ -56,25 +60,25 @@ Group name must refer to architecture, e.g. it should be `html_x86_64` for x86_6
## `upload` group
Remote synchronization settings:
Remote synchronization settings.
* `target` - list of synchronizations to be used, space separated list of strings, optional. Allowed values are `rsync`, `s3`.
### `rsync_*` group
### `rsync_*` groups
Group name must refer to architecture, e.g. it should be `rsync_x86_64` for x86_64 architecture. Requires `rsync` package to be installed. Do not forget to configure ssh for user `ahriman`.
* `remote` - remote server to rsync (e.g. `1.2.3.4:5678:path/to/sync`), string, required.
### `s3_*` group
### `s3_*` groups
Group name must refer to architecture, e.g. it should be `s3_x86_64` for x86_64 architecture. Requires `aws-cli` package to be installed. Do not forget to configure it for user `ahriman`.
* `bucket` - bucket name (e.g. `s3://bucket/path`), string, required.
## `web` group
## `web_*` groups
Web server settings. If any of `host`/`port` is not set, web intergration will be disabled.
Web server settings. If any of `host`/`port` is not set, web integration will be disabled. Group name must refer to architecture, e.g. it should be `web_x86_64` for x86_64 architecture.
* `host` - host to bind, string, optional.
* `port` - port to bind, int, optional.

View File

@ -5,9 +5,11 @@ Wrapper for managing custom repository inspired by [repo-scripts](https://github
## Features
* Install-configure-forget manager for own repository
* Multi-architecture support
* VCS packages support
* Sign support with gpg
* Synchronization to remote services and report generation
* Sign support with gpg (repository, package, per package settings)
* Synchronization to remote services (rsync, s3) and report generation (html)
* Dependency manager
* Repository status interface
## Installation and run
@ -16,14 +18,10 @@ Wrapper for managing custom repository inspired by [repo-scripts](https://github
* Change settings if required, see `CONFIGURING.md` for more details.
* Create `/var/lib/ahriman/.makepkg.conf` with `makepkg.conf` overrides if required (at least you might want to set `PACKAGER`).
* Configure build tools (it might be required if your package will use any custom repositories):
* create build command if required, e.g. `ln -s /usr/bin/archbuild /usr/local/bin/custom-x86_64-build` (you can choose any name for command);
* create build command, e.g. `ln -s /usr/bin/archbuild /usr/local/bin/custom-x86_64-build` (you can choose any name for command);
* create configuration file, e.g. `cp /usr/share/devtools/pacman-{extra,custom}.conf`;
* change configuration file: add your own repository, add multilib repository;
* set `build.build_command` to point to your command;
* configure `/etc/sudoers.d/ahriman` to allow to run command without password.
* change configuration file, add your own repository, add multilib repository etc;
* set `build.build_command` setting to point to your command;
* configure `/etc/sudoers.d/ahriman` to allow running command without password.
* Start and enable `ahriman.timer` via `systemctl`.
* Add packages by using `ahriman add {package}` command.
## Limitations
* It does not manage dependencies, so you have to add them before main package.

View File

@ -1,13 +1,13 @@
# Maintainer: Evgeniy Alekseev
pkgname='ahriman'
pkgver=0.10.0
pkgver=0.11.7
pkgrel=1
pkgdesc="ArcHlinux ReposItory MANager"
arch=('any')
url="https://github.com/arcan1s/ahriman"
license=('GPL3')
depends=('devtools' 'expac' 'git' 'python-aur' 'python-srcinfo')
depends=('devtools' 'git' 'pyalpm' 'python-aur' 'python-srcinfo')
makedepends=('python-pip')
optdepends=('aws-cli: sync to s3'
'breezy: -bzr packages support'
@ -21,11 +21,9 @@ optdepends=('aws-cli: sync to s3'
'rsync: sync by using rsync'
'subversion: -svn packages support')
source=("https://github.com/arcan1s/ahriman/releases/download/$pkgver/$pkgname-$pkgver-src.tar.xz"
'ahriman.sudoers'
'ahriman.sysusers'
'ahriman.tmpfiles')
sha512sums=('2c811060106aea6f8826cc6beac9f5733370386a43448b359051ea377e233218aae0c5ec3ef7b1ec399fa6a53c02059015b7398b0d88b5a2e7129f167d025539'
'8c9b5b63ac3f7b4d9debaf801a1e9c060877c33d3ecafe18010fcca778e5fa2f2e46909d3d0ff1b229ff8aa978445d8243fd36e1fc104117ed678d5e21901167'
sha512sums=('d35053c7a52e5cc2dd8a3dca6c9d9f18788296d0059683b16238a54318a74fb4c42544d0727460250e7d0f0ce1009aca96e88d3e52e6bdffab8d45e5e4901b7b'
'13718afec2c6786a18f0b223ef8e58dccf0688bca4cdbe203f14071f5031ed20120eb0ce38b52c76cfd6e8b6581a9c9eaa2743eb11abbaca637451a84c33f075'
'55b20f6da3d66e7bbf2add5d95a3b60632df121717d25a993e56e737d14f51fe063eb6f1b38bd81cc32e05db01c0c1d80aaa720c45cde87f238d8b46cdb8cbc4')
backup=('etc/ahriman.ini'
@ -42,7 +40,6 @@ package() {
python setup.py install --root="$pkgdir"
install -Dm400 "$srcdir/$pkgname.sudoers" "$pkgdir/etc/sudoers.d/$pkgname"
install -Dm644 "$srcdir/$pkgname.sysusers" "$pkgdir/usr/lib/sysusers.d/$pkgname.conf"
install -Dm644 "$srcdir/$pkgname.tmpfiles" "$pkgdir/usr/lib/tmpfiles.d/$pkgname.conf"
}

View File

@ -1,4 +0,0 @@
# Used by ArcHlinux ReposItory MANager with default settings
Cmnd_Alias ARCHBUILD_CMD = /usr/bin/extra-x86_64-build *, /usr/bin/multilib-build *
ahriman ALL=(ALL) NOPASSWD: ARCHBUILD_CMD

View File

@ -2,10 +2,13 @@
include = /etc/ahriman.ini.d
logging = /etc/ahriman.ini.d/logging.ini
[aur]
url = https://aur.archlinux.org
[alpm]
aur_url = https://aur.archlinux.org
database = /var/lib/pacman
repositories = core extra community multilib
root = /
[build_x86_64]
[build]
archbuild_flags =
build_command = extra-x86_64-build
ignore_packages =
@ -23,7 +26,7 @@ key =
[report]
target =
[html_x86_64]
[html]
path =
homepage =
link_path =
@ -32,10 +35,10 @@ template_path = /usr/share/ahriman/repo-index.jinja2
[upload]
target =
[rsync_x86_64]
[rsync]
remote =
[s3_x86_64]
[s3]
bucket =
[web]

View File

@ -1,11 +0,0 @@
[Unit]
Description=ArcHlinux ReposItory MANager web server
[Service]
Type=simple
ExecStart=/usr/bin/ahriman --architecture x86_64 web
User=ahriman
Group=ahriman
[Install]
WantedBy=multi-user.target

View File

@ -0,0 +1,15 @@
[Unit]
Description=ArcHlinux ReposItory MANager web server (%I architecture)
After=network.target
[Service]
Type=simple
ExecStart=/usr/bin/ahriman --architecture %i web
User=ahriman
Group=ahriman
KillSignal=SIGQUIT
SuccessExitStatus=SIGQUIT
[Install]
WantedBy=multi-user.target

View File

@ -1,7 +0,0 @@
[Unit]
Description=ArcHlinux ReposItory MANager
[Service]
ExecStart=/usr/bin/ahriman --architecture x86_64 update
User=ahriman
Group=ahriman

View File

@ -0,0 +1,7 @@
[Unit]
Description=ArcHlinux ReposItory MANager (%I architecture)
[Service]
ExecStart=/usr/bin/ahriman --architecture %i update
User=ahriman
Group=ahriman

View File

@ -1,5 +1,5 @@
[Unit]
Description=ArcHlinux ReposItory MANager timer
Description=ArcHlinux ReposItory MANager timer (%I architecture)
[Timer]
OnCalendar=daily

View File

@ -0,0 +1,42 @@
<!doctype html>
<html lang="en">
<head>
<title>{{ repository|e }}</title>
{% include "style.jinja2" %}
{% include "sorttable.jinja2" %}
{% include "search.jinja2" %}
</head>
<body>
<div class="root">
<h1>ahriman {{ version|e }} ({{ architecture|e }})</h1>
{% include "search-line.jinja2" %}
<section class="element">
<table class="sortable search-table">
<tr class="header">
<th>package base</th>
<th>packages</th>
<th>version</th>
<th>last update</th>
<th>status</th>
</tr>
{% for package in packages %}
<tr class="package">
<td class="include-search"><a href="{{ package.web_url|e }}" title="{{ package.base|e }}">{{ package.base|e }}</a></td>
<td class="include-search">{{ package.packages|join("<br>"|safe) }}</td>
<td>{{ package.version|e }}</td>
<td>{{ package.timestamp|e }}</td>
<td class="package-{{ package.status|e }}">{{ package.status|e }}</td>
</tr>
{% endfor %}
</table>
</section>
</div>
</body>
</html>

View File

@ -1,84 +0,0 @@
<!doctype html>
<html lang="en">
<head>
<title>{{ repository|e }}</title>
<style>
:root {
--color-building: 250, 255, 146;
--color-failed: 255, 94, 94;
--color-pending: 250, 255, 146;
--color-success: 121, 255, 94;
--color-unknown: 197, 197, 197;
}
@keyframes blink-building {
0% { background-color: rgba(var(--color-building), 1.0); }
10% { background-color: rgba(var(--color-building), 0.9); }
20% { background-color: rgba(var(--color-building), 0.8); }
30% { background-color: rgba(var(--color-building), 0.7); }
40% { background-color: rgba(var(--color-building), 0.6); }
50% { background-color: rgba(var(--color-building), 0.5); }
60% { background-color: rgba(var(--color-building), 0.4); }
70% { background-color: rgba(var(--color-building), 0.3); }
80% { background-color: rgba(var(--color-building), 0.2); }
90% { background-color: rgba(var(--color-building), 0.1); }
100% { background-color: rgba(var(--color-building), 0.0); }
}
table, th, td {
padding: 5px;
}
td.package {
font-weight: bolder;
}
td.package-unknown {
background-color: rgba(var(--color-unknown), 1.0);
}
td.package-pending {
background-color: rgba(var(--color-pending), 1.0);
}
td.package-building {
background-color: rgba(var(--color-building), 1.0);
animation-name: blink-building;
animation-duration: 1s;
animation-timing-function: linear;
animation-iteration-count: infinite;
animation-direction: alternate;
}
td.package-failed {
background-color: rgba(var(--color-failed), 1.0);
}
td.package-success {
background-color: rgba(var(--color-success), 1.0);
}
</style>
<script src="https://www.kryogenix.org/code/browser/sorttable/sorttable.js"></script>
</head>
<body>
<table class="sortable" id="builds">
<tr>
<th>package base</th>
<th>packages</th>
<th>version</th>
<th>architecture</th>
<th>timestamp</th>
<th>status</th>
</tr>
{% for package in packages %}
<tr>
<td class="package"><a href="{{ package.web_url|e }}" title="{{ package.base|e }}">{{ package.base|e }}</a></td>
<td>{{ package.packages|join("<br>"|safe) }}</td>
<td>{{ package.version|e }}</td>
<td>{{ architecture|e }}</td>
<td>{{ package.timestamp|e }}</td>
<td class="package-{{ package.status|e }}">{{ package.status|e }}</td>
</tr>
{% endfor %}
</table>
</body>
</html>

View File

@ -2,31 +2,51 @@
<html lang="en">
<head>
<title>{{ repository|e }}</title>
{% include "style.jinja2" %}
{% include "sorttable.jinja2" %}
{% include "search.jinja2" %}
</head>
<body>
<h1>Archlinux custom repository</h1>
<div class="root">
<h1>Archlinux user repository</h1>
{% if pgp_key is not none %}
<p>This repository is signed with <a href="http://keys.gnupg.net/pks/lookup?search=0x{{ pgp_key|e }}" title="key search">{{ pgp_key|e }}</a>.</p>
{% endif %}
<section class="element">
{% if pgp_key is not none %}
<p>This repository is signed with <a href="http://keys.gnupg.net/pks/lookup?search=0x{{ pgp_key|e }}&fingerprint=on&op=index" title="key search">{{ pgp_key|e }}</a> by default.</p>
{% endif %}
<code>
$ cat /etc/pacman.conf<br>
[{{ repository|e }}]<br>
Server = {{ link_path|e }}<br>
SigLevel = Database{% if has_repo_signed %}Required{% else %}Never{% endif %} Package{% if has_package_signed %}Required{% else %}Never{% endif %} TrustedOnly
</code>
<code>
$ cat /etc/pacman.conf<br>
[{{ repository|e }}]<br>
Server = {{ link_path|e }}<br>
SigLevel = Database{% if has_repo_signed %}Required{% else %}Never{% endif %} Package{% if has_package_signed %}Required{% else %}Never{% endif %} TrustedOnly
</code>
</section>
<p>Packages:</p>
<ul>
{% for package, package_url in packages.items() %}
<li><a href="{{ package_url|e }}" title="{{ package|e }}">{{ package|e }}</a></li>
{% endfor %}
</ul>
{% include "search-line.jinja2" %}
{% if homepage is not none %}
<footer><a href="{{ homepage|e }}" title="homepage">Homepage</a></footer>
{% endif %}
<section class="element">
<table class="sortable search-table">
<tr class="header">
<th>package</th>
<th>version</th>
</tr>
{% for package in packages %}
<tr class="package">
<td class="include-search"><a href="{{ link_path|e }}/{{ package.filename|e }}" title="{{ package.name|e }}">{{ package.name|e }}</a></td>
<td>{{ package.version|e }}</td>
</tr>
{% endfor %}
</table>
</section>
{% if homepage is not none %}
<footer><a href="{{ homepage|e }}" title="homepage">Homepage</a></footer>
{% endif %}
</div>
</body>
</html>

View File

@ -0,0 +1,3 @@
<section class="element">
<input type="search" id="search" onkeyup="searchInTable()" placeholder="search for package" title="search for package"/>
</section>

View File

@ -0,0 +1,25 @@
<script type="text/javascript">
function searchInTable() {
const input = document.getElementById("search");
const filter = input.value.toLowerCase();
const tables = document.getElementsByClassName("search-table");
for (let i = 0; i < tables.length; i++) {
const tr = tables[i].getElementsByTagName("tr");
// from 1 coz of header
for (let i = 1; i < tr.length; i++) {
let td = tr[i].getElementsByClassName("include-search");
let display = "none";
for (let j = 0; j < td.length; j++) {
if (td[j].tagName.toLowerCase() === "td") {
if (td[j].innerHTML.toLowerCase().indexOf(filter) > -1) {
display = "";
break;
}
}
}
tr[i].style.display = display;
}
}
}
</script>

View File

@ -0,0 +1 @@
<script src="https://www.kryogenix.org/code/browser/sorttable/sorttable.js"></script>

View File

@ -0,0 +1,78 @@
<style>
:root {
--color-building: 255, 255, 146;
--color-failed: 255, 94, 94;
--color-pending: 255, 255, 146;
--color-success: 94, 255, 94;
--color-unknown: 225, 225, 225;
}
@keyframes blink-building {
0% { background-color: rgba(var(--color-building), 1.0); }
10% { background-color: rgba(var(--color-building), 0.9); }
20% { background-color: rgba(var(--color-building), 0.8); }
30% { background-color: rgba(var(--color-building), 0.7); }
40% { background-color: rgba(var(--color-building), 0.6); }
50% { background-color: rgba(var(--color-building), 0.5); }
60% { background-color: rgba(var(--color-building), 0.4); }
70% { background-color: rgba(var(--color-building), 0.3); }
80% { background-color: rgba(var(--color-building), 0.2); }
90% { background-color: rgba(var(--color-building), 0.1); }
100% { background-color: rgba(var(--color-building), 0.0); }
}
div.root {
width: 70%;
padding: 15px 15% 0;
}
section.element {
width: 100%;
padding: 10px 0;
}
code, input, table {
width: inherit;
}
th, td {
padding: 5px;
}
tr.package:nth-child(odd) {
background-color: rgba(255, 255, 255, 1);
}
tr.package:nth-child(even) {
background-color: rgba(235, 235, 255, 1);
}
tr.package:hover {
background-color: rgba(255, 255, 225, 1);
}
tr.header{
background-color: rgba(200, 200, 255, 1);
}
td.package-unknown {
background-color: rgba(var(--color-unknown), 1.0);
}
td.package-pending {
background-color: rgba(var(--color-pending), 1.0);
}
td.package-building {
background-color: rgba(var(--color-building), 1.0);
animation-name: blink-building;
animation-duration: 1s;
animation-timing-function: linear;
animation-iteration-count: infinite;
animation-direction: alternate;
}
td.package-failed {
background-color: rgba(var(--color-failed), 1.0);
}
td.package-success {
background-color: rgba(var(--color-success), 1.0);
}
</style>

View File

@ -28,6 +28,7 @@ setup(
],
install_requires=[
'aur',
'pyalpm',
'srcinfo',
],
setup_requires=[
@ -49,13 +50,17 @@ setup(
'package/etc/ahriman.ini.d/logging.ini',
]),
('lib/systemd/system', [
'package/lib/systemd/system/ahriman.service',
'package/lib/systemd/system/ahriman.timer',
'package/lib/systemd/system/ahriman-web.service',
'package/lib/systemd/system/ahriman@.service',
'package/lib/systemd/system/ahriman@.timer',
'package/lib/systemd/system/ahriman-web@.service',
]),
('share/ahriman', [
'package/share/ahriman/index.jinja2',
'package/share/ahriman/build-status.jinja2',
'package/share/ahriman/repo-index.jinja2',
'package/share/ahriman/search.jinja2',
'package/share/ahriman/search-line.jinja2',
'package/share/ahriman/sorttable.jinja2',
'package/share/ahriman/style.jinja2',
]),
],

View File

@ -18,74 +18,40 @@
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
import argparse
import os
from typing import Optional
import ahriman.version as version
from ahriman.application.application import Application
from ahriman.application.lock import Lock
from ahriman.core.configuration import Configuration
def _get_app(args: argparse.Namespace) -> Application:
config = _get_config(args.config)
return Application(args.architecture, config)
def _get_config(config_path: str) -> Configuration:
config = Configuration()
config.load(config_path)
config.load_logging()
return config
def _lock_check(path: Optional[str]) -> None:
if path is None:
return
if os.path.exists(args.lock):
raise RuntimeError('Another application instance is run')
def _lock_create(path: Optional[str]) -> None:
if path is None:
return
open(path, 'w').close()
def _lock_remove(path: Optional[str]) -> None:
if path is None:
return
if os.path.exists(path):
os.remove(path)
def add(args: argparse.Namespace) -> None:
_get_app(args).add(args.package)
Application.from_args(args).add(args.package, args.without_dependencies)
def rebuild(args: argparse.Namespace) -> None:
app = _get_app(args)
app = Application.from_args(args)
packages = app.repository.packages()
app.update(packages)
def remove(args: argparse.Namespace) -> None:
_get_app(args).remove(args.package)
Application.from_args(args).remove(args.package)
def report(args: argparse.Namespace) -> None:
_get_app(args).report(args.target)
Application.from_args(args).report(args.target)
def sync(args: argparse.Namespace) -> None:
_get_app(args).sync(args.target)
Application.from_args(args).sync(args.target)
def update(args: argparse.Namespace) -> None:
app = _get_app(args)
app = Application.from_args(args)
log_fn = lambda line: print(line) if args.dry_run else app.logger.info(line)
packages = app.get_updates(args.no_aur, args.no_manual, args.no_vcs, log_fn)
packages = app.get_updates(args.package, args.no_aur, args.no_manual, args.no_vcs, log_fn)
if args.dry_run:
return
app.update(packages)
@ -93,9 +59,9 @@ def update(args: argparse.Namespace) -> None:
def web(args: argparse.Namespace) -> None:
from ahriman.web.web import run_server, setup_service
config = _get_config(args.config)
config = Configuration.from_path(args.config)
app = setup_service(args.architecture, config)
run_server(app)
run_server(app, args.architecture)
if __name__ == '__main__':
@ -109,9 +75,11 @@ if __name__ == '__main__':
add_parser = subparsers.add_parser('add', description='add package')
add_parser.add_argument('package', help='package name or archive path', nargs='+')
add_parser.add_argument('--without-dependencies', help='do not add dependencies', action='store_true')
add_parser.set_defaults(fn=add)
check_parser = subparsers.add_parser('check', description='check for updates')
check_parser.add_argument('package', help='filter check by packages', nargs='*')
check_parser.set_defaults(fn=update, no_aur=False, no_manual=True, no_vcs=False, dry_run=True)
rebuild_parser = subparsers.add_parser('rebuild', description='rebuild whole repository')
@ -130,6 +98,7 @@ if __name__ == '__main__':
sync_parser.set_defaults(fn=sync)
update_parser = subparsers.add_parser('update', description='run updates')
update_parser.add_argument('package', help='filter check by packages', nargs='*')
update_parser.add_argument('--dry-run', help='just perform check for updates, same as check command', action='store_true')
update_parser.add_argument('--no-aur', help='do not check for AUR updates', action='store_true')
update_parser.add_argument('--no-manual', help='do not include manual updates', action='store_true')
@ -140,18 +109,9 @@ if __name__ == '__main__':
web_parser.set_defaults(fn=web, lock=None)
args = parser.parse_args()
if args.force:
_lock_remove(args.lock)
_lock_check(args.lock)
if 'fn' not in args:
parser.print_help()
exit(1)
try:
_lock_create(args.lock)
with Lock(args.lock, args.architecture, args.force):
args.fn(args)
finally:
_lock_remove(args.lock)

View File

@ -17,15 +17,19 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
from __future__ import annotations
import argparse
import logging
import os
import shutil
from typing import Callable, List, Optional
from typing import Callable, Iterable, List, Optional, Set, Type
from ahriman.core.build_tools.task import Task
from ahriman.core.configuration import Configuration
from ahriman.core.repository import Repository
from ahriman.core.tree import Tree
from ahriman.models.package import Package
@ -37,16 +41,29 @@ class Application:
self.architecture = architecture
self.repository = Repository(architecture, config)
@classmethod
def from_args(cls: Type[Application], args: argparse.Namespace) -> Application:
config = Configuration.from_path(args.config)
return cls(args.architecture, config)
def _known_packages(self) -> Set[str]:
known_packages: Set[str] = set()
# local set
for package in self.repository.packages():
known_packages.update(package.packages.keys())
known_packages.update(self.repository.pacman.all_packages())
return known_packages
def _finalize(self) -> None:
self.report()
self.sync()
def get_updates(self, no_aur: bool, no_manual: bool, no_vcs: bool,
def get_updates(self, filter_packages: List[str], no_aur: bool, no_manual: bool, no_vcs: bool,
log_fn: Callable[[str], None]) -> List[Package]:
updates = []
if not no_aur:
updates.extend(self.repository.updates_aur(no_vcs))
updates.extend(self.repository.updates_aur(filter_packages, no_vcs))
if not no_manual:
updates.extend(self.repository.updates_manual())
@ -55,35 +72,60 @@ class Application:
return updates
def add(self, names: List[str]) -> None:
def add_manual(name: str) -> None:
package = Package.load(name, self.config.get('aur', 'url'))
Task.fetch(os.path.join(self.repository.paths.manual, package.base), package.git_url)
def add(self, names: Iterable[str], without_dependencies: bool) -> None:
known_packages = self._known_packages()
def add_manual(name: str) -> str:
package = Package.load(name, self.repository.pacman, self.config.get('alpm', 'aur_url'))
path = os.path.join(self.repository.paths.manual, package.base)
Task.fetch(path, package.git_url)
return path
def add_archive(src: str) -> None:
dst = os.path.join(self.repository.paths.packages, os.path.basename(src))
shutil.move(src, dst)
for name in names:
if os.path.isfile(name):
add_archive(name)
else:
add_manual(name)
def process_dependencies(path: str) -> None:
if without_dependencies:
return
dependencies = Package.dependencies(path)
self.add(dependencies.difference(known_packages), without_dependencies)
def remove(self, names: List[str]) -> None:
def process_single(name: str) -> None:
if not os.path.isfile(name):
path = add_manual(name)
process_dependencies(path)
else:
add_archive(name)
for name in names:
process_single(name)
def remove(self, names: Iterable[str]) -> None:
self.repository.process_remove(names)
self._finalize()
def report(self, target: Optional[List[str]] = None) -> None:
def report(self, target: Optional[Iterable[str]] = None) -> None:
targets = target or None
self.repository.process_report(targets)
def sync(self, target: Optional[List[str]] = None) -> None:
def sync(self, target: Optional[Iterable[str]] = None) -> None:
targets = target or None
self.repository.process_sync(targets)
def update(self, updates: List[Package]) -> None:
packages = self.repository.process_build(updates)
self.repository.process_update(packages)
self._finalize()
def update(self, updates: Iterable[Package]) -> None:
def process_update(paths: Iterable[str]) -> None:
self.repository.process_update(paths)
self._finalize()
# process built packages
packages = self.repository.packages_built()
process_update(packages)
# process manual packages
tree = Tree()
tree.load(updates)
for num, level in enumerate(tree.levels()):
self.logger.info(f'processing level #{num} {[package.base for package in level]}')
packages = self.repository.process_build(level)
process_update(packages)

View File

@ -0,0 +1,63 @@
#
# Copyright (c) 2021 Evgenii Alekseev.
#
# This file is part of ahriman
# (see https://github.com/arcan1s/ahriman).
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
from __future__ import annotations
import os
from types import TracebackType
from typing import Literal, Optional, Type
from ahriman.core.exceptions import DuplicateRun
class Lock:
def __init__(self, path: Optional[str], architecture: str, force: bool) -> None:
self.path = f'{path}_{architecture}' if path is not None else None
self.force = force
def __enter__(self) -> Lock:
if self.force:
self.remove()
self.check()
self.create()
return self
def __exit__(self, exc_type: Optional[Type[Exception]], exc_val: Optional[Exception],
exc_tb: TracebackType) -> Literal[False]:
self.remove()
return False
def check(self) -> None:
if self.path is None:
return
if os.path.exists(self.path):
raise DuplicateRun()
def create(self) -> None:
if self.path is None:
return
open(self.path, 'w').close()
def remove(self) -> None:
if self.path is None:
return
if os.path.exists(self.path):
os.remove(self.path)

View File

@ -0,0 +1,40 @@
#
# Copyright (c) 2021 Evgenii Alekseev.
#
# This file is part of ahriman
# (see https://github.com/arcan1s/ahriman).
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
from pyalpm import Handle # type: ignore
from typing import List, Set
from ahriman.core.configuration import Configuration
class Pacman:
def __init__(self, config: Configuration) -> None:
root = config.get('alpm', 'root')
pacman_root = config.get('alpm', 'database')
self.handle = Handle(root, pacman_root)
for repository in config.getlist('alpm', 'repositories'):
self.handle.register_syncdb(repository, 0) # 0 is pgp_level
def all_packages(self) -> List[str]:
result: Set[str] = set()
for database in self.handle.get_syncdbs():
result.update({package.name for package in database.pkgcache})
return list(result)

View File

@ -27,7 +27,7 @@ from ahriman.core.util import check_output
from ahriman.models.repository_paths import RepositoryPaths
class RepoWrapper:
class Repo:
def __init__(self, name: str, paths: RepositoryPaths, sign_args: List[str]) -> None:
self.logger = logging.getLogger('build_details')

View File

@ -39,10 +39,10 @@ class Task:
self.paths = paths
section = config.get_section_name('build', architecture)
self.archbuild_flags = config.get_list(section, 'archbuild_flags')
self.archbuild_flags = config.getlist(section, 'archbuild_flags')
self.build_command = config.get(section, 'build_command')
self.makepkg_flags = config.get_list(section, 'makepkg_flags')
self.makechrootpkg_flags = config.get_list(section, 'makechrootpkg_flags')
self.makepkg_flags = config.getlist(section, 'makepkg_flags')
self.makechrootpkg_flags = config.getlist(section, 'makechrootpkg_flags')
@property
def git_path(self) -> str:

View File

@ -17,11 +17,13 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
from __future__ import annotations
import configparser
import os
from logging.config import fileConfig
from typing import List, Optional, Set
from typing import List, Optional, Type
# built-in configparser extension
@ -35,7 +37,14 @@ class Configuration(configparser.RawConfigParser):
def include(self) -> str:
return self.get('settings', 'include')
def get_list(self, section: str, key: str) -> List[str]:
@classmethod
def from_path(cls: Type[Configuration], path: str) -> Configuration:
config = cls()
config.load(path)
config.load_logging()
return config
def getlist(self, section: str, key: str) -> List[str]:
raw = self.get(section, key, fallback=None)
if not raw: # empty string or none
return []
@ -52,8 +61,7 @@ class Configuration(configparser.RawConfigParser):
def load_includes(self) -> None:
try:
include_dir = self.include
for conf in filter(lambda p: p.endswith('.ini'), sorted(os.listdir(include_dir))):
for conf in filter(lambda p: p.endswith('.ini'), sorted(os.listdir(self.include))):
self.read(os.path.join(self.include, conf))
except (FileNotFoundError, configparser.NoOptionError):
pass

View File

@ -25,12 +25,17 @@ class BuildFailed(Exception):
Exception.__init__(self, f'Package {package} build failed, check logs for details')
class DuplicateRun(Exception):
def __init__(self) -> None:
Exception.__init__(self, 'Another application instance is run')
class InitializeException(Exception):
def __init__(self) -> None:
Exception.__init__(self, 'Could not load service')
class InvalidOptionException(Exception):
class InvalidOption(Exception):
def __init__(self, value: Any) -> None:
Exception.__init__(self, f'Invalid or unknown option value `{value}`')
@ -40,16 +45,11 @@ class InvalidPackageInfo(Exception):
Exception.__init__(self, f'There are errors during reading package information: `{details}`')
class MissingConfiguration(Exception):
def __init__(self, name: str) -> None:
Exception.__init__(self, f'No section `{name}` found')
class ReportFailed(Exception):
def __init__(self, cause: Exception) -> None:
Exception.__init__(self, f'Report failed with reason {cause}')
def __init__(self) -> None:
Exception.__init__(self, 'Report failed')
class SyncFailed(Exception):
def __init__(self, cause: Exception) -> None:
Exception.__init__(self, f'Sync failed with reason {cause}')
def __init__(self) -> None:
Exception.__init__(self, 'Sync failed')

View File

@ -20,11 +20,11 @@
import jinja2
import os
from typing import Dict
from typing import Callable, Dict, Iterable
from ahriman.core.configuration import Configuration
from ahriman.core.report.report import Report
from ahriman.core.util import package_like
from ahriman.models.package import Package
from ahriman.models.sign_settings import SignSettings
@ -32,37 +32,41 @@ class HTML(Report):
def __init__(self, architecture: str, config: Configuration) -> None:
Report.__init__(self, architecture, config)
section = self.config.get_section_name('html', self.architecture)
section = config.get_section_name('html', architecture)
self.report_path = config.get(section, 'path')
self.link_path = config.get(section, 'link_path')
self.template_path = config.get(section, 'template_path')
# base template vars
self.sign_targets = [SignSettings.from_option(opt) for opt in config.get_list('sign', 'target')]
self.pgp_key = config.get('sign', 'key', fallback=None)
self.homepage = config.get(section, 'homepage', fallback=None)
self.repository = config.get('repository', 'name')
def generate(self, path: str) -> None:
sign_section = config.get_section_name('sign', architecture)
self.sign_targets = [SignSettings.from_option(opt) for opt in config.getlist(sign_section, 'target')]
self.pgp_key = config.get(sign_section, 'key') if self.sign_targets else None
def generate(self, packages: Iterable[Package]) -> None:
# idea comes from https://stackoverflow.com/a/38642558
templates_dir, template_name = os.path.split(self.template_path)
loader = jinja2.FileSystemLoader(searchpath=templates_dir)
environment = jinja2.Environment(loader=loader)
template = environment.get_template(template_name)
packages: Dict[str, str] = {}
for fn in sorted(os.listdir(path)):
if not package_like(fn):
continue
packages[fn] = f'{self.link_path}/{fn}'
content = [
{
'filename': filename,
'name': package,
'version': base.version
} for base in packages for package, filename in base.packages.items()
]
comparator: Callable[[Dict[str, str]], str] = lambda item: item['filename']
html = template.render(
homepage=self.homepage,
link_path=self.link_path,
has_package_signed=SignSettings.SignPackages in self.sign_targets,
has_repo_signed=SignSettings.SignRepository in self.sign_targets,
packages=packages,
packages=sorted(content, key=comparator),
pgp_key=self.pgp_key,
repository=self.repository)

View File

@ -19,20 +19,23 @@
#
import logging
from typing import Iterable
from ahriman.core.configuration import Configuration
from ahriman.core.exceptions import ReportFailed
from ahriman.models.package import Package
from ahriman.models.report_settings import ReportSettings
class Report:
def __init__(self, architecture: str, config: Configuration) -> None:
self.logger = logging.getLogger('builder')
self.architecture = architecture
self.config = config
self.logger = logging.getLogger('builder')
@staticmethod
def run(architecture: str, config: Configuration, target: str, path: str) -> None:
def run(architecture: str, config: Configuration, target: str, packages: Iterable[Package]) -> None:
provider = ReportSettings.from_option(target)
if provider == ReportSettings.HTML:
from ahriman.core.report.html import HTML
@ -41,9 +44,10 @@ class Report:
report = Report(architecture, config)
try:
report.generate(path)
except Exception as e:
raise ReportFailed(e) from e
report.generate(packages)
except Exception:
report.logger.exception('report generation failed', exc_info=True)
raise ReportFailed()
def generate(self, path: str) -> None:
def generate(self, packages: Iterable[Package]) -> None:
pass

View File

@ -21,17 +21,17 @@ import logging
import os
import shutil
from typing import Dict, List, Optional
from typing import Dict, Iterable, List, Optional
from ahriman.core.alpm.pacman import Pacman
from ahriman.core.alpm.repo import Repo
from ahriman.core.build_tools.task import Task
from ahriman.core.configuration import Configuration
from ahriman.core.repo.repo_wrapper import RepoWrapper
from ahriman.core.report.report import Report
from ahriman.core.sign.gpg_wrapper import GPGWrapper
from ahriman.core.sign.gpg import GPG
from ahriman.core.upload.uploader import Uploader
from ahriman.core.util import package_like
from ahriman.core.watcher.client import Client
from ahriman.models.build_status import BuildStatusEnum
from ahriman.models.package import Package
from ahriman.models.repository_paths import RepositoryPaths
@ -43,16 +43,16 @@ class Repository:
self.architecture = architecture
self.config = config
self.aur_url = config.get('aur', 'url')
self.aur_url = config.get('alpm', 'aur_url')
self.name = config.get('repository', 'name')
self.paths = RepositoryPaths(config.get('repository', 'root'), self.architecture)
self.paths = RepositoryPaths(config.get('repository', 'root'), architecture)
self.paths.create_tree()
self.sign = GPGWrapper(config)
self.wrapper = RepoWrapper(self.name, self.paths, self.sign.repository_sign_args)
self.web_report = Client.load(config)
self.pacman = Pacman(config)
self.sign = GPG(architecture, config)
self.repo = Repo(self.name, self.paths, self.sign.repository_sign_args)
self.web = Client.load(architecture, config)
def _clear_build(self) -> None:
for package in os.listdir(self.paths.sources):
@ -63,8 +63,8 @@ class Repository:
shutil.rmtree(os.path.join(self.paths.manual, package))
def _clear_packages(self) -> None:
for package in os.listdir(self.paths.packages):
os.remove(os.path.join(self.paths.packages, package))
for package in self.packages_built():
os.remove(package)
def packages(self) -> List[Package]:
result: Dict[str, Package] = {}
@ -73,16 +73,22 @@ class Repository:
continue
full_path = os.path.join(self.paths.repository, fn)
try:
local = Package.load(full_path, self.aur_url)
local = Package.load(full_path, self.pacman, self.aur_url)
result.setdefault(local.base, local).packages.update(local.packages)
except Exception:
self.logger.exception(f'could not load package from {fn}', exc_info=True)
continue
return list(result.values())
def process_build(self, updates: List[Package]) -> List[str]:
def packages_built(self) -> List[str]:
return [
os.path.join(self.paths.packages, fn)
for fn in os.listdir(self.paths.packages)
]
def process_build(self, updates: Iterable[Package]) -> List[str]:
def build_single(package: Package) -> None:
self.web_report.update(package.base, BuildStatusEnum.Building)
self.web.set_building(package.base)
task = Task(package, self.architecture, self.config, self.paths)
task.clone()
built = task.build()
@ -94,84 +100,85 @@ class Repository:
try:
build_single(package)
except Exception:
self.web_report.update(package.base, BuildStatusEnum.Failed)
self.web.set_failed(package.base)
self.logger.exception(f'{package.base} ({self.architecture}) build exception', exc_info=True)
continue
self._clear_build()
return [
os.path.join(self.paths.packages, fn)
for fn in os.listdir(self.paths.packages)
]
return self.packages_built()
def process_remove(self, packages: List[str]) -> str:
def process_remove(self, packages: Iterable[str]) -> str:
def remove_single(package: str) -> None:
try:
self.wrapper.remove(package, package)
self.repo.remove(package, package)
except Exception:
self.logger.exception(f'could not remove {package}', exc_info=True)
requested = set(packages)
for local in self.packages():
if local.base in packages:
to_remove = local.packages
elif local.packages.intersection(packages):
to_remove = local.packages.intersection(packages)
to_remove = set(local.packages.keys())
elif requested.intersection(local.packages.keys()):
to_remove = requested.intersection(local.packages.keys())
else:
to_remove = set()
self.web_report.remove(local.base, to_remove)
self.web.remove(local.base, to_remove)
for package in to_remove:
remove_single(package)
return self.wrapper.repo_path
return self.repo.repo_path
def process_report(self, targets: Optional[List[str]]) -> None:
def process_report(self, targets: Optional[Iterable[str]]) -> None:
if targets is None:
targets = self.config.get_list('report', 'target')
targets = self.config.getlist('report', 'target')
for target in targets:
Report.run(self.architecture, self.config, target, self.paths.repository)
Report.run(self.architecture, self.config, target, self.packages())
def process_sync(self, targets: Optional[List[str]]) -> None:
def process_sync(self, targets: Optional[Iterable[str]]) -> None:
if targets is None:
targets = self.config.get_list('upload', 'target')
targets = self.config.getlist('upload', 'target')
for target in targets:
Uploader.run(self.architecture, self.config, target, self.paths.repository)
def process_update(self, packages: List[str]) -> str:
def process_update(self, packages: Iterable[str]) -> str:
for package in packages:
local = Package.load(package, self.aur_url) # we will use it for status reports
local = Package.load(package, self.pacman, self.aur_url) # we will use it for status reports
try:
files = self.sign.sign_package(package)
files = self.sign.sign_package(package, local.base)
for src in files:
dst = os.path.join(self.paths.repository, os.path.basename(src))
shutil.move(src, dst)
package_fn = os.path.join(self.paths.repository, os.path.basename(package))
self.wrapper.add(package_fn)
self.web_report.add(local, BuildStatusEnum.Success)
self.repo.add(package_fn)
self.web.set_success(local)
except Exception:
self.logger.exception(f'could not process {package}', exc_info=True)
self.web_report.update(local.base, BuildStatusEnum.Failed)
self.web.set_failed(local.base)
self._clear_packages()
return self.wrapper.repo_path
return self.repo.repo_path
def updates_aur(self, no_vcs: bool) -> List[Package]:
def updates_aur(self, filter_packages: Iterable[str], no_vcs: bool) -> List[Package]:
result: List[Package] = []
ignore_list = self.config.get_list(
self.config.get_section_name('build', self.architecture), 'ignore_packages')
build_section = self.config.get_section_name('build', self.architecture)
ignore_list = self.config.getlist(build_section, 'ignore_packages')
for local in self.packages():
if local.base in ignore_list:
continue
if local.is_vcs and no_vcs:
continue
if filter_packages and local.base not in filter_packages:
continue
try:
remote = Package.load(local.base, self.aur_url)
remote = Package.load(local.base, self.pacman, self.aur_url)
if local.is_outdated(remote):
result.append(remote)
self.web_report.update(local.base, BuildStatusEnum.Pending)
self.web.set_pending(local.base)
except Exception:
self.web_report.update(local.base, BuildStatusEnum.Failed)
self.web.set_failed(local.base)
self.logger.exception(f'could not load remote package {local.base}', exc_info=True)
continue
@ -182,9 +189,9 @@ class Repository:
for fn in os.listdir(self.paths.manual):
try:
local = Package.load(os.path.join(self.paths.manual, fn), self.aur_url)
local = Package.load(os.path.join(self.paths.manual, fn), self.pacman, self.aur_url)
result.append(local)
self.web_report.add(local, BuildStatusEnum.Unknown)
self.web.set_unknown(local)
except Exception:
self.logger.exception(f'could not add package from {fn}', exc_info=True)
self._clear_manual()

View File

@ -28,41 +28,42 @@ from ahriman.core.util import check_output
from ahriman.models.sign_settings import SignSettings
class GPGWrapper:
class GPG:
def __init__(self, config: Configuration) -> None:
def __init__(self, architecture: str, config: Configuration) -> None:
self.logger = logging.getLogger('build_details')
self.target = [SignSettings.from_option(opt) for opt in config.get_list('sign', 'target')]
self.key = config.get('sign', 'key') if self.target else None
self.config = config
self.section = config.get_section_name('sign', architecture)
self.target = [SignSettings.from_option(opt) for opt in config.getlist(self.section, 'target')]
self.default_key = config.get(self.section, 'key') if self.target else ''
@property
def repository_sign_args(self) -> List[str]:
if SignSettings.SignRepository not in self.target:
return []
return ['--sign', '--key', self.key] if self.key else ['--sign']
return ['--sign', '--key', self.default_key]
def process(self, path: str) -> List[str]:
def process(self, path: str, key: str) -> List[str]:
check_output(
*self.sign_cmd(path),
*self.sign_cmd(path, key),
exception=BuildFailed(path),
cwd=os.path.dirname(path),
logger=self.logger)
return [path, f'{path}.sig']
def sign_cmd(self, path: str) -> List[str]:
def sign_cmd(self, path: str, key: str) -> List[str]:
cmd = ['gpg']
if self.key is not None:
cmd.extend(['-u', self.key])
cmd.extend(['-u', key])
cmd.extend(['-b', path])
return cmd
def sign_package(self, path: str) -> List[str]:
def sign_package(self, path: str, base: str) -> List[str]:
if SignSettings.SignPackages not in self.target:
return [path]
return self.process(path)
key = self.config.get(self.section, f'key_{base}', fallback=self.default_key)
return self.process(path, key)
def sign_repository(self, path: str) -> List[str]:
if SignSettings.SignRepository not in self.target:
return [path]
return self.process(path)
return self.process(path, self.default_key)

79
src/ahriman/core/tree.py Normal file
View File

@ -0,0 +1,79 @@
#
# Copyright (c) 2021 Evgenii Alekseev.
#
# This file is part of ahriman
# (see https://github.com/arcan1s/ahriman).
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
from __future__ import annotations
import shutil
import tempfile
from typing import Iterable, List, Set
from ahriman.core.build_tools.task import Task
from ahriman.models.package import Package
class Tree:
def __init__(self) -> None:
self.packages: List[Leaf] = []
def levels(self) -> List[List[Package]]:
result: List[List[Package]] = []
unprocessed = [leaf for leaf in self.packages]
while unprocessed:
result.append([leaf.package for leaf in unprocessed if leaf.is_root(unprocessed)])
unprocessed = [leaf for leaf in unprocessed if not leaf.is_root(unprocessed)]
return result
def load(self, packages: Iterable[Package]) -> None:
for package in packages:
leaf = Leaf(package)
leaf.load_dependencies()
self.packages.append(leaf)
class Leaf:
def __init__(self, package: Package) -> None:
self.package = package
self.dependencies: Set[str] = set()
@property
def items(self) -> Iterable[str]:
return self.package.packages.keys()
def is_root(self, packages: Iterable[Leaf]) -> bool:
'''
:param packages:
:return: true if any of packages is dependency of the leaf, false otherwise
'''
for leaf in packages:
if self.dependencies.intersection(leaf.items):
return False
return True
def load_dependencies(self) -> None:
clone_dir = tempfile.mkdtemp()
try:
Task.fetch(clone_dir, self.package.git_url)
self.dependencies = Package.dependencies(clone_dir)
finally:
shutil.rmtree(clone_dir, ignore_errors=True)

View File

@ -26,8 +26,8 @@ class Rsync(Uploader):
def __init__(self, architecture: str, config: Configuration) -> None:
Uploader.__init__(self, architecture, config)
section = self.config.get_section_name('rsync', self.architecture)
self.remote = self.config.get(section, 'remote')
section = config.get_section_name('rsync', architecture)
self.remote = config.get(section, 'remote')
def sync(self, path: str) -> None:
check_output('rsync', '--archive', '--verbose', '--compress', '--partial', '--progress', '--delete', path, self.remote,

View File

@ -26,8 +26,8 @@ class S3(Uploader):
def __init__(self, architecture: str, config: Configuration) -> None:
Uploader.__init__(self, architecture, config)
section = self.config.get_section_name('s3', self.architecture)
self.bucket = self.config.get(section, 'bucket')
section = config.get_section_name('s3', architecture)
self.bucket = config.get(section, 'bucket')
def sync(self, path: str) -> None:
# TODO rewrite to boto, but it is bullshit

View File

@ -27,9 +27,9 @@ from ahriman.models.upload_settings import UploadSettings
class Uploader:
def __init__(self, architecture: str, config: Configuration) -> None:
self.logger = logging.getLogger('builder')
self.architecture = architecture
self.config = config
self.logger = logging.getLogger('builder')
@staticmethod
def run(architecture: str, config: Configuration, target: str, path: str) -> None:
@ -45,8 +45,9 @@ class Uploader:
try:
uploader.sync(path)
except Exception as e:
raise SyncFailed(e) from e
except Exception:
uploader.logger.exception('remote sync failed', exc_info=True)
raise SyncFailed()
def sync(self, path: str) -> None:
pass

View File

@ -24,7 +24,7 @@ from typing import Optional
def check_output(*args: str, exception: Optional[Exception],
cwd = None, stderr: int = subprocess.STDOUT,
cwd: Optional[str] = None, stderr: int = subprocess.STDOUT,
logger: Optional[Logger] = None) -> str:
try:
result = subprocess.check_output(args, cwd=cwd, stderr=stderr).decode('utf8').strip()

View File

@ -39,10 +39,26 @@ class Client:
def update(self, base: str, status: BuildStatusEnum) -> None:
pass
def set_building(self, base: str) -> None:
return self.update(base, BuildStatusEnum.Building)
def set_failed(self, base: str) -> None:
return self.update(base, BuildStatusEnum.Failed)
def set_pending(self, base: str) -> None:
return self.update(base, BuildStatusEnum.Pending)
def set_success(self, package: Package) -> None:
return self.add(package, BuildStatusEnum.Success)
def set_unknown(self, package: Package) -> None:
return self.add(package, BuildStatusEnum.Unknown)
@staticmethod
def load(config: Configuration) -> Client:
host = config.get('web', 'host', fallback=None)
port = config.getint('web', 'port', fallback=None)
def load(architecture: str, config: Configuration) -> Client:
section = config.get_section_name('web', architecture)
host = config.get(section, 'host', fallback=None)
port = config.getint(section, 'port', fallback=None)
if host is None or port is None:
return Client()
return WebClient(host, port)
@ -63,10 +79,12 @@ class WebClient(Client):
payload: Dict[str, Any] = {
'status': status.value,
'base': package.base,
'packages': [p for p in package.packages],
'version': package.version,
'url': package.web_url
'package': {
'base': package.base,
'packages': [p for p in package.packages],
'version': package.version,
'aur_url': package.aur_url
}
}
try:

View File

@ -19,16 +19,17 @@
#
from __future__ import annotations
import shutil
import aur
import aur # type: ignore
import os
import shutil
import tempfile
from dataclasses import dataclass, field
from srcinfo.parse import parse_srcinfo
from typing import Set, Type
from dataclasses import dataclass
from pyalpm import vercmp # type: ignore
from srcinfo.parse import parse_srcinfo # type: ignore
from typing import Dict, List, Optional, Set, Type
from ahriman.core.alpm.pacman import Pacman
from ahriman.core.exceptions import InvalidPackageInfo
from ahriman.core.util import check_output
@ -38,7 +39,7 @@ class Package:
base: str
version: str
aur_url: str
packages: Set[str] = field(default_factory=set)
packages: Dict[str, str] # map of package name to archive name
@property
def git_url(self) -> str:
@ -67,7 +68,7 @@ class Package:
try:
Task.fetch(clone_dir, self.git_url)
# update pkgver first
check_output('makepkg', '--nodeps', '--noprepare', '--nobuild',
check_output('makepkg', '--nodeps', '--nobuild',
exception=None, cwd=clone_dir)
# generate new .SRCINFO and put it to parser
src_info_source = check_output('makepkg', '--printsrcinfo',
@ -75,19 +76,19 @@ class Package:
src_info, errors = parse_srcinfo(src_info_source)
if errors:
raise InvalidPackageInfo(errors)
return f'{src_info["pkgver"]}-{src_info["pkgrel"]}'
return self.full_version(src_info.get('epoch'), src_info['pkgver'], src_info['pkgrel'])
finally:
shutil.rmtree(clone_dir, ignore_errors=True)
@classmethod
def from_archive(cls: Type[Package], path: str, aur_url: str) -> Package:
package, base, version = check_output('expac', '-p', '%n %e %v', path, exception=None).split()
return cls(base, version, aur_url, {package})
def from_archive(cls: Type[Package], path: str, pacman: Pacman, aur_url: str) -> Package:
package = pacman.handle.load_pkg(path)
return cls(package.base, package.version, aur_url, {package.name: os.path.basename(path)})
@classmethod
def from_aur(cls: Type[Package], name: str, aur_url: str)-> Package:
package = aur.info(name)
return cls(package.package_base, package.version, aur_url, {package.name})
return cls(package.package_base, package.version, aur_url, {package.name: ''})
@classmethod
def from_build(cls: Type[Package], path: str, aur_url: str) -> Package:
@ -95,17 +96,38 @@ class Package:
src_info, errors = parse_srcinfo(fn.read())
if errors:
raise InvalidPackageInfo(errors)
packages = set(src_info['packages'].keys())
packages = {key: '' for key in src_info['packages'].keys()}
version = cls.full_version(src_info.get('epoch'), src_info['pkgver'], src_info['pkgrel'])
return cls(src_info['pkgbase'], f'{src_info["pkgver"]}-{src_info["pkgrel"]}', aur_url, packages)
return cls(src_info['pkgbase'], version, aur_url, packages)
@staticmethod
def load(path: str, aur_url: str) -> Package:
def dependencies(path: str) -> Set[str]:
with open(os.path.join(path, '.SRCINFO')) as fn:
src_info, errors = parse_srcinfo(fn.read())
if errors:
raise InvalidPackageInfo(errors)
makedepends = src_info.get('makedepends', [])
# sum over each package
depends: List[str] = src_info.get('depends', [])
for package in src_info['packages'].values():
depends.extend(package.get('depends', []))
# we are not interested in dependencies inside pkgbase
packages = set(src_info['packages'].keys())
return set(depends + makedepends) - packages
@staticmethod
def full_version(epoch: Optional[str], pkgver: str, pkgrel: str) -> str:
prefix = f'{epoch}:' if epoch else ''
return f'{prefix}{pkgver}-{pkgrel}'
@staticmethod
def load(path: str, pacman: Pacman, aur_url: str) -> Package:
try:
if os.path.isdir(path):
package: Package = Package.from_build(path, aur_url)
elif os.path.exists(path):
package = Package.from_archive(path, aur_url)
package = Package.from_archive(path, pacman, aur_url)
else:
package = Package.from_aur(path, aur_url)
return package
@ -116,5 +138,5 @@ class Package:
def is_outdated(self, remote: Package) -> bool:
remote_version = remote.actual_version() # either normal version or updated VCS
result = check_output('vercmp', self.version, remote_version, exception=None)
return True if int(result) < 0 else False
result: int = vercmp(self.version, remote_version)
return result < 0

View File

@ -21,7 +21,7 @@ from __future__ import annotations
from enum import Enum, auto
from ahriman.core.exceptions import InvalidOptionException
from ahriman.core.exceptions import InvalidOption
class ReportSettings(Enum):
@ -31,4 +31,4 @@ class ReportSettings(Enum):
def from_option(value: str) -> ReportSettings:
if value.lower() in ('html',):
return ReportSettings.HTML
raise InvalidOptionException(value)
raise InvalidOption(value)

View File

@ -29,22 +29,37 @@ class RepositoryPaths:
@property
def chroot(self) -> str:
'''
:return: directory for devtools chroot
'''
return os.path.join(self.root, 'chroot')
@property
def manual(self) -> str:
'''
:return: directory for manual updates (i.e. from add command)
'''
return os.path.join(self.root, 'manual')
@property
def packages(self) -> str:
'''
:return: directory for built packages
'''
return os.path.join(self.root, 'packages')
@property
def repository(self) -> str:
'''
:return: repository directory
'''
return os.path.join(self.root, 'repository', self.architecture)
@property
def sources(self) -> str:
'''
:return: directory for downloaded PKGBUILDs for current build
'''
return os.path.join(self.root, 'sources')
def create_tree(self) -> None:

View File

@ -21,7 +21,7 @@ from __future__ import annotations
from enum import Enum, auto
from ahriman.core.exceptions import InvalidOptionException
from ahriman.core.exceptions import InvalidOption
class SignSettings(Enum):
@ -34,4 +34,4 @@ class SignSettings(Enum):
return SignSettings.SignPackages
elif value.lower() in ('repository', 'sign-repository'):
return SignSettings.SignRepository
raise InvalidOptionException(value)
raise InvalidOption(value)

View File

@ -21,7 +21,7 @@ from __future__ import annotations
from enum import Enum, auto
from ahriman.core.exceptions import InvalidOptionException
from ahriman.core.exceptions import InvalidOption
class UploadSettings(Enum):
@ -34,4 +34,4 @@ class UploadSettings(Enum):
return UploadSettings.Rsync
elif value.lower() in ('s3',):
return UploadSettings.S3
raise InvalidOptionException(value)
raise InvalidOption(value)

View File

@ -17,4 +17,4 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
__version__ = '0.10.0'
__version__ = '0.11.7'

View File

@ -17,16 +17,19 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
from aiohttp.web import middleware, Request, Response
from logging import Logger
from typing import Callable
from aiohttp.web import middleware, Request
from aiohttp.web_exceptions import HTTPClientError
from aiohttp.web_response import StreamResponse
from logging import Logger
from typing import Awaitable, Callable
def exception_handler(logger: Logger) -> Callable:
HandlerType = Callable[[Request], Awaitable[StreamResponse]]
def exception_handler(logger: Logger) -> Callable[[Request, HandlerType], Awaitable[StreamResponse]]:
@middleware
async def handle(request: Request, handler: Callable) -> Response:
async def handle(request: Request, handler: HandlerType) -> StreamResponse:
try:
return await handler(request)
except HTTPClientError:

View File

@ -22,8 +22,10 @@ from aiohttp.web import View
from ahriman.core.watcher.watcher import Watcher
# special class to make it typed
class BaseView(View):
@property
def service(self) -> Watcher:
return self.request.app['watcher']
watcher: Watcher = self.request.app['watcher']
return watcher

View File

@ -17,18 +17,20 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
import aiohttp_jinja2 # type: ignore
from typing import Any, Dict
from aiohttp_jinja2 import template
import ahriman.version as version
from ahriman.web.views.base import BaseView
class IndexView(BaseView):
@template("index.jinja2")
@aiohttp_jinja2.template("build-status.jinja2") # type: ignore
async def get(self) -> Dict[str, Any]:
# some magic to make it jinja-readable
# some magic to make it jinja-friendly
packages = [
{
'base': package.base,
@ -44,4 +46,5 @@ class IndexView(BaseView):
'architecture': self.service.architecture,
'packages': packages,
'repository': self.service.repository.name,
'version': version.__version__,
}

View File

@ -37,7 +37,7 @@ class PackageView(BaseView):
data = await self.request.json()
package = Package(**data['package']) if 'package' in data else None
status = BuildStatusEnum(data.get('status', 'unknown'))
status = BuildStatusEnum(data['status'])
self.service.update(base, status, package)
return HTTPOk()

View File

@ -17,7 +17,7 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
import aiohttp_jinja2
import aiohttp_jinja2 # type: ignore
import jinja2
import logging
@ -38,17 +38,19 @@ async def on_startup(app: web.Application) -> None:
app.logger.info('server started')
try:
app['watcher'].load()
except Exception as e:
except Exception:
app.logger.exception('could not load packages', exc_info=True)
raise InitializeException() from e
raise InitializeException()
def run_server(app: web.Application) -> None:
def run_server(app: web.Application, architecture: str) -> None:
app.logger.info('start server')
web.run_app(app,
host=app['config'].get('web', 'host'),
port=app['config'].getint('web', 'port'),
handle_signals=False)
section = app['config'].get_section_name('web', architecture)
host = app['config'].get(section, 'host')
port = app['config'].getint(section, 'port')
web.run_app(app, host=host, port=port, handle_signals=False)
def setup_service(architecture: str, config: Configuration) -> web.Application: