Compare commits

...

15 Commits

Author SHA1 Message Date
b0d1f3c091 Release 1.0.0 2021-04-10 01:38:55 +03:00
50e219fda5 import pgp key implementation (#17)
* import pgp key implementation

* do not ask confirmation for local sign. Also add argparser test

* superseed requests by python-aur package

* ...and drop --skippgpcheck makgepkg flag by default
2021-04-10 01:37:45 +03:00
75298d1b8a better naming for actions 2021-04-09 20:02:17 +03:00
8196dcc8a0 add search subparser (#15) 2021-04-09 11:57:06 +03:00
f634f1df58 Add web status route (#13)
* add status route

* typed status and get status at the start of application
2021-04-08 01:48:53 +03:00
32df4fc54f Move search line inside extended report option 2021-04-06 17:03:34 +03:00
11ae930c59 Release 0.22.1 2021-04-06 05:54:04 +03:00
9c332c23d2 format long line 2021-04-06 05:53:38 +03:00
4ed0a49a44 add ability to skip email report generation for empty update list 2021-04-06 05:51:50 +03:00
50f532a48a Release 0.22.0 2021-04-06 05:46:12 +03:00
c6ccf53768 Email report (#11)
* Demo email report implementation

* improved ssl mode

* correct default option spelling and more fields to be hidden for not
extended reports
2021-04-06 05:45:17 +03:00
ce0c07cbd9 Release 0.21.4 2021-04-05 02:28:38 +03:00
912a76d5cb drop changelog
the main reason is that it uses github to generate changelog. Thus it
will be updated AFTER release is created
2021-04-05 02:27:12 +03:00
76d0b0bc6d Release 0.21.3 2021-04-05 02:22:44 +03:00
27d018e721 update changelog at correct step
also fix commit filter and do not update sha anymore
2021-04-05 02:22:11 +03:00
66 changed files with 1345 additions and 330 deletions

View File

@ -1,4 +1,4 @@
name: create release
name: release
on:
push:
@ -20,7 +20,7 @@ jobs:
uses: jaywcjlove/changelog-generator@main
with:
token: ${{ secrets.GITHUB_TOKEN }}
filter: '[R|r]elease[d]\s+[v|V]\d(\.\d+){0,2}'
filter: 'Release \d+\.\d+\.\d+'
- name: create archive
run: make archive
env:

View File

@ -1,5 +1,4 @@
# based on https://github.com/actions/starter-workflows/blob/main/ci/python-app.yml
name: check commit
name: tests
on:
push:

View File

@ -1,126 +0,0 @@
# Changelog
## [0.21.1](https://github.com/arcan1s/ahriman/tree/0.21.1) (2021-04-04)
[Full Changelog](https://github.com/arcan1s/ahriman/compare/0.21.0...0.21.1)
## [0.21.0](https://github.com/arcan1s/ahriman/tree/0.21.0) (2021-04-04)
[Full Changelog](https://github.com/arcan1s/ahriman/compare/0.20.0...0.21.0)
## [0.20.0](https://github.com/arcan1s/ahriman/tree/0.20.0) (2021-03-31)
[Full Changelog](https://github.com/arcan1s/ahriman/compare/0.19.0...0.20.0)
## [0.19.0](https://github.com/arcan1s/ahriman/tree/0.19.0) (2021-03-30)
[Full Changelog](https://github.com/arcan1s/ahriman/compare/0.18.0...0.19.0)
## [0.18.0](https://github.com/arcan1s/ahriman/tree/0.18.0) (2021-03-29)
[Full Changelog](https://github.com/arcan1s/ahriman/compare/0.17.0...0.18.0)
## [0.17.0](https://github.com/arcan1s/ahriman/tree/0.17.0) (2021-03-29)
[Full Changelog](https://github.com/arcan1s/ahriman/compare/0.16.0...0.17.0)
**Implemented enhancements:**
- Sign command [\#7](https://github.com/arcan1s/ahriman/issues/7)
- Provide extended setup help command [\#6](https://github.com/arcan1s/ahriman/issues/6)
**Merged pull requests:**
- Setup command [\#9](https://github.com/arcan1s/ahriman/pull/9) ([arcan1s](https://github.com/arcan1s))
- add sign command [\#8](https://github.com/arcan1s/ahriman/pull/8) ([arcan1s](https://github.com/arcan1s))
## [0.16.0](https://github.com/arcan1s/ahriman/tree/0.16.0) (2021-03-28)
[Full Changelog](https://github.com/arcan1s/ahriman/compare/0.15.0...0.16.0)
**Implemented enhancements:**
- Review unsafe option [\#4](https://github.com/arcan1s/ahriman/issues/4)
- Split functions class to module package [\#3](https://github.com/arcan1s/ahriman/issues/3)
- Import from another repository [\#2](https://github.com/arcan1s/ahriman/issues/2)
- Tests [\#1](https://github.com/arcan1s/ahriman/issues/1)
**Merged pull requests:**
- Add tests \(\#1\) [\#5](https://github.com/arcan1s/ahriman/pull/5) ([arcan1s](https://github.com/arcan1s))
## [0.15.0](https://github.com/arcan1s/ahriman/tree/0.15.0) (2021-03-20)
[Full Changelog](https://github.com/arcan1s/ahriman/compare/0.14.1...0.15.0)
## [0.14.1](https://github.com/arcan1s/ahriman/tree/0.14.1) (2021-03-17)
[Full Changelog](https://github.com/arcan1s/ahriman/compare/0.14.0...0.14.1)
## [0.14.0](https://github.com/arcan1s/ahriman/tree/0.14.0) (2021-03-16)
[Full Changelog](https://github.com/arcan1s/ahriman/compare/0.13.0...0.14.0)
## [0.13.0](https://github.com/arcan1s/ahriman/tree/0.13.0) (2021-03-15)
[Full Changelog](https://github.com/arcan1s/ahriman/compare/0.12.2...0.13.0)
## [0.12.2](https://github.com/arcan1s/ahriman/tree/0.12.2) (2021-03-15)
[Full Changelog](https://github.com/arcan1s/ahriman/compare/0.12.1...0.12.2)
## [0.12.1](https://github.com/arcan1s/ahriman/tree/0.12.1) (2021-03-15)
[Full Changelog](https://github.com/arcan1s/ahriman/compare/0.12.0...0.12.1)
## [0.12.0](https://github.com/arcan1s/ahriman/tree/0.12.0) (2021-03-15)
[Full Changelog](https://github.com/arcan1s/ahriman/compare/0.11.7...0.12.0)
## [0.11.7](https://github.com/arcan1s/ahriman/tree/0.11.7) (2021-03-14)
[Full Changelog](https://github.com/arcan1s/ahriman/compare/0.11.6...0.11.7)
## [0.11.6](https://github.com/arcan1s/ahriman/tree/0.11.6) (2021-03-13)
[Full Changelog](https://github.com/arcan1s/ahriman/compare/0.11.5...0.11.6)
## [0.11.5](https://github.com/arcan1s/ahriman/tree/0.11.5) (2021-03-13)
[Full Changelog](https://github.com/arcan1s/ahriman/compare/0.11.4...0.11.5)
## [0.11.4](https://github.com/arcan1s/ahriman/tree/0.11.4) (2021-03-13)
[Full Changelog](https://github.com/arcan1s/ahriman/compare/0.11.3...0.11.4)
## [0.11.3](https://github.com/arcan1s/ahriman/tree/0.11.3) (2021-03-12)
[Full Changelog](https://github.com/arcan1s/ahriman/compare/0.11.2...0.11.3)
## [0.11.2](https://github.com/arcan1s/ahriman/tree/0.11.2) (2021-03-12)
[Full Changelog](https://github.com/arcan1s/ahriman/compare/0.11.1...0.11.2)
## [0.11.1](https://github.com/arcan1s/ahriman/tree/0.11.1) (2021-03-11)
[Full Changelog](https://github.com/arcan1s/ahriman/compare/0.11.0...0.11.1)
## [0.11.0](https://github.com/arcan1s/ahriman/tree/0.11.0) (2021-03-11)
[Full Changelog](https://github.com/arcan1s/ahriman/compare/0.10.0...0.11.0)
## [0.10.0](https://github.com/arcan1s/ahriman/tree/0.10.0) (2021-03-10)
[Full Changelog](https://github.com/arcan1s/ahriman/compare/0.9.1...0.10.0)
## [0.9.1](https://github.com/arcan1s/ahriman/tree/0.9.1) (2021-03-09)
[Full Changelog](https://github.com/arcan1s/ahriman/compare/0.9.0...0.9.1)
## [0.9.0](https://github.com/arcan1s/ahriman/tree/0.9.0) (2021-03-08)
[Full Changelog](https://github.com/arcan1s/ahriman/compare/53d21d6496ab902b664a678a2f2d3a7f4e96d8d1...0.9.0)
\* *This Changelog was automatically generated by [github_changelog_generator](https://github.com/github-changelog-generator/github-changelog-generator)*

View File

@ -47,7 +47,23 @@ Settings for signing packages or repository. Group name must refer to architectu
Report generation settings.
* `target` - list of reports to be generated, space separated list of strings, optional. Allowed values are `html`.
* `target` - list of reports to be generated, space separated list of strings, optional. Allowed values are `html`, `email`.
### `email:*` groups
Group name must refer to architecture, e.g. it should be `email:x86_64` for x86_64 architecture.
* `homepage` - link to homepage, string, optional.
* `host` - SMTP host for sending emails, string, required.
* `link_path` - prefix for HTML links, string, required.
* `no_empty_report` - skip report generation for empty packages list, boolean, optional, default `yes`.
* `password` - SMTP password to authenticate, string, optional.
* `port` - SMTP port for sending emails, int, required.
* `receivers` - SMTP receiver addresses, space separated list of strings, required.
* `sender` - SMTP sender address, string, required.
* `ssl` - SSL mode for SMTP connection, one of `ssl`, `starttls`, `disabled`, optional, default `disabled`.
* `template_path` - path to Jinja2 template, string, required.
* `user` - SMTP user to authenticate, string, optional.
### `html:*` groups

View File

@ -1,9 +1,9 @@
.PHONY: archive archive_directory archlinux changelog check clean directory push tests version
.PHONY: archive archive_directory archlinux check clean directory push tests version
.DEFAULT_GOAL := archlinux
PROJECT := ahriman
FILES := AUTHORS CHANGELOG.md COPYING CONFIGURING.md README.md package src setup.py
FILES := AUTHORS COPYING CONFIGURING.md README.md package src setup.py
TARGET_FILES := $(addprefix $(PROJECT)/, $(FILES))
IGNORE_FILES := package/archlinux src/.mypy_cache
@ -21,18 +21,11 @@ archive_directory: $(TARGET_FILES)
find "$(PROJECT)" -depth -type d -name "*.egg-info" -execdir rm -rf {} +
archlinux: archive
sed -i "/sha512sums=('[0-9A-Fa-f]*/s/[^'][^)]*/sha512sums=('$$(sha512sum $(PROJECT)-$(VERSION)-src.tar.xz | awk '{print $$1}')'/" package/archlinux/PKGBUILD
sed -i "s/pkgver=[0-9.]*/pkgver=$(VERSION)/" package/archlinux/PKGBUILD
changelog:
ifndef GITHUB_TOKEN
$(error GITHUB_TOKEN is required, but not set)
endif
docker run -it --rm -v "$(pwd)":/usr/local/src/your-app ferrarimarco/github-changelog-generator -u arcan1s -p ahriman -t $(GITHUB_TOKEN)
check: clean
cd src && mypy --implicit-reexport --strict -p "$(PROJECT)"
find "src/$(PROJECT)" tests -name "*.py" -execdir autopep8 --exit-code --max-line-length 120 -aa -i {} +
find "src/$(PROJECT)" "tests/$(PROJECT)" -name "*.py" -execdir autopep8 --exit-code --max-line-length 120 -aa -i {} +
cd src && pylint --rcfile=../.pylintrc "$(PROJECT)"
clean:
@ -42,11 +35,11 @@ clean:
directory: clean
mkdir "$(PROJECT)"
push: archlinux changelog
git add package/archlinux/PKGBUILD src/ahriman/version.py CHANGELOG.md
push: archlinux
git add package/archlinux/PKGBUILD src/ahriman/version.py
git commit -m "Release $(VERSION)"
git push
git tag "$(VERSION)"
git push
git push --tags
tests: clean

View File

@ -1,6 +1,6 @@
# ArcHlinux ReposItory MANager
![build status](https://github.com/arcan1s/ahriman/actions/workflows/python-app.yml/badge.svg)
[![build status](https://github.com/arcan1s/ahriman/actions/workflows/run-tests.yml/badge.svg)](https://github.com/arcan1s/ahriman/actions/workflows/run-tests.yml)
Wrapper for managing custom repository inspired by [repo-scripts](https://github.com/arcan1s/repo-scripts).

View File

@ -1,7 +1,7 @@
# Maintainer: Evgeniy Alekseev
pkgname='ahriman'
pkgver=0.21.2
pkgver=1.0.0
pkgrel=1
pkgdesc="ArcHlinux ReposItory MANager"
arch=('any')
@ -17,7 +17,6 @@ optdepends=('aws-cli: sync to s3'
'python-aiohttp: web server'
'python-aiohttp-jinja2: web server'
'python-jinja: html report generation'
'python-requests: web server'
'rsync: sync by using rsync'
'subversion: -svn packages support')
source=("https://github.com/arcan1s/ahriman/releases/download/$pkgver/$pkgname-$pkgver-src.tar.xz"

View File

@ -13,7 +13,7 @@ archbuild_flags =
build_command = extra-x86_64-build
ignore_packages =
makechrootpkg_flags =
makepkg_flags = --skippgpcheck
makepkg_flags =
[repository]
name = aur-clone
@ -25,17 +25,19 @@ target =
[report]
target =
[email]
no_empty_report = yes
template_path = /usr/share/ahriman/repo-index.jinja2
ssl = disabled
[html]
path =
homepage =
link_path =
template_path = /usr/share/ahriman/repo-index.jinja2
[upload]
target =
[rsync]
command = rsync --archive --verbose --compress --partial --delete
command = rsync --archive --compress --partial --delete
[s3]
command = aws s3 sync --quiet --delete

View File

@ -5,28 +5,32 @@
{% include "style.jinja2" %}
{% include "sorttable.jinja2" %}
{% include "search.jinja2" %}
{% if extended_report %}
{% include "sorttable.jinja2" %}
{% include "search.jinja2" %}
{% endif %}
</head>
<body>
<div class="root">
<h1>Archlinux user repository</h1>
{% if extended_report %}
<h1>Archlinux user repository</h1>
<section class="element">
{% if pgp_key is not none %}
<p>This repository is signed with <a href="http://keys.gnupg.net/pks/lookup?search=0x{{ pgp_key|e }}&fingerprint=on&op=index" title="key search">{{ pgp_key|e }}</a> by default.</p>
{% endif %}
<section class="element">
{% if pgp_key is not none %}
<p>This repository is signed with <a href="http://keys.gnupg.net/pks/lookup?search=0x{{ pgp_key|e }}&fingerprint=on&op=index" title="key search">{{ pgp_key|e }}</a> by default.</p>
{% endif %}
<code>
$ cat /etc/pacman.conf<br>
[{{ repository|e }}]<br>
Server = {{ link_path|e }}<br>
SigLevel = Database{% if has_repo_signed %}Required{% else %}Never{% endif %} Package{% if has_package_signed %}Required{% else %}Never{% endif %} TrustedOnly
</code>
</section>
{% include "search-line.jinja2" %}
<code>
$ cat /etc/pacman.conf<br>
[{{ repository|e }}]<br>
Server = {{ link_path|e }}<br>
SigLevel = Database{% if has_repo_signed %}Required{% else %}Never{% endif %} Package{% if has_package_signed %}Required{% else %}Never{% endif %} TrustedOnly
</code>
</section>
{% include "search-line.jinja2" %}
{% endif %}
<section class="element">
<table class="sortable search-table">
@ -50,13 +54,15 @@
</table>
</section>
<footer>
<ul class="navigation">
{% if homepage is not none %}
<li><a href="{{ homepage|e }}" title="homepage">Homepage</a></li>
{% endif %}
</ul>
</footer>
{% if extended_report %}
<footer>
<ul class="navigation">
{% if homepage is not none %}
<li><a href="{{ homepage|e }}" title="homepage">Homepage</a></li>
{% endif %}
</ul>
</footer>
{% endif %}
</div>
</body>
</html>

View File

@ -29,6 +29,7 @@ setup(
install_requires=[
"aur",
"pyalpm",
"requests",
"srcinfo",
],
setup_requires=[
@ -89,7 +90,6 @@ setup(
"Jinja2",
"aiohttp",
"aiohttp_jinja2",
"requests",
],
},
)

View File

@ -22,9 +22,8 @@ import sys
from pathlib import Path
import ahriman.application.handlers as handlers
import ahriman.version as version
from ahriman import version
from ahriman.application import handlers
from ahriman.models.build_status import BuildStatusEnum
from ahriman.models.sign_settings import SignSettings
@ -57,9 +56,11 @@ def _parser() -> argparse.ArgumentParser:
_set_check_parser(subparsers)
_set_clean_parser(subparsers)
_set_config_parser(subparsers)
_set_key_import_parser(subparsers)
_set_rebuild_parser(subparsers)
_set_remove_parser(subparsers)
_set_report_parser(subparsers)
_set_search_parser(subparsers)
_set_setup_parser(subparsers)
_set_sign_parser(subparsers)
_set_status_parser(subparsers)
@ -131,6 +132,21 @@ def _set_config_parser(root: SubParserAction) -> argparse.ArgumentParser:
return parser
def _set_key_import_parser(root: SubParserAction) -> argparse.ArgumentParser:
"""
add parser for key import subcommand
:param root: subparsers for the commands
:return: created argument parser
"""
parser = root.add_parser("key-import", help="import PGP key",
description="import PGP key from public sources to repository user",
formatter_class=argparse.ArgumentDefaultsHelpFormatter)
parser.add_argument("--key-server", help="key server for key import", default="keys.gnupg.net")
parser.add_argument("key", help="PGP key to import from public server")
parser.set_defaults(handler=handlers.KeyImport, lock=None, no_report=True)
return parser
def _set_rebuild_parser(root: SubParserAction) -> argparse.ArgumentParser:
"""
add parser for rebuild subcommand
@ -170,6 +186,18 @@ def _set_report_parser(root: SubParserAction) -> argparse.ArgumentParser:
return parser
def _set_search_parser(root: SubParserAction) -> argparse.ArgumentParser:
"""
add parser for search subcommand
:param root: subparsers for the commands
:return: created argument parser
"""
parser = root.add_parser("search", help="search for package", description="search for package in AUR using API")
parser.add_argument("search", help="search terms, can be specified multiple times", nargs="+")
parser.set_defaults(handler=handlers.Search, lock=None, no_report=True, unsafe=True)
return parser
def _set_setup_parser(root: SubParserAction) -> argparse.ArgumentParser:
"""
add parser for setup subcommand

View File

@ -51,12 +51,12 @@ class Application:
self.architecture = architecture
self.repository = Repository(architecture, configuration)
def _finalize(self) -> None:
def _finalize(self, built_packages: Iterable[Package]) -> None:
"""
generate report and sync to remote server
"""
self.report([])
self.sync([])
self.report([], built_packages)
self.sync([], built_packages)
def _known_packages(self) -> Set[str]:
"""
@ -160,15 +160,16 @@ class Application:
:param names: list of packages (either base or name) to remove
"""
self.repository.process_remove(names)
self._finalize()
self._finalize([])
def report(self, target: Iterable[str]) -> None:
def report(self, target: Iterable[str], built_packages: Iterable[Package]) -> None:
"""
generate report
:param target: list of targets to run (e.g. html)
:param built_packages: list of packages which has just been built
"""
targets = target or None
self.repository.process_report(targets)
self.repository.process_report(targets, built_packages)
def sign(self, packages: Iterable[str]) -> None:
"""
@ -191,15 +192,16 @@ class Application:
self.update([])
# sign repository database if set
self.repository.sign.sign_repository(self.repository.repo.repo_path)
self._finalize()
self._finalize([])
def sync(self, target: Iterable[str]) -> None:
def sync(self, target: Iterable[str], built_packages: Iterable[Package]) -> None:
"""
sync to remote server
:param target: list of targets to run (e.g. s3)
:param built_packages: list of packages which has just been built
"""
targets = target or None
self.repository.process_sync(targets)
self.repository.process_sync(targets, built_packages)
def update(self, updates: Iterable[Package]) -> None:
"""
@ -207,8 +209,9 @@ class Application:
:param updates: list of packages to update
"""
def process_update(paths: Iterable[Path]) -> None:
updated = [Package.load(path, self.repository.pacman, self.repository.aur_url) for path in paths]
self.repository.process_update(paths)
self._finalize()
self._finalize(updated)
# process built packages
packages = self.repository.packages_built()

View File

@ -22,9 +22,11 @@ from ahriman.application.handlers.handler import Handler
from ahriman.application.handlers.add import Add
from ahriman.application.handlers.clean import Clean
from ahriman.application.handlers.dump import Dump
from ahriman.application.handlers.key_import import KeyImport
from ahriman.application.handlers.rebuild import Rebuild
from ahriman.application.handlers.remove import Remove
from ahriman.application.handlers.report import Report
from ahriman.application.handlers.search import Search
from ahriman.application.handlers.setup import Setup
from ahriman.application.handlers.sign import Sign
from ahriman.application.handlers.status import Status

View File

@ -0,0 +1,42 @@
#
# Copyright (c) 2021 ahriman team.
#
# This file is part of ahriman
# (see https://github.com/arcan1s/ahriman).
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
import argparse
from typing import Type
from ahriman.application.application import Application
from ahriman.application.handlers.handler import Handler
from ahriman.core.configuration import Configuration
class KeyImport(Handler):
"""
key import packages handler
"""
@classmethod
def run(cls: Type[Handler], args: argparse.Namespace, architecture: str, configuration: Configuration) -> None:
"""
callback for command line
:param args: command line args
:param architecture: repository architecture
:param configuration: configuration instance
"""
Application(architecture, configuration).repository.sign.import_key(args.key_server, args.key)

View File

@ -39,4 +39,4 @@ class Report(Handler):
:param architecture: repository architecture
:param configuration: configuration instance
"""
Application(architecture, configuration).report(args.target)
Application(architecture, configuration).report(args.target, [])

View File

@ -0,0 +1,58 @@
#
# Copyright (c) 2021 ahriman team.
#
# This file is part of ahriman
# (see https://github.com/arcan1s/ahriman).
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
import argparse
import aur # type: ignore
from typing import Callable, Type
from ahriman.application.handlers.handler import Handler
from ahriman.core.configuration import Configuration
class Search(Handler):
"""
packages search handler
"""
@classmethod
def run(cls: Type[Handler], args: argparse.Namespace, architecture: str, configuration: Configuration) -> None:
"""
callback for command line
:param args: command line args
:param architecture: repository architecture
:param configuration: configuration instance
"""
search = " ".join(args.search)
packages = aur.search(search)
# it actually always should return string
# explicit cast to string just to avoid mypy warning for untyped library
comparator: Callable[[aur.Package], str] = lambda item: str(item.package_base)
for package in sorted(packages, key=comparator):
Search.log_fn(package)
@staticmethod
def log_fn(package: aur.Package) -> None:
"""
log package information
:param package: package object as from AUR
"""
print(f"=> {package.package_base} {package.version}")
print(f" {package.description}")

View File

@ -39,4 +39,4 @@ class Sync(Handler):
:param architecture: repository architecture
:param configuration: configuration instance
"""
Application(architecture, configuration).sync(args.target)
Application(architecture, configuration).sync(args.target, [])

View File

@ -20,12 +20,14 @@
from __future__ import annotations
import argparse
import logging
import os
from pathlib import Path
from types import TracebackType
from typing import Literal, Optional, Type
from ahriman import version
from ahriman.core.configuration import Configuration
from ahriman.core.exceptions import DuplicateRun, UnsafeRun
from ahriman.core.status.client import Client
@ -61,12 +63,13 @@ class Lock:
default workflow is the following:
check user UID
remove lock file if force flag is set
check if there is lock file
check web status watcher status
create lock file
report to web if enabled
"""
self.check_user()
self.check_version()
self.create()
self.reporter.update_self(BuildStatusEnum.Building)
return self
@ -85,6 +88,15 @@ class Lock:
self.reporter.update_self(status)
return False
def check_version(self) -> None:
"""
check web server version
"""
status = self.reporter.get_internal()
if status.version is not None and status.version != version.__version__:
logging.getLogger("root").warning(f"status watcher version mismatch, "
f"our {version.__version__}, their {status.version}")
def check_user(self) -> None:
"""
check if current user is actually owner of ahriman root

View File

@ -140,7 +140,7 @@ class Configuration(configparser.RawConfigParser):
if path == self.logging_path:
continue # we don't want to load logging explicitly
self.read(path)
except (FileNotFoundError, configparser.NoOptionError):
except (FileNotFoundError, configparser.NoOptionError, configparser.NoSectionError):
pass
def load_logging(self, logfile: bool) -> None:

View File

@ -0,0 +1,105 @@
#
# Copyright (c) 2021 ahriman team.
#
# This file is part of ahriman
# (see https://github.com/arcan1s/ahriman).
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
import datetime
import smtplib
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
from typing import Dict, Iterable
from ahriman.core.configuration import Configuration
from ahriman.core.report.jinja_template import JinjaTemplate
from ahriman.core.report.report import Report
from ahriman.core.util import pretty_datetime
from ahriman.models.package import Package
from ahriman.models.smtp_ssl_settings import SmtpSSLSettings
class Email(Report, JinjaTemplate):
"""
email report generator
:ivar host: SMTP host to connect
:ivar no_empty_report: skip empty report generation
:ivar password: password to authenticate via SMTP
:ivar port: SMTP port to connect
:ivar receivers: list of receivers emails
:ivar sender: sender email address
:ivar ssl: SSL mode for SMTP connection
:ivar user: username to authenticate via SMTP
"""
def __init__(self, architecture: str, configuration: Configuration) -> None:
"""
default constructor
:param architecture: repository architecture
:param configuration: configuration instance
"""
Report.__init__(self, architecture, configuration)
JinjaTemplate.__init__(self, "email", configuration)
# base smtp settings
self.host = configuration.get("email", "host")
self.no_empty_report = configuration.getboolean("email", "no_empty_report", fallback=True)
self.password = configuration.get("email", "password", fallback=None)
self.port = configuration.getint("email", "port")
self.receivers = configuration.getlist("email", "receivers")
self.sender = configuration.get("email", "sender")
self.ssl = SmtpSSLSettings.from_option(configuration.get("email", "ssl", fallback="disabled"))
self.user = configuration.get("email", "user", fallback=None)
def _send(self, text: str, attachment: Dict[str, str]) -> None:
"""
send email callback
:param text: email body text
:param attachment: map of attachment filename to attachment content
"""
message = MIMEMultipart()
message["From"] = self.sender
message["To"] = ", ".join(self.receivers)
message["Subject"] = f"{self.name} build report at {pretty_datetime(datetime.datetime.utcnow().timestamp())}"
message.attach(MIMEText(text, "html"))
for filename, content in attachment.items():
attach = MIMEText(content, "html")
attach.add_header("Content-Disposition", "attachment", filename=filename)
message.attach(attach)
if self.ssl != SmtpSSLSettings.SSL:
session = smtplib.SMTP(self.host, self.port)
if self.ssl == SmtpSSLSettings.STARTTLS:
session.starttls()
else:
session = smtplib.SMTP_SSL(self.host, self.port)
if self.user is not None and self.password is not None:
session.login(self.user, self.password)
session.sendmail(self.sender, self.receivers, message.as_string())
session.quit()
def generate(self, packages: Iterable[Package], built_packages: Iterable[Package]) -> None:
"""
generate report for the specified packages
:param packages: list of packages to generate report
:param built_packages: list of packages which has just been built
"""
if self.no_empty_report and not built_packages:
return
text = self.make_html(built_packages, False)
attachments = {"index.html": self.make_html(packages, True)}
self._send(text, attachments)

View File

@ -17,51 +17,18 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
import jinja2
from typing import Callable, Dict, Iterable
from typing import Iterable
from ahriman.core.configuration import Configuration
from ahriman.core.report.jinja_template import JinjaTemplate
from ahriman.core.report.report import Report
from ahriman.core.sign.gpg import GPG
from ahriman.core.util import pretty_datetime, pretty_size
from ahriman.models.package import Package
from ahriman.models.sign_settings import SignSettings
class HTML(Report):
class HTML(Report, JinjaTemplate):
"""
html report generator
It uses jinja2 templates for report generation, the following variables are allowed:
homepage - link to homepage, string, optional
link_path - prefix fo packages to download, string, required
has_package_signed - True in case if package sign enabled, False otherwise, required
has_repo_signed - True in case if repository database sign enabled, False otherwise, required
packages - sorted list of packages properties, required
* architecture, string
* archive_size, pretty printed size, string
* build_date, pretty printed datetime, string
* depends, sorted list of strings
* description, string
* filename, string,
* groups, sorted list of strings
* installed_size, pretty printed datetime, string
* licenses, sorted list of strings
* name, string
* url, string
* version, string
pgp_key - default PGP key ID, string, optional
repository - repository name, string, required
:ivar homepage: homepage link if any (for footer)
:ivar link_path: prefix fo packages to download
:ivar name: repository name
:ivar default_pgp_key: default PGP key
:ivar report_path: output path to html report
:ivar sign_targets: targets to sign enabled in configuration
:ivar template_path: path to directory with jinja templates
"""
def __init__(self, architecture: str, configuration: Configuration) -> None:
@ -71,51 +38,15 @@ class HTML(Report):
:param configuration: configuration instance
"""
Report.__init__(self, architecture, configuration)
JinjaTemplate.__init__(self, "html", configuration)
self.report_path = configuration.getpath("html", "path")
self.link_path = configuration.get("html", "link_path")
self.template_path = configuration.getpath("html", "template_path")
# base template vars
self.homepage = configuration.get("html", "homepage", fallback=None)
self.name = configuration.get("repository", "name")
self.sign_targets, self.default_pgp_key = GPG.sign_options(configuration)
def generate(self, packages: Iterable[Package]) -> None:
def generate(self, packages: Iterable[Package], built_packages: Iterable[Package]) -> None:
"""
generate report for the specified packages
:param packages: list of packages to generate report
:param built_packages: list of packages which has just been built
"""
# idea comes from https://stackoverflow.com/a/38642558
loader = jinja2.FileSystemLoader(searchpath=self.template_path.parent)
environment = jinja2.Environment(loader=loader)
template = environment.get_template(self.template_path.name)
content = [
{
"architecture": properties.architecture or "",
"archive_size": pretty_size(properties.archive_size),
"build_date": pretty_datetime(properties.build_date),
"depends": properties.depends,
"description": properties.description or "",
"filename": properties.filename,
"groups": properties.groups,
"installed_size": pretty_size(properties.installed_size),
"licenses": properties.licenses,
"name": package,
"url": properties.url or "",
"version": base.version
} for base in packages for package, properties in base.packages.items()
]
comparator: Callable[[Dict[str, str]], str] = lambda item: item["filename"]
html = template.render(
homepage=self.homepage,
link_path=self.link_path,
has_package_signed=SignSettings.Packages in self.sign_targets,
has_repo_signed=SignSettings.Repository in self.sign_targets,
packages=sorted(content, key=comparator),
pgp_key=self.default_pgp_key,
repository=self.name)
html = self.make_html(packages, True)
self.report_path.write_text(html)

View File

@ -0,0 +1,117 @@
#
# Copyright (c) 2021 ahriman team.
#
# This file is part of ahriman
# (see https://github.com/arcan1s/ahriman).
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
import jinja2
from typing import Callable, Dict, Iterable
from ahriman.core.configuration import Configuration
from ahriman.core.sign.gpg import GPG
from ahriman.core.util import pretty_datetime, pretty_size
from ahriman.models.package import Package
from ahriman.models.sign_settings import SignSettings
class JinjaTemplate:
"""
jinja based report generator
It uses jinja2 templates for report generation, the following variables are allowed:
homepage - link to homepage, string, optional
link_path - prefix fo packages to download, string, required
has_package_signed - True in case if package sign enabled, False otherwise, required
has_repo_signed - True in case if repository database sign enabled, False otherwise, required
packages - sorted list of packages properties, required
* architecture, string
* archive_size, pretty printed size, string
* build_date, pretty printed datetime, string
* depends, sorted list of strings
* description, string
* filename, string,
* groups, sorted list of strings
* installed_size, pretty printed datetime, string
* licenses, sorted list of strings
* name, string
* url, string
* version, string
pgp_key - default PGP key ID, string, optional
repository - repository name, string, required
:ivar homepage: homepage link if any (for footer)
:ivar link_path: prefix fo packages to download
:ivar name: repository name
:ivar default_pgp_key: default PGP key
:ivar sign_targets: targets to sign enabled in configuration
:ivar template_path: path to directory with jinja templates
"""
def __init__(self, section: str, configuration: Configuration) -> None:
"""
default constructor
:param section: settings section name
:param configuration: configuration instance
"""
self.link_path = configuration.get(section, "link_path")
self.template_path = configuration.getpath(section, "template_path")
# base template vars
self.homepage = configuration.get(section, "homepage", fallback=None)
self.name = configuration.get("repository", "name")
self.sign_targets, self.default_pgp_key = GPG.sign_options(configuration)
def make_html(self, packages: Iterable[Package], extended_report: bool) -> str:
"""
generate report for the specified packages
:param packages: list of packages to generate report
:param extended_report: include additional blocks to the report
"""
# idea comes from https://stackoverflow.com/a/38642558
loader = jinja2.FileSystemLoader(searchpath=self.template_path.parent)
environment = jinja2.Environment(loader=loader)
template = environment.get_template(self.template_path.name)
content = [
{
"architecture": properties.architecture or "",
"archive_size": pretty_size(properties.archive_size),
"build_date": pretty_datetime(properties.build_date),
"depends": properties.depends,
"description": properties.description or "",
"filename": properties.filename,
"groups": properties.groups,
"installed_size": pretty_size(properties.installed_size),
"licenses": properties.licenses,
"name": package,
"url": properties.url or "",
"version": base.version
} for base in packages for package, properties in base.packages.items()
]
comparator: Callable[[Dict[str, str]], str] = lambda item: item["filename"]
return template.render(
extended_report=extended_report,
homepage=self.homepage,
link_path=self.link_path,
has_package_signed=SignSettings.Packages in self.sign_targets,
has_repo_signed=SignSettings.Repository in self.sign_targets,
packages=sorted(content, key=comparator),
pgp_key=self.default_pgp_key,
repository=self.name)

View File

@ -60,21 +60,26 @@ class Report:
if provider == ReportSettings.HTML:
from ahriman.core.report.html import HTML
return HTML(architecture, configuration)
if provider == ReportSettings.Email:
from ahriman.core.report.email import Email
return Email(architecture, configuration)
return cls(architecture, configuration) # should never happen
def generate(self, packages: Iterable[Package]) -> None:
def generate(self, packages: Iterable[Package], built_packages: Iterable[Package]) -> None:
"""
generate report for the specified packages
:param packages: list of packages to generate report
:param built_packages: list of packages which has just been built
"""
def run(self, packages: Iterable[Package]) -> None:
def run(self, packages: Iterable[Package], built_packages: Iterable[Package]) -> None:
"""
run report generation
:param packages: list of packages to generate report
:param built_packages: list of packages which has just been built
"""
try:
self.generate(packages)
self.generate(packages, built_packages)
except Exception:
self.logger.exception("report generation failed")
raise ReportFailed()

View File

@ -100,27 +100,29 @@ class Executor(Cleaner):
return self.repo.repo_path
def process_report(self, targets: Optional[Iterable[str]]) -> None:
def process_report(self, targets: Optional[Iterable[str]], built_packages: Iterable[Package]) -> None:
"""
generate reports
:param targets: list of targets to generate reports. Configuration option will be used if it is not set
:param built_packages: list of packages which has just been built
"""
if targets is None:
targets = self.configuration.getlist("report", "target")
for target in targets:
runner = Report.load(self.architecture, self.configuration, target)
runner.run(self.packages())
runner.run(self.packages(), built_packages)
def process_sync(self, targets: Optional[Iterable[str]]) -> None:
def process_sync(self, targets: Optional[Iterable[str]], built_packages: Iterable[Package]) -> None:
"""
process synchronization to remote servers
:param targets: list of targets to sync. Configuration option will be used if it is not set
:param built_packages: list of packages which has just been built
"""
if targets is None:
targets = self.configuration.getlist("upload", "target")
for target in targets:
runner = Upload.load(self.architecture, self.configuration, target)
runner.run(self.paths.repository)
runner.run(self.paths.repository, built_packages)
def process_update(self, packages: Iterable[Path]) -> Path:
"""

View File

@ -37,9 +37,7 @@ class Repository(Executor, UpdateHandler):
:return: list of packages properties
"""
result: Dict[str, Package] = {}
for full_path in self.paths.repository.iterdir():
if not package_like(full_path):
continue
for full_path in filter(package_like, self.paths.repository.iterdir()):
try:
local = Package.load(full_path, self.pacman, self.aur_url)
result.setdefault(local.base, local).packages.update(local.packages)

View File

@ -18,13 +18,14 @@
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
import logging
import requests
from pathlib import Path
from typing import List, Optional, Set, Tuple
from ahriman.core.configuration import Configuration
from ahriman.core.exceptions import BuildFailed
from ahriman.core.util import check_output
from ahriman.core.util import check_output, exception_response_text
from ahriman.models.sign_settings import SignSettings
@ -87,6 +88,36 @@ class GPG:
default_key = configuration.get("sign", "key") if targets else None
return targets, default_key
def download_key(self, server: str, key: str) -> str:
"""
download key from public PGP server
:param server: public PGP server which will be used to download the key
:param key: key ID to download
:return: key as plain text
"""
key = key if key.startswith("0x") else f"0x{key}"
try:
response = requests.get(f"http://{server}/pks/lookup", params={
"op": "get",
"options": "mr",
"search": key
})
response.raise_for_status()
except requests.exceptions.HTTPError as e:
self.logger.exception(f"could not download key {key} from {server}: {exception_response_text(e)}")
raise
return response.text
def import_key(self, server: str, key: str) -> None:
"""
import key to current user and sign it locally
:param server: public PGP server which will be used to download the key
:param key: key ID to import
"""
key_body = self.download_key(server, key)
GPG._check_output("gpg", "--import", input_data=key_body, exception=None, logger=self.logger)
GPG._check_output("gpg", "--quick-lsign-key", key, exception=None, logger=self.logger)
def process(self, path: Path, key: str) -> List[Path]:
"""
gpg command wrapper

View File

@ -23,6 +23,7 @@ from typing import List, Optional, Tuple, Type
from ahriman.core.configuration import Configuration
from ahriman.models.build_status import BuildStatus, BuildStatusEnum
from ahriman.models.internal_status import InternalStatus
from ahriman.models.package import Package
@ -62,6 +63,14 @@ class Client:
del base
return []
# pylint: disable=no-self-use
def get_internal(self) -> InternalStatus:
"""
get internal service status
:return: current internal (web) service status
"""
return InternalStatus()
# pylint: disable=no-self-use
def get_self(self) -> BuildStatus:
"""

View File

@ -23,7 +23,9 @@ import requests
from typing import List, Optional, Tuple
from ahriman.core.status.client import Client
from ahriman.core.util import exception_response_text
from ahriman.models.build_status import BuildStatusEnum, BuildStatus
from ahriman.models.internal_status import InternalStatus
from ahriman.models.package import Package
@ -45,16 +47,6 @@ class WebClient(Client):
self.host = host
self.port = port
@staticmethod
def _exception_response_text(exception: requests.exceptions.HTTPError) -> str:
"""
safe response exception text generation
:param exception: exception raised
:return: text of the response if it is not None and empty string otherwise
"""
result: str = exception.response.text if exception.response is not None else ""
return result
def _ahriman_url(self) -> str:
"""
url generator
@ -70,6 +62,13 @@ class WebClient(Client):
"""
return f"http://{self.host}:{self.port}/api/v1/packages/{base}"
def _status_url(self) -> str:
"""
url generator
:return: full url for web service for status
"""
return f"http://{self.host}:{self.port}/api/v1/status"
def add(self, package: Package, status: BuildStatusEnum) -> None:
"""
add new package with status
@ -85,7 +84,7 @@ class WebClient(Client):
response = requests.post(self._package_url(package.base), json=payload)
response.raise_for_status()
except requests.exceptions.HTTPError as e:
self.logger.exception(f"could not add {package.base}: {WebClient._exception_response_text(e)}")
self.logger.exception(f"could not add {package.base}: {exception_response_text(e)}")
except Exception:
self.logger.exception(f"could not add {package.base}")
@ -105,11 +104,28 @@ class WebClient(Client):
for package in status_json
]
except requests.exceptions.HTTPError as e:
self.logger.exception(f"could not get {base}: {WebClient._exception_response_text(e)}")
self.logger.exception(f"could not get {base}: {exception_response_text(e)}")
except Exception:
self.logger.exception(f"could not get {base}")
return []
def get_internal(self) -> InternalStatus:
"""
get internal service status
:return: current internal (web) service status
"""
try:
response = requests.get(self._status_url())
response.raise_for_status()
status_json = response.json()
return InternalStatus.from_json(status_json)
except requests.exceptions.HTTPError as e:
self.logger.exception(f"could not get web service status: {exception_response_text(e)}")
except Exception:
self.logger.exception("could not get web service status")
return InternalStatus()
def get_self(self) -> BuildStatus:
"""
get ahriman status itself
@ -122,7 +138,7 @@ class WebClient(Client):
status_json = response.json()
return BuildStatus.from_json(status_json)
except requests.exceptions.HTTPError as e:
self.logger.exception(f"could not get service status: {WebClient._exception_response_text(e)}")
self.logger.exception(f"could not get service status: {exception_response_text(e)}")
except Exception:
self.logger.exception("could not get service status")
return BuildStatus()
@ -136,7 +152,7 @@ class WebClient(Client):
response = requests.delete(self._package_url(base))
response.raise_for_status()
except requests.exceptions.HTTPError as e:
self.logger.exception(f"could not delete {base}: {WebClient._exception_response_text(e)}")
self.logger.exception(f"could not delete {base}: {exception_response_text(e)}")
except Exception:
self.logger.exception(f"could not delete {base}")
@ -152,7 +168,7 @@ class WebClient(Client):
response = requests.post(self._package_url(base), json=payload)
response.raise_for_status()
except requests.exceptions.HTTPError as e:
self.logger.exception(f"could not update {base}: {WebClient._exception_response_text(e)}")
self.logger.exception(f"could not update {base}: {exception_response_text(e)}")
except Exception:
self.logger.exception(f"could not update {base}")
@ -167,6 +183,6 @@ class WebClient(Client):
response = requests.post(self._ahriman_url(), json=payload)
response.raise_for_status()
except requests.exceptions.HTTPError as e:
self.logger.exception(f"could not update service status: {WebClient._exception_response_text(e)}")
self.logger.exception(f"could not update service status: {exception_response_text(e)}")
except Exception:
self.logger.exception("could not update service status")

View File

@ -18,10 +18,12 @@
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
from pathlib import Path
from typing import Iterable
from ahriman.core.configuration import Configuration
from ahriman.core.upload.upload import Upload
from ahriman.core.util import check_output
from ahriman.models.package import Package
class Rsync(Upload):
@ -43,9 +45,10 @@ class Rsync(Upload):
self.command = configuration.getlist("rsync", "command")
self.remote = configuration.get("rsync", "remote")
def sync(self, path: Path) -> None:
def sync(self, path: Path, built_packages: Iterable[Package]) -> None:
"""
sync data to remote server
:param path: local path to sync
:param built_packages: list of packages which has just been built
"""
Rsync._check_output(*self.command, str(path), self.remote, exception=None, logger=self.logger)

View File

@ -18,10 +18,12 @@
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
from pathlib import Path
from typing import Iterable
from ahriman.core.configuration import Configuration
from ahriman.core.upload.upload import Upload
from ahriman.core.util import check_output
from ahriman.models.package import Package
class S3(Upload):
@ -43,10 +45,11 @@ class S3(Upload):
self.bucket = configuration.get("s3", "bucket")
self.command = configuration.getlist("s3", "command")
def sync(self, path: Path) -> None:
def sync(self, path: Path, built_packages: Iterable[Package]) -> None:
"""
sync data to remote server
:param path: local path to sync
:param built_packages: list of packages which has just been built
"""
# TODO rewrite to boto, but it is bullshit
S3._check_output(*self.command, str(path), self.bucket, exception=None, logger=self.logger)

View File

@ -22,10 +22,11 @@ from __future__ import annotations
import logging
from pathlib import Path
from typing import Type
from typing import Iterable, Type
from ahriman.core.configuration import Configuration
from ahriman.core.exceptions import SyncFailed
from ahriman.models.package import Package
from ahriman.models.upload_settings import UploadSettings
@ -65,19 +66,21 @@ class Upload:
return S3(architecture, configuration)
return cls(architecture, configuration) # should never happen
def run(self, path: Path) -> None:
def run(self, path: Path, built_packages: Iterable[Package]) -> None:
"""
run remote sync
:param path: local path to sync
:param built_packages: list of packages which has just been built
"""
try:
self.sync(path)
self.sync(path, built_packages)
except Exception:
self.logger.exception("remote sync failed")
raise SyncFailed()
def sync(self, path: Path) -> None:
def sync(self, path: Path, built_packages: Iterable[Package]) -> None:
"""
sync data to remote server
:param path: local path to sync
:param built_packages: list of packages which has just been built
"""

View File

@ -19,37 +19,51 @@
#
import datetime
import subprocess
import requests
from logging import Logger
from pathlib import Path
from typing import Optional
from typing import Optional, Union
from ahriman.core.exceptions import InvalidOption
def check_output(*args: str, exception: Optional[Exception],
cwd: Optional[Path] = None, logger: Optional[Logger] = None) -> str:
def check_output(*args: str, exception: Optional[Exception], cwd: Optional[Path] = None,
input_data: Optional[str] = None, logger: Optional[Logger] = None) -> str:
"""
subprocess wrapper
:param args: command line arguments
:param exception: exception which has to be reraised instead of default subprocess exception
:param cwd: current working directory
:param input_data: data which will be written to command stdin
:param logger: logger to log command result if required
:return: command output
"""
try:
result = subprocess.check_output(args, cwd=cwd, stderr=subprocess.STDOUT).decode("utf8").strip()
# universal_newlines is required to read input from string
result: str = subprocess.check_output(args, cwd=cwd, input=input_data, stderr=subprocess.STDOUT, # type: ignore
universal_newlines=True).strip()
if logger is not None:
for line in result.splitlines():
logger.debug(line)
except subprocess.CalledProcessError as e:
if e.output is not None and logger is not None:
for line in e.output.decode("utf8").splitlines():
for line in e.output.splitlines():
logger.debug(line)
raise exception or e
return result
def exception_response_text(exception: requests.exceptions.HTTPError) -> str:
"""
safe response exception text generation
:param exception: exception raised
:return: text of the response if it is not None and empty string otherwise
"""
result: str = exception.response.text if exception.response is not None else ""
return result
def package_like(filename: Path) -> bool:
"""
check if file looks like package
@ -60,7 +74,7 @@ def package_like(filename: Path) -> bool:
return ".pkg." in name and not name.endswith(".sig")
def pretty_datetime(timestamp: Optional[int]) -> str:
def pretty_datetime(timestamp: Optional[Union[float, int]]) -> str:
"""
convert datetime object to string
:param timestamp: datetime to convert

View File

@ -0,0 +1,71 @@
#
# Copyright (c) 2021 ahriman team.
#
# This file is part of ahriman
# (see https://github.com/arcan1s/ahriman).
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
from __future__ import annotations
from dataclasses import dataclass, fields
from typing import Any, Dict, List, Tuple, Type
from ahriman.models.build_status import BuildStatus
from ahriman.models.package import Package
@dataclass
class Counters:
"""
package counters
:ivar total: total packages count
:ivar unknown: packages in unknown status count
:ivar pending: packages in pending status count
:ivar building: packages in building status count
:ivar failed: packages in failed status count
:ivar success: packages in success status count
"""
total: int
unknown: int = 0
pending: int = 0
building: int = 0
failed: int = 0
success: int = 0
@classmethod
def from_json(cls: Type[Counters], dump: Dict[str, Any]) -> Counters:
"""
construct counters from json dump
:param dump: json dump body
:return: status counters
"""
# filter to only known fields
known_fields = [pair.name for pair in fields(cls)]
dump = {key: value for key, value in dump.items() if key in known_fields}
return cls(**dump)
@classmethod
def from_packages(cls: Type[Counters], packages: List[Tuple[Package, BuildStatus]]) -> Counters:
"""
construct counters from packages statuses
:param packages: list of package and their status as per watcher property
:return: status counters
"""
per_status = {"total": len(packages)}
for _, status in packages:
key = status.status.name.lower()
per_status.setdefault(key, 0)
per_status[key] += 1
return cls(**per_status)

View File

@ -0,0 +1,60 @@
#
# Copyright (c) 2021 ahriman team.
#
# This file is part of ahriman
# (see https://github.com/arcan1s/ahriman).
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
from __future__ import annotations
from dataclasses import asdict, dataclass, field
from typing import Any, Dict, Optional, Type
from ahriman.models.counters import Counters
@dataclass
class InternalStatus:
"""
internal server status
:ivar architecture: repository architecture
:ivar packages: packages statuses counter object
:ivar repository: repository name
:ivar version: service version
"""
architecture: Optional[str] = None
packages: Counters = field(default=Counters(total=0))
repository: Optional[str] = None
version: Optional[str] = None
@classmethod
def from_json(cls: Type[InternalStatus], dump: Dict[str, Any]) -> InternalStatus:
"""
construct internal status from json dump
:param dump: json dump body
:return: internal status
"""
counters = Counters.from_json(dump["packages"]) if "packages" in dump else Counters(total=0)
return cls(architecture=dump.get("architecture"),
packages=counters,
repository=dump.get("repository"),
version=dump.get("version"))
def view(self) -> Dict[str, Any]:
"""
generate json status view
:return: json-friendly dictionary
"""
return asdict(self)

View File

@ -28,11 +28,14 @@ from ahriman.core.exceptions import InvalidOption
class ReportSettings(Enum):
"""
report targets enumeration
:cvar Disabled: option which generates no report for testing purpose
:cvar HTML: html report generation
:cvar Email: email report generation
"""
Disabled = auto() # for testing purpose
HTML = auto()
Email = auto()
@classmethod
def from_option(cls: Type[ReportSettings], value: str) -> ReportSettings:
@ -43,4 +46,6 @@ class ReportSettings(Enum):
"""
if value.lower() in ("html",):
return cls.HTML
if value.lower() in ("email",):
return cls.Email
raise InvalidOption(value)

View File

@ -0,0 +1,49 @@
#
# Copyright (c) 2021 ahriman team.
#
# This file is part of ahriman
# (see https://github.com/arcan1s/ahriman).
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
from __future__ import annotations
from enum import Enum, auto
from typing import Type
class SmtpSSLSettings(Enum):
"""
SMTP SSL mode enumeration
:cvar Disabled: no SSL enabled
:cvar SSL: use SMTP_SSL instead of normal SMTP client
:cvar STARTTLS: use STARTTLS in normal SMTP client
"""
Disabled = auto()
SSL = auto()
STARTTLS = auto()
@classmethod
def from_option(cls: Type[SmtpSSLSettings], value: str) -> SmtpSSLSettings:
"""
construct value from configuration
:param value: configuration value
:return: parsed value
"""
if value.lower() in ("ssl", "ssl/tls"):
return cls.SSL
if value.lower() in ("starttls",):
return cls.STARTTLS
return cls.Disabled

View File

@ -28,6 +28,7 @@ from ahriman.core.exceptions import InvalidOption
class UploadSettings(Enum):
"""
remote synchronization targets enumeration
:cvar Disabled: no sync will be performed, required for testing purpose
:cvar Rsync: sync via rsync
:cvar S3: sync to Amazon S3
"""

View File

@ -17,4 +17,4 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
__version__ = "0.21.2"
__version__ = "1.0.0"

View File

@ -23,6 +23,7 @@ from ahriman.web.views.ahriman import AhrimanView
from ahriman.web.views.index import IndexView
from ahriman.web.views.package import PackageView
from ahriman.web.views.packages import PackagesView
from ahriman.web.views.status import StatusView
def setup_routes(application: Application) -> None:
@ -44,6 +45,8 @@ def setup_routes(application: Application) -> None:
GET /api/v1/package/:base get package base status
POST /api/v1/package/:base update package base status
GET /api/v1/status get web service status itself
:param application: web application instance
"""
application.router.add_get("/", IndexView)
@ -58,3 +61,5 @@ def setup_routes(application: Application) -> None:
application.router.add_delete("/api/v1/packages/{package}", PackageView)
application.router.add_get("/api/v1/packages/{package}", PackageView)
application.router.add_post("/api/v1/packages/{package}", PackageView)
application.router.add_get("/api/v1/status", StatusView)

View File

@ -21,8 +21,7 @@ import aiohttp_jinja2
from typing import Any, Dict
import ahriman.version as version
from ahriman import version
from ahriman.core.util import pretty_datetime
from ahriman.web.views.base import BaseView

View File

@ -0,0 +1,45 @@
#
# Copyright (c) 2021 ahriman team.
#
# This file is part of ahriman
# (see https://github.com/arcan1s/ahriman).
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
from aiohttp.web import Response, json_response
from ahriman import version
from ahriman.models.counters import Counters
from ahriman.models.internal_status import InternalStatus
from ahriman.web.views.base import BaseView
class StatusView(BaseView):
"""
web service status web view
"""
async def get(self) -> Response:
"""
get current service status
:return: 200 with service status object
"""
counters = Counters.from_packages(self.service.packages)
status = InternalStatus(
architecture=self.service.architecture,
packages=counters,
repository=self.service.repository.name,
version=version.__version__)
return json_response(status.view())

View File

@ -1,4 +1,5 @@
import argparse
import aur
import pytest
from pytest_mock import MockerFixture
@ -7,6 +8,7 @@ from ahriman.application.ahriman import _parser
from ahriman.application.application import Application
from ahriman.application.lock import Lock
from ahriman.core.configuration import Configuration
from ahriman.models.package import Package
@pytest.fixture
@ -20,6 +22,26 @@ def args() -> argparse.Namespace:
return argparse.Namespace(lock=None, force=False, unsafe=False, no_report=True)
@pytest.fixture
def aur_package_ahriman(package_ahriman: Package) -> aur.Package:
return aur.Package(
num_votes=None,
description=package_ahriman.packages[package_ahriman.base].description,
url_path=package_ahriman.web_url,
last_modified=None,
name=package_ahriman.base,
out_of_date=None,
id=None,
first_submitted=None,
maintainer=None,
version=package_ahriman.version,
license=package_ahriman.packages[package_ahriman.base].licenses,
url=None,
package_base=package_ahriman.base,
package_base_id=None,
category_id=None)
@pytest.fixture
def lock(args: argparse.Namespace, configuration: Configuration) -> Lock:
return Lock(args, "x86_64", configuration)

View File

@ -0,0 +1,24 @@
import argparse
from pytest_mock import MockerFixture
from ahriman.application.handlers import KeyImport
from ahriman.core.configuration import Configuration
def _default_args(args: argparse.Namespace) -> argparse.Namespace:
args.key = "0xE989490C"
args.key_server = "keys.gnupg.net"
return args
def test_run(args: argparse.Namespace, configuration: Configuration, mocker: MockerFixture) -> None:
"""
must run command
"""
args = _default_args(args)
mocker.patch("pathlib.Path.mkdir")
application_mock = mocker.patch("ahriman.core.sign.gpg.GPG.import_key")
KeyImport.run(args, "x86_64", configuration)
application_mock.assert_called_once()

View File

@ -0,0 +1,50 @@
import argparse
import aur
from pytest_mock import MockerFixture
from ahriman.application.handlers import Search
from ahriman.core.configuration import Configuration
def _default_args(args: argparse.Namespace) -> argparse.Namespace:
args.search = ["ahriman"]
return args
def test_run(args: argparse.Namespace, configuration: Configuration, aur_package_ahriman: aur.Package,
mocker: MockerFixture) -> None:
"""
must run command
"""
args = _default_args(args)
mocker.patch("aur.search", return_value=[aur_package_ahriman])
log_mock = mocker.patch("ahriman.application.handlers.search.Search.log_fn")
Search.run(args, "x86_64", configuration)
log_mock.assert_called_once()
def test_run_multiple_search(args: argparse.Namespace, configuration: Configuration, mocker: MockerFixture) -> None:
"""
must run command with multiple search arguments
"""
args = _default_args(args)
args.search = ["ahriman", "is", "cool"]
search_mock = mocker.patch("aur.search")
Search.run(args, "x86_64", configuration)
search_mock.assert_called_with(" ".join(args.search))
def test_log_fn(args: argparse.Namespace, configuration: Configuration, aur_package_ahriman: aur.Package,
mocker: MockerFixture) -> None:
"""
log function must call print built-in
"""
args = _default_args(args)
mocker.patch("aur.search", return_value=[aur_package_ahriman])
print_mock = mocker.patch("builtins.print")
Search.run(args, "x86_64", configuration)
print_mock.assert_called() # we don't really care about call details tbh

View File

@ -71,6 +71,25 @@ def test_subparsers_config(parser: argparse.ArgumentParser) -> None:
assert args.unsafe
def test_subparsers_key_import(parser: argparse.ArgumentParser) -> None:
"""
key-import command must imply lock and no_report
"""
args = parser.parse_args(["-a", "x86_64", "key-import", "key"])
assert args.lock is None
assert args.no_report
def test_subparsers_search(parser: argparse.ArgumentParser) -> None:
"""
search command must imply lock, no_report and unsafe
"""
args = parser.parse_args(["-a", "x86_64", "search", "ahriman"])
assert args.lock is None
assert args.no_report
assert args.unsafe
def test_subparsers_setup(parser: argparse.ArgumentParser) -> None:
"""
setup command must imply lock, no_report and unsafe

View File

@ -15,7 +15,7 @@ def test_finalize(application: Application, mocker: MockerFixture) -> None:
report_mock = mocker.patch("ahriman.application.application.Application.report")
sync_mock = mocker.patch("ahriman.application.application.Application.sync")
application._finalize()
application._finalize([])
report_mock.assert_called_once()
sync_mock.assert_called_once()
@ -218,7 +218,7 @@ def test_report(application: Application, mocker: MockerFixture) -> None:
must generate report
"""
executor_mock = mocker.patch("ahriman.core.repository.executor.Executor.process_report")
application.report([])
application.report([], [])
executor_mock.assert_called_once()
@ -279,7 +279,7 @@ def test_sync(application: Application, mocker: MockerFixture) -> None:
must sync to remote
"""
executor_mock = mocker.patch("ahriman.core.repository.executor.Executor.process_sync")
application.sync([])
application.sync([], [])
executor_mock.assert_called_once()
@ -292,6 +292,7 @@ def test_update(application: Application, package_ahriman: Package, mocker: Mock
mocker.patch("ahriman.core.tree.Tree.load", return_value=tree)
mocker.patch("ahriman.core.repository.repository.Repository.packages_built", return_value=[])
mocker.patch("ahriman.models.package.Package.load", return_value=package_ahriman)
build_mock = mocker.patch("ahriman.core.repository.executor.Executor.process_build", return_value=paths)
update_mock = mocker.patch("ahriman.core.repository.executor.Executor.process_update")
finalize_mock = mocker.patch("ahriman.application.application.Application._finalize")
@ -299,4 +300,4 @@ def test_update(application: Application, package_ahriman: Package, mocker: Mock
application.update([package_ahriman])
build_mock.assert_called_once()
update_mock.assert_has_calls([mock.call([]), mock.call(paths)])
finalize_mock.assert_has_calls([mock.call(), mock.call()])
finalize_mock.assert_has_calls([mock.call([]), mock.call([package_ahriman])])

View File

@ -5,9 +5,11 @@ from pathlib import Path
from pytest_mock import MockerFixture
from unittest import mock
from ahriman import version
from ahriman.application.lock import Lock
from ahriman.core.exceptions import DuplicateRun, UnsafeRun
from ahriman.models.build_status import BuildStatusEnum
from ahriman.models.internal_status import InternalStatus
def test_enter(lock: Lock, mocker: MockerFixture) -> None:
@ -15,6 +17,7 @@ def test_enter(lock: Lock, mocker: MockerFixture) -> None:
must process with context manager
"""
check_user_mock = mocker.patch("ahriman.application.lock.Lock.check_user")
check_version_mock = mocker.patch("ahriman.application.lock.Lock.check_version")
clear_mock = mocker.patch("ahriman.application.lock.Lock.clear")
create_mock = mocker.patch("ahriman.application.lock.Lock.create")
update_status_mock = mocker.patch("ahriman.core.status.client.Client.update_self")
@ -24,6 +27,7 @@ def test_enter(lock: Lock, mocker: MockerFixture) -> None:
check_user_mock.assert_called_once()
clear_mock.assert_called_once()
create_mock.assert_called_once()
check_version_mock.assert_called_once()
update_status_mock.assert_has_calls([
mock.call(BuildStatusEnum.Building),
mock.call(BuildStatusEnum.Success)
@ -48,6 +52,30 @@ def test_exit_with_exception(lock: Lock, mocker: MockerFixture) -> None:
])
def test_check_version(lock: Lock, mocker: MockerFixture) -> None:
"""
must check version correctly
"""
mocker.patch("ahriman.core.status.client.Client.get_internal",
return_value=InternalStatus(version=version.__version__))
logging_mock = mocker.patch("logging.Logger.warning")
lock.check_version()
logging_mock.assert_not_called()
def test_check_version_mismatch(lock: Lock, mocker: MockerFixture) -> None:
"""
must check version correctly
"""
mocker.patch("ahriman.core.status.client.Client.get_internal",
return_value=InternalStatus(version="version"))
logging_mock = mocker.patch("logging.Logger.warning")
lock.check_version()
logging_mock.assert_called_once()
def test_check_user(lock: Lock, mocker: MockerFixture) -> None:
"""
must check user correctly

View File

@ -0,0 +1,130 @@
from pytest_mock import MockerFixture
from ahriman.core.configuration import Configuration
from ahriman.core.report.email import Email
from ahriman.models.package import Package
def test_send(configuration: Configuration, mocker: MockerFixture) -> None:
"""
must send an email with attachment
"""
smtp_mock = mocker.patch("smtplib.SMTP")
report = Email("x86_64", configuration)
report._send("a text", {"attachment.html": "an attachment"})
smtp_mock.return_value.starttls.assert_not_called()
smtp_mock.return_value.login.assert_not_called()
smtp_mock.return_value.sendmail.assert_called_once()
smtp_mock.return_value.quit.assert_called_once()
def test_send_auth(configuration: Configuration, mocker: MockerFixture) -> None:
"""
must send an email with attachment with auth
"""
configuration.set("email", "user", "username")
configuration.set("email", "password", "password")
smtp_mock = mocker.patch("smtplib.SMTP")
report = Email("x86_64", configuration)
report._send("a text", {"attachment.html": "an attachment"})
smtp_mock.return_value.login.assert_called_once()
def test_send_auth_no_password(configuration: Configuration, mocker: MockerFixture) -> None:
"""
must send an email with attachment without auth if no password supplied
"""
configuration.set("email", "user", "username")
smtp_mock = mocker.patch("smtplib.SMTP")
report = Email("x86_64", configuration)
report._send("a text", {"attachment.html": "an attachment"})
smtp_mock.return_value.login.assert_not_called()
def test_send_auth_no_user(configuration: Configuration, mocker: MockerFixture) -> None:
"""
must send an email with attachment without auth if no user supplied
"""
configuration.set("email", "password", "password")
smtp_mock = mocker.patch("smtplib.SMTP")
report = Email("x86_64", configuration)
report._send("a text", {"attachment.html": "an attachment"})
smtp_mock.return_value.login.assert_not_called()
def test_send_ssl_tls(configuration: Configuration, mocker: MockerFixture) -> None:
"""
must send an email with attachment with ssl/tls
"""
configuration.set("email", "ssl", "ssl")
smtp_mock = mocker.patch("smtplib.SMTP_SSL")
report = Email("x86_64", configuration)
report._send("a text", {"attachment.html": "an attachment"})
smtp_mock.return_value.starttls.assert_not_called()
smtp_mock.return_value.login.assert_not_called()
smtp_mock.return_value.sendmail.assert_called_once()
smtp_mock.return_value.quit.assert_called_once()
def test_send_starttls(configuration: Configuration, mocker: MockerFixture) -> None:
"""
must send an email with attachment with starttls
"""
configuration.set("email", "ssl", "starttls")
smtp_mock = mocker.patch("smtplib.SMTP")
report = Email("x86_64", configuration)
report._send("a text", {"attachment.html": "an attachment"})
smtp_mock.return_value.starttls.assert_called_once()
def test_generate(configuration: Configuration, package_ahriman: Package, mocker: MockerFixture) -> None:
"""
must generate report
"""
send_mock = mocker.patch("ahriman.core.report.email.Email._send")
report = Email("x86_64", configuration)
report.generate([package_ahriman], [])
send_mock.assert_called_once()
def test_generate_with_built(configuration: Configuration, package_ahriman: Package, mocker: MockerFixture) -> None:
"""
must generate report with built packages
"""
send_mock = mocker.patch("ahriman.core.report.email.Email._send")
report = Email("x86_64", configuration)
report.generate([package_ahriman], [package_ahriman])
send_mock.assert_called_once()
def test_generate_no_empty(configuration: Configuration, package_ahriman: Package, mocker: MockerFixture) -> None:
"""
must not generate report with built packages if no_empty_report is set
"""
configuration.set("email", "no_empty_report", "yes")
send_mock = mocker.patch("ahriman.core.report.email.Email._send")
report = Email("x86_64", configuration)
report.generate([package_ahriman], [])
send_mock.assert_not_called()
def test_generate_no_empty_with_built(configuration: Configuration, package_ahriman: Package,
mocker: MockerFixture) -> None:
"""
must generate report with built packages if no_empty_report is set
"""
configuration.set("email", "no_empty_report", "yes")
send_mock = mocker.patch("ahriman.core.report.email.Email._send")
report = Email("x86_64", configuration)
report.generate([package_ahriman], [package_ahriman])
send_mock.assert_called_once()

View File

@ -12,5 +12,5 @@ def test_generate(configuration: Configuration, package_ahriman: Package, mocker
write_mock = mocker.patch("pathlib.Path.write_text")
report = HTML("x86_64", configuration)
report.generate([package_ahriman])
report.generate([package_ahriman], [])
write_mock.assert_called_once()

View File

@ -0,0 +1,19 @@
from ahriman.core.configuration import Configuration
from ahriman.core.report.jinja_template import JinjaTemplate
from ahriman.models.package import Package
def test_generate(configuration: Configuration, package_ahriman: Package) -> None:
"""
must generate html report
"""
report = JinjaTemplate("html", configuration)
assert report.make_html([package_ahriman], extended_report=False)
def test_generate_extended(configuration: Configuration, package_ahriman: Package) -> None:
"""
must generate extended html report
"""
report = JinjaTemplate("html", configuration)
assert report.make_html([package_ahriman], extended_report=True)

View File

@ -15,7 +15,7 @@ def test_report_failure(configuration: Configuration, mocker: MockerFixture) ->
"""
mocker.patch("ahriman.core.report.html.HTML.generate", side_effect=Exception())
with pytest.raises(ReportFailed):
Report.load("x86_64", configuration, ReportSettings.HTML.name).run(Path("path"))
Report.load("x86_64", configuration, ReportSettings.HTML.name).run(Path("path"), [])
def test_report_dummy(configuration: Configuration, mocker: MockerFixture) -> None:
@ -24,7 +24,16 @@ def test_report_dummy(configuration: Configuration, mocker: MockerFixture) -> No
"""
mocker.patch("ahriman.models.report_settings.ReportSettings.from_option", return_value=ReportSettings.Disabled)
report_mock = mocker.patch("ahriman.core.report.report.Report.generate")
Report.load("x86_64", configuration, ReportSettings.Disabled.name).run(Path("path"))
Report.load("x86_64", configuration, ReportSettings.Disabled.name).run(Path("path"), [])
report_mock.assert_called_once()
def test_report_email(configuration: Configuration, mocker: MockerFixture) -> None:
"""
must generate email report
"""
report_mock = mocker.patch("ahriman.core.report.email.Email.generate")
Report.load("x86_64", configuration, ReportSettings.Email.name).run(Path("path"), [])
report_mock.assert_called_once()
@ -33,5 +42,5 @@ def test_report_html(configuration: Configuration, mocker: MockerFixture) -> Non
must generate html report
"""
report_mock = mocker.patch("ahriman.core.report.html.HTML.generate")
Report.load("x86_64", configuration, ReportSettings.HTML.name).run(Path("path"))
Report.load("x86_64", configuration, ReportSettings.HTML.name).run(Path("path"), [])
report_mock.assert_called_once()

View File

@ -133,7 +133,7 @@ def test_process_report(executor: Executor, package_ahriman: Package, mocker: Mo
mocker.patch("ahriman.core.report.report.Report.load", return_value=Report("x86_64", executor.configuration))
report_mock = mocker.patch("ahriman.core.report.report.Report.run")
executor.process_report(["dummy"])
executor.process_report(["dummy"], [])
report_mock.assert_called_once()
@ -143,7 +143,7 @@ def test_process_report_auto(executor: Executor, mocker: MockerFixture) -> None:
"""
configuration_getlist_mock = mocker.patch("ahriman.core.configuration.Configuration.getlist")
executor.process_report(None)
executor.process_report(None, [])
configuration_getlist_mock.assert_called_once()
@ -154,7 +154,7 @@ def test_process_upload(executor: Executor, mocker: MockerFixture) -> None:
mocker.patch("ahriman.core.upload.upload.Upload.load", return_value=Upload("x86_64", executor.configuration))
upload_mock = mocker.patch("ahriman.core.upload.upload.Upload.run")
executor.process_sync(["dummy"])
executor.process_sync(["dummy"], [])
upload_mock.assert_called_once()
@ -164,7 +164,7 @@ def test_process_upload_auto(executor: Executor, mocker: MockerFixture) -> None:
"""
configuration_getlist_mock = mocker.patch("ahriman.core.configuration.Configuration.getlist")
executor.process_sync(None)
executor.process_sync(None, [])
configuration_getlist_mock.assert_called_once()

View File

@ -1,5 +1,9 @@
import pytest
import requests
from pathlib import Path
from pytest_mock import MockerFixture
from unittest import mock
from ahriman.core.sign.gpg import GPG
from ahriman.models.sign_settings import SignSettings
@ -60,6 +64,38 @@ def test_sign_command(gpg_with_key: GPG) -> None:
assert gpg_with_key.sign_command(Path("a"), gpg_with_key.default_key)
def test_download_key(gpg: GPG, mocker: MockerFixture) -> None:
"""
must download the key from public server
"""
requests_mock = mocker.patch("requests.get")
gpg.download_key("keys.gnupg.net", "0xE989490C")
requests_mock.assert_called_once()
def test_download_key_failure(gpg: GPG, mocker: MockerFixture) -> None:
"""
must download the key from public server and log error if any (and raise it again)
"""
mocker.patch("requests.get", side_effect=requests.exceptions.HTTPError())
with pytest.raises(requests.exceptions.HTTPError):
gpg.download_key("keys.gnupg.net", "0xE989490C")
def test_import_key(gpg: GPG, mocker: MockerFixture) -> None:
"""
must import PGP key from the server
"""
mocker.patch("ahriman.core.sign.gpg.GPG.download_key", return_value="key")
check_output_mock = mocker.patch("ahriman.core.sign.gpg.GPG._check_output")
gpg.import_key("keys.gnupg.net", "0xE989490C")
check_output_mock.assert_has_calls([
mock.call("gpg", "--import", input_data="key", exception=None, logger=pytest.helpers.anyvar(int)),
mock.call("gpg", "--quick-lsign-key", "0xE989490C", exception=None, logger=pytest.helpers.anyvar(int))
])
def test_process(gpg_with_key: GPG, mocker: MockerFixture) -> None:
"""
must call process method correctly

View File

@ -4,6 +4,7 @@ from ahriman.core.configuration import Configuration
from ahriman.core.status.client import Client
from ahriman.core.status.web_client import WebClient
from ahriman.models.build_status import BuildStatusEnum
from ahriman.models.internal_status import InternalStatus
from ahriman.models.package import Package
@ -38,6 +39,13 @@ def test_get(client: Client, package_ahriman: Package) -> None:
assert client.get(None) == []
def test_get_internal(client: Client) -> None:
"""
must return dummy status for web service
"""
assert client.get_internal() == InternalStatus()
def test_get_self(client: Client) -> None:
"""
must return unknown status for service

View File

@ -7,6 +7,7 @@ from requests import Response
from ahriman.core.status.web_client import WebClient
from ahriman.models.build_status import BuildStatus, BuildStatusEnum
from ahriman.models.internal_status import InternalStatus
from ahriman.models.package import Package
@ -26,6 +27,14 @@ def test_package_url(web_client: WebClient, package_ahriman: Package) -> None:
assert web_client._package_url(package_ahriman.base).endswith(f"/api/v1/packages/{package_ahriman.base}")
def test_status_url(web_client: WebClient) -> None:
"""
must generate service status url correctly
"""
assert web_client._status_url().startswith(f"http://{web_client.host}:{web_client.port}")
assert web_client._status_url().endswith("/api/v1/status")
def test_add(web_client: WebClient, package_ahriman: Package, mocker: MockerFixture) -> None:
"""
must process package addition
@ -103,6 +112,37 @@ def test_get_single(web_client: WebClient, package_ahriman: Package, mocker: Moc
assert (package_ahriman, BuildStatusEnum.Unknown) in [(package, status.status) for package, status in result]
def test_get_internal(web_client: WebClient, mocker: MockerFixture) -> None:
"""
must return web service status
"""
response_obj = Response()
response_obj._content = json.dumps(InternalStatus(architecture="x86_64").view()).encode("utf8")
response_obj.status_code = 200
requests_mock = mocker.patch("requests.get", return_value=response_obj)
result = web_client.get_internal()
requests_mock.assert_called_once()
assert result.architecture == "x86_64"
def test_get_internal_failed(web_client: WebClient, mocker: MockerFixture) -> None:
"""
must suppress any exception happened during web service status getting
"""
mocker.patch("requests.get", side_effect=Exception())
assert web_client.get_internal() == InternalStatus()
def test_get_internal_failed_http_error(web_client: WebClient, mocker: MockerFixture) -> None:
"""
must suppress any exception happened during web service status getting
"""
mocker.patch("requests.get", side_effect=requests.exceptions.HTTPError())
assert web_client.get_internal() == InternalStatus()
def test_get_self(web_client: WebClient, mocker: MockerFixture) -> None:
"""
must return service status

View File

@ -113,6 +113,14 @@ def test_load_includes_no_option(configuration: Configuration) -> None:
configuration.load_includes()
def test_load_includes_no_section(configuration: Configuration) -> None:
"""
must not fail if no option set
"""
configuration.remove_section("settings")
configuration.load_includes()
def test_load_logging_fallback(configuration: Configuration, mocker: MockerFixture) -> None:
"""
must fallback to stderr without errors

View File

@ -12,5 +12,5 @@ def test_sync(configuration: Configuration, mocker: MockerFixture) -> None:
check_output_mock = mocker.patch("ahriman.core.upload.rsync.Rsync._check_output")
upload = Rsync("x86_64", configuration)
upload.sync(Path("path"))
upload.sync(Path("path"), [])
check_output_mock.assert_called_once()

View File

@ -12,5 +12,5 @@ def test_sync(configuration: Configuration, mocker: MockerFixture) -> None:
check_output_mock = mocker.patch("ahriman.core.upload.s3.S3._check_output")
upload = S3("x86_64", configuration)
upload.sync(Path("path"))
upload.sync(Path("path"), [])
check_output_mock.assert_called_once()

View File

@ -15,7 +15,7 @@ def test_upload_failure(configuration: Configuration, mocker: MockerFixture) ->
"""
mocker.patch("ahriman.core.upload.rsync.Rsync.sync", side_effect=Exception())
with pytest.raises(SyncFailed):
Upload.load("x86_64", configuration, UploadSettings.Rsync.name).run(Path("path"))
Upload.load("x86_64", configuration, UploadSettings.Rsync.name).run(Path("path"), [])
def test_report_dummy(configuration: Configuration, mocker: MockerFixture) -> None:
@ -24,7 +24,7 @@ def test_report_dummy(configuration: Configuration, mocker: MockerFixture) -> No
"""
mocker.patch("ahriman.models.upload_settings.UploadSettings.from_option", return_value=UploadSettings.Disabled)
upload_mock = mocker.patch("ahriman.core.upload.upload.Upload.sync")
Upload.load("x86_64", configuration, UploadSettings.Disabled.name).run(Path("path"))
Upload.load("x86_64", configuration, UploadSettings.Disabled.name).run(Path("path"), [])
upload_mock.assert_called_once()
@ -33,7 +33,7 @@ def test_upload_rsync(configuration: Configuration, mocker: MockerFixture) -> No
must upload via rsync
"""
upload_mock = mocker.patch("ahriman.core.upload.rsync.Rsync.sync")
Upload.load("x86_64", configuration, UploadSettings.Rsync.name).run(Path("path"))
Upload.load("x86_64", configuration, UploadSettings.Rsync.name).run(Path("path"), [])
upload_mock.assert_called_once()
@ -42,5 +42,5 @@ def test_upload_s3(configuration: Configuration, mocker: MockerFixture) -> None:
must upload via s3
"""
upload_mock = mocker.patch("ahriman.core.upload.s3.S3.sync")
Upload.load("x86_64", configuration, UploadSettings.S3.name).run(Path("path"))
Upload.load("x86_64", configuration, UploadSettings.S3.name).run(Path("path"), [])
upload_mock.assert_called_once()

View File

@ -2,7 +2,10 @@ import pytest
from unittest.mock import MagicMock, PropertyMock
from ahriman import version
from ahriman.models.build_status import BuildStatus, BuildStatusEnum
from ahriman.models.counters import Counters
from ahriman.models.internal_status import InternalStatus
from ahriman.models.package import Package
from ahriman.models.package_description import PackageDescription
@ -12,6 +15,24 @@ def build_status_failed() -> BuildStatus:
return BuildStatus(BuildStatusEnum.Failed, 42)
@pytest.fixture
def counters() -> Counters:
return Counters(total=10,
unknown=1,
pending=2,
building=3,
failed=4,
success=0)
@pytest.fixture
def internal_status(counters: Counters) -> InternalStatus:
return InternalStatus(architecture="x86_64",
packages=counters,
version=version.__version__,
repository="aur-clone")
@pytest.fixture
def package_tpacpi_bat_git() -> Package:
return Package(

View File

@ -0,0 +1,31 @@
from dataclasses import asdict
from ahriman.models.build_status import BuildStatus, BuildStatusEnum
from ahriman.models.counters import Counters
from ahriman.models.package import Package
def test_counters_from_json_view(counters: Counters) -> None:
"""
must construct same object from json
"""
assert Counters.from_json(asdict(counters)) == counters
def test_counters_from_packages(package_ahriman: Package, package_python_schedule: Package) -> None:
"""
must construct object from list of packages with their statuses
"""
payload = [
(package_ahriman, BuildStatus(status=BuildStatusEnum.Success)),
(package_python_schedule, BuildStatus(status=BuildStatusEnum.Failed)),
]
counters = Counters.from_packages(payload)
assert counters.total == 2
assert counters.success == 1
assert counters.failed == 1
json = asdict(counters)
total = json.pop("total")
assert total == sum(i for i in json.values())

View File

@ -0,0 +1,8 @@
from ahriman.models.internal_status import InternalStatus
def test_internal_status_from_json_view(internal_status: InternalStatus) -> None:
"""
must construct same object from json
"""
assert InternalStatus.from_json(internal_status.view()) == internal_status

View File

@ -18,3 +18,6 @@ def test_from_option_valid() -> None:
"""
assert ReportSettings.from_option("html") == ReportSettings.HTML
assert ReportSettings.from_option("HTML") == ReportSettings.HTML
assert ReportSettings.from_option("email") == ReportSettings.Email
assert ReportSettings.from_option("EmAil") == ReportSettings.Email

View File

@ -0,0 +1,21 @@
from ahriman.models.smtp_ssl_settings import SmtpSSLSettings
def test_from_option_invalid() -> None:
"""
must return disabled value on invalid option
"""
assert SmtpSSLSettings.from_option("invalid") == SmtpSSLSettings.Disabled
def test_from_option_valid() -> None:
"""
must return value from valid options
"""
assert SmtpSSLSettings.from_option("ssl") == SmtpSSLSettings.SSL
assert SmtpSSLSettings.from_option("SSL") == SmtpSSLSettings.SSL
assert SmtpSSLSettings.from_option("ssl/tls") == SmtpSSLSettings.SSL
assert SmtpSSLSettings.from_option("SSL/TLS") == SmtpSSLSettings.SSL
assert SmtpSSLSettings.from_option("starttls") == SmtpSSLSettings.STARTTLS
assert SmtpSSLSettings.from_option("STARTTLS") == SmtpSSLSettings.STARTTLS

View File

@ -0,0 +1,22 @@
from pytest_aiohttp import TestClient
import ahriman.version as version
from ahriman.models.build_status import BuildStatusEnum
from ahriman.models.package import Package
async def test_get(client: TestClient, package_ahriman: Package) -> None:
"""
must generate web service status correctly)
"""
await client.post(f"/api/v1/packages/{package_ahriman.base}",
json={"status": BuildStatusEnum.Success.value, "package": package_ahriman.view()})
response = await client.get("/api/v1/status")
assert response.status == 200
json = await response.json()
assert json["version"] == version.__version__
assert json["packages"]
assert json["packages"]["total"] == 1

View File

@ -25,6 +25,15 @@ target =
[report]
target =
[email]
host = 0.0.0.0
link_path =
no_empty_report = no
port = 587
receivers = mail@example.com
sender = mail@example.com
template_path = ../web/templates/repo-index.jinja2
[html]
path =
homepage =