Release 1.6.4

* handle packages which have been removed from the repository

* manually remove packages which have been removed from the base

docs/configuration.md

@@ -2,6 +2,13 @@
 
 Some groups can be specified for each architecture separately. E.g. if there are `build` and `build:x86_64` groups it will use the option from `build:x86_64` for the `x86_64` architecture and `build` for any other (architecture specific group has higher priority). In case if both groups are presented, architecture specific options will be merged into global ones overriding them.
 
+Some values have list of strings type. Those values will be read in the same way as shell does:
+
+* By default, it splits value by spaces excluding empty elements. 
+* In case if quotation mark (`"` or `'`) will be found, any spaces inside will be ignored.
+* In order to use quotation mark inside value it is required to put it to another quotation mark, e.g. `wor"'"d "with quote"` will be parsed as `["wor'd", "with quote"]` and vice versa.
+* Unclosed quotation mark is not allowed and will rise an exception.
+
 ## `settings` group
 
 Base configuration settings.
@@ -38,11 +45,11 @@ Authorization mapping. Group name must refer to user access level, i.e. it shoul
 Key is always username (case-insensitive), option value depends on authorization provider:
 
 * `OAuth` - by default requires only usernames and ignores values. But in case of direct login method call (via POST request) it will act as `Mapping` authorization method.
-* `Mapping` (default) - reads salted password hashes from values, uses SHA512 in order to hash passwords. Password can be set by using `create-user` subcommand.
+* `Mapping` (default) - reads salted password hashes from values, uses SHA512 in order to hash passwords. Password can be set by using `user-add` subcommand.
 
 ## `build:*` groups
 
-Build related configuration. Group name must refer to architecture, e.g. it should be `build:x86_64` for x86_64 architecture.
+Build related configuration. Group name can refer to architecture, e.g. `build:x86_64` can be used for x86_64 architecture specific settings.
 
 * `archbuild_flags` - additional flags passed to `archbuild` command, space separated list of strings, optional.
 * `build_command` - default build command, string, required.
@@ -59,7 +66,7 @@ Base repository settings.
 
 ## `sign:*` groups
 
-Settings for signing packages or repository. Group name must refer to architecture, e.g. it should be `sign:x86_64` for x86_64 architecture.
+Settings for signing packages or repository. Group name can refer to architecture, e.g. `sign:x86_64` can be used for x86_64 architecture specific settings.
 
 * `target` - configuration flag to enable signing, space separated list of strings, required. Allowed values are `package` (sign each package separately), `repository` (sign repository database file).
 * `key` - default PGP key, string, required. This key will also be used for database signing if enabled.
@@ -69,7 +76,7 @@ Settings for signing packages or repository. Group name must refer to architectu
 
 Report generation settings.
 
-* `target` - list of reports to be generated, space separated list of strings, required. It must point to valid section (or to section with architecture), e.g. `somerandomname` must point to existing section, `email` must point to one of `email` of `email:x86_64` (with architecture it has higher priority). 
+* `target` - list of reports to be generated, space separated list of strings, required. It must point to valid section (or to section with architecture), e.g. `somerandomname` must point to existing section, `email` must point to one of `email` of `email:x86_64` (the one with architecture has higher priority). 
 
 Type will be read from several ways:
 
@@ -152,7 +159,7 @@ Requires `boto3` library to be installed. Section name must be either `s3` (plus
 
 ## `web:*` groups
 
-Web server settings. If any of `host`/`port` is not set, web integration will be disabled. Group name must refer to architecture, e.g. it should be `web:x86_64` for x86_64 architecture. This feature requires `aiohttp` libraries to be installed.
+Web server settings. If any of `host`/`port` is not set, web integration will be disabled. Group name can refer to architecture, e.g. `web:x86_64` can be used for x86_64 architecture specific settings. This feature requires `aiohttp` libraries to be installed.
 
 * `address` - optional address in form `proto://host:port` (`port` can be omitted in case of default `proto` ports), will be used instead of `http://{host}:{port}` in case if set, string, optional. This option is required in case if `OAuth` provider is used.
 * `debug` - enable debug toolbar, boolean, optional, default `no`.

docs/faq.md

@@ -163,6 +163,41 @@ Server = file:///var/lib/ahriman/repository/x86_64
 
 (You might need to add `SigLevel` option according to the pacman documentation.)
 
+
+### I would like to serve the repository
+
+Easy. For example, nginx configuration (without SSL) will look like:
+
+```
+server {
+    listen 80;
+    server_name repo.example.com;
+
+    location / {
+        autoindex on;
+        root /var/lib/ahriman/repository;
+    }
+}
+```
+
+Example of the status page configuration is the following (status service is using 8080 port):
+
+```
+server {
+    listen 80;
+    server_name builds.example.com;
+
+    location / {
+        proxy_set_header Host $host;
+        proxy_set_header X-Real-IP $remote_addr;
+        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarder-Proto $scheme;
+
+        proxy_pass http://127.0.0.1:8080;
+    }
+}
+```
+
 ## Remote synchronization
 
 ### Wait I would like to use the repository from another server

package/archlinux/PKGBUILD

@@ -1,7 +1,7 @@
 # Maintainer: Evgeniy Alekseev
 
 pkgname='ahriman'
-pkgver=1.6.1
+pkgver=1.6.4
 pkgrel=1
 pkgdesc="ArcH Linux ReposItory MANager"
 arch=('any')

src/ahriman/application/application/repository.py

@@ -105,27 +105,37 @@ class Repository(Properties):
         targets = target or None
         self.repository.process_sync(targets, built_packages)
 
-    def unknown(self) -> List[Package]:
+    def unknown(self) -> List[str]:
         """
         get packages which were not found in AUR
-        :return: unknown package list
+        :return: unknown package archive list
         """
-        def has_aur(package_base: str, aur_url: str) -> bool:
+        def has_local(probe: Package) -> bool:
-            try:
+            cache_dir = self.repository.paths.cache_for(probe.base)
-                _ = Package.from_aur(package_base, aur_url)
-            except Exception:
-                return False
-            return True
-
-        def has_local(package_base: str) -> bool:
-            cache_dir = self.repository.paths.cache_for(package_base)
             return cache_dir.is_dir() and not Sources.has_remotes(cache_dir)
 
-        return [
+        def unknown_aur(probe: Package) -> List[str]:
-            package
+            packages: List[str] = []
-            for package in self.repository.packages()
+            for single in probe.packages:
-            if not has_aur(package.base, package.aur_url) and not has_local(package.base)
+                try:
-        ]
+                    _ = Package.from_aur(single, probe.aur_url)
+                except Exception:
+                    packages.append(single)
+            return packages
+
+        def unknown_local(probe: Package) -> List[str]:
+            cache_dir = self.repository.paths.cache_for(probe.base)
+            local = Package.from_build(cache_dir, probe.aur_url)
+            packages = set(probe.packages.keys()).difference(local.packages.keys())
+            return list(packages)
+
+        result = []
+        for package in self.repository.packages():
+            if has_local(package):
+                result.extend(unknown_local(package))  # there is local package
+            else:
+                result.extend(unknown_aur(package))  # local package not found
+        return result
 
     def update(self, updates: Iterable[Package]) -> None:
         """

src/ahriman/application/formatters/status_printer.py

@@ -17,11 +17,10 @@
 # You should have received a copy of the GNU General Public License
 # along with this program. If not, see <http://www.gnu.org/licenses/>.
 #
-from typing import List, Optional
+from typing import Optional
 
 from ahriman.application.formatters.printer import Printer
 from ahriman.models.build_status import BuildStatus
-from ahriman.models.property import Property
 
 
 class StatusPrinter(Printer):
@@ -36,13 +35,6 @@ class StatusPrinter(Printer):
         """
         self.content = status
 
-    def properties(self) -> List[Property]:
-        """
-        convert content into printable data
-        :return: list of content properties
-        """
-        return []
-
     def title(self) -> Optional[str]:
         """
         generate entry title from content

src/ahriman/application/formatters/string_printer.py

@@ -0,0 +1,42 @@
+#
+# Copyright (c) 2021 ahriman team.
+#
+# This file is part of ahriman
+# (see https://github.com/arcan1s/ahriman).
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+from typing import Optional
+
+from ahriman.application.formatters.printer import Printer
+
+
+class StringPrinter(Printer):
+    """
+    print content of the random string
+    """
+
+    def __init__(self, content: str) -> None:
+        """
+        default constructor
+        :param content: any content string
+        """
+        self.content = content
+
+    def title(self) -> Optional[str]:
+        """
+        generate entry title from content
+        :return: content title if it can be generated and None otherwise
+        """
+        return self.content

src/ahriman/application/handlers/remove_unknown.py

@@ -22,10 +22,9 @@ import argparse
 from typing import Type
 
 from ahriman.application.application import Application
-from ahriman.application.formatters.package_printer import PackagePrinter
+from ahriman.application.formatters.string_printer import StringPrinter
 from ahriman.application.handlers.handler import Handler
 from ahriman.core.configuration import Configuration
-from ahriman.models.build_status import BuildStatus
 
 
 class RemoveUnknown(Handler):
@@ -46,8 +45,8 @@ class RemoveUnknown(Handler):
         application = Application(architecture, configuration, no_report)
         unknown_packages = application.unknown()
         if args.dry_run:
-            for package in unknown_packages:
+            for package in sorted(unknown_packages):
-                PackagePrinter(package, BuildStatus()).print(args.info)
+                StringPrinter(package).print(args.info)
             return
 
-        application.remove(package.base for package in unknown_packages)
+        application.remove(unknown_packages)

src/ahriman/core/configuration.py

@@ -24,7 +24,7 @@ import logging
 
 from logging.config import fileConfig
 from pathlib import Path
-from typing import Any, Dict, List, Optional, Tuple, Type
+from typing import Any, Dict, Generator, List, Optional, Tuple, Type
 
 from ahriman.core.exceptions import InitializeException
 
@@ -49,7 +49,7 @@ class Configuration(configparser.RawConfigParser):
         default constructor. In the most cases must not be called directly
         """
         configparser.RawConfigParser.__init__(self, allow_no_value=True, converters={
-            "list": lambda value: value.split(),
+            "list": self.__convert_list,
             "path": self.__convert_path,
         })
         self.architecture: Optional[str] = None
@@ -84,6 +84,32 @@ class Configuration(configparser.RawConfigParser):
         config.load_logging(quiet)
         return config
 
+    @staticmethod
+    def __convert_list(value: str) -> List[str]:
+        """
+        convert string value to list of strings
+        :param value: string configuration value
+        :return: list of string from the parsed string
+        """
+        def generator() -> Generator[str, None, None]:
+            quote_mark = None
+            word = ""
+            for char in value:
+                if char in ("'", "\"") and quote_mark is None:  # quoted part started, store quote and do nothing
+                    quote_mark = char
+                elif char == quote_mark:  # quoted part ended, reset quotation
+                    quote_mark = None
+                elif char == " " and quote_mark is None:  # found space outside of the quotation, yield the word
+                    yield word
+                    word = ""
+                else:  # append character to the buffer
+                    word += char
+            if quote_mark:  # there is unmatched quote
+                raise ValueError(f"unmatched quote in {value}")
+            yield word  # sequence done, return whatever we found
+
+        return [word for word in generator() if word]
+
     @staticmethod
     def section_name(section: str, suffix: str) -> str:
         """
@@ -204,6 +230,8 @@ class Configuration(configparser.RawConfigParser):
         """
         if self.path is None or self.architecture is None:
             raise InitializeException("Configuration path and/or architecture are not set")
+        for section in self.sections():  # clear current content
+            self.remove_section(section)
         self.load(self.path)
         self.merge_sections(self.architecture)
 

src/ahriman/core/exceptions.py

@@ -153,8 +153,12 @@ class UnknownPackage(ValueError):
     exception for status watcher which will be thrown on unknown package
     """
 
-    def __init__(self, base: str) -> None:
+    def __init__(self, package_base: str) -> None:
-        ValueError.__init__(self, f"Package base {base} is unknown")
+        """
+        default constructor
+        :param package_base: package base name
+        """
+        ValueError.__init__(self, f"Package base {package_base} is unknown")
 
 
 class UnsafeRun(RuntimeError):
@@ -165,9 +169,9 @@ class UnsafeRun(RuntimeError):
     def __init__(self, current_uid: int, root_uid: int) -> None:
         """
         default constructor
+        :param current_uid: current user ID
+        :param root_uid: ID of the owner of root directory
         """
-        RuntimeError.__init__(
+        RuntimeError.__init__(self, f"Current UID {current_uid} differs from root owner {root_uid}. "
-            self,
+                                    f"Note that for the most actions it is unsafe to run application as different user."
-            f"""Current UID {current_uid} differs from root owner {root_uid}.
+                                    f" If you are 100% sure that it must be there try --unsafe option")
-Note that for the most actions it is unsafe to run application as different user.
-If you are 100% sure that it must be there try --unsafe option""")

src/ahriman/core/repository/executor.py

@@ -20,14 +20,13 @@
 import shutil
 
 from pathlib import Path
-from typing import Dict, Iterable, List, Optional
+from typing import Iterable, List, Optional, Set
 
 from ahriman.core.build_tools.task import Task
 from ahriman.core.report.report import Report
 from ahriman.core.repository.cleaner import Cleaner
 from ahriman.core.upload.upload import Upload
 from ahriman.models.package import Package
-from ahriman.models.package_source import PackageSource
 
 
 class Executor(Cleaner):
@@ -35,6 +34,14 @@ class Executor(Cleaner):
     trait for common repository update processes
     """
 
+    def load_archives(self, packages: Iterable[Path]) -> List[Package]:
+        """
+        load packages from list of archives
+        :param packages: paths to package archives
+        :return: list of read packages
+        """
+        raise NotImplementedError
+
     def packages(self) -> List[Package]:
         """
         generate list of repository packages
@@ -152,23 +159,24 @@ class Executor(Cleaner):
             package_path = self.paths.repository / name
             self.repo.add(package_path)
 
-        # we are iterating over bases, not single packages
+        current_packages = self.packages()
-        updates: Dict[str, Package] = {}
+        removed_packages: List[str] = []  # list of packages which have been removed from the base
-        for filename in packages:
+        updates = self.load_archives(packages)
-            try:
-                local = Package.load(str(filename), PackageSource.Archive, self.pacman, self.aur_url)
-                updates.setdefault(local.base, local).packages.update(local.packages)
-            except Exception:
-                self.logger.exception("could not load package from %s", filename)
 
-        for local in updates.values():
+        for local in updates:
             try:
                 for description in local.packages.values():
                     update_single(description.filename, local.base)
                 self.reporter.set_success(local)
+
+                current_package_archives: Set[str] = next(
+                    (set(current.packages) for current in current_packages if current.base == local.base), set())
+                removed_packages.extend(current_package_archives.difference(local.packages))
             except Exception:
                 self.reporter.set_failed(local.base)
                 self.logger.exception("could not process %s", local.base)
         self.clear_packages()
 
+        self.process_remove(removed_packages)
+
         return self.repo.repo_path

src/ahriman/core/repository/repository.py

@@ -18,7 +18,7 @@
 # along with this program. If not, see <http://www.gnu.org/licenses/>.
 #
 from pathlib import Path
-from typing import Dict, List
+from typing import Dict, Iterable, List
 
 from ahriman.core.repository.executor import Executor
 from ahriman.core.repository.update_handler import UpdateHandler
@@ -32,20 +32,35 @@ class Repository(Executor, UpdateHandler):
     base repository control class
     """
 
+    def load_archives(self, packages: Iterable[Path]) -> List[Package]:
+        """
+        load packages from list of archives
+        :param packages: paths to package archives
+        :return: list of read packages
+        """
+        result: Dict[str, Package] = {}
+        # we are iterating over bases, not single packages
+        for full_path in packages:
+            try:
+                local = Package.load(str(full_path), PackageSource.Archive, self.pacman, self.aur_url)
+                current = result.setdefault(local.base, local)
+                if current.version != local.version:
+                    # force version to max of them
+                    self.logger.warning("version of %s differs, found %s and %s",
+                                        current.base, current.version, local.version)
+                    if current.is_outdated(local, self.paths, calculate_version=False):
+                        current.version = local.version
+                current.packages.update(local.packages)
+            except Exception:
+                self.logger.exception("could not load package from %s", full_path)
+        return list(result.values())
+
     def packages(self) -> List[Package]:
         """
         generate list of repository packages
         :return: list of packages properties
         """
-        result: Dict[str, Package] = {}
+        return self.load_archives(filter(package_like, self.paths.repository.iterdir()))
-        for full_path in filter(package_like, self.paths.repository.iterdir()):
-            try:
-                local = Package.load(str(full_path), PackageSource.Archive, self.pacman, self.aur_url)
-                result.setdefault(local.base, local).packages.update(local.packages)
-            except Exception:
-                self.logger.exception("could not load package from %s", full_path)
-                continue
-        return list(result.values())
 
     def packages_built(self) -> List[Path]:
         """

src/ahriman/models/build_status.py

@@ -21,7 +21,7 @@ from __future__ import annotations
 
 import datetime
 
-from dataclasses import dataclass, fields
+from dataclasses import dataclass, field, fields
 from enum import Enum
 from typing import Any, Dict, Type
 
@@ -84,7 +84,7 @@ class BuildStatus:
     """
 
     status: BuildStatusEnum = BuildStatusEnum.Unknown
-    timestamp: int = int(datetime.datetime.utcnow().timestamp())
+    timestamp: int = field(default_factory=lambda: int(datetime.datetime.utcnow().timestamp()))
 
     def __post_init__(self) -> None:
         """

src/ahriman/models/package.py

@@ -257,14 +257,15 @@ class Package:
 
         return self.version
 
-    def is_outdated(self, remote: Package, paths: RepositoryPaths) -> bool:
+    def is_outdated(self, remote: Package, paths: RepositoryPaths, calculate_version: bool = True) -> bool:
         """
         check if package is out-of-dated
         :param remote: package properties from remote source
         :param paths: repository paths instance. Required for VCS packages cache
+        :param calculate_version: expand version to actual value (by calculating git versions)
         :return: True if the package is out-of-dated and False otherwise
         """
-        remote_version = remote.actual_version(paths)  # either normal version or updated VCS
+        remote_version = remote.actual_version(paths) if calculate_version else remote.version
         result: int = vercmp(self.version, remote_version)
         return result < 0
 

src/ahriman/version.py

@@ -17,4 +17,4 @@
 # You should have received a copy of the GNU General Public License
 # along with this program. If not, see <http://www.gnu.org/licenses/>.
 #
-__version__ = "1.6.1"
+__version__ = "1.6.4"

tests/ahriman/application/application/test_application_repository.py

@@ -147,6 +147,7 @@ def test_unknown_no_aur(application_repository: Repository, package_ahriman: Pac
     """
     mocker.patch("ahriman.core.repository.repository.Repository.packages", return_value=[package_ahriman])
     mocker.patch("ahriman.models.package.Package.from_aur", side_effect=Exception())
+    mocker.patch("ahriman.models.package.Package.from_build", return_value=package_ahriman)
     mocker.patch("pathlib.Path.is_dir", return_value=True)
     mocker.patch("ahriman.core.build_tools.sources.Sources.has_remotes", return_value=False)
 
@@ -163,7 +164,7 @@ def test_unknown_no_aur_no_local(application_repository: Repository, package_ahr
     mocker.patch("pathlib.Path.is_dir", return_value=False)
 
     packages = application_repository.unknown()
-    assert packages == [package_ahriman]
+    assert packages == list(package_ahriman.packages.keys())
 
 
 def test_unknown_no_local(application_repository: Repository, package_ahriman: Package, mocker: MockerFixture) -> None:

tests/ahriman/application/formatters/conftest.py

@@ -5,6 +5,7 @@ from ahriman.application.formatters.aur_printer import AurPrinter
 from ahriman.application.formatters.configuration_printer import ConfigurationPrinter
 from ahriman.application.formatters.package_printer import PackagePrinter
 from ahriman.application.formatters.status_printer import StatusPrinter
+from ahriman.application.formatters.string_printer import StringPrinter
 from ahriman.application.formatters.update_printer import UpdatePrinter
 from ahriman.models.build_status import BuildStatus
 from ahriman.models.package import Package
@@ -48,6 +49,15 @@ def status_printer() -> StatusPrinter:
     return StatusPrinter(BuildStatus())
 
 
+@pytest.fixture
+def string_printer() -> StringPrinter:
+    """
+    fixture for any string printer
+    :return: any string printer test instance
+    """
+    return StringPrinter("hello, world")
+
+
 @pytest.fixture
 def update_printer(package_ahriman: Package) -> UpdatePrinter:
     """

tests/ahriman/application/formatters/test_string_printer.py

@@ -0,0 +1,15 @@
+from ahriman.application.formatters.string_printer import StringPrinter
+
+
+def test_properties(string_printer: StringPrinter) -> None:
+    """
+    must return empty properties list
+    """
+    assert not string_printer.properties()
+
+
+def test_title(string_printer: StringPrinter) -> None:
+    """
+    must return non empty title
+    """
+    assert string_printer.title() is not None

tests/ahriman/core/repository/test_executor.py

@@ -11,6 +11,14 @@ from ahriman.core.upload.upload import Upload
 from ahriman.models.package import Package
 
 
+def test_load_archives(executor: Executor) -> None:
+    """
+    must raise NotImplemented for missing load_archives method
+    """
+    with pytest.raises(NotImplementedError):
+        executor.load_archives([])
+
+
 def test_packages(executor: Executor) -> None:
     """
     must raise NotImplemented for missing method
@@ -182,11 +190,13 @@ def test_process_update(executor: Executor, package_ahriman: Package, mocker: Mo
     """
     must run update process
     """
-    mocker.patch("ahriman.models.package.Package.load", return_value=package_ahriman)
+    mocker.patch("ahriman.core.repository.executor.Executor.load_archives", return_value=[package_ahriman])
+    mocker.patch("ahriman.core.repository.executor.Executor.packages", return_value=[package_ahriman])
     move_mock = mocker.patch("shutil.move")
     repo_add_mock = mocker.patch("ahriman.core.alpm.repo.Repo.add")
     sign_package_mock = mocker.patch("ahriman.core.sign.gpg.GPG.process_sign_package", side_effect=lambda fn, _: [fn])
     status_client_mock = mocker.patch("ahriman.core.status.client.Client.set_success")
+    remove_mock = mocker.patch("ahriman.core.repository.executor.Executor.process_remove")
 
     # must return complete
     assert executor.process_update([package.filepath for package in package_ahriman.packages.values()])
@@ -201,6 +211,8 @@ def test_process_update(executor: Executor, package_ahriman: Package, mocker: Mo
     # must clear directory
     from ahriman.core.repository.cleaner import Cleaner
     Cleaner.clear_packages.assert_called_once()
+    # clear removed packages
+    remove_mock.assert_called_once_with([])
 
 
 def test_process_update_group(executor: Executor, package_python_schedule: Package,
@@ -209,9 +221,11 @@ def test_process_update_group(executor: Executor, package_python_schedule: Packa
     must group single packages under one base
     """
     mocker.patch("shutil.move")
-    mocker.patch("ahriman.models.package.Package.load", return_value=package_python_schedule)
+    mocker.patch("ahriman.core.repository.executor.Executor.load_archives", return_value=[package_python_schedule])
+    mocker.patch("ahriman.core.repository.executor.Executor.packages", return_value=[package_python_schedule])
     repo_add_mock = mocker.patch("ahriman.core.alpm.repo.Repo.add")
     status_client_mock = mocker.patch("ahriman.core.status.client.Client.set_success")
+    remove_mock = mocker.patch("ahriman.core.repository.executor.Executor.process_remove")
 
     executor.process_update([package.filepath for package in package_python_schedule.packages.values()])
     repo_add_mock.assert_has_calls([
@@ -219,6 +233,7 @@ def test_process_update_group(executor: Executor, package_python_schedule: Packa
         for package in package_python_schedule.packages.values()
     ], any_order=True)
     status_client_mock.assert_called_once_with(package_python_schedule)
+    remove_mock.assert_called_once_with([])
 
 
 def test_process_empty_filename(executor: Executor, package_ahriman: Package, mocker: MockerFixture) -> None:
@@ -226,7 +241,8 @@ def test_process_empty_filename(executor: Executor, package_ahriman: Package, mo
     must skip update for package which does not have path
     """
     package_ahriman.packages[package_ahriman.base].filename = None
-    mocker.patch("ahriman.models.package.Package.load", return_value=package_ahriman)
+    mocker.patch("ahriman.core.repository.executor.Executor.load_archives", return_value=[package_ahriman])
+    mocker.patch("ahriman.core.repository.executor.Executor.packages", return_value=[package_ahriman])
     executor.process_update([package.filepath for package in package_ahriman.packages.values()])
 
 
@@ -235,18 +251,27 @@ def test_process_update_failed(executor: Executor, package_ahriman: Package, moc
     must process update for failed package
     """
     mocker.patch("shutil.move", side_effect=Exception())
-    mocker.patch("ahriman.models.package.Package.load", return_value=package_ahriman)
+    mocker.patch("ahriman.core.repository.executor.Executor.load_archives", return_value=[package_ahriman])
+    mocker.patch("ahriman.core.repository.executor.Executor.packages", return_value=[package_ahriman])
     status_client_mock = mocker.patch("ahriman.core.status.client.Client.set_failed")
 
     executor.process_update([package.filepath for package in package_ahriman.packages.values()])
     status_client_mock.assert_called_once()
 
 
-def test_process_update_failed_on_load(executor: Executor, package_ahriman: Package, mocker: MockerFixture) -> None:
+def test_process_update_removed_package(executor: Executor, package_python_schedule: Package,
+                                        mocker: MockerFixture) -> None:
     """
-    must process update even with failed package load
+    must remove packages which have been removed from the new base
     """
-    mocker.patch("shutil.move")
+    without_python2 = Package.from_json(package_python_schedule.view())
-    mocker.patch("ahriman.models.package.Package.load", side_effect=Exception())
+    del without_python2.packages["python2-schedule"]
 
-    assert executor.process_update([package.filepath for package in package_ahriman.packages.values()])
+    mocker.patch("shutil.move")
+    mocker.patch("ahriman.core.alpm.repo.Repo.add")
+    mocker.patch("ahriman.core.repository.executor.Executor.load_archives", return_value=[without_python2])
+    mocker.patch("ahriman.core.repository.executor.Executor.packages", return_value=[package_python_schedule])
+    remove_mock = mocker.patch("ahriman.core.repository.executor.Executor.process_remove")
+
+    executor.process_update([package.filepath for package in without_python2.packages.values()])
+    remove_mock.assert_called_once_with(["python2-schedule"])

tests/ahriman/core/repository/test_repository.py

@@ -5,7 +5,7 @@ from ahriman.core.repository import Repository
 from ahriman.models.package import Package
 
 
-def test_packages(package_ahriman: Package, package_python_schedule: Package,
+def test_load_archives(package_ahriman: Package, package_python_schedule: Package,
                        repository: Repository, mocker: MockerFixture) -> None:
     """
     must return all packages grouped by package base
@@ -17,12 +17,9 @@ def test_packages(package_ahriman: Package, package_python_schedule: Package,
                 packages={package: props})
         for package, props in package_python_schedule.packages.items()
     ] + [package_ahriman]
-
-    mocker.patch("pathlib.Path.iterdir",
-                 return_value=[Path("a.pkg.tar.xz"), Path("b.pkg.tar.xz"), Path("c.pkg.tar.xz")])
     mocker.patch("ahriman.models.package.Package.load", side_effect=single_packages)
 
-    packages = repository.packages()
+    packages = repository.load_archives([Path("a.pkg.tar.xz"), Path("b.pkg.tar.xz"), Path("c.pkg.tar.xz")])
     assert len(packages) == 2
     assert {package.base for package in packages} == {package_ahriman.base, package_python_schedule.base}
 
@@ -33,21 +30,48 @@ def test_packages(package_ahriman: Package, package_python_schedule: Package,
     assert set(archives) == expected
 
 
-def test_packages_failed(repository: Repository, mocker: MockerFixture) -> None:
+def test_load_archives_failed(repository: Repository, mocker: MockerFixture) -> None:
     """
     must skip packages which cannot be loaded
     """
-    mocker.patch("pathlib.Path.iterdir", return_value=[Path("a.pkg.tar.xz")])
     mocker.patch("ahriman.models.package.Package.load", side_effect=Exception())
-    assert not repository.packages()
+    assert not repository.load_archives([Path("a.pkg.tar.xz")])
 
 
-def test_packages_not_package(repository: Repository, mocker: MockerFixture) -> None:
+def test_load_archives_not_package(repository: Repository) -> None:
     """
     must skip not packages from iteration
     """
-    mocker.patch("pathlib.Path.iterdir", return_value=[Path("a.tar.xz")])
+    assert not repository.load_archives([Path("a.tar.xz")])
-    assert not repository.packages()
+
+
+def test_load_archives_different_version(repository: Repository, package_python_schedule: Package,
+                                         mocker: MockerFixture) -> None:
+    """
+    must load packages with different versions choosing maximal
+    """
+    single_packages = [
+        Package(base=package_python_schedule.base,
+                version=package_python_schedule.version,
+                aur_url=package_python_schedule.aur_url,
+                packages={package: props})
+        for package, props in package_python_schedule.packages.items()
+    ]
+    single_packages[0].version = "0.0.1-1"
+    mocker.patch("ahriman.models.package.Package.load", side_effect=single_packages)
+
+    packages = repository.load_archives([Path("a.pkg.tar.xz"), Path("b.pkg.tar.xz")])
+    assert len(packages) == 1
+    assert packages[0].version == package_python_schedule.version
+
+
+def test_packages(repository: Repository, mocker: MockerFixture) -> None:
+    """
+    must return repository packages
+    """
+    load_mock = mocker.patch("ahriman.core.repository.repository.Repository.load_archives")
+    repository.packages()
+    load_mock.assert_called_once()  # it uses filter object so we cannot verity argument list =/
 
 
 def test_packages_built(repository: Repository, mocker: MockerFixture) -> None:

tests/ahriman/core/test_configuration.py

@@ -4,6 +4,7 @@ import pytest
 
 from pathlib import Path
 from pytest_mock import MockerFixture
+from unittest import mock
 
 from ahriman.core.configuration import Configuration
 from ahriman.core.exceptions import InitializeException
@@ -54,6 +55,64 @@ def test_section_name(configuration: Configuration) -> None:
     assert configuration.section_name("build", "x86_64") == "build:x86_64"
 
 
+def test_getlist(configuration: Configuration) -> None:
+    """
+    must return list of string correctly
+    """
+    configuration.set_option("build", "test_list", "a b c")
+    assert configuration.getlist("build", "test_list") == ["a", "b", "c"]
+
+
+def test_getlist_empty(configuration: Configuration) -> None:
+    """
+    must return list of string correctly for non-existing option
+    """
+    assert configuration.getlist("build", "test_list", fallback=[]) == []
+    configuration.set_option("build", "test_list", "")
+    assert configuration.getlist("build", "test_list") == []
+
+
+def test_getlist_single(configuration: Configuration) -> None:
+    """
+    must return list of strings for single string
+    """
+    configuration.set_option("build", "test_list", "a")
+    assert configuration.getlist("build", "test_list") == ["a"]
+    assert configuration.getlist("build", "test_list") == ["a"]
+
+
+def test_getlist_with_spaces(configuration: Configuration) -> None:
+    """
+    must return list of string if there is string with spaces in quotes
+    """
+    configuration.set_option("build", "test_list", """"ahriman is" cool""")
+    assert configuration.getlist("build", "test_list") == ["""ahriman is""", """cool"""]
+    configuration.set_option("build", "test_list", """'ahriman is' cool""")
+    assert configuration.getlist("build", "test_list") == ["""ahriman is""", """cool"""]
+
+
+def test_getlist_with_quotes(configuration: Configuration) -> None:
+    """
+    must return list of string if there is string with quote inside quote
+    """
+    configuration.set_option("build", "test_list", """"ahriman is" c"'"ool""")
+    assert configuration.getlist("build", "test_list") == ["""ahriman is""", """c'ool"""]
+    configuration.set_option("build", "test_list", """'ahriman is' c'"'ool""")
+    assert configuration.getlist("build", "test_list") == ["""ahriman is""", """c"ool"""]
+
+
+def test_getlist_unmatched_quote(configuration: Configuration) -> None:
+    """
+    must raise exception on unmatched quote in string value
+    """
+    configuration.set_option("build", "test_list", """ahri"man is cool""")
+    with pytest.raises(ValueError):
+        configuration.getlist("build", "test_list")
+    configuration.set_option("build", "test_list", """ahri'man is cool""")
+    with pytest.raises(ValueError):
+        configuration.getlist("build", "test_list")
+
+
 def test_getpath_absolute_to_absolute(configuration: Configuration) -> None:
     """
     must not change path for absolute path in settings
@@ -94,32 +153,6 @@ def test_getpath_without_fallback(configuration: Configuration) -> None:
         assert configuration.getpath("build", "option")
 
 
-def test_getlist(configuration: Configuration) -> None:
-    """
-    must return list of string correctly
-    """
-    configuration.set_option("build", "test_list", "a b c")
-    assert configuration.getlist("build", "test_list") == ["a", "b", "c"]
-
-
-def test_getlist_empty(configuration: Configuration) -> None:
-    """
-    must return list of string correctly for non-existing option
-    """
-    assert configuration.getlist("build", "test_list", fallback=[]) == []
-    configuration.set_option("build", "test_list", "")
-    assert configuration.getlist("build", "test_list") == []
-
-
-def test_getlist_single(configuration: Configuration) -> None:
-    """
-    must return list of strings for single string
-    """
-    configuration.set_option("build", "test_list", "a")
-    assert configuration.getlist("build", "test_list") == ["a"]
-    assert configuration.getlist("build", "test_list") == ["a"]
-
-
 def test_gettype(configuration: Configuration) -> None:
     """
     must extract type from variable
@@ -222,6 +255,17 @@ def test_reload(configuration: Configuration, mocker: MockerFixture) -> None:
     merge_mock.assert_called_once()
 
 
+def test_reload_clear(configuration: Configuration, mocker: MockerFixture) -> None:
+    """
+    must clear current settings before configuration reload
+    """
+    clear_mock = mocker.patch("ahriman.core.configuration.Configuration.remove_section")
+    sections = configuration.sections()
+
+    configuration.reload()
+    clear_mock.assert_has_calls([mock.call(section) for section in sections])
+
+
 def test_reload_no_architecture(configuration: Configuration) -> None:
     """
     must raise exception on reload if no architecture set

tests/ahriman/models/test_build_status.py

@@ -1,4 +1,5 @@
 import datetime
+import time
 
 from ahriman.models.build_status import BuildStatus, BuildStatusEnum
 
@@ -45,6 +46,17 @@ def test_build_status_init_2(build_status_failed: BuildStatus) -> None:
     assert status == build_status_failed
 
 
+def test_build_status_init_empty_timestamp() -> None:
+    """
+    must st current timestamp when not set
+    """
+    first = BuildStatus()
+    time.sleep(1)
+    second = BuildStatus()
+    # well technically it just should increase
+    assert first.timestamp < second.timestamp
+
+
 def test_build_status_from_json_view(build_status_failed: BuildStatus) -> None:
     """
     must construct same object from json