feat: replace scan paths options to single one

It has been found that previous system didn't allow to configure
specific cases (e.g. a whitelisted directory inside /usr/lib/cmake). The
current solution replaces two options to single one, which also allows a
regular expressions

Also PackageArchive class has been moved to core package, because it is
more about service rather than model

CONTRIBUTING.md

@@ -92,7 +92,7 @@ Again, the most checks can be performed by `tox` command, though some additional
     ```
 
 * Type annotations are the must, even for local functions. For the function argument `self` (for instance methods) and `cls` (for class methods) should not be annotated.
-* For collection types built-in classes must be used if possible (e.g. `dict` instead of `typing.Dict`, `tuple` instead of `typing.Tuple`). In case if built-in type is not available, but `collections.abc` provides interface, it must be used (e.g. `collections.abc.Awaitable` instead of `typing.Awaitable`, `collections.abc.Iterable` instead of `typing.Iterable`). For union classes, the bar operator (`|`) must be used (e.g. `float | int` instead of `typing.Union[float, int]`), which also includes `typinng.Optional` (e.g. `str | None` instead of `Optional[str]`).
+* For collection types built-in classes must be used if possible (e.g. `dict` instead of `typing.Dict`, `tuple` instead of `typing.Tuple`). In case if built-in type is not available, but `collections.abc` provides interface, it must be used (e.g. `collections.abc.Awaitable` instead of `typing.Awaitable`, `collections.abc.Iterable` instead of `typing.Iterable`). For union classes, the bar operator (`|`) must be used (e.g. `float | int` instead of `typing.Union[float, int]`), which also includes `typing.Optional` (e.g. `str | None` instead of `Optional[str]`).
 * `classmethod` should (almost) always return `Self`. In case of mypy warning (e.g. if there is a branch in which function doesn't return the instance of `cls`) consider using `staticmethod` instead.
 * Recommended order of function definitions in class:
 
@@ -132,7 +132,7 @@ Again, the most checks can be performed by `tox` command, though some additional
 * For any path interactions `pathlib.Path` must be used.
 * Configuration interactions must go through `ahriman.core.configuration.Configuration` class instance.
 * In case if class load requires some actions, it is recommended to create class method which can be used for class instantiating.
-* The most (expected) exceptions must be handled and printed to log, allowing service to continue work. However, fatal and (in some cases) unexpected exceptions may lead to the application termination.
+* The code must follow the exception safety, unless it is explicitly asked by end user. It means that most exceptions must be handled and printed to log, no other actions must be done (e.g. raising another exception).
 * Exceptions without parameters should be raised without parentheses, e.g.:
 
     ```python

README.md

@@ -2,7 +2,7 @@
 
 [![tests status](https://github.com/arcan1s/ahriman/actions/workflows/tests.yml/badge.svg)](https://github.com/arcan1s/ahriman/actions/workflows/tests.yml)
 [![setup status](https://github.com/arcan1s/ahriman/actions/workflows/setup.yml/badge.svg)](https://github.com/arcan1s/ahriman/actions/workflows/setup.yml)
-[![Docker Image Version (latest semver)](https://img.shields.io/docker/v/arcan1s/ahriman?label=Docker%20image)](https://hub.docker.com/r/arcan1s/ahriman)
+[![Docker Image Version (latest semver)](https://img.shields.io/docker/v/arcan1s/ahriman?label=Docker%20image&sort=semver)](https://hub.docker.com/r/arcan1s/ahriman)
 [![CodeFactor](https://www.codefactor.io/repository/github/arcan1s/ahriman/badge)](https://www.codefactor.io/repository/github/arcan1s/ahriman)
 [![Documentation Status](https://readthedocs.org/projects/ahriman/badge/?version=latest)](https://ahriman.readthedocs.io)
 
@@ -40,5 +40,3 @@ The application provides reasonable defaults which allow to use it out-of-box; h
 * [Build status page](https://ahriman-demo.arcanis.me). You can log in as `demo` user by using `demo` password. However, you will not be able to run tasks. [HTTP API documentation](https://ahriman-demo.arcanis.me/api-docs) is also available.
 * [Repository index](https://repo.arcanis.me/arcanisrepo/x86_64/).
 * [Telegram feed](https://t.me/arcanisrepo).
-
-Do you have any success story? You can [share it](https://github.com/arcan1s/ahriman/issues/new?template=04-discussion.md)!

docs/ahriman.core.build_tools.rst

@@ -4,6 +4,14 @@ ahriman.core.build\_tools package
 Submodules
 ----------
 
+ahriman.core.build\_tools.package\_archive module
+-------------------------------------------------
+
+.. automodule:: ahriman.core.build_tools.package_archive
+   :members:
+   :no-undoc-members:
+   :show-inheritance:
+
 ahriman.core.build\_tools.sources module
 ----------------------------------------
 

docs/ahriman.models.rst

@@ -124,14 +124,6 @@ ahriman.models.package module
    :no-undoc-members:
    :show-inheritance:
 
-ahriman.models.package\_archive module
---------------------------------------
-
-.. automodule:: ahriman.models.package_archive
-   :members:
-   :no-undoc-members:
-   :show-inheritance:
-
 ahriman.models.package\_description module
 ------------------------------------------
 

docs/configuration.rst

@@ -81,14 +81,13 @@ Authorized users are stored inside internal database, if any of external provide
 
 Build related configuration. Group name can refer to architecture, e.g. ``build:x86_64`` can be used for x86_64 architecture specific settings.
 
-* ``allowed_scan_paths`` - paths to be used for implicit dependencies scan, scape separated list of paths, optional.
 * ``archbuild_flags`` - additional flags passed to ``archbuild`` command, space separated list of strings, optional.
-* ``blacklisted_scan_paths`` - paths to be excluded for implicit dependencies scan, scape separated list of paths, optional. Normally all elements of this option must be child paths of any of ``allowed_scan_paths`` element.
 * ``build_command`` - default build command, string, required.
 * ``ignore_packages`` - list packages to ignore during a regular update (manual update will still work), space separated list of strings, optional.
 * ``include_debug_packages`` - distribute debug packages, boolean, optional, default ``yes``.
 * ``makepkg_flags`` - additional flags passed to ``makepkg`` command, space separated list of strings, optional.
 * ``makechrootpkg_flags`` - additional flags passed to ``makechrootpkg`` command, space separated list of strings, optional.
+* ``scan_paths`` - paths to be used for implicit dependencies scan, space separated list of strings, optional. If any of those paths is matched against the path, it will be added to the allowed list.
 * ``triggers`` - list of ``ahriman.core.triggers.Trigger`` class implementation (e.g. ``ahriman.core.report.ReportTrigger ahriman.core.upload.UploadTrigger``) which will be loaded and run at the end of processing, space separated list of strings, optional. You can also specify triggers by their paths, e.g. ``/usr/lib/python3.10/site-packages/ahriman/core/report/report.py.ReportTrigger``. Triggers are run in the order of definition.
 * ``triggers_known`` - optional list of ``ahriman.core.triggers.Trigger`` class implementations which are not run automatically and used only for trigger discovery and configuration validation.
 * ``vcs_allowed_age`` - maximal age in seconds of the VCS packages before their version will be updated with its remote source, integer, optional, default is 7 days.

docs/faq/general.rst

@@ -379,7 +379,7 @@ After the success build the application extracts all linked libraries and used d
 
 In order to disable this check completely, the ``--no-check-files`` flag can be used.
 
-In addition, there is possibility to control paths which will be used for checking, by using options ``build.allowed_scan_paths`` and ``build.blacklisted_scan_paths``. Leaving ``build.allowed_scan_paths`` blank will effectively disable any check too.
+In addition, there is possibility to control paths which will be used for checking, by using option ``build.scan_paths``, which supports regular expressions. Leaving ``build.scan_paths`` blank will effectively disable any check too.
 
 How to install built packages
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

package/archlinux/PKGBUILD

@@ -1,7 +1,7 @@
 # Maintainer: Evgeniy Alekseev
 
 pkgname='ahriman'
-pkgver=2.14.1
+pkgver=2.14.0
 pkgrel=1
 pkgdesc="ArcH linux ReposItory MANager"
 arch=('any')

package/share/ahriman/settings/ahriman.ini

@@ -50,12 +50,8 @@ allow_read_only = yes
 ;salt =
 
 [build]
-; List of paths to be used for implicit dependency scan
-allowed_scan_paths = /usr/lib
 ; List of additional flags passed to archbuild command.
 ;archbuild_flags =
-; List of paths to be excluded for implicit dependency scan. Usually they should be subpaths of allowed_scan_paths
-blacklisted_scan_paths = /usr/lib/cmake
 ; Path to build command
 ;build_command =
 ; List of packages to be ignored during automatic updates.
@@ -66,6 +62,8 @@ blacklisted_scan_paths = /usr/lib/cmake
 ;makechrootpkg_flags =
 ; List of additional flags passed to makepkg command.
 makepkg_flags = --nocolor --ignorearch
+; List of paths to be used for implicit dependency scan. Regular expressions are supported
+scan_paths = ^usr/lib(?!/cmake).*$
 ; List of enabled triggers in the order of calls.
 triggers = ahriman.core.gitremote.RemotePullTrigger ahriman.core.report.ReportTrigger ahriman.core.upload.UploadTrigger ahriman.core.gitremote.RemotePushTrigger
 ; List of well-known triggers. Used only for configuration purposes.

package/share/man/man1/ahriman.1

@@ -1,4 +1,4 @@
-.TH AHRIMAN "1" "2024\-09\-04" "ahriman" "Generated Python Manual"
+.TH AHRIMAN "1" "2024\-08\-23" "ahriman" "Generated Python Manual"
 .SH NAME
 ahriman
 .SH SYNOPSIS

src/ahriman/__init__.py

@@ -17,4 +17,4 @@
 # You should have received a copy of the GNU General Public License
 # along with this program. If not, see <http://www.gnu.org/licenses/>.
 #
-__version__ = "2.14.1"
+__version__ = "2.14.0"

src/ahriman/application/lock.py

@@ -75,9 +75,7 @@ class Lock(LazyLogging):
         """
         self.path: Path | None = None
         if args.lock is not None:
-            self.path = args.lock
+            self.path = args.lock.with_stem(f"{args.lock.stem}_{repository_id.id}")
-            if not repository_id.is_empty:
-                self.path = self.path.with_stem(f"{args.lock.stem}_{repository_id.id}")
             if not self.path.is_absolute():
                 # prepend full path to the lock file
                 self.path = Path("/") / "run" / "ahriman" / self.path

src/ahriman/core/alpm/pacman.py

@@ -43,7 +43,7 @@ class Pacman(LazyLogging):
         configuration(Configuration): configuration instance
         refresh_database(PacmanSynchronization): synchronize local cache to remote
         repository_id(RepositoryId): repository unique identifier
-        repository_paths(RepositoryPaths): repository paths instance
+        repository_path(RepositoryPaths): repository paths instance
     """
 
     def __init__(self, repository_id: RepositoryId, configuration: Configuration, *,
@@ -188,8 +188,8 @@ class Pacman(LazyLogging):
         Returns:
             dict[str, set[str]]: map of package name to its list of files
         """
-        def extract(tar: tarfile.TarFile, versions: dict[str, str]) -> Generator[tuple[str, set[str]], None, None]:
+        def extract(tar: tarfile.TarFile, package_names: dict[str, str]) -> Generator[tuple[str, set[str]], None, None]:
-            for package_name, version in versions.items():
+            for package_name, version in package_names.items():
                 path = Path(f"{package_name}-{version}") / "files"
                 try:
                     content = tar.extractfile(str(path))

src/ahriman/core/alpm/pacman_database.py

@@ -59,8 +59,7 @@ class PacmanDatabase(SyncHttpClient):
 
         self.sync_files_database = configuration.getboolean("alpm", "sync_files_database")
 
-    @staticmethod
+    def copy(self, remote_path: Path, local_path: Path) -> None:
-    def copy(remote_path: Path, local_path: Path) -> None:
         """
         copy local database file
 

src/ahriman/core/build_tools/package_archive.py

@@ -17,7 +17,6 @@
 # You should have received a copy of the GNU General Public License
 # along with this program. If not, see <http://www.gnu.org/licenses/>.
 #
-from dataclasses import dataclass
 from elftools.elf.dynamic import DynamicSection
 from elftools.elf.elffile import ELFFile
 from pathlib import Path
@@ -33,7 +32,6 @@ from ahriman.models.package import Package
 from ahriman.models.scan_paths import ScanPaths
 
 
-@dataclass
 class PackageArchive:
     """
     helper for package archives
@@ -45,10 +43,20 @@ class PackageArchive:
         scan_paths(ScanPaths): scan paths holder
     """
 
-    root: Path
+    def __init__(self, root: Path, package: Package, pacman: Pacman, scan_paths: ScanPaths) -> None:
-    package: Package
+        """
-    pacman: Pacman
+        default constructor
-    scan_paths: ScanPaths
+
+        Args:
+            root(Path): path to root filesystem
+            package(Package): package descriptor
+            pacman(Pacman): alpm wrapper instance
+            scan_paths(ScanPaths): scan paths holder
+        """
+        self.root = root
+        self.package = package
+        self.pacman = pacman
+        self.scan_paths = scan_paths
 
     @staticmethod
     def dynamic_needed(binary_path: Path) -> list[str]:

src/ahriman/core/configuration/schema.py

@@ -169,14 +169,6 @@ CONFIGURATION_SCHEMA: ConfigurationSchema = {
     "build": {
         "type": "dict",
         "schema": {
-            "allowed_scan_paths": {
-                "type": "list",
-                "coerce": "list",
-                "schema": {
-                    "type": "path",
-                    "coerce": "absolute_path",
-                },
-            },
             "archbuild_flags": {
                 "type": "list",
                 "coerce": "list",
@@ -185,14 +177,6 @@ CONFIGURATION_SCHEMA: ConfigurationSchema = {
                     "empty": False,
                 },
             },
-            "blacklisted_scan_paths": {
-                "type": "list",
-                "coerce": "list",
-                "schema": {
-                    "type": "path",
-                    "coerce": "absolute_path",
-                },
-            },
             "build_command": {
                 "type": "string",
                 "required": True,
@@ -226,6 +210,14 @@ CONFIGURATION_SCHEMA: ConfigurationSchema = {
                     "empty": False,
                 },
             },
+            "scan_paths": {
+                "type": "list",
+                "coerce": "list",
+                "schema": {
+                    "type": "string",
+                    "empty": False,
+                },
+            },
             "triggers": {
                 "type": "list",
                 "coerce": "list",

src/ahriman/core/database/sqlite.py

@@ -27,7 +27,6 @@ from ahriman.core.configuration import Configuration
 from ahriman.core.database.migrations import Migrations
 from ahriman.core.database.operations import AuthOperations, BuildOperations, ChangesOperations, \
     DependenciesOperations, LogsOperations, PackageOperations, PatchOperations
-from ahriman.models.repository_id import RepositoryId
 
 
 # pylint: disable=too-many-ancestors
@@ -103,26 +102,23 @@ class SQLite(
             self.with_connection(lambda connection: Migrations.migrate(connection, configuration))
         paths.chown(self.path)
 
-    def package_clear(self, package_base: str, repository_id: RepositoryId | None = None) -> None:
+    def package_clear(self, package_base: str) -> None:
         """
         completely remove package from all tables
 
         Args:
             package_base(str): package base to remove
-            repository_id(RepositoryId, optional): repository unique identifier override (Default value = None)
 
         Examples:
             This method completely removes the package from all tables and must be used, e.g. on package removal::
 
             >>> database.package_clear("ahriman")
         """
-        self.build_queue_clear(package_base, repository_id)
+        self.build_queue_clear(package_base)
-        self.patches_remove(package_base, None)
+        self.patches_remove(package_base, [])
-        self.logs_remove(package_base, None, repository_id)
+        self.logs_remove(package_base, None)
-        self.changes_remove(package_base, repository_id)
+        self.changes_remove(package_base)
-        self.dependencies_remove(package_base, repository_id)
+        self.dependencies_remove(package_base)
-
-        self.package_remove(package_base, repository_id)
 
         # remove local cache too
         self._repository_paths.tree_clear(package_base)

src/ahriman/core/repository/executor.py

@@ -23,13 +23,13 @@ from collections.abc import Iterable
 from pathlib import Path
 from tempfile import TemporaryDirectory
 
+from ahriman.core.build_tools.package_archive import PackageArchive
 from ahriman.core.build_tools.task import Task
 from ahriman.core.repository.cleaner import Cleaner
 from ahriman.core.repository.package_info import PackageInfo
 from ahriman.core.utils import safe_filename
 from ahriman.models.changes import Changes
 from ahriman.models.package import Package
-from ahriman.models.package_archive import PackageArchive
 from ahriman.models.package_description import PackageDescription
 from ahriman.models.packagers import Packagers
 from ahriman.models.result import Result

src/ahriman/core/repository/repository_properties.py

@@ -80,10 +80,7 @@ class RepositoryProperties(LazyLogging):
         self.reporter = Client.load(repository_id, configuration, database, report=report)
         self.triggers = TriggerLoader.load(repository_id, configuration)
 
-        self.scan_paths = ScanPaths(
+        self.scan_paths = ScanPaths(configuration.getlist("build", "scan_paths", fallback=[]))
-            allowed_paths=configuration.getpathlist("build", "allowed_scan_paths", fallback=[]),
-            blacklisted_paths=configuration.getpathlist("build", "blacklisted_scan_paths", fallback=[]),
-        )
 
     @property
     def architecture(self) -> str:

src/ahriman/core/status/client.py

@@ -310,7 +310,7 @@ class Client:
     def set_unknown(self, package: Package) -> None:
         """
         set package status to unknown. Unlike other methods, this method also checks if package is known,
-        and - in case if it is - it silently skips update
+        and - in case if it is - it silently skips updatd
 
         Args:
             package(Package): current package properties

src/ahriman/core/status/local_client.py

@@ -184,7 +184,7 @@ class LocalClient(Client):
         Args:
             package_base(str): package base to remove
         """
-        self.database.package_clear(package_base, self.repository_id)
+        self.database.package_clear(package_base)
 
     def package_status_update(self, package_base: str, status: BuildStatusEnum) -> None:
         """

src/ahriman/core/status/watcher.py

@@ -140,6 +140,7 @@ class Watcher(LazyLogging):
         with self._lock:
             self._known.pop(package_base, None)
         self.client.package_remove(package_base)
+        self.package_logs_remove(package_base, None)
 
     def package_status_update(self, package_base: str, status: BuildStatusEnum) -> None:
         """

src/ahriman/models/repository_id.py

@@ -41,12 +41,9 @@ class RepositoryId:
 
         Returns:
             str: unique id for this repository
-
-        Raises:
-            ValueError: if repository identifier is empty
         """
         if self.is_empty:
-            raise ValueError("Repository ID is called on empty repository identifier")
+            return ""
         return f"{self.architecture}-{self.name}"  # basically the same as used for command line
 
     @property

src/ahriman/models/repository_paths.py

@@ -113,7 +113,7 @@ class RepositoryPaths(LazyLogging):
         Returns:
             Path: full patch to devtools chroot directory
         """
-        # for the chroot directory devtools will create own tree, and we don't have to specify architecture here
+        # for the chroot directory devtools will create own tree, and we don"t have to specify architecture here
         return self.root / "chroot" / self.repository_id.name
 
     @property

src/ahriman/models/scan_paths.py

@@ -17,29 +17,33 @@
 # You should have received a copy of the GNU General Public License
 # along with this program. If not, see <http://www.gnu.org/licenses/>.
 #
+import re
+
 from dataclasses import dataclass
+from functools import cached_property
 from pathlib import Path
 
 
-@dataclass(frozen=True, kw_only=True)
+@dataclass(frozen=True)
 class ScanPaths:
     """
     paths used for scan filesystem
 
     Attributes:
-        allowed_paths(list[Path]): list of whitelisted paths
+        paths(list[str]): list of regular expressions to be used to match paths
-        blacklisted_paths(list[Path]): list of paths to be skipped from scan
     """
 
-    allowed_paths: list[Path]
+    paths: list[str]
-    blacklisted_paths: list[Path]
 
-    def __post_init__(self) -> None:
+    @cached_property
+    def patterns(self) -> list[re.Pattern[str]]:
         """
-        compute relative to / paths
+        compiled regular expressions
+
+        Returns:
+            list[re.Pattern]: a list of compiled regular expressions
         """
-        object.__setattr__(self, "allowed_paths", [path.relative_to("/") for path in self.allowed_paths])
+        return [re.compile(path) for path in self.paths]
-        object.__setattr__(self, "blacklisted_paths", [path.relative_to("/") for path in self.blacklisted_paths])
 
     def is_allowed(self, path: Path) -> bool:
         """
@@ -49,10 +53,7 @@ class ScanPaths:
             path(Path): path to be checked
 
         Returns:
-            bool: ``True`` in case if :attr:`allowed_paths` contains element which is parent for the path and
+            bool: ``True`` in case if :attr:`paths` contains at least one element to which the path is matched
-            :attr:`blacklisted_paths` doesn't and ``False`` otherwise
+            and ``False`` otherwise
         """
-        if any(path.is_relative_to(blacklisted) for blacklisted in self.blacklisted_paths):
+        return any(pattern.match(str(path)) for pattern in self.patterns)
-            return False  # path is blacklisted
-        # check if we actually have to check this path
-        return any(path.is_relative_to(allowed) for allowed in self.allowed_paths)

tests/ahriman/application/test_lock.py

@@ -14,7 +14,6 @@ from ahriman.core.configuration import Configuration
 from ahriman.core.exceptions import DuplicateRunError, UnsafeRunError
 from ahriman.models.build_status import BuildStatus, BuildStatusEnum
 from ahriman.models.internal_status import InternalStatus
-from ahriman.models.repository_id import RepositoryId
 
 
 def test_path(args: argparse.Namespace, configuration: Configuration) -> None:
@@ -31,8 +30,6 @@ def test_path(args: argparse.Namespace, configuration: Configuration) -> None:
     args.lock = Path("ahriman.pid")
     assert Lock(args, repository_id, configuration).path == Path("/run/ahriman/ahriman_x86_64-aur-clone.pid")
 
-    assert Lock(args, RepositoryId("", ""), configuration).path == Path("/run/ahriman/ahriman.pid")
-
     with pytest.raises(ValueError):
         args.lock = Path("/")
         assert Lock(args, repository_id, configuration).path  # special case

tests/ahriman/conftest.py

@@ -25,6 +25,7 @@ from ahriman.models.remote_source import RemoteSource
 from ahriman.models.repository_id import RepositoryId
 from ahriman.models.repository_paths import RepositoryPaths
 from ahriman.models.result import Result
+from ahriman.models.scan_paths import ScanPaths
 from ahriman.models.user import User
 from ahriman.models.user_access import UserAccess
 
@@ -587,6 +588,20 @@ def result(package_ahriman: Package) -> Result:
     return result
 
 
+@pytest.fixture
+def scan_paths(configuration: Configuration) -> ScanPaths:
+    """
+    scan paths fixture
+
+    Args:
+        configuration(Configuration): configuration test instance
+
+    Returns:
+        ScanPaths: scan paths test instance
+    """
+    return ScanPaths(configuration.getlist("build", "scan_paths", fallback=[]))
+
+
 @pytest.fixture
 def spawner(configuration: Configuration) -> Spawn:
     """

tests/ahriman/core/alpm/test_pacman_database.py

@@ -8,12 +8,12 @@ from ahriman.core.alpm.pacman_database import PacmanDatabase
 from ahriman.core.exceptions import PacmanError
 
 
-def test_copy(mocker: MockerFixture) -> None:
+def test_copy(pacman_database: PacmanDatabase, mocker: MockerFixture) -> None:
     """
     must copy loca database file
     """
     copy_mock = mocker.patch("shutil.copy")
-    PacmanDatabase.copy(Path("remote"), Path("local"))
+    pacman_database.copy(Path("remote"), Path("local"))
     copy_mock.assert_called_once_with(Path("remote"), Path("local"))
 
 

tests/ahriman/core/build_tools/conftest.py

@@ -1,6 +1,35 @@
 import pytest
 
+from pytest_mock import MockerFixture
+from typing import Any
+
+from ahriman.core.alpm.pacman import Pacman
+from ahriman.core.build_tools.package_archive import PackageArchive
 from ahriman.core.build_tools.sources import Sources
+from ahriman.models.package import Package
+from ahriman.models.repository_paths import RepositoryPaths
+from ahriman.models.scan_paths import ScanPaths
+
+
+@pytest.fixture
+def package_archive_ahriman(package_ahriman: Package, repository_paths: RepositoryPaths, pacman: Pacman,
+                            scan_paths: ScanPaths, passwd: Any, mocker: MockerFixture) -> PackageArchive:
+    """
+    package archive fixture
+
+    Args:
+        package_ahriman(Package): package test instance
+        repository_paths(RepositoryPaths): repository paths test instance
+        pacman(Pacman): pacman test instance
+        scan_paths(ScanPaths): scan paths test instance
+        passwd(Any): passwd structure test instance
+        mocker(MockerFixture): mocker object
+
+    Returns:
+        PackageArchive: package archive test instance
+    """
+    mocker.patch("ahriman.models.repository_paths.getpwuid", return_value=passwd)
+    return PackageArchive(repository_paths.build_directory, package_ahriman, pacman, scan_paths)
 
 
 @pytest.fixture

tests/ahriman/core/build_tools/test_package_archive.py

@@ -3,16 +3,16 @@ from pathlib import Path
 from pytest_mock import MockerFixture
 from unittest.mock import MagicMock, PropertyMock
 
+from ahriman.core.build_tools.package_archive import PackageArchive
 from ahriman.core.exceptions import UnknownPackageError
 from ahriman.models.filesystem_package import FilesystemPackage
-from ahriman.models.package_archive import PackageArchive
 
 
 def test_dynamic_needed(mocker: MockerFixture) -> None:
     """
     must correctly define list of dynamically linked libraries
     """
-    mocker.patch("ahriman.models.package_archive.PackageArchive.is_elf", return_value=True)
+    mocker.patch("ahriman.core.build_tools.package_archive.PackageArchive.is_elf", return_value=True)
 
     linked = PackageArchive.dynamic_needed(Path(".tox") / "tests" / "bin" / "python")
     assert linked
@@ -24,7 +24,7 @@ def test_dynamic_needed_not_elf(mocker: MockerFixture) -> None:
     """
     must skip checking if not an elf file
     """
-    mocker.patch("ahriman.models.package_archive.PackageArchive.is_elf", return_value=False)
+    mocker.patch("ahriman.core.build_tools.package_archive.PackageArchive.is_elf", return_value=False)
     assert not PackageArchive.dynamic_needed(Path(".tox") / "tests" / "bin" / "python")
 
 
@@ -32,7 +32,7 @@ def test_dynamic_needed_no_section(mocker: MockerFixture) -> None:
     """
     must skip checking if there was no dynamic section found
     """
-    mocker.patch("ahriman.models.package_archive.PackageArchive.is_elf", return_value=True)
+    mocker.patch("ahriman.core.build_tools.package_archive.PackageArchive.is_elf", return_value=True)
     mocker.patch("elftools.elf.elffile.ELFFile.iter_sections", return_value=[])
     assert not PackageArchive.dynamic_needed(Path(".tox") / "tests" / "bin" / "python")
 
@@ -109,8 +109,8 @@ def test_raw_dependencies_packages(package_archive_ahriman: PackageArchive, mock
             files=[Path("package2") / "file4", Path("package2") / "file3"],
         ),
     }
-    mocker.patch("ahriman.models.package_archive.PackageArchive.installed_packages", return_value=packages)
+    mocker.patch("ahriman.core.build_tools.package_archive.PackageArchive.installed_packages", return_value=packages)
-    mocker.patch("ahriman.models.package_archive.PackageArchive.depends_on_paths", return_value=(
+    mocker.patch("ahriman.core.build_tools.package_archive.PackageArchive.depends_on_paths", return_value=(
         {"file1", "file3"},
         {Path("usr") / "dir2", Path("dir3"), Path("package2") / "dir4"},
     ))
@@ -165,17 +165,19 @@ def test_depends_on(package_archive_ahriman: PackageArchive, mocker: MockerFixtu
     """
     must extract packages and files which are dependencies for the package
     """
-    raw_mock = mocker.patch("ahriman.models.package_archive.PackageArchive._raw_dependencies_packages",
+    raw_mock = mocker.patch("ahriman.core.build_tools.package_archive.PackageArchive._raw_dependencies_packages",
                             return_value="1")
-    refined_mock = mocker.patch("ahriman.models.package_archive.PackageArchive._refine_dependencies", return_value={
+    refined_mock = mocker.patch(
-        Path("package1") / "file1": [FilesystemPackage(package_name="package1", depends=set(), opt_depends=set())],
+        "ahriman.core.build_tools.package_archive.PackageArchive._refine_dependencies", return_value={
-        Path("package2") / "file3": [FilesystemPackage(package_name="package2", depends=set(), opt_depends=set())],
+            Path("package1") / "file1": [FilesystemPackage(package_name="package1", depends=set(), opt_depends=set())],
-        Path("package2") / "dir4": [FilesystemPackage(package_name="package2", depends=set(), opt_depends=set())],
+            Path("package2") / "file3": [FilesystemPackage(package_name="package2", depends=set(), opt_depends=set())],
-        Path("usr") / "dir2": [
+            Path("package2") / "dir4": [FilesystemPackage(package_name="package2", depends=set(), opt_depends=set())],
-            FilesystemPackage(package_name="package1", depends=set(), opt_depends=set()),
+            Path("usr") / "dir2": [
-            FilesystemPackage(package_name="package2", depends=set(), opt_depends=set()),
+                FilesystemPackage(package_name="package1", depends=set(), opt_depends=set()),
-        ],
+                FilesystemPackage(package_name="package2", depends=set(), opt_depends=set()),
-    })
+            ],
+        }
+    )
 
     result = package_archive_ahriman.depends_on()
     raw_mock.assert_called_once_with()
@@ -194,8 +196,9 @@ def test_depends_on_paths(package_archive_ahriman: PackageArchive, mocker: Mocke
     """
     package_dir = package_archive_ahriman.root / "build" / \
         package_archive_ahriman.package.base / "pkg" / package_archive_ahriman.package.base
-    dynamic_mock = mocker.patch("ahriman.models.package_archive.PackageArchive.dynamic_needed", return_value=["lib"])
+    dynamic_mock = mocker.patch("ahriman.core.build_tools.package_archive.PackageArchive.dynamic_needed",
-    walk_mock = mocker.patch("ahriman.models.package_archive.walk", return_value=[
+                                return_value=["lib"])
+    walk_mock = mocker.patch("ahriman.core.build_tools.package_archive.walk", return_value=[
         package_dir / "root" / "file",
         Path("directory"),
     ])
@@ -213,7 +216,7 @@ def test_installed_packages(package_archive_ahriman: PackageArchive, mocker: Moc
     """
     must load list of installed packages and their files
     """
-    walk_mock = mocker.patch("ahriman.models.package_archive.walk", return_value=[
+    walk_mock = mocker.patch("ahriman.core.build_tools.package_archive.walk", return_value=[
         Path("ahriman-2.13.3-1") / "desc",
         Path("ahriman-2.13.3-1") / "files",
     ])

tests/ahriman/core/database/test_sqlite.py

@@ -4,7 +4,6 @@ from pytest_mock import MockerFixture
 
 from ahriman.core.configuration import Configuration
 from ahriman.core.database import SQLite
-from ahriman.models.repository_id import RepositoryId
 
 
 def test_load(configuration: Configuration, mocker: MockerFixture) -> None:
@@ -36,7 +35,7 @@ def test_init_skip_migration(database: SQLite, configuration: Configuration, moc
     migrate_schema_mock.assert_not_called()
 
 
-def test_package_clear(database: SQLite, repository_id: RepositoryId, mocker: MockerFixture) -> None:
+def test_package_clear(database: SQLite, mocker: MockerFixture) -> None:
     """
     must clear package data
     """
@@ -45,14 +44,12 @@ def test_package_clear(database: SQLite, repository_id: RepositoryId, mocker: Mo
     logs_mock = mocker.patch("ahriman.core.database.SQLite.logs_remove")
     changes_mock = mocker.patch("ahriman.core.database.SQLite.changes_remove")
     dependencies_mock = mocker.patch("ahriman.core.database.SQLite.dependencies_remove")
-    package_mock = mocker.patch("ahriman.core.database.SQLite.package_remove")
     tree_clear_mock = mocker.patch("ahriman.models.repository_paths.RepositoryPaths.tree_clear")
 
-    database.package_clear("package", repository_id)
+    database.package_clear("package")
-    build_queue_mock.assert_called_once_with("package", repository_id)
+    build_queue_mock.assert_called_once_with("package")
-    patches_mock.assert_called_once_with("package", None)
+    patches_mock.assert_called_once_with("package", [])
-    logs_mock.assert_called_once_with("package", None, repository_id)
+    logs_mock.assert_called_once_with("package", None)
-    changes_mock.assert_called_once_with("package", repository_id)
+    changes_mock.assert_called_once_with("package")
-    dependencies_mock.assert_called_once_with("package", repository_id)
+    dependencies_mock.assert_called_once_with("package")
-    package_mock.assert_called_once_with("package", repository_id)
     tree_clear_mock.assert_called_once_with("package")

tests/ahriman/core/repository/test_executor.py

@@ -24,7 +24,7 @@ def test_process_build(executor: Executor, package_ahriman: Package, passwd: Any
     move_mock = mocker.patch("shutil.move")
     status_client_mock = mocker.patch("ahriman.core.status.Client.set_building")
     commit_sha_mock = mocker.patch("ahriman.core.status.local_client.LocalClient.package_changes_update")
-    depends_on_mock = mocker.patch("ahriman.models.package_archive.PackageArchive.depends_on",
+    depends_on_mock = mocker.patch("ahriman.core.build_tools.package_archive.PackageArchive.depends_on",
                                    return_value=Dependencies())
     dependencies_mock = mocker.patch("ahriman.core.status.local_client.LocalClient.package_dependencies_update")
 

tests/ahriman/core/status/test_local_client.py

@@ -158,7 +158,7 @@ def test_package_remove(local_client: LocalClient, package_ahriman: Package, moc
     """
     package_mock = mocker.patch("ahriman.core.database.SQLite.package_clear")
     local_client.package_remove(package_ahriman.base)
-    package_mock.assert_called_once_with(package_ahriman.base, local_client.repository_id)
+    package_mock.assert_called_once_with(package_ahriman.base)
 
 
 def test_package_status_update(local_client: LocalClient, package_ahriman: Package, mocker: MockerFixture) -> None:

tests/ahriman/core/status/test_watcher.py

@@ -101,11 +101,13 @@ def test_package_remove(watcher: Watcher, package_ahriman: Package, mocker: Mock
     must remove package base
     """
     cache_mock = mocker.patch("ahriman.core.status.local_client.LocalClient.package_remove")
+    logs_mock = mocker.patch("ahriman.core.status.watcher.Watcher.package_logs_remove", create=True)
     watcher._known = {package_ahriman.base: (package_ahriman, BuildStatus())}
 
     watcher.package_remove(package_ahriman.base)
     assert not watcher._known
     cache_mock.assert_called_once_with(package_ahriman.base)
+    logs_mock.assert_called_once_with(package_ahriman.base, None)
 
 
 def test_package_remove_unknown(watcher: Watcher, package_ahriman: Package, mocker: MockerFixture) -> None:

tests/ahriman/models/conftest.py

@@ -1,24 +1,17 @@
 import pytest
 
-from typing import Any
 from unittest.mock import MagicMock, PropertyMock
-from pytest_mock import MockerFixture
 
 from ahriman import __version__
-from ahriman.core.alpm.pacman import Pacman
 from ahriman.core.alpm.remote import AUR
-from ahriman.core.configuration import Configuration
 from ahriman.models.build_status import BuildStatus, BuildStatusEnum
 from ahriman.models.counters import Counters
 from ahriman.models.filesystem_package import FilesystemPackage
 from ahriman.models.internal_status import InternalStatus
 from ahriman.models.package import Package
-from ahriman.models.package_archive import PackageArchive
 from ahriman.models.package_description import PackageDescription
 from ahriman.models.package_source import PackageSource
 from ahriman.models.remote_source import RemoteSource
-from ahriman.models.repository_paths import RepositoryPaths
-from ahriman.models.scan_paths import ScanPaths
 
 
 @pytest.fixture
@@ -77,27 +70,6 @@ def internal_status(counters: Counters) -> InternalStatus:
                           repository="aur-clone")
 
 
-@pytest.fixture
-def package_archive_ahriman(package_ahriman: Package, repository_paths: RepositoryPaths, pacman: Pacman,
-                            scan_paths: ScanPaths, passwd: Any, mocker: MockerFixture) -> PackageArchive:
-    """
-    package archive fixture
-
-    Args:
-        package_ahriman(Package): package test instance
-        repository_paths(RepositoryPaths): repository paths test instance
-        pacman(Pacman): pacman test instance
-        scan_paths(ScanPaths): scan paths test instance
-        passwd(Any): passwd structure test instance
-        mocker(MockerFixture): mocker object
-
-    Returns:
-        PackageArchive: package archive test instance
-    """
-    mocker.patch("ahriman.models.repository_paths.getpwuid", return_value=passwd)
-    return PackageArchive(repository_paths.build_directory, package_ahriman, pacman, scan_paths)
-
-
 @pytest.fixture
 def package_tpacpi_bat_git() -> Package:
     """
@@ -161,20 +133,3 @@ def pyalpm_package_description_ahriman(package_description_ahriman: PackageDescr
     type(mock).provides = PropertyMock(return_value=package_description_ahriman.provides)
     type(mock).url = PropertyMock(return_value=package_description_ahriman.url)
     return mock
-
-
-@pytest.fixture
-def scan_paths(configuration: Configuration) -> ScanPaths:
-    """
-    scan paths fixture
-
-    Args:
-        configuration(Configuration): configuration test instance
-
-    Returns:
-        ScanPaths: scan paths test instance
-    """
-    return ScanPaths(
-        allowed_paths=configuration.getpathlist("build", "allowed_scan_paths"),
-        blacklisted_paths=configuration.getpathlist("build", "blacklisted_scan_paths"),
-    )

tests/ahriman/models/test_repository_id.py

@@ -7,17 +7,10 @@ def test_id() -> None:
     """
     must correctly generate id
     """
+    assert RepositoryId("", "").id == ""
     assert RepositoryId("arch", "repo").id == "arch-repo"
 
 
-def test_id_empty() -> None:
-    """
-    must raise exception on empty identifier
-    """
-    with pytest.raises(ValueError):
-        assert RepositoryId("", "").id
-
-
 def test_is_empty() -> None:
     """
     must check if repository id is empty or not

tests/ahriman/models/test_scan_paths.py

@@ -3,40 +3,30 @@ from pathlib import Path
 from ahriman.models.scan_paths import ScanPaths
 
 
-def test_post_init(scan_paths: ScanPaths) -> None:
-    """
-    must convert paths to / relative
-    """
-    assert all(not path.is_absolute() for path in scan_paths.allowed_paths)
-    assert all(not path.is_absolute() for path in scan_paths.blacklisted_paths)
-
-
 def test_is_allowed() -> None:
     """
     must check if path is subpath of one in allowed list
     """
-    assert ScanPaths(allowed_paths=[Path("/") / "usr"], blacklisted_paths=[]).is_allowed(Path("usr"))
+    assert ScanPaths(["usr"]).is_allowed(Path("usr"))
-    assert ScanPaths(allowed_paths=[Path("/") / "usr"], blacklisted_paths=[]).is_allowed(Path("usr") / "lib")
+    assert ScanPaths(["usr"]).is_allowed(Path("usr") / "lib")
-    assert not ScanPaths(allowed_paths=[Path("/") / "usr"], blacklisted_paths=[]).is_allowed(Path("var"))
+    assert not ScanPaths(["usr"]).is_allowed(Path("var"))
+
+    assert ScanPaths(["usr(?!/lib)"]).is_allowed(Path("usr"))
+    assert ScanPaths(["usr(?!/lib)", "var"]).is_allowed(Path("var"))
+    assert not ScanPaths(["usr(?!/lib)"]).is_allowed(Path("usr") / "lib")
 
 
-def test_is_blacklisted() -> None:
+def test_is_allowed_default(scan_paths: ScanPaths) -> None:
     """
-    must check if path is not subpath of one in blacklist
+    must provide expected default configuration
     """
-    assert ScanPaths(
+    assert not scan_paths.is_allowed(Path("usr"))
-        allowed_paths=[Path("/") / "usr"],
+    assert not scan_paths.is_allowed(Path("var"))
-        blacklisted_paths=[Path("/") / "usr" / "lib"],
+
-    ).is_allowed(Path("usr"))
+    assert scan_paths.is_allowed(Path("usr") / "lib")
-    assert ScanPaths(
+    assert scan_paths.is_allowed(Path("usr") / "lib" / "libm.so")
-        allowed_paths=[Path("/") / "usr", Path("/") / "var"],
+
-        blacklisted_paths=[Path("/") / "usr" / "lib"],
+    # cmake case
-    ).is_allowed(Path("var"))
+    assert scan_paths.is_allowed(Path("usr") / "lib" / "libcmake.so")
-    assert not ScanPaths(
+    assert not scan_paths.is_allowed(Path("usr") / "lib" / "cmake")
-        allowed_paths=[Path("/") / "usr"],
+    assert not scan_paths.is_allowed(Path("usr") / "lib" / "cmake" / "file.cmake")
-        blacklisted_paths=[Path("/") / "usr" / "lib"],
-    ).is_allowed(Path(" usr") / "lib")
-    assert not ScanPaths(
-        allowed_paths=[Path("/") / "usr"],
-        blacklisted_paths=[Path("/") / "usr" / "lib"],
-    ).is_allowed(Path("usr") / "lib" / "qt")

tests/ahriman/web/views/test_view_base.py

@@ -201,7 +201,7 @@ def test_service_not_found(base: BaseView) -> None:
     must raise HTTPNotFound if no repository found
     """
     with pytest.raises(HTTPNotFound):
-        base.service(RepositoryId("repo", "arch"))
+        base.service(RepositoryId("", ""))
 
 
 def test_service_package(base: BaseView, repository_id: RepositoryId, mocker: MockerFixture) -> None:

tests/testresources/core/ahriman.ini

@@ -20,13 +20,12 @@ salt = salt
 allow_read_only = no
 
 [build]
-allowed_scan_paths = /usr/lib
 archbuild_flags =
-blacklisted_scan_paths = /usr/lib/cmake
 build_command = extra-x86_64-build
 ignore_packages =
 makechrootpkg_flags =
 makepkg_flags = --skippgpcheck
+scan_paths = ^usr/lib(?!/cmake).*$
 triggers = ahriman.core.report.ReportTrigger ahriman.core.upload.UploadTrigger
 triggers_known = ahriman.core.distributed.WorkerLoaderTrigger ahriman.core.distributed.WorkerRegisterTrigger ahriman.core.distributed.WorkerTrigger ahriman.core.distributed.WorkerUnregisterTrigger ahriman.core.gitremote.RemotePullTrigger ahriman.core.gitremote.RemotePushTrigger ahriman.core.report.ReportTrigger ahriman.core.upload.UploadTrigger ahriman.core.support.KeyringTrigger ahriman.core.support.MirrorlistTrigger
 

tox.ini

@@ -11,7 +11,6 @@ flags = --implicit-reexport --strict --allow-untyped-decorators --allow-subclass
 
 [pytest]
 addopts = --cov=ahriman --cov-report=term-missing:skip-covered --no-cov-on-fail --cov-fail-under=100 --spec
-asyncio_default_fixture_loop_scope = function
 asyncio_mode = auto
 spec_test_format = {result} {docstring_summary}