feat: add blacklisted paths to implicit dependencies processing

It has been found that in some cases additional packages have been added
as dependencies, like usr/share/applications, usr/lib/cmake, etc

This commit adds an ability to blacklist specific paths from processing

docs/ahriman.models.rst

@@ -228,6 +228,14 @@ ahriman.models.result module
    :no-undoc-members:
    :show-inheritance:
 
+ahriman.models.scan\_paths module
+---------------------------------
+
+.. automodule:: ahriman.models.scan_paths
+   :members:
+   :no-undoc-members:
+   :show-inheritance:
+
 ahriman.models.sign\_settings module
 ------------------------------------
 

docs/configuration.rst

@@ -81,7 +81,9 @@ Authorized users are stored inside internal database, if any of external provide
 
 Build related configuration. Group name can refer to architecture, e.g. ``build:x86_64`` can be used for x86_64 architecture specific settings.
 
+* ``allowed_scan_paths`` - paths to be used for implicit dependencies scan, scape separated list of paths, optional.
 * ``archbuild_flags`` - additional flags passed to ``archbuild`` command, space separated list of strings, optional.
+* ``blacklisted_scan_paths`` - paths to be excluded for implicit dependencies scan, scape separated list of paths, optional. Normally all elements of this option must be child paths of any of ``allowed_scan_paths`` element.
 * ``build_command`` - default build command, string, required.
 * ``ignore_packages`` - list packages to ignore during a regular update (manual update will still work), space separated list of strings, optional.
 * ``include_debug_packages`` - distribute debug packages, boolean, optional, default ``yes``.
@@ -132,7 +134,7 @@ Web server settings. This feature requires ``aiohttp`` libraries to be installed
 * ``port`` - port to bind, integer, optional.
 * ``service_only`` - disable status routes (including logs), boolean, optional, default ``no``.
 * ``static_path`` - path to directory with static files, string, required.
-* ``templates`` - path to templates directories, space separated list of strings, required.
+* ``templates`` - path to templates directories, space separated list of paths, required.
 * ``unix_socket`` - path to the listening unix socket, string, optional. If set, server will create the socket on the specified address which can (and will) be used by application. Note, that unlike usual host/port configuration, unix socket allows to perform requests without authorization.
 * ``unix_socket_unsafe`` - set unsafe (o+w) permissions to unix socket, boolean, optional, default ``yes``. This option is enabled by default, because it is supposed that unix socket is created in safe environment (only web service is supposed to be used in unsafe), but it can be disabled by configuration.
 * ``wait_timeout`` - wait timeout in seconds, maximum amount of time to be waited before lock will be free, integer, optional.
@@ -254,7 +256,7 @@ Section name must be either ``email`` (plus optional architecture name, e.g. ``e
 * ``ssl`` - SSL mode for SMTP connection, one of ``ssl``, ``starttls``, ``disabled``, optional, default ``disabled``.
 * ``template`` - Jinja2 template name, string, required.
 * ``template_full`` - Jinja2 template name for full package description index, string, optional.
-* ``templates`` - path to templates directories, space separated list of strings, required.
+* ``templates`` - path to templates directories, space separated list of paths, required.
 * ``user`` - SMTP user to authenticate, string, optional.
 
 ``html`` type
@@ -267,7 +269,7 @@ Section name must be either ``html`` (plus optional architecture name, e.g. ``ht
 * ``link_path`` - prefix for HTML links, string, required.
 * ``path`` - path to html report file, string, required.
 * ``template`` - Jinja2 template name, string, required.
-* ``templates`` - path to templates directories, space separated list of strings, required.
+* ``templates`` - path to templates directories, space separated list of paths, required.
 
 ``remote-call`` type
 ^^^^^^^^^^^^^^^^^^^^
@@ -292,7 +294,7 @@ Section name must be either ``telegram`` (plus optional architecture name, e.g.
 * ``link_path`` - prefix for HTML links, string, required.
 * ``template`` - Jinja2 template name, string, required.
 * ``template_type`` - ``parse_mode`` to be passed to telegram API, one of ``MarkdownV2``, ``HTML``, ``Markdown``, string, optional, default ``HTML``.
-* ``templates`` - path to templates directories, space separated list of strings, required.
+* ``templates`` - path to templates directories, space separated list of paths, required.
 * ``timeout`` - HTTP request timeout in seconds, integer, optional, default is ``30``.
 
 ``upload`` group

docs/faq/general.rst

@@ -370,7 +370,16 @@ TL;DR
 
 You can even rebuild the whole repository (which is particular useful in case if you would like to change packager) if you do not supply ``--depends-on`` option. This action will automatically increment ``pkgrel`` value; in case if you don't want to, the ``--no-increment`` option has to be supplied.
 
-However, note that you do not need to rebuild repository in case if you just changed signing option, just use ``repo-sign`` command instead. 
+However, note that you do not need to rebuild repository in case if you just changed signing option, just use ``repo-sign`` command instead.
+
+Automated broken dependencies detection
+"""""""""""""""""""""""""""""""""""""""
+
+After the success build the application extracts all linked libraries and used directories and stores them in database. During the check process, the application extracts pacman databases and checks if file names have been changed (e.g. new python release caused ``/usr/lib/python3.x`` directory renaming to ``/usr/lib/python3.y`` or soname for a linked library has been changed). In case if broken dependencies have been detected, the package will be added to the rebuild queue.
+
+In order to disable this check completely, the ``--no-check-files`` flag can be used.
+
+In addition, there is possibility to control paths which will be used for checking, by using options ``build.allowed_scan_paths`` and ``build.blacklisted_scan_paths``. Leaving ``build.allowed_scan_paths`` blank will effectively disable any check too.
 
 How to install built packages
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

package/share/ahriman/settings/ahriman.ini

@@ -50,8 +50,12 @@ allow_read_only = yes
 ;salt =
 
 [build]
+; List of paths to be used for implicit dependency scan
+allowed_scan_paths = /usr/lib
 ; List of additional flags passed to archbuild command.
 ;archbuild_flags =
+; List of paths to be excluded for implicit dependency scan. Usually they should be subpaths of allowed_scan_paths
+blacklisted_scan_paths = /usr/lib/cmake
 ; Path to build command
 ;build_command =
 ; List of packages to be ignored during automatic updates.

src/ahriman/core/configuration/schema.py

@@ -169,6 +169,14 @@ CONFIGURATION_SCHEMA: ConfigurationSchema = {
     "build": {
         "type": "dict",
         "schema": {
+            "allowed_scan_paths": {
+                "type": "list",
+                "coerce": "list",
+                "schema": {
+                    "type": "path",
+                    "coerce": "absolute_path",
+                },
+            },
             "archbuild_flags": {
                 "type": "list",
                 "coerce": "list",
@@ -177,6 +185,14 @@ CONFIGURATION_SCHEMA: ConfigurationSchema = {
                     "empty": False,
                 },
             },
+            "blacklisted_scan_paths": {
+                "type": "list",
+                "coerce": "list",
+                "schema": {
+                    "type": "path",
+                    "coerce": "absolute_path",
+                },
+            },
             "build_command": {
                 "type": "string",
                 "required": True,

src/ahriman/core/repository/executor.py

@@ -80,7 +80,8 @@ class Executor(PackageInfo, Cleaner):
                     # clear changes and update commit hash
                     self.reporter.package_changes_update(single.base, Changes(last_commit_sha))
                     # update dependencies list
-                    dependencies = PackageArchive(self.paths.build_directory, single, self.pacman).depends_on()
+                    package_archive = PackageArchive(self.paths.build_directory, single, self.pacman, self.scan_paths)
+                    dependencies = package_archive.depends_on()
                     self.reporter.package_dependencies_update(single.base, dependencies)
                     # update result set
                     result.add_updated(single)

src/ahriman/core/repository/repository_properties.py

@@ -29,6 +29,7 @@ from ahriman.models.packagers import Packagers
 from ahriman.models.pacman_synchronization import PacmanSynchronization
 from ahriman.models.repository_id import RepositoryId
 from ahriman.models.repository_paths import RepositoryPaths
+from ahriman.models.scan_paths import ScanPaths
 from ahriman.models.user import User
 from ahriman.models.user_access import UserAccess
 
@@ -46,6 +47,7 @@ class RepositoryProperties(LazyLogging):
         repo(Repo): repo commands wrapper instance
         reporter(Client): build status reporter instance
         repository_id(RepositoryId): repository unique identifier
+        scan_paths(ScanPaths): scan paths for the implicit dependencies
         sign(GPG): GPG wrapper instance
         triggers(TriggerLoader): triggers holder
         vcs_allowed_age(int): maximal age of the VCS packages before they will be checked
@@ -78,6 +80,11 @@ class RepositoryProperties(LazyLogging):
         self.reporter = Client.load(repository_id, configuration, database, report=report)
         self.triggers = TriggerLoader.load(repository_id, configuration)
 
+        self.scan_paths = ScanPaths(
+            allowed_paths=configuration.getpathlist("build", "allowed_scan_paths", fallback=[]),
+            blacklisted_paths=configuration.getpathlist("build", "blacklisted_scan_paths", fallback=[]),
+        )
+
     @property
     def architecture(self) -> str:
         """

src/ahriman/models/package_archive.py

@@ -30,6 +30,7 @@ from ahriman.core.utils import walk
 from ahriman.models.dependencies import Dependencies
 from ahriman.models.filesystem_package import FilesystemPackage
 from ahriman.models.package import Package
+from ahriman.models.scan_paths import ScanPaths
 
 
 @dataclass
@@ -39,13 +40,15 @@ class PackageArchive:
 
     Attributes:
         package(Package): package descriptor
-        root(Path): path to root filesystem
         pacman(Pacman): alpm wrapper instance
+        root(Path): path to root filesystem
+        scan_paths(ScanPaths): scan paths holder
     """
 
     root: Path
     package: Package
     pacman: Pacman
+    scan_paths: ScanPaths
 
     @staticmethod
     def dynamic_needed(binary_path: Path) -> list[str]:
@@ -165,6 +168,10 @@ class PackageArchive:
             if any(package.package_name in base_packages for package in packages):
                 continue
 
+            # check path against the black/white listed
+            if not self.scan_paths.is_allowed(path):
+                continue
+
             # remove explicit dependencies
             packages = [package for package in packages if package.is_root_package(packages, include_optional=False)]
             # remove optional dependencies

src/ahriman/models/scan_paths.py

@@ -0,0 +1,58 @@
+#
+# Copyright (c) 2021-2024 ahriman team.
+#
+# This file is part of ahriman
+# (see https://github.com/arcan1s/ahriman).
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+from dataclasses import dataclass
+from pathlib import Path
+
+
+@dataclass(frozen=True, kw_only=True)
+class ScanPaths:
+    """
+    paths used for scan filesystem
+
+    Attributes:
+        allowed_paths(list[Path]): list of whitelisted paths
+        blacklisted_paths(list[Path]): list of paths to be skipped from scan
+    """
+
+    allowed_paths: list[Path]
+    blacklisted_paths: list[Path]
+
+    def __post_init__(self) -> None:
+        """
+        compute relative to / paths
+        """
+        object.__setattr__(self, "allowed_paths", [path.relative_to("/") for path in self.allowed_paths])
+        object.__setattr__(self, "blacklisted_paths", [path.relative_to("/") for path in self.blacklisted_paths])
+
+    def is_allowed(self, path: Path) -> bool:
+        """
+        check whether path is allowed to scan or not
+
+        Args:
+            path(Path): path to be checked
+
+        Returns:
+            bool: ``True`` in case if :attr:`allowed_paths` contains element which is parent for the path and
+            :attr:`blacklisted_paths` doesn't and ``False`` otherwise
+        """
+        if any(path.is_relative_to(blacklisted) for blacklisted in self.blacklisted_paths):
+            return False  # path is blacklisted
+        # check if we actually have to check this path
+        return any(path.is_relative_to(allowed) for allowed in self.allowed_paths)

tests/ahriman/models/conftest.py

@@ -7,6 +7,7 @@ from pytest_mock import MockerFixture
 from ahriman import __version__
 from ahriman.core.alpm.pacman import Pacman
 from ahriman.core.alpm.remote import AUR
+from ahriman.core.configuration import Configuration
 from ahriman.models.build_status import BuildStatus, BuildStatusEnum
 from ahriman.models.counters import Counters
 from ahriman.models.filesystem_package import FilesystemPackage
@@ -17,6 +18,7 @@ from ahriman.models.package_description import PackageDescription
 from ahriman.models.package_source import PackageSource
 from ahriman.models.remote_source import RemoteSource
 from ahriman.models.repository_paths import RepositoryPaths
+from ahriman.models.scan_paths import ScanPaths
 
 
 @pytest.fixture
@@ -77,7 +79,7 @@ def internal_status(counters: Counters) -> InternalStatus:
 
 @pytest.fixture
 def package_archive_ahriman(package_ahriman: Package, repository_paths: RepositoryPaths, pacman: Pacman,
-                            passwd: Any, mocker: MockerFixture) -> PackageArchive:
+                            scan_paths: ScanPaths, passwd: Any, mocker: MockerFixture) -> PackageArchive:
     """
     package archive fixture
 
@@ -85,6 +87,7 @@ def package_archive_ahriman(package_ahriman: Package, repository_paths: Reposito
         package_ahriman(Package): package test instance
         repository_paths(RepositoryPaths): repository paths test instance
         pacman(Pacman): pacman test instance
+        scan_paths(ScanPaths): scan paths test instance
         passwd(Any): passwd structure test instance
         mocker(MockerFixture): mocker object
 
@@ -92,7 +95,7 @@ def package_archive_ahriman(package_ahriman: Package, repository_paths: Reposito
         PackageArchive: package archive test instance
     """
     mocker.patch("ahriman.models.repository_paths.getpwuid", return_value=passwd)
-    return PackageArchive(repository_paths.build_directory, package_ahriman, pacman)
+    return PackageArchive(repository_paths.build_directory, package_ahriman, pacman, scan_paths)
 
 
 @pytest.fixture
@@ -158,3 +161,20 @@ def pyalpm_package_description_ahriman(package_description_ahriman: PackageDescr
     type(mock).provides = PropertyMock(return_value=package_description_ahriman.provides)
     type(mock).url = PropertyMock(return_value=package_description_ahriman.url)
     return mock
+
+
+@pytest.fixture
+def scan_paths(configuration: Configuration) -> ScanPaths:
+    """
+    scan paths fixture
+
+    Args:
+        configuration(Configuration): configuration test instance
+
+    Returns:
+        ScanPaths: scan paths test instance
+    """
+    return ScanPaths(
+        allowed_paths=configuration.getpathlist("build", "allowed_scan_paths"),
+        blacklisted_paths=configuration.getpathlist("build", "blacklisted_scan_paths"),
+    )

tests/ahriman/models/test_package_archive.py

@@ -134,8 +134,10 @@ def test_refine_dependencies(package_archive_ahriman: PackageArchive, mocker: Mo
 
     path1 = Path("usr") / "lib" / "python3.12"
     path2 = path1 / "site-packages"
-    path3 = Path("etc")
-    path4 = Path("var") / "lib" / "whatever"
+    path3 = Path("usr") / "lib" / "path"
+    path4 = Path("usr") / "lib" / "whatever"
+    path5 = Path("usr") / "share" / "applications"
+    path6 = Path("etc")
 
     package1 = FilesystemPackage(package_name="package1", depends={"package5"}, opt_depends={"package2"})
     package2 = FilesystemPackage(package_name="package2", depends={"package1"}, opt_depends=set())
@@ -149,6 +151,8 @@ def test_refine_dependencies(package_archive_ahriman: PackageArchive, mocker: Mo
         path2: [package1, package2, package3, package5],
         path3: [package1, package4],
         path4: [package1],
+        path5: [package1],
+        path6: [package1],
     }) == {
         path1: [package6],
         path2: [package1, package5],

tests/ahriman/models/test_scan_paths.py

@@ -0,0 +1,42 @@
+from pathlib import Path
+
+from ahriman.models.scan_paths import ScanPaths
+
+
+def test_post_init(scan_paths: ScanPaths) -> None:
+    """
+    must convert paths to / relative
+    """
+    assert all(not path.is_absolute() for path in scan_paths.allowed_paths)
+    assert all(not path.is_absolute() for path in scan_paths.blacklisted_paths)
+
+
+def test_is_allowed() -> None:
+    """
+    must check if path is subpath of one in allowed list
+    """
+    assert ScanPaths(allowed_paths=[Path("/") / "usr"], blacklisted_paths=[]).is_allowed(Path("usr"))
+    assert ScanPaths(allowed_paths=[Path("/") / "usr"], blacklisted_paths=[]).is_allowed(Path("usr") / "lib")
+    assert not ScanPaths(allowed_paths=[Path("/") / "usr"], blacklisted_paths=[]).is_allowed(Path("var"))
+
+
+def test_is_blacklisted() -> None:
+    """
+    must check if path is not subpath of one in blacklist
+    """
+    assert ScanPaths(
+        allowed_paths=[Path("/") / "usr"],
+        blacklisted_paths=[Path("/") / "usr" / "lib"],
+    ).is_allowed(Path("usr"))
+    assert ScanPaths(
+        allowed_paths=[Path("/") / "usr", Path("/") / "var"],
+        blacklisted_paths=[Path("/") / "usr" / "lib"],
+    ).is_allowed(Path("var"))
+    assert not ScanPaths(
+        allowed_paths=[Path("/") / "usr"],
+        blacklisted_paths=[Path("/") / "usr" / "lib"],
+    ).is_allowed(Path(" usr") / "lib")
+    assert not ScanPaths(
+        allowed_paths=[Path("/") / "usr"],
+        blacklisted_paths=[Path("/") / "usr" / "lib"],
+    ).is_allowed(Path("usr") / "lib" / "qt")

tests/testresources/core/ahriman.ini

@@ -20,7 +20,9 @@ salt = salt
 allow_read_only = no
 
 [build]
+allowed_scan_paths = /usr/lib
 archbuild_flags =
+blacklisted_scan_paths = /usr/lib/cmake
 build_command = extra-x86_64-build
 ignore_packages =
 makechrootpkg_flags =