diff --git a/README.md b/README.md index 436d0ff8..3f9a3072 100644 --- a/README.md +++ b/README.md @@ -13,8 +13,9 @@ Wrapper for managing custom repository inspired by [repo-scripts](https://github * Install-configure-forget manager for own repository. * Multi-architecture support. * VCS packages support. +* Official repository support. * Sign support with gpg (repository, package, per package settings). -* Synchronization to remote services (rsync, s3 and github) and report generation (email, html, telegram) and even ability to write own extensions. +* Triggers for repository updates, e.g. synchronization to remote services (rsync, s3 and github) and report generation (email, html, telegram). * Dependency manager. * Ability to patch AUR packages and even create package from local PKGBUILDs. * Repository status interface with optional authorization and control options: diff --git a/docs/advanced-usage.rst b/docs/advanced-usage.rst index 617bca4b..b431fcb3 100644 --- a/docs/advanced-usage.rst +++ b/docs/advanced-usage.rst @@ -37,7 +37,6 @@ And the ``repository`` instance can be used to perform repository maintenance built_packages = repository.packages_built() update_result = repository.process_update(built_packages) - repository.process_report(None, update_result) - repository.process_sync(None, update_result.success) + repository.process_triggers(update_result) For the more info please refer to the classes documentation. diff --git a/docs/architecture.rst b/docs/architecture.rst index b6cdaa51..07b7b62c 100644 --- a/docs/architecture.rst +++ b/docs/architecture.rst @@ -19,9 +19,9 @@ Full dependency diagram: ``ahriman.application`` package ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -This package contains application (aka executable) related classes and everything for that. It also contains package called ``ahriman.application.handlers`` in which all available subcommands are described as separated classes derived from base ``ahriman.application.handlers.handler.Handler`` class. +This package contains application (aka executable) related classes and everything for that. It also contains package called ``ahriman.application.handlers`` in which all available subcommands are described as separated classes derived from base ``ahriman.application.handlers.Handler`` class. -``ahriman.application.application.application.Application`` (god class) is used for any interaction from parsers with repository, web etc. It is divided into multiple traits by functions (package related and repository related) in the same package. +``ahriman.application.application.Application`` (god class) is used for any interaction from parsers with repository, web etc. It is divided into multiple traits by functions (package related and repository related) in the same package. ``ahriman.application.ahriman`` contains only command line parses and executes specified ``Handler`` on success, ``ahriman.application.lock.Lock`` is additional class which provides file-based lock and also performs some common checks. @@ -31,15 +31,16 @@ This package contains application (aka executable) related classes and everythin This package contains everything which is required for any time of application run and separated to several packages: * ``ahriman.core.alpm`` package controls pacman related functions. It provides wrappers for ``pyalpm`` library and safe calls for repository tools (``repo-add`` and ``repo-remove``). Also this package contains ``ahriman.core.alpm.remote`` package which provides wrapper for remote sources (e.g. AUR RPC and official repositories RPC). -* ``ahriman.core.auth`` package provides classes for authorization methods used by web mostly. Base class is ``ahriman.core.auth.auth.Auth`` which must be called by ``load`` method. +* ``ahriman.core.auth`` package provides classes for authorization methods used by web mostly. Base class is ``ahriman.core.auth.Auth`` which must be called by ``load`` method. * ``ahriman.core.build_tools`` is a package which provides wrapper for ``devtools`` commands. * ``ahriman.core.database`` is everything including data and schema migrations for database. * ``ahriman.core.formatters`` package provides ``Printer`` sub-classes for printing data (e.g. package properties) to stdout which are used by some handlers. -* ``ahriman.core.report`` is a package with reporting classes. Usually it must be called by ``ahriman.core.report.report.Report.load`` method. -* ``ahriman.core.repository`` contains several traits and base repository (``ahriman.core.repository.repository.Repository`` class) implementation. +* ``ahriman.core.report`` is a package with reporting classes. Usually it must be called by ``ahriman.core.report.Report.load`` method. +* ``ahriman.core.repository`` contains several traits and base repository (``ahriman.core.repository.Repository`` class) implementation. * ``ahriman.core.sign`` package provides sign feature (only gpg calls are available). * ``ahriman.core.status`` contains helpers and watcher class which are required for web application. Reporter must be initialized by using ``ahriman.core.status.client.Client.load`` method. -* ``ahriman.core.upload`` package provides sync feature, must be called by ``ahriman.core.upload.upload.Upload.load`` method. +* ``ahriman.core.triggers`` package contains base trigger classes. Classes from this package must be imported in order to implement user extensions. In fact, ``ahriman.core.report`` and ``ahriman.core.upload`` uses this package. +* ``ahriman.core.upload`` package provides sync feature, must be called by ``ahriman.core.upload.Upload.load`` method. This package also provides some generic functions and classes which may be used by other packages: @@ -86,14 +87,14 @@ The service uses SQLite database in order to store some internal info. Database instance ^^^^^^^^^^^^^^^^^ -All methods related to specific part of database (basically operations per table) are split into different traits located inside ``ahriman.core.database.operations`` package. The base trait ``ahriman.core.database.operations.operations.Operations`` also provides generic methods for database access (e.g. row converters and transactional support). +All methods related to specific part of database (basically operations per table) are split into different traits located inside ``ahriman.core.database.operations`` package. The base trait ``ahriman.core.database.operations.Operations`` also provides generic methods for database access (e.g. row converters and transactional support). -The ``ahriman.core.database.sqlite.SQLite`` class itself derives from all of these traits and implements methods for initialization, including migrations. +The ``ahriman.core.database.SQLite`` class itself derives from all of these traits and implements methods for initialization, including migrations. Schema and data migrations ^^^^^^^^^^^^^^^^^^^^^^^^^^ -The schema migration are applied according to current ``pragma user_info`` values, located at ``ahriman.core.database.migrations`` package and named as ``m000_migration_name.py`` (the preceding ``m`` is required in order to import migration content for tests). Additional class ``ahriman.core.database.migrations.Migrations`` reads all migrations autmatically and applies them in alphabetical order. +The schema migration are applied according to current ``pragma user_info`` values, located at ``ahriman.core.database.migrations`` package and named as ``m000_migration_name.py`` (the preceding ``m`` is required in order to import migration content for tests). Additional class ``ahriman.core.database.migrations.Migrations`` reads all migrations automatically and applies them in alphabetical order. There are also data migrations which are located at ``ahriman.core.database.data`` package and move data from old-style (e.g. json files in filesystem, directory trees, etc) to the database. They are also part of migration and (unlike schema migrations) are applied only at specific version breakpoints (e.g. if ``user_version`` is more than 0 no initial migration will be applied). @@ -154,6 +155,13 @@ Configuration ``ahriman.core.configuration.Configuration`` class provides some additional methods (e.g. ``getpath`` and ``getlist``) and also combines multiple files into single configuration dictionary using architecture overrides. It is the recommended way to deal with settings. +Enumerations +^^^^^^^^^^^^ + +All enumerations are derived from ``str`` and ``enum.Enum``. Integer enumerations are not allowed, because most of operations require conversions from string variable. Derivation from string class is required to make json conversions implicitly (e.g. during calling ``json.dumps`` methods). + +In addition, some enumerations provide ``from_option`` class methods in order to allow some flexibility while reading configuration options. + Utils ^^^^^ @@ -262,7 +270,7 @@ Requests and scopes Service provides optional authorization which can be turned on in settings. In order to control user access there are two levels of authorization - read-only (only GET-like requests) and write (anything) which are provided by each web view directly. -If this feature is configured any request will be prohibited without authentication. In addition, configuration flag ``auth.safe_build_status`` can be used in order to allow seeing main page without authorization. +If this feature is configured any request will be prohibited without authentication. In addition, configuration flag ``auth.allow_read_only`` can be used in order to allow read-only operations - reading index page and packages - without authorization. For authenticated users it uses encrypted session cookies to store tokens; encryption key is generated each time at the start of the application. It also stores expiration time of the session inside. diff --git a/docs/index.rst b/docs/index.rst index 4ae90145..f8adc7c4 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -9,8 +9,9 @@ Features * Install-configure-forget manager for own repository. * Multi-architecture support. * VCS packages support. +* Official repository support. * Sign support with gpg (repository, package, per package settings). -* Synchronization to remote services (rsync, s3 and github) and report generation (email, html, telegram) and even ability to write own extensions. +* Triggers for repository updates, e.g. synchronization to remote services (rsync, s3 and github) and report generation (email, html, telegram). * Dependency manager. * Ability to patch AUR packages and even create package from local PKGBUILDs. * Repository status interface with optional authorization and control options. diff --git a/package/archlinux/PKGBUILD b/package/archlinux/PKGBUILD index 54335f23..053b338b 100644 --- a/package/archlinux/PKGBUILD +++ b/package/archlinux/PKGBUILD @@ -3,7 +3,7 @@ pkgname='ahriman' pkgver=2.0.0rc11 pkgrel=1 -pkgdesc="ArcH Linux ReposItory MANager" +pkgdesc="ArcH linux ReposItory MANager" arch=('any') url="https://github.com/arcan1s/ahriman" license=('GPL3') diff --git a/package/archlinux/ahriman.sysusers b/package/archlinux/ahriman.sysusers index 6fd64e0a..d86820a8 100644 --- a/package/archlinux/ahriman.sysusers +++ b/package/archlinux/ahriman.sysusers @@ -1 +1 @@ -u ahriman 643 "ArcH Linux ReposItory MANager" /var/lib/ahriman \ No newline at end of file +u ahriman 643 "ArcH linux ReposItory MANager" /var/lib/ahriman \ No newline at end of file diff --git a/package/lib/systemd/system/ahriman-web@.service b/package/lib/systemd/system/ahriman-web@.service index d720b864..d135cfc7 100644 --- a/package/lib/systemd/system/ahriman-web@.service +++ b/package/lib/systemd/system/ahriman-web@.service @@ -1,5 +1,5 @@ [Unit] -Description=ArcH Linux ReposItory MANager web server (%I architecture) +Description=ArcH linux ReposItory MANager web server (%I architecture) After=network.target [Service] diff --git a/package/lib/systemd/system/ahriman@.service b/package/lib/systemd/system/ahriman@.service index 6bb84407..69aac3d9 100644 --- a/package/lib/systemd/system/ahriman@.service +++ b/package/lib/systemd/system/ahriman@.service @@ -1,5 +1,5 @@ [Unit] -Description=ArcH Linux ReposItory MANager (%I architecture) +Description=ArcH linux ReposItory MANager (%I architecture) [Service] ExecStart=/usr/bin/ahriman --architecture %i update diff --git a/package/lib/systemd/system/ahriman@.timer b/package/lib/systemd/system/ahriman@.timer index 653b2ee1..61164a83 100644 --- a/package/lib/systemd/system/ahriman@.timer +++ b/package/lib/systemd/system/ahriman@.timer @@ -1,5 +1,5 @@ [Unit] -Description=ArcH Linux ReposItory MANager timer (%I architecture) +Description=ArcH linux ReposItory MANager timer (%I architecture) [Timer] OnCalendar=daily diff --git a/setup.py b/setup.py index 50546948..772b9b6b 100644 --- a/setup.py +++ b/setup.py @@ -1,5 +1,5 @@ from pathlib import Path -from setuptools import setup, find_packages +from setuptools import find_packages, setup from typing import Any, Dict @@ -15,7 +15,7 @@ setup( version=metadata["__version__"], zip_safe=False, - description="ArcH Linux ReposItory MANager", + description="ArcH linux ReposItory MANager", author="ahriman team", author_email="", diff --git a/src/ahriman/application/ahriman.py b/src/ahriman/application/ahriman.py index f8705e99..f7c4de67 100644 --- a/src/ahriman/application/ahriman.py +++ b/src/ahriman/application/ahriman.py @@ -63,7 +63,7 @@ def _parser() -> argparse.ArgumentParser: Returns: argparse.ArgumentParser: command line parser for the application """ - parser = argparse.ArgumentParser(prog="ahriman", description="ArcH Linux ReposItory MANager", + parser = argparse.ArgumentParser(prog="ahriman", description="ArcH linux ReposItory MANager", epilog="Argument list can also be read from file by using @ prefix.", fromfile_prefix_chars="@", formatter_class=_formatter) parser.add_argument("-a", "--architecture", help="target architectures (can be used multiple times)", diff --git a/src/ahriman/core/database/data/package_remotes.py b/src/ahriman/core/database/data/package_remotes.py index d7cc7d4c..8b4da113 100644 --- a/src/ahriman/core/database/data/package_remotes.py +++ b/src/ahriman/core/database/data/package_remotes.py @@ -36,7 +36,7 @@ def migrate_package_remotes(connection: Connection, paths: RepositoryPaths) -> N connection(Connection): database connection paths(RepositoryPaths): repository paths instance """ - from ahriman.core.database.operations.package_operations import PackageOperations + from ahriman.core.database.operations import PackageOperations def insert_remote(base: str, remote: RemoteSource) -> None: connection.execute( diff --git a/src/ahriman/core/database/operations/__init__.py b/src/ahriman/core/database/operations/__init__.py index 680676f1..88d69f53 100644 --- a/src/ahriman/core/database/operations/__init__.py +++ b/src/ahriman/core/database/operations/__init__.py @@ -17,3 +17,9 @@ # You should have received a copy of the GNU General Public License # along with this program. If not, see . # +from ahriman.core.database.operations.operations import Operations + +from ahriman.core.database.operations.auth_operations import AuthOperations +from ahriman.core.database.operations.build_operations import BuildOperations +from ahriman.core.database.operations.package_operations import PackageOperations +from ahriman.core.database.operations.patch_operations import PatchOperations diff --git a/src/ahriman/core/database/operations/auth_operations.py b/src/ahriman/core/database/operations/auth_operations.py index 86454003..094291e3 100644 --- a/src/ahriman/core/database/operations/auth_operations.py +++ b/src/ahriman/core/database/operations/auth_operations.py @@ -20,7 +20,7 @@ from sqlite3 import Connection from typing import List, Optional -from ahriman.core.database.operations.operations import Operations +from ahriman.core.database.operations import Operations from ahriman.models.user import User from ahriman.models.user_access import UserAccess diff --git a/src/ahriman/core/database/operations/build_operations.py b/src/ahriman/core/database/operations/build_operations.py index 65d2bcd7..17e6904e 100644 --- a/src/ahriman/core/database/operations/build_operations.py +++ b/src/ahriman/core/database/operations/build_operations.py @@ -20,7 +20,7 @@ from sqlite3 import Connection from typing import List, Optional -from ahriman.core.database.operations.operations import Operations +from ahriman.core.database.operations import Operations from ahriman.models.package import Package diff --git a/src/ahriman/core/database/operations/package_operations.py b/src/ahriman/core/database/operations/package_operations.py index 5d52f254..d8498e14 100644 --- a/src/ahriman/core/database/operations/package_operations.py +++ b/src/ahriman/core/database/operations/package_operations.py @@ -20,7 +20,7 @@ from sqlite3 import Connection from typing import Dict, Generator, Iterable, List, Tuple -from ahriman.core.database.operations.operations import Operations +from ahriman.core.database.operations import Operations from ahriman.models.build_status import BuildStatus from ahriman.models.package import Package from ahriman.models.package_description import PackageDescription diff --git a/src/ahriman/core/database/operations/patch_operations.py b/src/ahriman/core/database/operations/patch_operations.py index c18d569e..9dcebfe4 100644 --- a/src/ahriman/core/database/operations/patch_operations.py +++ b/src/ahriman/core/database/operations/patch_operations.py @@ -20,7 +20,7 @@ from sqlite3 import Connection from typing import Dict, Optional -from ahriman.core.database.operations.operations import Operations +from ahriman.core.database.operations import Operations class PatchOperations(Operations): diff --git a/src/ahriman/core/database/sqlite.py b/src/ahriman/core/database/sqlite.py index 42bc36cf..842941cb 100644 --- a/src/ahriman/core/database/sqlite.py +++ b/src/ahriman/core/database/sqlite.py @@ -27,10 +27,7 @@ from typing import Type from ahriman.core.configuration import Configuration from ahriman.core.database.migrations import Migrations -from ahriman.core.database.operations.auth_operations import AuthOperations -from ahriman.core.database.operations.build_operations import BuildOperations -from ahriman.core.database.operations.package_operations import PackageOperations -from ahriman.core.database.operations.patch_operations import PatchOperations +from ahriman.core.database.operations import AuthOperations, BuildOperations, PackageOperations, PatchOperations class SQLite(AuthOperations, BuildOperations, PackageOperations, PatchOperations): diff --git a/tests/ahriman/conftest.py b/tests/ahriman/conftest.py index 8b431659..19368c93 100644 --- a/tests/ahriman/conftest.py +++ b/tests/ahriman/conftest.py @@ -101,7 +101,7 @@ def aur_package_ahriman() -> AURPackage: package_base_id=165427, package_base="ahriman", version="1.7.0-1", - description="ArcH Linux ReposItory MANager", + description="ArcH linux ReposItory MANager", num_votes=0, popularity=0, first_submitted=datetime.datetime.utcfromtimestamp(1618008285), @@ -299,7 +299,7 @@ def package_description_ahriman() -> PackageDescription: "python-passlib", "python-srcinfo", ], - description="ArcH Linux ReposItory MANager", + description="ArcH linux ReposItory MANager", filename="ahriman-1.7.0-1-any.pkg.tar.zst", groups=[], installed_size=4200000, diff --git a/tests/ahriman/core/database/data/test_package_remotes.py b/tests/ahriman/core/database/data/test_package_remotes.py index 280e9a79..51a6fe25 100644 --- a/tests/ahriman/core/database/data/test_package_remotes.py +++ b/tests/ahriman/core/database/data/test_package_remotes.py @@ -14,7 +14,7 @@ def test_migrate_package_remotes(package_ahriman: Package, connection: Connectio must put package remotes to database """ mocker.patch( - "ahriman.core.database.operations.package_operations.PackageOperations._packages_get_select_package_bases", + "ahriman.core.database.operations.PackageOperations._packages_get_select_package_bases", return_value={package_ahriman.base: package_ahriman}) mocker.patch("pathlib.Path.exists", return_value=False) @@ -28,7 +28,7 @@ def test_migrate_package_remotes_has_local(package_ahriman: Package, connection: must skip processing for packages which have local cache """ mocker.patch( - "ahriman.core.database.operations.package_operations.PackageOperations._packages_get_select_package_bases", + "ahriman.core.database.operations.PackageOperations._packages_get_select_package_bases", return_value={package_ahriman.base: package_ahriman}) mocker.patch("pathlib.Path.exists", return_value=True) @@ -42,7 +42,7 @@ def test_migrate_package_remotes_vcs(package_ahriman: Package, connection: Conne must process VCS packages with local cache """ mocker.patch( - "ahriman.core.database.operations.package_operations.PackageOperations._packages_get_select_package_bases", + "ahriman.core.database.operations.PackageOperations._packages_get_select_package_bases", return_value={package_ahriman.base: package_ahriman}) mocker.patch("pathlib.Path.exists", return_value=True) mocker.patch.object(Package, "is_vcs", True) @@ -57,7 +57,7 @@ def test_migrate_package_remotes_no_remotes(package_ahriman: Package, connection must skip processing in case if no remotes generated (should never happen) """ mocker.patch( - "ahriman.core.database.operations.package_operations.PackageOperations._packages_get_select_package_bases", + "ahriman.core.database.operations.PackageOperations._packages_get_select_package_bases", return_value={package_ahriman.base: package_ahriman}) mocker.patch("pathlib.Path.exists", return_value=False) mocker.patch("ahriman.models.remote_source.RemoteSource.from_source", return_value=None) diff --git a/tests/ahriman/core/upload/test_http_upload.py b/tests/ahriman/core/upload/test_http_upload.py index e332d10d..c6bbb9b6 100644 --- a/tests/ahriman/core/upload/test_http_upload.py +++ b/tests/ahriman/core/upload/test_http_upload.py @@ -21,7 +21,7 @@ def test_calculate_hash_small(resource_path_root: Path) -> None: must calculate checksum for path which is single chunk """ path = resource_path_root / "models" / "package_ahriman_srcinfo" - assert HttpUpload.calculate_hash(path) == "c0aaf6ebf95ca9206dc8ba1d8ff10af3" + assert HttpUpload.calculate_hash(path) == "fcfc0f2522b0ee92de89fcedc7e56010" def test_get_body_get_hashes() -> None: diff --git a/tests/ahriman/core/upload/test_s3.py b/tests/ahriman/core/upload/test_s3.py index f9d71469..0480909c 100644 --- a/tests/ahriman/core/upload/test_s3.py +++ b/tests/ahriman/core/upload/test_s3.py @@ -31,7 +31,7 @@ def test_calculate_etag_small(resource_path_root: Path) -> None: must calculate checksum for path which is single chunk """ path = resource_path_root / "models" / "package_ahriman_srcinfo" - assert S3.calculate_etag(path, _chunk_size) == "c0aaf6ebf95ca9206dc8ba1d8ff10af3" + assert S3.calculate_etag(path, _chunk_size) == "fcfc0f2522b0ee92de89fcedc7e56010" def test_files_remove(s3_remote_objects: List[Any]) -> None: diff --git a/tests/testresources/models/package_ahriman_aur b/tests/testresources/models/package_ahriman_aur index 585df47a..cf5bf80a 100644 --- a/tests/testresources/models/package_ahriman_aur +++ b/tests/testresources/models/package_ahriman_aur @@ -10,7 +10,7 @@ "python-passlib", "python-srcinfo" ], - "Description": "ArcH Linux ReposItory MANager", + "Description": "ArcH linux ReposItory MANager", "FirstSubmitted": 1618008285, "ID": 1009791, "Keywords": [], diff --git a/tests/testresources/models/package_ahriman_srcinfo b/tests/testresources/models/package_ahriman_srcinfo index 2529d2ef..4a6c8b45 100644 --- a/tests/testresources/models/package_ahriman_srcinfo +++ b/tests/testresources/models/package_ahriman_srcinfo @@ -1,5 +1,5 @@ pkgbase = ahriman - pkgdesc = ArcH Linux ReposItory MANager + pkgdesc = ArcH linux ReposItory MANager pkgver = 1.7.0 pkgrel = 1 url = https://github.com/arcan1s/ahriman