Add gitremote triggers (#68)

* add gitremote pull trigger

* add push gitremote trigger

* docs update

docs/ahriman.1

@@ -391,7 +391,7 @@ just perform check for packages without rebuild process itself
 .TP
 \fB\-\-from\-database\fR
 read packages from database instead of filesystem. This feature in particular is required in case if you would like to
-restore repository from another repository instance. Note however that in order to restore packages you need to have
+restore repository from another repository instance. Note, however, that in order to restore packages you need to have
 original ahriman instance run with web service and have run repo\-update at least once.
 
 .TP

docs/ahriman.core.gitremote.rst

@@ -0,0 +1,29 @@
+ahriman.core.gitremote package
+==============================
+
+Submodules
+----------
+
+ahriman.core.gitremote.remote\_pull\_trigger module
+---------------------------------------------------
+
+.. automodule:: ahriman.core.gitremote.remote_pull_trigger
+   :members:
+   :no-undoc-members:
+   :show-inheritance:
+
+ahriman.core.gitremote.remote\_push\_trigger module
+---------------------------------------------------
+
+.. automodule:: ahriman.core.gitremote.remote_push_trigger
+   :members:
+   :no-undoc-members:
+   :show-inheritance:
+
+Module contents
+---------------
+
+.. automodule:: ahriman.core.gitremote
+   :members:
+   :no-undoc-members:
+   :show-inheritance:

docs/ahriman.core.rst

@@ -12,6 +12,7 @@ Subpackages
    ahriman.core.build_tools
    ahriman.core.database
    ahriman.core.formatters
+   ahriman.core.gitremote
    ahriman.core.report
    ahriman.core.repository
    ahriman.core.sign

docs/architecture.rst

@@ -35,12 +35,13 @@ This package contains everything which is required for any time of application r
 * ``ahriman.core.build_tools`` is a package which provides wrapper for ``devtools`` commands.
 * ``ahriman.core.database`` is everything including data and schema migrations for database.
 * ``ahriman.core.formatters`` package provides ``Printer`` sub-classes for printing data (e.g. package properties) to stdout which are used by some handlers.
-* ``ahriman.core.report`` is a package with reporting classes. Usually it must be called by ``ahriman.core.report.Report.load`` method.
+* ``ahriman.core.gitremote`` is a package with remote PKGBUILD triggers. Should not be called directly.
+* ``ahriman.core.report`` is a package with reporting triggers. Should not be called directly.
 * ``ahriman.core.repository`` contains several traits and base repository (``ahriman.core.repository.Repository`` class) implementation.
 * ``ahriman.core.sign`` package provides sign feature (only gpg calls are available).
 * ``ahriman.core.status`` contains helpers and watcher class which are required for web application. Reporter must be initialized by using ``ahriman.core.status.client.Client.load`` method.
 * ``ahriman.core.triggers`` package contains base trigger classes. Classes from this package must be imported in order to implement user extensions. In fact, ``ahriman.core.report`` and ``ahriman.core.upload`` use this package.
-* ``ahriman.core.upload`` package provides sync feature, must be called by ``ahriman.core.upload.Upload.load`` method.
+* ``ahriman.core.upload`` package provides sync feature, should not be called directly.
 
 This package also provides some generic functions and classes which may be used by other packages:
 
@@ -202,12 +203,14 @@ In order to configure users there are special commands.
 Triggers
 ^^^^^^^^
 
-Triggers are extensions which can be used in order to perform any actions after the update process. The package provides two default extensions - one is report generation and another one is remote upload feature.
+Triggers are extensions which can be used in order to perform any actions on application start, after the update process and, finally, before the application exit. The package provides two default extensions - one is report generation and another one is remote upload feature.
 
 The main idea is to load classes by their full path (e.g. ``ahriman.core.upload.UploadTrigger``) by using ``importlib``: get the last part of the import and treat it as class name, join remain part by ``.`` and interpret as module path, import module and extract attribute from it.
 
 The loaded triggers will be called with ``ahriman.models.result.Result`` and ``List[Packages]`` arguments, which describes the process result and current repository packages respectively. Any exception raised will be suppressed and will generate an exception message in logs.
 
+In addition triggers can implement ``on_start`` and ``on_stop`` actions which will be called on the application start and right before the application exit. The ``on_start`` action is usually being called from handlers directly in order to make sure that no trigger will be run when it is not required (e.g. on user management). As soon as ``on_start`` action is called, the additional flag will be set; ``ahriman.core.triggers.TriggerLoader`` class implements ``__del__`` method in which, if the flag is set, the ``on_stop`` actions will be called.
+
 For more details how to deal with the triggers, refer to :doc:`documentation <triggers>` and modules descriptions.
 
 Remote synchronization

docs/configuration.rst

@@ -75,6 +75,18 @@ Settings for signing packages or repository. Group name can refer to architectur
 * ``key`` - default PGP key, string, required. This key will also be used for database signing if enabled.
 * ``key_*`` settings - PGP key which will be used for specific packages, string, optional. For example, if there is ``key_yay`` option the specified key will be used for yay package and default key for others.
 
+``gitremote`` group
+-------------------
+
+Remote git source synchronization settings. Unlike ``Upload`` triggers those triggers are used for PKGBUILD synchronization - e.g. fetch from remote repository PKGBUILDs before updating process or pulling updated PKGBUILDs to the remote repository.
+
+Both urls support authorization; to do so you'd need to prefix the url with authorization part, e.g. ``https://key:token@github.com/arcan1s/ahriman.git``. It is highly recommended to use application tokens instead of your user authorization details.
+
+* ``pull_url`` - url of the remote repository from which PKGBUILDs can be pulled before build process, string, required.
+* ``pull_branch`` - branch of the remote repository from which PKGBUILDs can be pulled before build process, string, optional, default is ``master``.
+* ``push_url`` - url of the remote repository to which PKGBUILDs should be pushed after build process, string, required.
+* ``push_branch`` - branch of the remote repository to which PKGBUILDs should be pushed after build process, string, optional, default is ``master``.
+
 ``report`` group
 ----------------
 

docs/faq.rst

@@ -112,6 +112,53 @@ TL;DR
 
 Before using this command you will need to create local directory, put ``PKGBUILD`` there and generate ``.SRCINFO`` by using ``makepkg --printsrcinfo > .SRCINFO`` command. These packages will be stored locally and *will be ignored* during automatic update; in order to update the package you will need to run ``package-add`` command again.
 
+
+Err, I have remote repository with PKGBUILDs and would like to get versions from there automatically
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+For that purpose you could use ``RemotePullTrigger`` trigger. To do so you will need:
+
+#.
+   Append ``triggers`` option in ``build`` section with the following line:
+
+   .. code-block:: ini
+
+      [build]
+      triggers = ahriman.core.gitremote.RemotePullTrigger
+
+#.
+   Configure trigger as following:
+
+   .. code-block:: ini
+
+      [gitremote]
+      pull_url = https://github.com/username/repository
+
+During the next application run it will fetch repository from the specified url and will try to find packages there which can be used as local sources.
+
+I would like to push PKGBUILDs to the remote repository
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+For that purpose you'd need to use another trigger called ``RemotePushTrigger``. Configure it as following:
+
+#.
+   Append ``triggers`` option in ``build`` section with the trigger name:
+
+   .. code-block:: ini
+
+      [build]
+      triggers = ahriman.core.gitremote.RemotePushTrigger
+
+#.
+   Configure trigger as following:
+
+   .. code-block:: ini
+
+      [gitremote]
+      push_url = https://github.com/username/repository
+
+Unlike ``RemotePullTrigger`` trigger, the ``RemotePushTrigger`` more likely will require authorization. It is highly recommended to use application tokens for that instead of using your password (e.g. for Github you can generate tokens `here <https://github.com/settings/tokens>`_ with scope ``public_repo``). Authorization can be supplied by using authorization part of the url, e.g. ``https://key:token@github.com/username/repository``.
+
 But I just wanted to change PKGBUILD from AUR a bit!
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 

docs/triggers.rst

@@ -1,12 +1,45 @@
 Triggers
 ========
 
-The package provides ability to write custom extensions which will be run on (the most) actions, e.g. after updates. By default ahriman provides two types of extensions - reporting and files uploading. Each extension must derive from the ``ahriman.core.triggers.Trigger`` class and implement ``run`` method
+The package provides ability to write custom extensions which will be run on (the most) actions, e.g. after updates. By default ahriman provides three types of extensions - reporting, files uploading and PKGBUILD syncronization. Each extension must derive from the ``ahriman.core.triggers.Trigger`` class and should implement at least one of the abstract methods:
+
+* ``on_result`` - trigger action which will be called after build process, the build result and the list of repository packages will be supplied as arguments.
+* ``on_start`` - trigger action which will be called right before the start of the application process.
+* ``on_stop`` - action which will be called right before the exit.
+
+Note, it isn't required to implement all of those methods (or even one of them), however, it is highly recommended to avoid trigger actions in ``__init__`` method as it will be run on any application start (e.g. even if you are just searching in AUR).
+
+Built-in triggers
+-----------------
+
+For the configuration details and settings explanation kindly refer to the :doc:`documentation <configuration>`.
+
+``ahriman.core.gitremote.RemotePullTrigger``
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+This trigger will be called before any action (``on_start``) and pulls remote PKGBUILD repository locally; after that it copies found PKGBUILDs from the cloned repository to the local cache. It is useful in case if you have patched PGKBUILDs (or even missing in AUR) which you would like to use for package building and, technically, just simplifies the local package building.
+
+In order to update those packages you would need to clone your repository separately, make changes in PKGBUILD (e.g. bump version and update checksums), commit them and push back. On the next ahriman's repository update, it will pull changes you commited and will perform package update.
+
+``ahriman.core.gitremote.RemotePushTrigger``
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+This trigger will be called right after build process (``on_result``). It will pick PKGBUILDs for the updated packages, pull them (together with any other files) and commit and push changes to remote repository. No real use cases, but the most of user repositories do it.
+
+``ahriman.core.report.ReportTrigger``
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Trigger which can be used for reporting. It implements ``on_result`` method and thus being called on each build update and generates report (e.g. html, telegram etc) according to the current settings.
+
+``ahriman.core.upload.UploadTrigger``
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+This trigger takes build result (``on_result``) and performs syncing of the local packages to the remote mirror (e.g. S3 or just by rsync).
 
 Trigger example
 ---------------
 
-Lets consider example of reporting trigger (e.g. `slack <https://slack.com/>`_, which provides easy HTTP API for integration triggers).
+Lets consider example of reporting trigger (e.g. `slack <https://slack.com/>`_, which provides easy HTTP API for integration triggers).gre
 
 In order to post message to slack we will need a specific trigger url (something like ``https://hooks.slack.com/services/company_id/trigger_id``), channel (e.g. ``#archrepo``) and username (``repo-bot``).
 
@@ -49,12 +82,12 @@ Obviously you can implement the specified method in class, but for guide purpose
            self.channel = configuration.get("slack", "channel")
            self.username = configuration.get("slack", "username")
 
-       def run(self, result, packages):
+       def on_result(self, result, packages):
            notify(result, self.slack_url, self.channel, self.username)
 
 Setup the trigger
------------------
+^^^^^^^^^^^^^^^^^
 
-First, put the trigger in any path it can be exported, e.g. by packing the resource into python package (which will lead to import path as ``package.slack_reporter.SlackReporter``) or just put file somewhere it can be accessed by application (e.g. ``/usr/local/lib/slack_reporter.py.SlackReporter``.
+First, put the trigger in any path it can be exported, e.g. by packing the resource into python package (which will lead to import path as ``package.slack_reporter.SlackReporter``) or just put file somewhere it can be accessed by application (e.g. ``/usr/local/lib/slack_reporter.py.SlackReporter``).
 
 After that run application as usual and receive notification in your slack channel.