feat: add workers autodicsovery feature (#121)

* add workers autodicsovery feature

* suppress erros while retrieving worker list

* update recipes

* fix tests and update docs

* filter health checks

* ping based workers

docs/ahriman.core.distributed.rst

@@ -0,0 +1,45 @@
+ahriman.core.distributed package
+================================
+
+Submodules
+----------
+
+ahriman.core.distributed.distributed\_system module
+---------------------------------------------------
+
+.. automodule:: ahriman.core.distributed.distributed_system
+   :members:
+   :no-undoc-members:
+   :show-inheritance:
+
+ahriman.core.distributed.worker\_loader\_trigger module
+-------------------------------------------------------
+
+.. automodule:: ahriman.core.distributed.worker_loader_trigger
+   :members:
+   :no-undoc-members:
+   :show-inheritance:
+
+ahriman.core.distributed.worker\_trigger module
+-----------------------------------------------
+
+.. automodule:: ahriman.core.distributed.worker_trigger
+   :members:
+   :no-undoc-members:
+   :show-inheritance:
+
+ahriman.core.distributed.workers\_cache module
+----------------------------------------------
+
+.. automodule:: ahriman.core.distributed.workers_cache
+   :members:
+   :no-undoc-members:
+   :show-inheritance:
+
+Module contents
+---------------
+
+.. automodule:: ahriman.core.distributed
+   :members:
+   :no-undoc-members:
+   :show-inheritance:

docs/ahriman.core.rst

@@ -12,6 +12,7 @@ Subpackages
    ahriman.core.build_tools
    ahriman.core.configuration
    ahriman.core.database
+   ahriman.core.distributed
    ahriman.core.formatters
    ahriman.core.gitremote
    ahriman.core.http

docs/ahriman.web.schemas.rst

@@ -260,6 +260,14 @@ ahriman.web.schemas.versioned\_log\_schema module
    :no-undoc-members:
    :show-inheritance:
 
+ahriman.web.schemas.worker\_schema module
+-----------------------------------------
+
+.. automodule:: ahriman.web.schemas.worker_schema
+   :members:
+   :no-undoc-members:
+   :show-inheritance:
+
 Module contents
 ---------------
 

docs/ahriman.web.views.v1.distributed.rst

@@ -0,0 +1,21 @@
+ahriman.web.views.v1.distributed package
+========================================
+
+Submodules
+----------
+
+ahriman.web.views.v1.distributed.workers module
+-----------------------------------------------
+
+.. automodule:: ahriman.web.views.v1.distributed.workers
+   :members:
+   :no-undoc-members:
+   :show-inheritance:
+
+Module contents
+---------------
+
+.. automodule:: ahriman.web.views.v1.distributed
+   :members:
+   :no-undoc-members:
+   :show-inheritance:

docs/ahriman.web.views.v1.rst

@@ -7,6 +7,7 @@ Subpackages
 .. toctree::
    :maxdepth: 4
 
+   ahriman.web.views.v1.distributed
    ahriman.web.views.v1.service
    ahriman.web.views.v1.status
    ahriman.web.views.v1.user

docs/architecture.rst

@@ -37,6 +37,7 @@ This package contains everything required for the most of application actions an
 * ``ahriman.core.build_tools`` is a package which provides wrapper for ``devtools`` commands.
 * ``ahriman.core.configuration`` contains extension for standard ``configparser`` library and some validation related classes.
 * ``ahriman.core.database`` is everything for database, including data and schema migrations.
+* ``ahriman.core.distributed`` package with triggers and helpers for distributed build system.
 * ``ahriman.core.formatters`` package provides ``Printer`` sub-classes for printing data (e.g. package properties) to stdout which are used by some handlers.
 * ``ahriman.core.gitremote`` is a package with remote PKGBUILD triggers. Should not be called directly.
 * ``ahriman.core.http`` package provides HTTP clients which can be used later by other classes.

docs/configuration.rst

@@ -86,7 +86,7 @@ Build related configuration. Group name can refer to architecture, e.g. ``build:
 * ``triggers`` - list of ``ahriman.core.triggers.Trigger`` class implementation (e.g. ``ahriman.core.report.ReportTrigger ahriman.core.upload.UploadTrigger``) which will be loaded and run at the end of processing, space separated list of strings, optional. You can also specify triggers by their paths, e.g. ``/usr/lib/python3.10/site-packages/ahriman/core/report/report.py.ReportTrigger``. Triggers are run in the order of definition.
 * ``triggers_known`` - optional list of ``ahriman.core.triggers.Trigger`` class implementations which are not run automatically and used only for trigger discovery and configuration validation.
 * ``vcs_allowed_age`` - maximal age in seconds of the VCS packages before their version will be updated with its remote source, integer, optional, default is 7 days.
-* ``workers`` - list of worker nodes addresses used for build process, space separated list of strings, optional. Each worker address must be valid and reachable url, e.g. ``https://10.0.0.1:8080``. If none set, the build process will be run on the current node.
+* ``workers`` - list of worker nodes addresses used for build process, space separated list of strings, optional. Each worker address must be valid and reachable url, e.g. ``https://10.0.0.1:8080``. If none set, the build process will be run on the current node. There is also special trigger which loads this value based on the list of the discovered nodes.
 
 ``repository`` group
 --------------------
@@ -351,4 +351,13 @@ Requires ``boto3`` library to be installed. Section name must be either ``s3`` (
 * ``chunk_size`` - chunk size for calculating entity tags, integer, optional, default 8 * 1024 * 1024.
 * ``object_path`` - path prefix for stored objects, string, optional. If none set, the prefix as in repository tree will be used.
 * ``region`` - bucket region (e.g. ``eu-central-1``), string, required.
-* ``secret_key`` - AWS secret access key, string, required.
+* ``secret_key`` - AWS secret access key, string, required.
+
+``worker`` group
+----------------
+
+This section controls settings for ``ahriman.core.distributed.WorkerTrigger`` plugin.
+
+* ``address`` - address of the instance, string, required. Must be reachable for the master instance.
+* ``identifier`` - unique identifier of the instance, string, optional.
+* ``time_to_live`` - amount of time which remote worker will be considered alive in seconds, integer, optional, default is ``60``. The ping interval will be set automatically equal this value divided by 4.

docs/faq.rst

@@ -1168,6 +1168,11 @@ Addition of new package, package removal, repository update
 
 In all scenarios, update process must be run only on ``master`` node. Unlike the manually distributed packages described above, automatic update must be enabled only for ``master`` node.
 
+Automatic worker nodes discovery
+""""""""""""""""""""""""""""""""
+
+Instead of setting ``build.workers`` option it is also possible to configure services to load worker list dynamically. To do so, the ``ahriman.core.distributed.WorkerLoaderTrigger`` and ``ahriman.core.distributed.WorkerTrigger`` must be used for ``master`` and ``worker`` nodes repsectively. See recipes for more details.
+
 Known limitations
 """""""""""""""""
 

docs/triggers.rst

@@ -14,6 +14,16 @@ Built-in triggers
 
 For the configuration details and settings explanation kindly refer to the :doc:`documentation <configuration>`.
 
+``ahriman.core.distributed.WorkerLoaderTrigger``
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Special trigger to be used to load workers from database on the start of the application rather than configuration. If the option is already set, it will skip processing.
+
+``ahriman.core.distributed.WorkerTrigger``
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Another trigger for the distributed system, which registers itself as remote worker, calling remote service periodically.
+
 ``ahriman.core.gitremote.RemotePullTrigger``
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^