fix tests and update docs

docs/ahriman.core.database.migrations.rst

@@ -108,6 +108,14 @@ ahriman.core.database.migrations.m012\_last\_commit\_sha module
    :no-undoc-members:
    :show-inheritance:
 
+ahriman.core.database.migrations.m013\_workers module
+-----------------------------------------------------
+
+.. automodule:: ahriman.core.database.migrations.m013_workers
+   :members:
+   :no-undoc-members:
+   :show-inheritance:
+
 Module contents
 ---------------
 

docs/ahriman.core.database.operations.rst

@@ -60,6 +60,14 @@ ahriman.core.database.operations.patch\_operations module
    :no-undoc-members:
    :show-inheritance:
 
+ahriman.core.database.operations.workers\_operations module
+-----------------------------------------------------------
+
+.. automodule:: ahriman.core.database.operations.workers_operations
+   :members:
+   :no-undoc-members:
+   :show-inheritance:
+
 Module contents
 ---------------
 

docs/ahriman.core.distributed.rst

@@ -0,0 +1,53 @@
+ahriman.core.distributed package
+================================
+
+Submodules
+----------
+
+ahriman.core.distributed.distributed\_system module
+---------------------------------------------------
+
+.. automodule:: ahriman.core.distributed.distributed_system
+   :members:
+   :no-undoc-members:
+   :show-inheritance:
+
+ahriman.core.distributed.worker\_loader\_trigger module
+-------------------------------------------------------
+
+.. automodule:: ahriman.core.distributed.worker_loader_trigger
+   :members:
+   :no-undoc-members:
+   :show-inheritance:
+
+ahriman.core.distributed.worker\_register\_trigger module
+---------------------------------------------------------
+
+.. automodule:: ahriman.core.distributed.worker_register_trigger
+   :members:
+   :no-undoc-members:
+   :show-inheritance:
+
+ahriman.core.distributed.worker\_trigger module
+-----------------------------------------------
+
+.. automodule:: ahriman.core.distributed.worker_trigger
+   :members:
+   :no-undoc-members:
+   :show-inheritance:
+
+ahriman.core.distributed.worker\_unregister\_trigger module
+-----------------------------------------------------------
+
+.. automodule:: ahriman.core.distributed.worker_unregister_trigger
+   :members:
+   :no-undoc-members:
+   :show-inheritance:
+
+Module contents
+---------------
+
+.. automodule:: ahriman.core.distributed
+   :members:
+   :no-undoc-members:
+   :show-inheritance:

docs/ahriman.core.rst

@@ -12,6 +12,7 @@ Subpackages
    ahriman.core.build_tools
    ahriman.core.configuration
    ahriman.core.database
+   ahriman.core.distributed
    ahriman.core.formatters
    ahriman.core.gitremote
    ahriman.core.http

docs/ahriman.web.schemas.rst

@@ -260,6 +260,22 @@ ahriman.web.schemas.versioned\_log\_schema module
    :no-undoc-members:
    :show-inheritance:
 
+ahriman.web.schemas.worker\_id\_schema module
+---------------------------------------------
+
+.. automodule:: ahriman.web.schemas.worker_id_schema
+   :members:
+   :no-undoc-members:
+   :show-inheritance:
+
+ahriman.web.schemas.worker\_schema module
+-----------------------------------------
+
+.. automodule:: ahriman.web.schemas.worker_schema
+   :members:
+   :no-undoc-members:
+   :show-inheritance:
+
 Module contents
 ---------------
 

docs/ahriman.web.views.v1.distributed.rst

@@ -0,0 +1,29 @@
+ahriman.web.views.v1.distributed package
+========================================
+
+Submodules
+----------
+
+ahriman.web.views.v1.distributed.worker module
+----------------------------------------------
+
+.. automodule:: ahriman.web.views.v1.distributed.worker
+   :members:
+   :no-undoc-members:
+   :show-inheritance:
+
+ahriman.web.views.v1.distributed.workers module
+-----------------------------------------------
+
+.. automodule:: ahriman.web.views.v1.distributed.workers
+   :members:
+   :no-undoc-members:
+   :show-inheritance:
+
+Module contents
+---------------
+
+.. automodule:: ahriman.web.views.v1.distributed
+   :members:
+   :no-undoc-members:
+   :show-inheritance:

docs/ahriman.web.views.v1.rst

@@ -7,6 +7,7 @@ Subpackages
 .. toctree::
    :maxdepth: 4
 
+   ahriman.web.views.v1.distributed
    ahriman.web.views.v1.service
    ahriman.web.views.v1.status
    ahriman.web.views.v1.user

docs/configuration.rst

@@ -86,7 +86,7 @@ Build related configuration. Group name can refer to architecture, e.g. ``build:
 * ``triggers`` - list of ``ahriman.core.triggers.Trigger`` class implementation (e.g. ``ahriman.core.report.ReportTrigger ahriman.core.upload.UploadTrigger``) which will be loaded and run at the end of processing, space separated list of strings, optional. You can also specify triggers by their paths, e.g. ``/usr/lib/python3.10/site-packages/ahriman/core/report/report.py.ReportTrigger``. Triggers are run in the order of definition.
 * ``triggers_known`` - optional list of ``ahriman.core.triggers.Trigger`` class implementations which are not run automatically and used only for trigger discovery and configuration validation.
 * ``vcs_allowed_age`` - maximal age in seconds of the VCS packages before their version will be updated with its remote source, integer, optional, default is 7 days.
-* ``workers`` - list of worker nodes addresses used for build process, space separated list of strings, optional. Each worker address must be valid and reachable url, e.g. ``https://10.0.0.1:8080``. If none set, the build process will be run on the current node.
+* ``workers`` - list of worker nodes addresses used for build process, space separated list of strings, optional. Each worker address must be valid and reachable url, e.g. ``https://10.0.0.1:8080``. If none set, the build process will be run on the current node. There is also special trigger which loads this value based on the list of the discovered nodes.
 
 ``repository`` group
 --------------------
@@ -351,4 +351,13 @@ Requires ``boto3`` library to be installed. Section name must be either ``s3`` (
 * ``chunk_size`` - chunk size for calculating entity tags, integer, optional, default 8 * 1024 * 1024.
 * ``object_path`` - path prefix for stored objects, string, optional. If none set, the prefix as in repository tree will be used.
 * ``region`` - bucket region (e.g. ``eu-central-1``), string, required.
-* ``secret_key`` - AWS secret access key, string, required.
+* ``secret_key`` - AWS secret access key, string, required.
+
+``worker`` group
+----------------
+
+This section controls settings for ``ahriman.core.distributed.WorkerTrigger`` plugin.
+
+* ``address`` - address of the instance, string, required. Must be reachable for the master instance.
+* ``identifier`` - unique identifier of the instance, string, optional. If none set, the random uuid will be generated on each run automatically.
+* ``identifier_path`` - path to lock file, string, optional, default is ``/tmp/ahriman-worker-identifier``.

docs/faq.rst

@@ -1168,6 +1168,11 @@ Addition of new package, package removal, repository update
 
 In all scenarios, update process must be run only on ``master`` node. Unlike the manually distributed packages described above, automatic update must be enabled only for ``master`` node.
 
+Automatic worker nodes discovery
+""""""""""""""""""""""""""""""""
+
+Instead of setting ``build.workers`` option it is also possible to configure services to load worker list dynamically. To do so, the ``ahriman.core.distributed.WorkerLoaderTrigger`` and ``ahriman.core.distributed.WorkerTrigger`` must be used for ``master`` and ``worker`` nodes repsectively. See recipes for more details.
+
 Known limitations
 """""""""""""""""
 

docs/triggers.rst

@@ -14,6 +14,18 @@ Built-in triggers
 
 For the configuration details and settings explanation kindly refer to the :doc:`documentation <configuration>`.
 
+``ahriman.core.distributed.WorkerLoaderTrigger``
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Special trigger to be used to load workers from database on the start of the application rather than configuration. If the option is already set, it will skip processing.
+
+``ahriman.core.distributed.WorkerTrigger``
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Another trigger for the distributed system, which registers itself as remote worker. It calls the remote server on start (if no lock file found) and, later, it deregister itself before the stop.
+
+There are also two triggers which performs only registration and removal (``ahriman.core.distributed.WorkerRegisterTrigger`` and ``ahriman.core.distributed.WorkerUnregisterTrigger`` respectively), but they are not meant to be called directly.
+
 ``ahriman.core.gitremote.RemotePullTrigger``
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^