mirror of
https://github.com/arcan1s/ahriman.git
synced 2025-06-28 06:41:43 +00:00
Remote call trigger support (#105)
* add support of remote task tracking * add remote call trigger implementation * docs update * add cross-service upload * add notes about user * add more ability to control upload * multipart upload with signatures as well as safe file save * configuration reference update * rename watcher methods * erase logs based on current package version Old implementation has used process id instead, but it leads to log removal in case of remote process trigger * add --server flag for setup command * restore behavior of the httploghandler
This commit is contained in:
GitHub
@ -20,6 +20,14 @@ ahriman.core.configuration.schema module
|
||||
:no-undoc-members:
|
||||
:show-inheritance:
|
||||
|
||||
ahriman.core.configuration.shell\_interpolator module
|
||||
-----------------------------------------------------
|
||||
|
||||
.. automodule:: ahriman.core.configuration.shell_interpolator
|
||||
:members:
|
||||
:no-undoc-members:
|
||||
:show-inheritance:
|
||||
|
||||
ahriman.core.configuration.validator module
|
||||
-------------------------------------------
|
||||
|
||||
|
||||
@ -84,6 +84,14 @@ ahriman.core.database.migrations.m009\_local\_source module
|
||||
:no-undoc-members:
|
||||
:show-inheritance:
|
||||
|
||||
ahriman.core.database.migrations.m010\_version\_based\_logs\_removal module
|
||||
---------------------------------------------------------------------------
|
||||
|
||||
.. automodule:: ahriman.core.database.migrations.m010_version_based_logs_removal
|
||||
:members:
|
||||
:no-undoc-members:
|
||||
:show-inheritance:
|
||||
|
||||
Module contents
|
||||
---------------
|
||||
|
||||
|
||||
@ -36,6 +36,14 @@ ahriman.core.report.jinja\_template module
|
||||
:no-undoc-members:
|
||||
:show-inheritance:
|
||||
|
||||
ahriman.core.report.remote\_call module
|
||||
---------------------------------------
|
||||
|
||||
.. automodule:: ahriman.core.report.remote_call
|
||||
:members:
|
||||
:no-undoc-members:
|
||||
:show-inheritance:
|
||||
|
||||
ahriman.core.report.report module
|
||||
---------------------------------
|
||||
|
||||
|
||||
@ -20,6 +20,14 @@ ahriman.core.upload.http\_upload module
|
||||
:no-undoc-members:
|
||||
:show-inheritance:
|
||||
|
||||
ahriman.core.upload.remote\_service module
|
||||
------------------------------------------
|
||||
|
||||
.. automodule:: ahriman.core.upload.remote_service
|
||||
:members:
|
||||
:no-undoc-members:
|
||||
:show-inheritance:
|
||||
|
||||
ahriman.core.upload.rsync module
|
||||
--------------------------------
|
||||
|
||||
|
||||
@ -220,6 +220,14 @@ ahriman.models.user\_access module
|
||||
:no-undoc-members:
|
||||
:show-inheritance:
|
||||
|
||||
ahriman.models.waiter module
|
||||
----------------------------
|
||||
|
||||
.. automodule:: ahriman.models.waiter
|
||||
:members:
|
||||
:no-undoc-members:
|
||||
:show-inheritance:
|
||||
|
||||
Module contents
|
||||
---------------
|
||||
|
||||
|
||||
@ -36,6 +36,14 @@ ahriman.web.schemas.error\_schema module
|
||||
:no-undoc-members:
|
||||
:show-inheritance:
|
||||
|
||||
ahriman.web.schemas.file\_schema module
|
||||
---------------------------------------
|
||||
|
||||
.. automodule:: ahriman.web.schemas.file_schema
|
||||
:members:
|
||||
:no-undoc-members:
|
||||
:show-inheritance:
|
||||
|
||||
ahriman.web.schemas.internal\_status\_schema module
|
||||
---------------------------------------------------
|
||||
|
||||
@ -132,6 +140,22 @@ ahriman.web.schemas.pgp\_key\_schema module
|
||||
:no-undoc-members:
|
||||
:show-inheritance:
|
||||
|
||||
ahriman.web.schemas.process\_id\_schema module
|
||||
----------------------------------------------
|
||||
|
||||
.. automodule:: ahriman.web.schemas.process_id_schema
|
||||
:members:
|
||||
:no-undoc-members:
|
||||
:show-inheritance:
|
||||
|
||||
ahriman.web.schemas.process\_schema module
|
||||
------------------------------------------
|
||||
|
||||
.. automodule:: ahriman.web.schemas.process_schema
|
||||
:members:
|
||||
:no-undoc-members:
|
||||
:show-inheritance:
|
||||
|
||||
ahriman.web.schemas.remote\_schema module
|
||||
-----------------------------------------
|
||||
|
||||
@ -156,6 +180,14 @@ ahriman.web.schemas.status\_schema module
|
||||
:no-undoc-members:
|
||||
:show-inheritance:
|
||||
|
||||
ahriman.web.schemas.update\_flags\_schema module
|
||||
------------------------------------------------
|
||||
|
||||
.. automodule:: ahriman.web.schemas.update_flags_schema
|
||||
:members:
|
||||
:no-undoc-members:
|
||||
:show-inheritance:
|
||||
|
||||
Module contents
|
||||
---------------
|
||||
|
||||
|
||||
@ -20,6 +20,14 @@ ahriman.web.views.service.pgp module
|
||||
:no-undoc-members:
|
||||
:show-inheritance:
|
||||
|
||||
ahriman.web.views.service.process module
|
||||
----------------------------------------
|
||||
|
||||
.. automodule:: ahriman.web.views.service.process
|
||||
:members:
|
||||
:no-undoc-members:
|
||||
:show-inheritance:
|
||||
|
||||
ahriman.web.views.service.rebuild module
|
||||
----------------------------------------
|
||||
|
||||
@ -60,6 +68,14 @@ ahriman.web.views.service.update module
|
||||
:no-undoc-members:
|
||||
:show-inheritance:
|
||||
|
||||
ahriman.web.views.service.upload module
|
||||
---------------------------------------
|
||||
|
||||
.. automodule:: ahriman.web.views.service.upload
|
||||
:members:
|
||||
:no-undoc-members:
|
||||
:show-inheritance:
|
||||
|
||||
Module contents
|
||||
---------------
|
||||
|
||||
|
||||
@ -106,8 +106,10 @@ Web server settings. If any of ``host``/``port`` is not set, web integration wil
|
||||
* ``debug`` - enable debug toolbar, boolean, optional, default ``no``.
|
||||
* ``debug_check_host`` - check hosts to access debug toolbar, boolean, optional, default ``no``.
|
||||
* ``debug_allowed_hosts`` - allowed hosts to get access to debug toolbar, space separated list of string, optional.
|
||||
* ``enable_archive_upload`` - allow to upload packages via HTTP (i.e. call of ``/api/v1/service/upload`` uri), boolean, optional, default ``no``.
|
||||
* ``host`` - host to bind, string, optional.
|
||||
* ``index_url`` - full url of the repository index page, string, optional.
|
||||
* ``max_body_size`` - max body size in bytes to be validated for archive upload, integer, optional. If not set, validation will be disabled.
|
||||
* ``password`` - password to authorize in web service in order to update service status, string, required in case if authorization enabled.
|
||||
* ``port`` - port to bind, int, optional.
|
||||
* ``static_path`` - path to directory with static files, string, required.
|
||||
@ -115,6 +117,7 @@ Web server settings. If any of ``host``/``port`` is not set, web integration wil
|
||||
* ``unix_socket`` - path to the listening unix socket, string, optional. If set, server will create the socket on the specified address which can (and will) be used by application. Note, that unlike usual host/port configuration, unix socket allows to perform requests without authorization.
|
||||
* ``unix_socket_unsafe`` - set unsafe (o+w) permissions to unix socket, boolean, optional, default ``yes``. This option is enabled by default, because it is supposed that unix socket is created in safe environment (only web service is supposed to be used in unsafe), but it can be disabled by configuration.
|
||||
* ``username`` - username to authorize in web service in order to update service status, string, required in case if authorization enabled.
|
||||
* ``wait_timeout`` - wait timeout in seconds, maximum amount of time to be waited before lock will be free, int, optional.
|
||||
|
||||
``keyring`` group
|
||||
--------------------
|
||||
@ -246,6 +249,17 @@ Section name must be either ``html`` (plus optional architecture name, e.g. ``ht
|
||||
* ``path`` - path to html report file, string, required.
|
||||
* ``template_path`` - path to Jinja2 template, string, required.
|
||||
|
||||
``remote-call`` type
|
||||
^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Section name must be either ``remote-call`` (plus optional architecture name, e.g. ``remote-call:x86_64``) or random name with ``type`` set.
|
||||
|
||||
* ``type`` - type of the report, string, optional, must be set to ``remote-call`` if exists.
|
||||
* ``aur`` - check for AUR packages updates, boolean, optional, default ``no``.
|
||||
* ``local`` - check for local packages updates, boolean, optional, default ``no``.
|
||||
* ``manual`` - update manually built packages, boolean, optional, default ``no``.
|
||||
* ``wait_timeout`` - maximum amount of time in seconds to be waited before remote process will be terminated, int, optional, default ``-1``.
|
||||
|
||||
``telegram`` type
|
||||
^^^^^^^^^^^^^^^^^
|
||||
|
||||
@ -291,6 +305,13 @@ This feature requires Github key creation (see below). Section name must be eith
|
||||
* ``timeout`` - HTTP request timeout in seconds, int, optional, default is ``30``.
|
||||
* ``username`` - Github authorization user, string, required. Basically the same as ``owner``.
|
||||
|
||||
``remote-service`` type
|
||||
^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Section name must be either ``remote-service`` (plus optional architecture name, e.g. ``remote-service:x86_64``) or random name with ``type`` set.
|
||||
|
||||
* ``type`` - type of the report, string, optional, must be set to ``remote-service`` if exists.
|
||||
|
||||
``rsync`` type
|
||||
^^^^^^^^^^^^^^
|
||||
|
||||
|
||||
201
docs/faq.rst
201
docs/faq.rst
@ -396,6 +396,7 @@ The following environment variables are supported:
|
||||
* ``AHRIMAN_PACMAN_MIRROR`` - override pacman mirror server if set.
|
||||
* ``AHRIMAN_PORT`` - HTTP server port if any, default is empty.
|
||||
* ``AHRIMAN_REPOSITORY`` - repository name, default is ``aur-clone``.
|
||||
* ``AHRIMAN_REPOSITORY_SERVER`` - optional override for the repository url. Useful if you would like to download packages from remote instead of local filesystem.
|
||||
* ``AHRIMAN_REPOSITORY_ROOT`` - repository root. Because of filesystem rights it is required to override default repository root. By default, it uses ``ahriman`` directory inside ahriman's home, which can be passed as mount volume.
|
||||
* ``AHRIMAN_UNIX_SOCKET`` - full path to unix socket which is used by web server, default is empty. Note that more likely you would like to put it inside ``AHRIMAN_REPOSITORY_ROOT`` directory (e.g. ``/var/lib/ahriman/ahriman/ahriman-web.sock``) or to ``/tmp``.
|
||||
* ``AHRIMAN_USER`` - ahriman user, usually must not be overwritten, default is ``ahriman``.
|
||||
@ -722,8 +723,7 @@ How to post build report to telegram
|
||||
#.
|
||||
Optionally (if you want to post message in chat):
|
||||
|
||||
|
||||
#. Create telegram channel.
|
||||
#. Create telegram channel.
|
||||
#. Invite your bot into the channel.
|
||||
#. Make your channel public
|
||||
|
||||
@ -753,6 +753,203 @@ If you did everything fine you should receive the message with the next update.
|
||||
|
||||
(replace ``${CHAT_ID}`` and ``${API_KEY}`` with the values from configuration).
|
||||
|
||||
Distributed builds
|
||||
------------------
|
||||
|
||||
The service allows to run build on multiple machines and collect packages on main node. There are multiple ways to achieve it, this section describes officially supported methods.
|
||||
|
||||
Remote synchronization and remote server call
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
This setup requires at least two instances of the service:
|
||||
|
||||
#. Web service (with opt-in authorization enabled), later will be referenced as ``master`` node.
|
||||
#. Application instances responsible for build, later will be referenced as ``worker`` nodes.
|
||||
|
||||
In this example the following settings are assumed:
|
||||
|
||||
* Repository architecture is ``x86_64``.
|
||||
* Master node address is ``master.example.com``.
|
||||
|
||||
Master node configuration
|
||||
"""""""""""""""""""""""""
|
||||
|
||||
The only requirements for the master node is that API must be available for worker nodes to call (e.g. port must be exposed to internet, or local network in case of VPN, etc) and file upload must be enabled:
|
||||
|
||||
.. code-block:: ini
|
||||
|
||||
[web]
|
||||
enable_archive_upload = yes
|
||||
|
||||
In addition, the following settings are recommended for the master node:
|
||||
|
||||
*
|
||||
As it has been mentioned above, it is recommended to enable authentication (see `How to enable basic authorization`_) and create system user which will be used later. Later this user (if any) will be referenced as ``worker-user``.
|
||||
|
||||
*
|
||||
In order to be able to spawn multiple processes at the same time, wait timeout must be configured:
|
||||
|
||||
.. code-block:: ini
|
||||
|
||||
[web]
|
||||
wait_timeout = 0
|
||||
|
||||
Worker nodes configuration
|
||||
""""""""""""""""""""""""""
|
||||
|
||||
#.
|
||||
First of all, in this setup you need to split your repository into chunks manually, e.g. if you have repository on master node with packages ``A``, ``B`` and ``C``, you need to split them between all available workers, as example:
|
||||
|
||||
* Worker #1: ``A``.
|
||||
* Worker #2: ``B`` and ``C``.
|
||||
|
||||
#.
|
||||
Each worker must be configured to upload files to master node:
|
||||
|
||||
.. code-block:: ini
|
||||
|
||||
[upload]
|
||||
target = remote-service
|
||||
|
||||
[remote-service]
|
||||
|
||||
#.
|
||||
Worker must be configured to access web on master node:
|
||||
|
||||
.. code-block:: ini
|
||||
|
||||
[web]
|
||||
address = master.example.com
|
||||
username = worker-user
|
||||
password = very-secure-password
|
||||
|
||||
As it has been mentioned above, ``web.address`` must be available for workers. In case if unix socket is used, it can be passed as ``web.unix_socket`` variable as usual. Optional ``web.username``/``web.password`` can be supplied in case if authentication was enabled on master node.
|
||||
|
||||
#.
|
||||
Each worker must call master node on success:
|
||||
|
||||
.. code-block:: ini
|
||||
|
||||
[report]
|
||||
target = remote-call
|
||||
|
||||
[remote-call]
|
||||
manual = yes
|
||||
|
||||
After success synchronization (see above), the built packages will be put into directory, from which they will be read during manual update, thus ``remote-call.manual`` flag is required.
|
||||
|
||||
#.
|
||||
Change order of trigger runs. This step is required, because by default the report trigger is called before the upload trigger and we would like to achieve the opposite:
|
||||
|
||||
.. code-block:: ini
|
||||
|
||||
[build]
|
||||
triggers = ahriman.core.gitremote.RemotePullTrigger ahriman.core.upload.UploadTrigger ahriman.core.report.ReportTrigger ahriman.core.gitremote.RemotePushTrigger
|
||||
|
||||
In addition, the following settings are recommended for workers:
|
||||
|
||||
*
|
||||
You might want to wait until report trigger will be completed; in this case the following option must be set:
|
||||
|
||||
.. code-block:: ini
|
||||
|
||||
[remote-call]
|
||||
wait_timeout = 0
|
||||
|
||||
Dependency management
|
||||
"""""""""""""""""""""
|
||||
|
||||
By default worker nodes don't know anything about master nodes packages, thus it will try to build each dependency by its own. However, using ``AHRIMAN_REPOSITORY_SERVER`` docker variable (or ``--server`` flag for setup command), it is possible to specify address of the master node for devtools configuration.
|
||||
|
||||
Repository and packages signing
|
||||
"""""""""""""""""""""""""""""""
|
||||
|
||||
You can sign packages on worker nodes and then signatures will be synced to master node. In order to do so, you need to configure worker node as following, e.g.:
|
||||
|
||||
.. code-block:: ini
|
||||
|
||||
[sign]
|
||||
target = package
|
||||
key = 8BE91E5A773FB48AC05CC1EDBED105AED6246B39
|
||||
|
||||
Note, however, that in this case, signatures will not be validated on master node and just will be copied to repository tree.
|
||||
|
||||
If you would like to sign only database files (aka repository sign), it has to be configured on master node only as usual, e.g.:
|
||||
|
||||
.. code-block:: ini
|
||||
|
||||
[sign]
|
||||
target = repository
|
||||
key = 8BE91E5A773FB48AC05CC1EDBED105AED6246B39
|
||||
|
||||
Double node minimal docker example
|
||||
""""""""""""""""""""""""""""""""""
|
||||
|
||||
Master node config (``master.ini``) as:
|
||||
|
||||
.. code-block:: ini
|
||||
|
||||
[auth]
|
||||
target = mapping
|
||||
|
||||
[web]
|
||||
enable_archive_upload = yes
|
||||
wait_timeout = 0
|
||||
|
||||
|
||||
Command to run master node:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
docker run --privileged -p 8080:8080 -e AHRIMAN_PORT=8080 -v master.ini:/etc/ahriman.ini.d/overrides.ini arcan1s/ahriman:latest web
|
||||
|
||||
The user ``worker-user`` has been created additionally. Worker node config (``worker.ini``) as:
|
||||
|
||||
.. code-block:: ini
|
||||
|
||||
[web]
|
||||
address = http://172.17.0.1:8080
|
||||
username = worker-user
|
||||
password = very-secure-password
|
||||
|
||||
[upload]
|
||||
target = remote-service
|
||||
|
||||
[remote-service]
|
||||
|
||||
[report]
|
||||
target = remote-call
|
||||
|
||||
[remote-call]
|
||||
manual = yes
|
||||
wait_timeout = 0
|
||||
|
||||
[build]
|
||||
triggers = ahriman.core.gitremote.RemotePullTrigger ahriman.core.upload.UploadTrigger ahriman.core.report.ReportTrigger ahriman.core.gitremote.RemotePushTrigger
|
||||
|
||||
The address above (``http://172.17.0.1:8080``) is something available for worker container.
|
||||
|
||||
Command to run worker node:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
docker run --privileged -v worker.ini:/etc/ahriman.ini.d/overrides.ini -it arcan1s/ahriman:latest package-add arhiman --now
|
||||
|
||||
The command above will successfully build ``ahriman`` package, upload it on master node and, finally, will update master node repository.
|
||||
|
||||
Addition of new package and repository update
|
||||
"""""""""""""""""""""""""""""""""""""""""""""
|
||||
|
||||
Just run on worker command as usual, the built packages will be automatically uploaded to master node. Note that automatic update process must be disabled on master node.
|
||||
|
||||
Package removal
|
||||
"""""""""""""""
|
||||
|
||||
This action must be done in two steps:
|
||||
|
||||
#. Remove package on worker.
|
||||
#. Remove package on master node.
|
||||
|
||||
Maintenance packages
|
||||
--------------------
|
||||
|
||||
|
||||
Reference in New Issue
Block a user