Compare commits

...

8 Commits

Author SHA1 Message Date
b3ce545517 docs: restore docs for the view 2023-12-15 16:17:19 +02:00
e51d91740d feat: add ability to disable specific routes (#119) 2023-12-15 14:34:03 +02:00
5ddc08fce7 feat: add ability to run build process to remote instances (#118) 2023-12-13 15:38:51 +02:00
f2f6f6df70 fix: correct url for update requests in remote-call trigger 2023-12-11 15:43:28 +02:00
2760b36977 feat: changes screen implementation (#117)
Add support of changes generation. Changes will be generated (unless explicitly asked not to) automatically during check process (i.e. `repo-update --dry-run` and aliases) and uploaded to the remote server. Changes can be reviewed either by web interface or by special subcommands.

Changes will be automatically cleared during next successful build
2023-11-30 14:56:41 +02:00
a689448854 fix: use event instead of chained timer for daemon
Old solution causes amount of thread to be growing as well as stack is
increased during each iteration. Instead of cycle-free implementation,
this commit just uses while cycle
2023-11-30 13:40:59 +02:00
aef3cb95bc type: update to the typed aiohttp release 2023-11-23 15:35:38 +02:00
d72677aa29 feat: forbid form data in html
It has been a while since all pages have moved to json instead of form
data, except for login page. This commit changes login to json data
instead of form one
2023-11-16 16:42:27 +02:00
145 changed files with 3436 additions and 633 deletions

View File

@ -1,6 +1,14 @@
ahriman.application.application package
=======================================
Subpackages
-----------
.. toctree::
:maxdepth: 4
ahriman.application.application.workers
Submodules
----------

View File

@ -0,0 +1,37 @@
ahriman.application.application.workers package
===============================================
Submodules
----------
ahriman.application.application.workers.local\_updater module
-------------------------------------------------------------
.. automodule:: ahriman.application.application.workers.local_updater
:members:
:no-undoc-members:
:show-inheritance:
ahriman.application.application.workers.remote\_updater module
--------------------------------------------------------------
.. automodule:: ahriman.application.application.workers.remote_updater
:members:
:no-undoc-members:
:show-inheritance:
ahriman.application.application.workers.updater module
------------------------------------------------------
.. automodule:: ahriman.application.application.workers.updater
:members:
:no-undoc-members:
:show-inheritance:
Module contents
---------------
.. automodule:: ahriman.application.application.workers
:members:
:no-undoc-members:
:show-inheritance:

View File

@ -20,6 +20,14 @@ ahriman.application.handlers.backup module
:no-undoc-members:
:show-inheritance:
ahriman.application.handlers.change module
------------------------------------------
.. automodule:: ahriman.application.handlers.change
:members:
:no-undoc-members:
:show-inheritance:
ahriman.application.handlers.clean module
-----------------------------------------

View File

@ -100,6 +100,14 @@ ahriman.core.database.migrations.m011\_repository\_name module
:no-undoc-members:
:show-inheritance:
ahriman.core.database.migrations.m012\_last\_commit\_sha module
---------------------------------------------------------------
.. automodule:: ahriman.core.database.migrations.m012_last_commit_sha
:members:
:no-undoc-members:
:show-inheritance:
Module contents
---------------

View File

@ -20,6 +20,14 @@ ahriman.core.database.operations.build\_operations module
:no-undoc-members:
:show-inheritance:
ahriman.core.database.operations.changes\_operations module
-----------------------------------------------------------
.. automodule:: ahriman.core.database.operations.changes_operations
:members:
:no-undoc-members:
:show-inheritance:
ahriman.core.database.operations.logs\_operations module
--------------------------------------------------------

View File

@ -20,6 +20,14 @@ ahriman.core.formatters.build\_printer module
:no-undoc-members:
:show-inheritance:
ahriman.core.formatters.changes\_printer module
-----------------------------------------------
.. automodule:: ahriman.core.formatters.changes_printer
:members:
:no-undoc-members:
:show-inheritance:
ahriman.core.formatters.configuration\_paths\_printer module
------------------------------------------------------------

View File

@ -20,6 +20,14 @@ ahriman.core.repository.executor module
:no-undoc-members:
:show-inheritance:
ahriman.core.repository.package\_info module
--------------------------------------------
.. automodule:: ahriman.core.repository.package_info
:members:
:no-undoc-members:
:show-inheritance:
ahriman.core.repository.repository module
-----------------------------------------

View File

@ -36,6 +36,14 @@ ahriman.models.build\_status module
:no-undoc-members:
:show-inheritance:
ahriman.models.changes module
-----------------------------
.. automodule:: ahriman.models.changes
:members:
:no-undoc-members:
:show-inheritance:
ahriman.models.context\_key module
----------------------------------
@ -244,6 +252,14 @@ ahriman.models.waiter module
:no-undoc-members:
:show-inheritance:
ahriman.models.worker module
----------------------------
.. automodule:: ahriman.models.worker
:members:
:no-undoc-members:
:show-inheritance:
Module contents
---------------

View File

@ -20,6 +20,22 @@ ahriman.web.schemas.auth\_schema module
:no-undoc-members:
:show-inheritance:
ahriman.web.schemas.build\_options\_schema module
-------------------------------------------------
.. automodule:: ahriman.web.schemas.build_options_schema
:members:
:no-undoc-members:
:show-inheritance:
ahriman.web.schemas.changes\_schema module
------------------------------------------
.. automodule:: ahriman.web.schemas.changes_schema
:members:
:no-undoc-members:
:show-inheritance:
ahriman.web.schemas.counters\_schema module
-------------------------------------------

View File

@ -38,6 +38,14 @@ ahriman.web.views.static module
:no-undoc-members:
:show-inheritance:
ahriman.web.views.status\_view\_guard module
--------------------------------------------
.. automodule:: ahriman.web.views.status_view_guard
:members:
:no-undoc-members:
:show-inheritance:
Module contents
---------------

View File

@ -4,6 +4,14 @@ ahriman.web.views.v1.status package
Submodules
----------
ahriman.web.views.v1.status.changes module
------------------------------------------
.. automodule:: ahriman.web.views.v1.status.changes
:members:
:no-undoc-members:
:show-inheritance:
ahriman.web.views.v1.status.logs module
---------------------------------------

View File

@ -86,6 +86,7 @@ Build related configuration. Group name can refer to architecture, e.g. ``build:
* ``triggers`` - list of ``ahriman.core.triggers.Trigger`` class implementation (e.g. ``ahriman.core.report.ReportTrigger ahriman.core.upload.UploadTrigger``) which will be loaded and run at the end of processing, space separated list of strings, optional. You can also specify triggers by their paths, e.g. ``/usr/lib/python3.10/site-packages/ahriman/core/report/report.py.ReportTrigger``. Triggers are run in the order of mention.
* ``triggers_known`` - optional list of ``ahriman.core.triggers.Trigger`` class implementations which are not run automatically and used only for trigger discovery and configuration validation.
* ``vcs_allowed_age`` - maximal age in seconds of the VCS packages before their version will be updated with its remote source, integer, optional, default ``604800``.
* ``workers`` - list of worker nodes addresses used for build process, space separated list of strings, optional. Each worker address must be valid and reachable url, e.g. ``https://10.0.0.1:8080``. If none set, the build process will be run on the current node.
``repository`` group
--------------------
@ -128,6 +129,7 @@ Web server settings. If any of ``host``/``port`` is not set, web integration wil
* ``index_url`` - full url of the repository index page, string, optional.
* ``max_body_size`` - max body size in bytes to be validated for archive upload, integer, optional. If not set, validation will be disabled.
* ``port`` - port to bind, integer, optional.
* ``service_only`` - disable status routes (including logs), boolean, optional, default ``no``.
* ``static_path`` - path to directory with static files, string, required.
* ``templates`` - path to templates directories, space separated list of strings, required.
* ``unix_socket`` - path to the listening unix socket, string, optional. If set, server will create the socket on the specified address which can (and will) be used by application. Note, that unlike usual host/port configuration, unix socket allows to perform requests without authorization.

View File

@ -114,8 +114,8 @@ But for some cases you would like to have multiple different reports with the sa
type = email
...
How do I add new package
^^^^^^^^^^^^^^^^^^^^^^^^
How to add new package
^^^^^^^^^^^^^^^^^^^^^^
.. code-block:: shell
@ -237,6 +237,27 @@ Normally the service handles VCS packages correctly, however it requires additio
pacman -S breezy darcs mercurial subversion
How to review changes before build
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
In this scenario, the update process must be separated to several stages. First, it is required to check updates:
.. code-block:: shell
sudo -u ahriman ahriman repo-check
During the check process, the service will generate changes from the last known commit and will send it to remote service. In order to verify source files changes, the web interface or special subcommand can be used:
.. code-block:: shell
ahriman package-changes ahriman
After validation, the operator can run update process with approved list of packages, e.g.:
.. code-block:: shell
sudo -u ahriman ahriman repo-update ahriman
How to remove package
^^^^^^^^^^^^^^^^^^^^^
@ -1001,6 +1022,119 @@ This action must be done in two steps:
#. Remove package on worker.
#. Remove package on master node.
Delegate builds to remote workers
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
This setup heavily uses upload feature described above and, in addition, also delegates build process automatically to build machines. Same as above, there must be at least two instances available (``master`` and ``worker``), however, all ``worker`` nodes must be run in the web service mode.
Master node configuration
"""""""""""""""""""""""""
In addition to the configuration above, the worker list must be defined in configuration file (``build.workers`` option), i.e.:
.. code-block:: ini
[build]
workers = https://worker1.example.com https://worker2.example.com
[web]
enable_archive_upload = yes
wait_timeout = 0
In the example above, ``https://worker1.example.com`` and ``https://worker2.example.com`` are remote ``worker`` node addresses available for ``master`` node.
In case if authentication is required (which is recommended way to setup it), it can be set by using ``status`` section as usual.
Worker nodes configuration
""""""""""""""""""""""""""
It is required to point to the master node repository, otherwise internal dependencies will not be handled correctly. In order to do so, the ``--server`` argument (or ``AHRIMAN_REPOSITORY_SERVER`` environment variable for docker images) can be used.
Also, in case if authentication is enabled, the same user with the same password must be created for all workers.
It is also recommended to set ``web.wait_timeout`` to infinite in case of multiple conflicting runs and ``service_only`` to ``yes`` in order to disable status endpoints.
Other settings are the same as mentioned above.
Triple node minimal docker example
""""""""""""""""""""""""""""""""""
In this example, all instances are run on the same machine with address ``172.17.0.1`` with ports available outside of container. Master node config (``master.ini``) as:
.. code-block:: ini
[auth]
target = mapping
[status]
username = builder-user
password = very-secure-password
[build]
workers = http://172.17.0.1:8081 http://172.17.0.1:8082
[web]
enable_archive_upload = yes
wait_timeout = 0
Command to run master node:
.. code-block:: shell
docker run --privileged -p 8080:8080 -e AHRIMAN_PORT=8080 -v master.ini:/etc/ahriman.ini.d/overrides.ini arcan1s/ahriman:latest web
Worker nodes (applicable for all workers) config (``worker.ini``) as:
.. code-block:: ini
[auth]
target = mapping
[status]
address = http://172.17.0.1:8080
username = builder-user
password = very-secure-password
[upload]
target = remote-service
[remote-service]
[report]
target = remote-call
[remote-call]
manual = yes
wait_timeout = 0
[web]
service_only = yes
[build]
triggers = ahriman.core.upload.UploadTrigger ahriman.core.report.ReportTrigger
Command to run worker nodes (considering there will be two workers, one is on ``8081`` port and other is on ``8082``):
.. code-block:: ini
docker run --privileged -p 8081:8081 -e AHRIMAN_PORT=8081 -v worker.ini:/etc/ahriman.ini.d/overrides.ini arcan1s/ahriman:latest web
docker run --privileged -p 8082:8082 -e AHRIMAN_PORT=8082 -v worker.ini:/etc/ahriman.ini.d/overrides.ini arcan1s/ahriman:latest web
Unlike the previous setup, it doesn't require to mount repository root for ``worker`` nodes, because ``worker`` nodes don't use it anyway.
Addition of new package, package removal, repository update
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
In all scenarios, update process must be run only on ``master`` node. Unlike the setup described above, automatic update must be enabled only for ``master`` node also.
Known limitations
"""""""""""""""""
* Workers don't support local packages. However, it is possible to build custom packages by providing sources by using ``ahriman.core.gitremote.RemotePullTrigger`` trigger.
* No dynamic nodes discovery. In case if one of worker nodes is unavailable, the build process will fail.
* No pkgrel bump on conflicts. Well, it works, however, it isn't guaranteed.
* The identical user must be created for all workers. However, the ``master`` node user can be different from this one.
Maintenance packages
--------------------

View File

@ -2,6 +2,6 @@
Description=ArcH linux ReposItory MANager (%i)
[Service]
ExecStart=/usr/bin/ahriman --repository-id "%I" repo-update --refresh
ExecStart=/usr/bin/ahriman --repository-id "%I" repo-update --no-changes --refresh
User=ahriman
Group=ahriman

View File

@ -1,7 +1,7 @@
<script>
const alertPlaceholder = $("#alert-placeholder");
function createAlert(title, message, clz) {
function createAlert(title, message, clz, action) {
const wrapper = document.createElement("div");
wrapper.classList.add("toast", clz);
wrapper.role = "alert";
@ -23,7 +23,7 @@
const toast = new bootstrap.Toast(wrapper);
wrapper.addEventListener("hidden.bs.toast", () => {
wrapper.remove(); // bootstrap doesn't remove elements
reload();
(action || reload)();
});
toast.show();
}
@ -38,8 +38,8 @@
createAlert(title, description(details), "text-bg-danger");
}
function showSuccess(title, description) {
createAlert(title, description, "text-bg-success");
function showSuccess(title, description, action) {
createAlert(title, description, "text-bg-success", action);
}
</script>

View File

@ -1,7 +1,7 @@
<div id="login-modal" tabindex="-1" role="dialog" class="modal fade">
<div class="modal-dialog" role="document">
<div class="modal-content">
<form action="/api/v1/login" method="post">
<form id="login-form" onsubmit="return false">
<div class="modal-header">
<h4 class="modal-title">Login</h4>
<button type="button" class="btn-close" data-bs-dismiss="modal" aria-label="close"></button>
@ -26,7 +26,7 @@
</div>
</div>
<div class="modal-footer">
<button class="btn btn-primary"><i class="bi bi-person"></i> login</button>
<button type="submit" class="btn btn-primary" onclick="login()"><i class="bi bi-person"></i> login</button>
</div>
</form>
</div>
@ -34,16 +34,45 @@
</div>
<script>
const passwordInput = $("#login-password");
const loginModal = $("#login-modal");
const loginForm = $("#login-form");
loginModal.on("hidden.bs.modal", () => {
loginForm.trigger("reset");
});
const loginPasswordInput = $("#login-password");
const loginUsernameInput = $("#login-username");
const showHidePasswordButton = $("#login-show-hide-password-button");
function login() {
const password = loginPasswordInput.val();
const username = loginUsernameInput.val();
if (username && password) {
$.ajax({
url: "/api/v1/login",
data: JSON.stringify({username: username, password: password}),
type: "POST",
contentType: "application/json",
success: _ => {
loginModal.modal("hide");
showSuccess("Logged in", `Successfully logged in as ${username}`, () => location.href = "/");
},
error: (jqXHR, _, errorThrown) => {
const message = _ => `Could not login as ${username}`;
showFailure("Login error", message, jqXHR, errorThrown);
},
});
}
}
function showPassword() {
if (passwordInput.attr("type") === "password") {
passwordInput.attr("type", "text");
if (loginPasswordInput.attr("type") === "password") {
loginPasswordInput.attr("type", "text");
showHidePasswordButton.removeClass("bi-eye");
showHidePasswordButton.addClass("bi-eye-slash");
} else {
passwordInput.attr("type", "password");
loginPasswordInput.attr("type", "password");
showHidePasswordButton.removeClass("bi-eye-slash");
showHidePasswordButton.addClass("bi-eye");
}

View File

@ -36,13 +36,27 @@
<hr class="col-12">
<h3>Environment variables</h3>
<div id="package-info-variables-div" class="form-group row"></div>
<div id="package-info-variables-block" hidden>
<h3>Environment variables</h3>
<div id="package-info-variables-div" class="form-group row"></div>
<hr class="col-12">
<hr class="col-12">
</div>
<h3>Build logs</h3>
<pre class="language-logs"><samp id="package-info-logs-input" class="pre-scrollable language-logs"></samp><button id="package-info-logs-copy-button" type="button" class="btn language-logs" onclick="copyLogs()"><i class="bi bi-clipboard"></i> copy</button></pre>
<nav>
<div class="nav nav-tabs" role="tablist">
<button id="package-info-logs-button" class="nav-link active" data-bs-toggle="tab" data-bs-target="#package-info-logs" type="button" role="tab" aria-controls="package-info-logs" aria-selected="true"><h3>Build logs</h3></button>
<button id="package-info-changes-button" class="nav-link" data-bs-toggle="tab" data-bs-target="#package-info-changes" type="button" role="tab" aria-controls="package-info-changes" aria-selected="false"><h3>Changes</h3></button>
</div>
</nav>
<div class="tab-content" id="nav-tabContent">
<div id="package-info-logs" class="tab-pane fade show active" role="tabpanel" aria-labelledby="package-info-logs-button" tabindex="0">
<pre class="language-console"><code id="package-info-logs-input" class="pre-scrollable language-console"></code><button id="package-info-logs-copy-button" type="button" class="btn language-console" onclick="copyLogs()"><i class="bi bi-clipboard"></i> copy</button></pre>
</div>
<div id="package-info-changes" class="tab-pane fade" role="tabpanel" aria-labelledby="package-info-changes-button" tabindex="0">
<pre class="language-diff"><code id="package-info-changes-input" class="pre-scrollable language-diff"></code><button id="package-info-changes-copy-button" type="button" class="btn language-diff" onclick="copyChanges()"><i class="bi bi-clipboard"></i> copy</button></pre>
</div>
</div>
</div>
<div class="modal-footer">
<button id="package-info-update-button" type="submit" class="btn btn-success" onclick="packageInfoUpdate()" data-bs-dismiss="modal" hidden><i class="bi bi-play"></i><span class="d-none d-sm-inline"> update</span></button>
@ -68,9 +82,11 @@
packageInfoUpstreamUrl.empty();
packageInfoVersion.empty();
packageInfoVariablesBlock.attr("hidden", true);
packageInfoVariablesDiv.empty();
packageInfoLogsInput.empty();
packageInfoChangesInput.empty();
packageInfoModal.trigger("reset");
@ -80,6 +96,9 @@
const packageInfoLogsInput = $("#package-info-logs-input");
const packageInfoLogsCopyButton = $("#package-info-logs-copy-button");
const packageInfoChangesInput = $("#package-info-changes-input");
const packageInfoChangesCopyButton = $("#package-info-changes-copy-button");
const packageInfoAurUrl = $("#package-info-aur-url");
const packageInfoDepends = $("#package-info-depends");
const packageInfoGroups = $("#package-info-groups");
@ -89,8 +108,14 @@
const packageInfoUpstreamUrl = $("#package-info-upstream-url");
const packageInfoVersion = $("#package-info-version");
const packageInfoVariablesBlock = $("#package-info-variables-block");
const packageInfoVariablesDiv = $("#package-info-variables-div");
async function copyChanges() {
const changes = packageInfoChangesInput.text();
await copyToClipboard(changes, packageInfoChangesCopyButton);
}
async function copyLogs() {
const logs = packageInfoLogsInput.text();
await copyToClipboard(logs, packageInfoLogsCopyButton);
@ -142,6 +167,24 @@
packageInfoVariablesDiv.append(variableInput);
}
function loadChanges(packageBase, onFailure) {
$.ajax({
url: `/api/v1/packages/${packageBase}/changes`,
data: {
architecture: repository.architecture,
repository: repository.repository,
},
type: "GET",
dataType: "json",
success: response => {
const changes = response.changes;
packageInfoChangesInput.text(changes || "");
packageInfoChangesInput.map((_, el) => hljs.highlightElement(el));
},
error: onFailure,
});
}
function loadLogs(packageBase, onFailure) {
$.ajax({
url: `/api/v2/packages/${packageBase}/logs`,
@ -156,6 +199,7 @@
return `[${new Date(1000 * log_record.created).toISOString()}] ${log_record.message}`;
});
packageInfoLogsInput.text(logs.join("\n"));
packageInfoLogsInput.map((_, el) => hljs.highlightElement(el));
},
error: onFailure,
});
@ -228,6 +272,7 @@
success: response => {
packageInfoVariablesDiv.empty();
response.map(patch => insertVariable(packageBase, patch));
packageInfoVariablesBlock.attr("hidden", response.length === 0);
},
error: onFailure,
});
@ -260,6 +305,7 @@
loadPackage(packageBase, onFailure);
loadPatches(packageBase, onFailure);
loadLogs(packageBase, onFailure);
loadChanges(packageBase, onFailure)
if (isPackageBaseSet) packageInfoModal.modal("show");
}

View File

@ -15,6 +15,8 @@
<script src="https://cdn.jsdelivr.net/npm/bootstrap-table@1.22.1/dist/extensions/resizable/bootstrap-table-resizable.js" integrity="sha384-wd8Vc6Febikdnsnk9vthRWRvMwffw246vhqiqNO3aSNe1maTEA07Vh3zAQiSyDji" crossorigin="anonymous" type="application/javascript"></script>
<script src="https://cdn.jsdelivr.net/npm/bootstrap-table@1.22.1/dist/extensions/filter-control/bootstrap-table-filter-control.js" integrity="sha384-NIqcjpr/3eZI1iNzz7hgT5rgp70qFUzkZffeCgVva9gi80B5vqcm7gn+8QvlWxko" crossorigin="anonymous" type="application/javascript"></script>
<script src="https://cdn.jsdelivr.net/gh/highlightjs/cdn-release@11.9.0/build/highlight.min.js" integrity="sha384-F/bZzf7p3Joyp5psL90p/p89AZJsndkSoGwRpXcZhleCWhd8SnRuoYo4d0yirjJp" crossorigin="anonymous" type="application/javascript"></script>
<script>
async function copyToClipboard(text, button) {
if (navigator.clipboard === undefined) {

View File

@ -11,6 +11,8 @@
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/daterangepicker@3.1.0/daterangepicker.css" integrity="sha384-zLkQsiLfAQqGeIJeKLC+rcCR1YoYaQFLCL7cLDUoKE1ajKJzySpjzWGfYS2vjSG+" crossorigin="anonymous" type="text/css">
<link rel="stylesheet" href="https://cdn.jsdelivr.net/gh/highlightjs/cdn-release@11.9.0/build/styles/github.min.css" integrity="sha384-eFTL69TLRZTkNfYZOLM+G04821K1qZao/4QLJbet1pP4tcF+fdXq/9CdqAbWRl/L" crossorigin="anonymous" type="text/css">
<style>
.pre-scrollable {
display: block;

View File

@ -1,6 +1,6 @@
# AUTOMATICALLY GENERATED by `shtab`
_shtab_ahriman_subparsers=('aur-search' 'search' 'help-commands-unsafe' 'help' 'help-updates' 'help-version' 'version' 'package-add' 'add' 'package-update' 'package-remove' 'remove' 'package-status' 'status' 'package-status-remove' 'package-status-update' 'status-update' 'patch-add' 'patch-list' 'patch-remove' 'patch-set-add' 'repo-backup' 'repo-check' 'check' 'repo-create-keyring' 'repo-create-mirrorlist' 'repo-daemon' 'daemon' 'repo-rebuild' 'rebuild' 'repo-remove-unknown' 'remove-unknown' 'repo-report' 'report' 'repo-restore' 'repo-sign' 'sign' 'repo-status-update' 'repo-sync' 'sync' 'repo-tree' 'repo-triggers' 'repo-update' 'update' 'service-clean' 'clean' 'repo-clean' 'service-config' 'config' 'repo-config' 'service-config-validate' 'config-validate' 'repo-config-validate' 'service-key-import' 'key-import' 'service-repositories' 'service-run' 'run' 'service-setup' 'init' 'repo-init' 'repo-setup' 'setup' 'service-shell' 'shell' 'service-tree-migrate' 'user-add' 'user-list' 'user-remove' 'web')
_shtab_ahriman_subparsers=('aur-search' 'search' 'help-commands-unsafe' 'help' 'help-updates' 'help-version' 'version' 'package-add' 'add' 'package-update' 'package-changes' 'package-changes-remove' 'package-remove' 'remove' 'package-status' 'status' 'package-status-remove' 'package-status-update' 'status-update' 'patch-add' 'patch-list' 'patch-remove' 'patch-set-add' 'repo-backup' 'repo-check' 'check' 'repo-create-keyring' 'repo-create-mirrorlist' 'repo-daemon' 'daemon' 'repo-rebuild' 'rebuild' 'repo-remove-unknown' 'remove-unknown' 'repo-report' 'report' 'repo-restore' 'repo-sign' 'sign' 'repo-status-update' 'repo-sync' 'sync' 'repo-tree' 'repo-triggers' 'repo-update' 'update' 'service-clean' 'clean' 'repo-clean' 'service-config' 'config' 'repo-config' 'service-config-validate' 'config-validate' 'repo-config-validate' 'service-key-import' 'key-import' 'service-repositories' 'service-run' 'run' 'service-setup' 'init' 'repo-init' 'repo-setup' 'setup' 'service-shell' 'shell' 'service-tree-migrate' 'user-add' 'user-list' 'user-remove' 'web')
_shtab_ahriman_option_strings=('-h' '--help' '-a' '--architecture' '-c' '--configuration' '--force' '-l' '--lock' '--log-handler' '-q' '--quiet' '--report' '--no-report' '-r' '--repository' '--unsafe' '-V' '--version' '--wait-timeout')
_shtab_ahriman_aur_search_option_strings=('-h' '--help' '-e' '--exit-code' '--info' '--no-info' '--sort-by')
@ -13,6 +13,8 @@ _shtab_ahriman_version_option_strings=('-h' '--help')
_shtab_ahriman_package_add_option_strings=('-h' '--help' '--dependencies' '--no-dependencies' '-e' '--exit-code' '--increment' '--no-increment' '-n' '--now' '-y' '--refresh' '-s' '--source' '-u' '--username' '-v' '--variable')
_shtab_ahriman_add_option_strings=('-h' '--help' '--dependencies' '--no-dependencies' '-e' '--exit-code' '--increment' '--no-increment' '-n' '--now' '-y' '--refresh' '-s' '--source' '-u' '--username' '-v' '--variable')
_shtab_ahriman_package_update_option_strings=('-h' '--help' '--dependencies' '--no-dependencies' '-e' '--exit-code' '--increment' '--no-increment' '-n' '--now' '-y' '--refresh' '-s' '--source' '-u' '--username' '-v' '--variable')
_shtab_ahriman_package_changes_option_strings=('-h' '--help' '-e' '--exit-code')
_shtab_ahriman_package_changes_remove_option_strings=('-h' '--help')
_shtab_ahriman_package_remove_option_strings=('-h' '--help')
_shtab_ahriman_remove_option_strings=('-h' '--help')
_shtab_ahriman_package_status_option_strings=('-h' '--help' '--ahriman' '-e' '--exit-code' '--info' '--no-info' '-s' '--status')
@ -25,12 +27,12 @@ _shtab_ahriman_patch_list_option_strings=('-h' '--help' '-e' '--exit-code' '-v'
_shtab_ahriman_patch_remove_option_strings=('-h' '--help' '-v' '--variable')
_shtab_ahriman_patch_set_add_option_strings=('-h' '--help' '-t' '--track')
_shtab_ahriman_repo_backup_option_strings=('-h' '--help')
_shtab_ahriman_repo_check_option_strings=('-h' '--help' '-e' '--exit-code' '--vcs' '--no-vcs' '-y' '--refresh')
_shtab_ahriman_check_option_strings=('-h' '--help' '-e' '--exit-code' '--vcs' '--no-vcs' '-y' '--refresh')
_shtab_ahriman_repo_check_option_strings=('-h' '--help' '--changes' '--no-changes' '-e' '--exit-code' '--vcs' '--no-vcs' '-y' '--refresh')
_shtab_ahriman_check_option_strings=('-h' '--help' '--changes' '--no-changes' '-e' '--exit-code' '--vcs' '--no-vcs' '-y' '--refresh')
_shtab_ahriman_repo_create_keyring_option_strings=('-h' '--help')
_shtab_ahriman_repo_create_mirrorlist_option_strings=('-h' '--help')
_shtab_ahriman_repo_daemon_option_strings=('-h' '--help' '-i' '--interval' '--aur' '--no-aur' '--dependencies' '--no-dependencies' '--local' '--no-local' '--manual' '--no-manual' '--vcs' '--no-vcs' '-y' '--refresh')
_shtab_ahriman_daemon_option_strings=('-h' '--help' '-i' '--interval' '--aur' '--no-aur' '--dependencies' '--no-dependencies' '--local' '--no-local' '--manual' '--no-manual' '--vcs' '--no-vcs' '-y' '--refresh')
_shtab_ahriman_repo_daemon_option_strings=('-h' '--help' '-i' '--interval' '--aur' '--no-aur' '--changes' '--no-changes' '--dependencies' '--no-dependencies' '--dry-run' '--local' '--no-local' '--manual' '--no-manual' '--vcs' '--no-vcs' '-y' '--refresh')
_shtab_ahriman_daemon_option_strings=('-h' '--help' '-i' '--interval' '--aur' '--no-aur' '--changes' '--no-changes' '--dependencies' '--no-dependencies' '--dry-run' '--local' '--no-local' '--manual' '--no-manual' '--vcs' '--no-vcs' '-y' '--refresh')
_shtab_ahriman_repo_rebuild_option_strings=('-h' '--help' '--depends-on' '--dry-run' '--from-database' '--increment' '--no-increment' '-e' '--exit-code' '-s' '--status' '-u' '--username')
_shtab_ahriman_rebuild_option_strings=('-h' '--help' '--depends-on' '--dry-run' '--from-database' '--increment' '--no-increment' '-e' '--exit-code' '-s' '--status' '-u' '--username')
_shtab_ahriman_repo_remove_unknown_option_strings=('-h' '--help' '--dry-run')
@ -45,8 +47,8 @@ _shtab_ahriman_repo_sync_option_strings=('-h' '--help')
_shtab_ahriman_sync_option_strings=('-h' '--help')
_shtab_ahriman_repo_tree_option_strings=('-h' '--help' '-p' '--partitions')
_shtab_ahriman_repo_triggers_option_strings=('-h' '--help')
_shtab_ahriman_repo_update_option_strings=('-h' '--help' '--aur' '--no-aur' '--dependencies' '--no-dependencies' '--dry-run' '-e' '--exit-code' '--increment' '--no-increment' '--local' '--no-local' '--manual' '--no-manual' '-u' '--username' '--vcs' '--no-vcs' '-y' '--refresh')
_shtab_ahriman_update_option_strings=('-h' '--help' '--aur' '--no-aur' '--dependencies' '--no-dependencies' '--dry-run' '-e' '--exit-code' '--increment' '--no-increment' '--local' '--no-local' '--manual' '--no-manual' '-u' '--username' '--vcs' '--no-vcs' '-y' '--refresh')
_shtab_ahriman_repo_update_option_strings=('-h' '--help' '--aur' '--no-aur' '--changes' '--no-changes' '--dependencies' '--no-dependencies' '--dry-run' '-e' '--exit-code' '--increment' '--no-increment' '--local' '--no-local' '--manual' '--no-manual' '-u' '--username' '--vcs' '--no-vcs' '-y' '--refresh')
_shtab_ahriman_update_option_strings=('-h' '--help' '--aur' '--no-aur' '--changes' '--no-changes' '--dependencies' '--no-dependencies' '--dry-run' '-e' '--exit-code' '--increment' '--no-increment' '--local' '--no-local' '--manual' '--no-manual' '-u' '--username' '--vcs' '--no-vcs' '-y' '--refresh')
_shtab_ahriman_service_clean_option_strings=('-h' '--help' '--cache' '--no-cache' '--chroot' '--no-chroot' '--manual' '--no-manual' '--packages' '--no-packages' '--pacman' '--no-pacman')
_shtab_ahriman_clean_option_strings=('-h' '--help' '--cache' '--no-cache' '--chroot' '--no-chroot' '--manual' '--no-manual' '--packages' '--no-packages' '--pacman' '--no-pacman')
_shtab_ahriman_repo_clean_option_strings=('-h' '--help' '--cache' '--no-cache' '--chroot' '--no-chroot' '--manual' '--no-manual' '--packages' '--no-packages' '--pacman' '--no-pacman')
@ -76,7 +78,7 @@ _shtab_ahriman_web_option_strings=('-h' '--help')
_shtab_ahriman_pos_0_choices=('aur-search' 'search' 'help-commands-unsafe' 'help' 'help-updates' 'help-version' 'version' 'package-add' 'add' 'package-update' 'package-remove' 'remove' 'package-status' 'status' 'package-status-remove' 'package-status-update' 'status-update' 'patch-add' 'patch-list' 'patch-remove' 'patch-set-add' 'repo-backup' 'repo-check' 'check' 'repo-create-keyring' 'repo-create-mirrorlist' 'repo-daemon' 'daemon' 'repo-rebuild' 'rebuild' 'repo-remove-unknown' 'remove-unknown' 'repo-report' 'report' 'repo-restore' 'repo-sign' 'sign' 'repo-status-update' 'repo-sync' 'sync' 'repo-tree' 'repo-triggers' 'repo-update' 'update' 'service-clean' 'clean' 'repo-clean' 'service-config' 'config' 'repo-config' 'service-config-validate' 'config-validate' 'repo-config-validate' 'service-key-import' 'key-import' 'service-repositories' 'service-run' 'run' 'service-setup' 'init' 'repo-init' 'repo-setup' 'setup' 'service-shell' 'shell' 'service-tree-migrate' 'user-add' 'user-list' 'user-remove' 'web')
_shtab_ahriman_pos_0_choices=('aur-search' 'search' 'help-commands-unsafe' 'help' 'help-updates' 'help-version' 'version' 'package-add' 'add' 'package-update' 'package-changes' 'package-changes-remove' 'package-remove' 'remove' 'package-status' 'status' 'package-status-remove' 'package-status-update' 'status-update' 'patch-add' 'patch-list' 'patch-remove' 'patch-set-add' 'repo-backup' 'repo-check' 'check' 'repo-create-keyring' 'repo-create-mirrorlist' 'repo-daemon' 'daemon' 'repo-rebuild' 'rebuild' 'repo-remove-unknown' 'remove-unknown' 'repo-report' 'report' 'repo-restore' 'repo-sign' 'sign' 'repo-status-update' 'repo-sync' 'sync' 'repo-tree' 'repo-triggers' 'repo-update' 'update' 'service-clean' 'clean' 'repo-clean' 'service-config' 'config' 'repo-config' 'service-config-validate' 'config-validate' 'repo-config-validate' 'service-key-import' 'key-import' 'service-repositories' 'service-run' 'run' 'service-setup' 'init' 'repo-init' 'repo-setup' 'setup' 'service-shell' 'shell' 'service-tree-migrate' 'user-add' 'user-list' 'user-remove' 'web')
_shtab_ahriman___log_handler_choices=('console' 'syslog' 'journald')
_shtab_ahriman_aur_search___sort_by_choices=('description' 'first_submitted' 'id' 'last_modified' 'maintainer' 'name' 'num_votes' 'out_of_date' 'package_base' 'package_base_id' 'popularity' 'repository' 'submitter' 'url' 'url_path' 'version')
_shtab_ahriman_search___sort_by_choices=('description' 'first_submitted' 'id' 'last_modified' 'maintainer' 'name' 'num_votes' 'out_of_date' 'package_base' 'package_base_id' 'popularity' 'repository' 'submitter' 'url' 'url_path' 'version')
@ -187,6 +189,12 @@ _shtab_ahriman_package_update__n_nargs=0
_shtab_ahriman_package_update___now_nargs=0
_shtab_ahriman_package_update__y_nargs=0
_shtab_ahriman_package_update___refresh_nargs=0
_shtab_ahriman_package_changes__h_nargs=0
_shtab_ahriman_package_changes___help_nargs=0
_shtab_ahriman_package_changes__e_nargs=0
_shtab_ahriman_package_changes___exit_code_nargs=0
_shtab_ahriman_package_changes_remove__h_nargs=0
_shtab_ahriman_package_changes_remove___help_nargs=0
_shtab_ahriman_package_remove_pos_0_nargs=+
_shtab_ahriman_package_remove__h_nargs=0
_shtab_ahriman_package_remove___help_nargs=0
@ -233,6 +241,8 @@ _shtab_ahriman_repo_backup___help_nargs=0
_shtab_ahriman_repo_check_pos_0_nargs=*
_shtab_ahriman_repo_check__h_nargs=0
_shtab_ahriman_repo_check___help_nargs=0
_shtab_ahriman_repo_check___changes_nargs=0
_shtab_ahriman_repo_check___no_changes_nargs=0
_shtab_ahriman_repo_check__e_nargs=0
_shtab_ahriman_repo_check___exit_code_nargs=0
_shtab_ahriman_repo_check___vcs_nargs=0
@ -242,6 +252,8 @@ _shtab_ahriman_repo_check___refresh_nargs=0
_shtab_ahriman_check_pos_0_nargs=*
_shtab_ahriman_check__h_nargs=0
_shtab_ahriman_check___help_nargs=0
_shtab_ahriman_check___changes_nargs=0
_shtab_ahriman_check___no_changes_nargs=0
_shtab_ahriman_check__e_nargs=0
_shtab_ahriman_check___exit_code_nargs=0
_shtab_ahriman_check___vcs_nargs=0
@ -256,8 +268,11 @@ _shtab_ahriman_repo_daemon__h_nargs=0
_shtab_ahriman_repo_daemon___help_nargs=0
_shtab_ahriman_repo_daemon___aur_nargs=0
_shtab_ahriman_repo_daemon___no_aur_nargs=0
_shtab_ahriman_repo_daemon___changes_nargs=0
_shtab_ahriman_repo_daemon___no_changes_nargs=0
_shtab_ahriman_repo_daemon___dependencies_nargs=0
_shtab_ahriman_repo_daemon___no_dependencies_nargs=0
_shtab_ahriman_repo_daemon___dry_run_nargs=0
_shtab_ahriman_repo_daemon___local_nargs=0
_shtab_ahriman_repo_daemon___no_local_nargs=0
_shtab_ahriman_repo_daemon___manual_nargs=0
@ -270,8 +285,11 @@ _shtab_ahriman_daemon__h_nargs=0
_shtab_ahriman_daemon___help_nargs=0
_shtab_ahriman_daemon___aur_nargs=0
_shtab_ahriman_daemon___no_aur_nargs=0
_shtab_ahriman_daemon___changes_nargs=0
_shtab_ahriman_daemon___no_changes_nargs=0
_shtab_ahriman_daemon___dependencies_nargs=0
_shtab_ahriman_daemon___no_dependencies_nargs=0
_shtab_ahriman_daemon___dry_run_nargs=0
_shtab_ahriman_daemon___local_nargs=0
_shtab_ahriman_daemon___no_local_nargs=0
_shtab_ahriman_daemon___manual_nargs=0
@ -330,6 +348,8 @@ _shtab_ahriman_repo_update__h_nargs=0
_shtab_ahriman_repo_update___help_nargs=0
_shtab_ahriman_repo_update___aur_nargs=0
_shtab_ahriman_repo_update___no_aur_nargs=0
_shtab_ahriman_repo_update___changes_nargs=0
_shtab_ahriman_repo_update___no_changes_nargs=0
_shtab_ahriman_repo_update___dependencies_nargs=0
_shtab_ahriman_repo_update___no_dependencies_nargs=0
_shtab_ahriman_repo_update___dry_run_nargs=0
@ -350,6 +370,8 @@ _shtab_ahriman_update__h_nargs=0
_shtab_ahriman_update___help_nargs=0
_shtab_ahriman_update___aur_nargs=0
_shtab_ahriman_update___no_aur_nargs=0
_shtab_ahriman_update___changes_nargs=0
_shtab_ahriman_update___no_changes_nargs=0
_shtab_ahriman_update___dependencies_nargs=0
_shtab_ahriman_update___no_dependencies_nargs=0
_shtab_ahriman_update___dry_run_nargs=0
@ -568,6 +590,15 @@ _set_new_action() {
# ${!x} -> ${hello} -> "world"
_shtab_ahriman() {
local completing_word="${COMP_WORDS[COMP_CWORD]}"
local completed_positional_actions
local current_action
local current_action_args_start_index
local current_action_choices
local current_action_compgen
local current_action_is_positional
local current_action_nargs
local current_option_strings
local sub_parsers
COMPREPLY=()
local prefix=_shtab_ahriman

View File

@ -1,9 +1,9 @@
.TH AHRIMAN "1" "2023\-11\-13" "ahriman" "Generated Python Manual"
.TH AHRIMAN "1" "2023\-12\-08" "ahriman" "Generated Python Manual"
.SH NAME
ahriman
.SH SYNOPSIS
.B ahriman
[-h] [-a ARCHITECTURE] [-c CONFIGURATION] [--force] [-l LOCK] [--log-handler {console,syslog,journald}] [-q] [--report | --no-report] [-r REPOSITORY] [--unsafe] [-V] [--wait-timeout WAIT_TIMEOUT] {aur-search,search,help-commands-unsafe,help,help-updates,help-version,version,package-add,add,package-update,package-remove,remove,package-status,status,package-status-remove,package-status-update,status-update,patch-add,patch-list,patch-remove,patch-set-add,repo-backup,repo-check,check,repo-create-keyring,repo-create-mirrorlist,repo-daemon,daemon,repo-rebuild,rebuild,repo-remove-unknown,remove-unknown,repo-report,report,repo-restore,repo-sign,sign,repo-status-update,repo-sync,sync,repo-tree,repo-triggers,repo-update,update,service-clean,clean,repo-clean,service-config,config,repo-config,service-config-validate,config-validate,repo-config-validate,service-key-import,key-import,service-repositories,service-run,run,service-setup,init,repo-init,repo-setup,setup,service-shell,shell,service-tree-migrate,user-add,user-list,user-remove,web} ...
[-h] [-a ARCHITECTURE] [-c CONFIGURATION] [--force] [-l LOCK] [--log-handler {console,syslog,journald}] [-q] [--report | --no-report] [-r REPOSITORY] [--unsafe] [-V] [--wait-timeout WAIT_TIMEOUT] {aur-search,search,help-commands-unsafe,help,help-updates,help-version,version,package-add,add,package-update,package-changes,package-changes-remove,package-remove,remove,package-status,status,package-status-remove,package-status-update,status-update,patch-add,patch-list,patch-remove,patch-set-add,repo-backup,repo-check,check,repo-create-keyring,repo-create-mirrorlist,repo-daemon,daemon,repo-rebuild,rebuild,repo-remove-unknown,remove-unknown,repo-report,report,repo-restore,repo-sign,sign,repo-status-update,repo-sync,sync,repo-tree,repo-triggers,repo-update,update,service-clean,clean,repo-clean,service-config,config,repo-config,service-config-validate,config-validate,repo-config-validate,service-key-import,key-import,service-repositories,service-run,run,service-setup,init,repo-init,repo-setup,setup,service-shell,shell,service-tree-migrate,user-add,user-list,user-remove,web} ...
.SH DESCRIPTION
ArcH linux ReposItory MANager
@ -74,6 +74,12 @@ application version
\fBahriman\fR \fI\,package\-add\/\fR
add package
.TP
\fBahriman\fR \fI\,package\-changes\/\fR
get package changes
.TP
\fBahriman\fR \fI\,package\-changes\-remove\/\fR
remove package changes
.TP
\fBahriman\fR \fI\,package\-remove\/\fR
remove package
.TP
@ -285,6 +291,29 @@ build as user
\fB\-v\fR \fI\,VARIABLE\/\fR, \fB\-\-variable\fR \fI\,VARIABLE\/\fR
apply specified makepkg variables to the next build
.SH COMMAND \fI\,'ahriman package\-changes'\/\fR
usage: ahriman package\-changes [\-h] [\-e] package
retrieve package changes stored in database
.TP
\fBpackage\fR
package base
.SH OPTIONS \fI\,'ahriman package\-changes'\/\fR
.TP
\fB\-e\fR, \fB\-\-exit\-code\fR
return non\-zero exit status if result is empty
.SH COMMAND \fI\,'ahriman package\-changes\-remove'\/\fR
usage: ahriman package\-changes\-remove [\-h] package
remove the package changes stored remotely
.TP
\fBpackage\fR
package base
.SH COMMAND \fI\,'ahriman package\-remove'\/\fR
usage: ahriman package\-remove [\-h] package [package ...]
@ -418,7 +447,7 @@ backup repository settings and database
path of the output archive
.SH COMMAND \fI\,'ahriman repo\-check'\/\fR
usage: ahriman repo\-check [\-h] [\-e] [\-\-vcs | \-\-no\-vcs] [\-y] [package ...]
usage: ahriman repo\-check [\-h] [\-\-changes | \-\-no\-changes] [\-e] [\-\-vcs | \-\-no\-vcs] [\-y] [package ...]
check for packages updates. Same as repo\-update \-\-dry\-run \-\-no\-manual
@ -427,6 +456,10 @@ check for packages updates. Same as repo\-update \-\-dry\-run \-\-no\-manual
filter check by package base
.SH OPTIONS \fI\,'ahriman repo\-check'\/\fR
.TP
\fB\-\-changes\fR, \fB\-\-no\-changes\fR
calculate changes from the latest known commit if available. Only applicable in dry run mode
.TP
\fB\-e\fR, \fB\-\-exit\-code\fR
return non\-zero exit status if result is empty
@ -450,8 +483,9 @@ usage: ahriman repo\-create\-mirrorlist [\-h]
create package which contains list of available mirrors as set by configuration. Note, that this action will only create package, the package itself has to be built manually
.SH COMMAND \fI\,'ahriman repo\-daemon'\/\fR
usage: ahriman repo\-daemon [\-h] [\-i INTERVAL] [\-\-aur | \-\-no\-aur] [\-\-dependencies | \-\-no\-dependencies]
[\-\-local | \-\-no\-local] [\-\-manual | \-\-no\-manual] [\-\-vcs | \-\-no\-vcs] [\-y]
usage: ahriman repo\-daemon [\-h] [\-i INTERVAL] [\-\-aur | \-\-no\-aur] [\-\-changes | \-\-no\-changes]
[\-\-dependencies | \-\-no\-dependencies] [\-\-dry\-run] [\-\-local | \-\-no\-local]
[\-\-manual | \-\-no\-manual] [\-\-vcs | \-\-no\-vcs] [\-y]
start process which periodically will run update process
@ -464,10 +498,18 @@ interval between runs in seconds
\fB\-\-aur\fR, \fB\-\-no\-aur\fR
enable or disable checking for AUR updates
.TP
\fB\-\-changes\fR, \fB\-\-no\-changes\fR
calculate changes from the latest known commit if available. Only applicable in dry run mode
.TP
\fB\-\-dependencies\fR, \fB\-\-no\-dependencies\fR
process missing package dependencies
.TP
\fB\-\-dry\-run\fR
just perform check for updates, same as check command
.TP
\fB\-\-local\fR, \fB\-\-no\-local\fR
enable or disable checking of local packages for updates
@ -594,9 +636,9 @@ run triggers on empty build result as configured by settings
instead of running all triggers as set by configuration, just process specified ones in order of mention
.SH COMMAND \fI\,'ahriman repo\-update'\/\fR
usage: ahriman repo\-update [\-h] [\-\-aur | \-\-no\-aur] [\-\-dependencies | \-\-no\-dependencies] [\-\-dry\-run] [\-e]
[\-\-increment | \-\-no\-increment] [\-\-local | \-\-no\-local] [\-\-manual | \-\-no\-manual] [\-u USERNAME]
[\-\-vcs | \-\-no\-vcs] [\-y]
usage: ahriman repo\-update [\-h] [\-\-aur | \-\-no\-aur] [\-\-changes | \-\-no\-changes] [\-\-dependencies | \-\-no\-dependencies]
[\-\-dry\-run] [\-e] [\-\-increment | \-\-no\-increment] [\-\-local | \-\-no\-local]
[\-\-manual | \-\-no\-manual] [\-u USERNAME] [\-\-vcs | \-\-no\-vcs] [\-y]
[package ...]
check for packages updates and run build process if requested
@ -610,6 +652,10 @@ filter check by package base
\fB\-\-aur\fR, \fB\-\-no\-aur\fR
enable or disable checking for AUR updates
.TP
\fB\-\-changes\fR, \fB\-\-no\-changes\fR
calculate changes from the latest known commit if available. Only applicable in dry run mode
.TP
\fB\-\-dependencies\fR, \fB\-\-no\-dependencies\fR
process missing package dependencies

View File

@ -19,6 +19,8 @@ _shtab_ahriman_commands() {
"init:create initial service configuration, requires root"
"key-import:import PGP key from public sources to the repository user"
"package-add:add existing or new package to the build queue"
"package-changes:retrieve package changes stored in database"
"package-changes-remove:remove the package changes stored remotely"
"package-remove:remove package from the repository"
"package-status:request status of the package"
"package-status-remove:remove the package from the status page"
@ -117,6 +119,7 @@ _shtab_ahriman_aur_search_options=(
_shtab_ahriman_check_options=(
"(- : *)"{-h,--help}"[show this help message and exit]"
{--changes,--no-changes}"[calculate changes from the latest known commit if available. Only applicable in dry run mode (default\: True)]:changes:"
{-e,--exit-code}"[return non-zero exit status if result is empty (default\: False)]"
{--vcs,--no-vcs}"[fetch actual version of VCS packages (default\: True)]:vcs:"
"*"{-y,--refresh}"[download fresh package databases from the mirror before actions, -yy to force refresh even if up to date (default\: False)]"
@ -149,7 +152,9 @@ _shtab_ahriman_daemon_options=(
"(- : *)"{-h,--help}"[show this help message and exit]"
{-i,--interval}"[interval between runs in seconds (default\: 43200)]:interval:"
{--aur,--no-aur}"[enable or disable checking for AUR updates (default\: True)]:aur:"
{--changes,--no-changes}"[calculate changes from the latest known commit if available. Only applicable in dry run mode (default\: True)]:changes:"
{--dependencies,--no-dependencies}"[process missing package dependencies (default\: True)]:dependencies:"
"--dry-run[just perform check for updates, same as check command (default\: False)]"
{--local,--no-local}"[enable or disable checking of local packages for updates (default\: True)]:local:"
{--manual,--no-manual}"[include or exclude manual updates (default\: True)]:manual:"
{--vcs,--no-vcs}"[fetch actual version of VCS packages (default\: True)]:vcs:"
@ -210,6 +215,17 @@ _shtab_ahriman_package_add_options=(
"(*):package source (base name, path to local files, remote URL):"
)
_shtab_ahriman_package_changes_options=(
"(- : *)"{-h,--help}"[show this help message and exit]"
{-e,--exit-code}"[return non-zero exit status if result is empty (default\: False)]"
":package base:"
)
_shtab_ahriman_package_changes_remove_options=(
"(- : *)"{-h,--help}"[show this help message and exit]"
":package base:"
)
_shtab_ahriman_package_remove_options=(
"(- : *)"{-h,--help}"[show this help message and exit]"
"(*):package name or base:"
@ -302,6 +318,7 @@ _shtab_ahriman_repo_backup_options=(
_shtab_ahriman_repo_check_options=(
"(- : *)"{-h,--help}"[show this help message and exit]"
{--changes,--no-changes}"[calculate changes from the latest known commit if available. Only applicable in dry run mode (default\: True)]:changes:"
{-e,--exit-code}"[return non-zero exit status if result is empty (default\: False)]"
{--vcs,--no-vcs}"[fetch actual version of VCS packages (default\: True)]:vcs:"
"*"{-y,--refresh}"[download fresh package databases from the mirror before actions, -yy to force refresh even if up to date (default\: False)]"
@ -342,7 +359,9 @@ _shtab_ahriman_repo_daemon_options=(
"(- : *)"{-h,--help}"[show this help message and exit]"
{-i,--interval}"[interval between runs in seconds (default\: 43200)]:interval:"
{--aur,--no-aur}"[enable or disable checking for AUR updates (default\: True)]:aur:"
{--changes,--no-changes}"[calculate changes from the latest known commit if available. Only applicable in dry run mode (default\: True)]:changes:"
{--dependencies,--no-dependencies}"[process missing package dependencies (default\: True)]:dependencies:"
"--dry-run[just perform check for updates, same as check command (default\: False)]"
{--local,--no-local}"[enable or disable checking of local packages for updates (default\: True)]:local:"
{--manual,--no-manual}"[include or exclude manual updates (default\: True)]:manual:"
{--vcs,--no-vcs}"[fetch actual version of VCS packages (default\: True)]:vcs:"
@ -434,6 +453,7 @@ _shtab_ahriman_repo_triggers_options=(
_shtab_ahriman_repo_update_options=(
"(- : *)"{-h,--help}"[show this help message and exit]"
{--aur,--no-aur}"[enable or disable checking for AUR updates (default\: True)]:aur:"
{--changes,--no-changes}"[calculate changes from the latest known commit if available. Only applicable in dry run mode (default\: True)]:changes:"
{--dependencies,--no-dependencies}"[process missing package dependencies (default\: True)]:dependencies:"
"--dry-run[just perform check for updates, same as check command (default\: False)]"
{-e,--exit-code}"[return non-zero exit status if result is empty (default\: False)]"
@ -574,6 +594,7 @@ _shtab_ahriman_sync_options=(
_shtab_ahriman_update_options=(
"(- : *)"{-h,--help}"[show this help message and exit]"
{--aur,--no-aur}"[enable or disable checking for AUR updates (default\: True)]:aur:"
{--changes,--no-changes}"[calculate changes from the latest known commit if available. Only applicable in dry run mode (default\: True)]:changes:"
{--dependencies,--no-dependencies}"[process missing package dependencies (default\: True)]:dependencies:"
"--dry-run[just perform check for updates, same as check command (default\: False)]"
{-e,--exit-code}"[return non-zero exit status if result is empty (default\: False)]"
@ -644,6 +665,8 @@ _shtab_ahriman() {
init) _arguments -C -s $_shtab_ahriman_init_options ;;
key-import) _arguments -C -s $_shtab_ahriman_key_import_options ;;
package-add) _arguments -C -s $_shtab_ahriman_package_add_options ;;
package-changes) _arguments -C -s $_shtab_ahriman_package_changes_options ;;
package-changes-remove) _arguments -C -s $_shtab_ahriman_package_changes_remove_options ;;
package-remove) _arguments -C -s $_shtab_ahriman_package_remove_options ;;
package-status) _arguments -C -s $_shtab_ahriman_package_status_options ;;
package-status-remove) _arguments -C -s $_shtab_ahriman_package_status_remove_options ;;

View File

@ -101,6 +101,8 @@ def _parser() -> argparse.ArgumentParser:
_set_help_updates_parser(subparsers)
_set_help_version_parser(subparsers)
_set_package_add_parser(subparsers)
_set_package_changes_parser(subparsers)
_set_package_changes_remove_parser(subparsers)
_set_package_remove_parser(subparsers)
_set_package_status_parser(subparsers)
_set_package_status_remove_parser(subparsers)
@ -281,6 +283,44 @@ def _set_package_add_parser(root: SubParserAction) -> argparse.ArgumentParser:
return parser
def _set_package_changes_parser(root: SubParserAction) -> argparse.ArgumentParser:
"""
add parser for package changes subcommand
Args:
root(SubParserAction): subparsers for the commands
Returns:
argparse.ArgumentParser: created argument parser
"""
parser = root.add_parser("package-changes", help="get package changes",
description="retrieve package changes stored in database",
epilog="This feature requests package status from the web interface if it is available.",
formatter_class=_formatter)
parser.add_argument("package", help="package base")
parser.add_argument("-e", "--exit-code", help="return non-zero exit status if result is empty", action="store_true")
parser.set_defaults(handler=handlers.Change, action=Action.List, lock=None, quiet=True, report=False, unsafe=True)
return parser
def _set_package_changes_remove_parser(root: SubParserAction) -> argparse.ArgumentParser:
"""
add parser for package change remove subcommand
Args:
root(SubParserAction): subparsers for the commands
Returns:
argparse.ArgumentParser: created argument parser
"""
parser = root.add_parser("package-changes-remove", help="remove package changes",
description="remove the package changes stored remotely",
formatter_class=_formatter)
parser.add_argument("package", help="package base")
parser.set_defaults(handler=handlers.Change, action=Action.Remove, lock=None, quiet=True, report=False, unsafe=True)
return parser
def _set_package_remove_parser(root: SubParserAction) -> argparse.ArgumentParser:
"""
add parser for package removal subcommand
@ -493,6 +533,9 @@ def _set_repo_check_parser(root: SubParserAction) -> argparse.ArgumentParser:
description="check for packages updates. Same as repo-update --dry-run --no-manual",
formatter_class=_formatter)
parser.add_argument("package", help="filter check by package base", nargs="*")
parser.add_argument("--changes", help="calculate changes from the latest known commit if available. "
"Only applicable in dry run mode",
action=argparse.BooleanOptionalAction, default=True)
parser.add_argument("-e", "--exit-code", help="return non-zero exit status if result is empty", action="store_true")
parser.add_argument("--vcs", help="fetch actual version of VCS packages",
action=argparse.BooleanOptionalAction, default=True)
@ -558,8 +601,12 @@ def _set_repo_daemon_parser(root: SubParserAction) -> argparse.ArgumentParser:
parser.add_argument("-i", "--interval", help="interval between runs in seconds", type=int, default=60 * 60 * 12)
parser.add_argument("--aur", help="enable or disable checking for AUR updates",
action=argparse.BooleanOptionalAction, default=True)
parser.add_argument("--changes", help="calculate changes from the latest known commit if available. "
"Only applicable in dry run mode",
action=argparse.BooleanOptionalAction, default=True)
parser.add_argument("--dependencies", help="process missing package dependencies",
action=argparse.BooleanOptionalAction, default=True)
parser.add_argument("--dry-run", help="just perform check for updates, same as check command", action="store_true")
parser.add_argument("--local", help="enable or disable checking of local packages for updates",
action=argparse.BooleanOptionalAction, default=True)
parser.add_argument("--manual", help="include or exclude manual updates",
@ -569,7 +616,7 @@ def _set_repo_daemon_parser(root: SubParserAction) -> argparse.ArgumentParser:
parser.add_argument("-y", "--refresh", help="download fresh package databases from the mirror before actions, "
"-yy to force refresh even if up to date",
action="count", default=False)
parser.set_defaults(handler=handlers.Daemon, dry_run=False, exit_code=False, package=[])
parser.set_defaults(handler=handlers.Daemon, exit_code=False, package=[])
return parser
@ -769,6 +816,9 @@ def _set_repo_update_parser(root: SubParserAction) -> argparse.ArgumentParser:
parser.add_argument("package", help="filter check by package base", nargs="*")
parser.add_argument("--aur", help="enable or disable checking for AUR updates",
action=argparse.BooleanOptionalAction, default=True)
parser.add_argument("--changes", help="calculate changes from the latest known commit if available. "
"Only applicable in dry run mode",
action=argparse.BooleanOptionalAction, default=True)
parser.add_argument("--dependencies", help="process missing package dependencies",
action=argparse.BooleanOptionalAction, default=True)
parser.add_argument("--dry-run", help="just perform check for updates, same as check command", action="store_true")

View File

@ -117,7 +117,7 @@ class Application(ApplicationPackages, ApplicationRepository):
Returns:
list[Package]: updated packages list. Packager for dependencies will be copied from
original package
original package
"""
def missing_dependencies(source: Iterable[Package]) -> dict[str, str | None]:
# append list of known packages with packages which are in current sources
@ -150,7 +150,7 @@ class Application(ApplicationPackages, ApplicationRepository):
with_dependencies[package.base] = package
# register package in local database
self.database.remote_update(package)
self.database.package_base_update(package)
self.repository.reporter.set_unknown(package)
return list(with_dependencies.values())

View File

@ -65,7 +65,7 @@ class ApplicationPackages(ApplicationProperties):
"""
package = Package.from_aur(source, username)
self.database.build_queue_insert(package)
self.database.remote_update(package)
self.database.package_base_update(package)
def _add_directory(self, source: str, *_: Any) -> None:
"""
@ -139,7 +139,7 @@ class ApplicationPackages(ApplicationProperties):
"""
package = Package.from_official(source, self.repository.pacman, username)
self.database.build_queue_insert(package)
self.database.remote_update(package)
self.database.package_base_update(package)
def add(self, names: Iterable[str], source: PackageSource, username: str | None = None) -> None:
"""

View File

@ -18,11 +18,10 @@
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
from collections.abc import Iterable
from pathlib import Path
from ahriman.application.application.application_properties import ApplicationProperties
from ahriman.application.application.workers import Updater
from ahriman.core.build_tools.sources import Sources
from ahriman.core.tree import Tree
from ahriman.models.package import Package
from ahriman.models.packagers import Packagers
from ahriman.models.result import Result
@ -33,6 +32,23 @@ class ApplicationRepository(ApplicationProperties):
repository control class
"""
def changes(self, packages: Iterable[Package]) -> None:
"""
generate and update package changes
Args:
packages(Iterable[Package]): list of packages to retrieve changes
"""
last_commit_hashes = self.database.hashes_get()
for package in packages:
last_commit_sha = last_commit_hashes.get(package.base)
if last_commit_sha is None:
continue # skip check in case if we can't calculate diff
changes = self.repository.package_changes(package, last_commit_sha)
self.repository.reporter.package_changes_set(package.base, changes)
def clean(self, *, cache: bool, chroot: bool, manual: bool, packages: bool, pacman: bool) -> None:
"""
run all clean methods. Warning: some functions might not be available under non-root
@ -137,26 +153,25 @@ class ApplicationRepository(ApplicationProperties):
Returns:
Result: update result
"""
def process_update(paths: Iterable[Path], result: Result) -> None:
if not paths:
return # don't need to process if no update supplied
update_result = self.repository.process_update(paths, packagers)
self.on_result(result.merge(update_result))
result = Result()
# process built packages
build_result = Result()
packages = self.repository.packages_built()
process_update(packages, build_result)
# process already built packages if any
built_packages = self.repository.packages_built()
if built_packages: # speedup a bit
build_result = self.repository.process_update(built_packages, packagers)
result.merge(build_result)
self.on_result(result.merge(build_result))
# process manual packages
tree = Tree.resolve(updates)
for num, level in enumerate(tree):
self.logger.info("processing level #%i %s", num, [package.base for package in level])
build_result = self.repository.process_build(level, packagers, bump_pkgrel=bump_pkgrel)
packages = self.repository.packages_built()
process_update(packages, build_result)
builder = Updater.load(self.repository_id, self.configuration, self.repository)
return build_result
# ok so for now we split all packages into chunks and process each chunk accordingly
partitions = builder.partition(updates)
for num, partition in enumerate(partitions):
self.logger.info("processing chunk #%i %s", num, [package.base for package in partition])
build_result = builder.update(partition, packagers, bump_pkgrel=bump_pkgrel)
self.on_result(result.merge(build_result))
return result
def updates(self, filter_packages: Iterable[str], *,
aur: bool, local: bool, manual: bool, vcs: bool) -> list[Package]:

View File

@ -0,0 +1,20 @@
#
# Copyright (c) 2021-2023 ahriman team.
#
# This file is part of ahriman
# (see https://github.com/arcan1s/ahriman).
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
from ahriman.application.application.workers.updater import Updater

View File

@ -0,0 +1,77 @@
#
# Copyright (c) 2021-2023 ahriman team.
#
# This file is part of ahriman
# (see https://github.com/arcan1s/ahriman).
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
from collections.abc import Iterable
from ahriman.application.application.workers.updater import Updater
from ahriman.core.repository import Repository
from ahriman.core.tree import Tree
from ahriman.models.package import Package
from ahriman.models.packagers import Packagers
from ahriman.models.result import Result
class LocalUpdater(Updater):
"""
local build process implementation
Attributes:
repository(Repository): repository instance
"""
def __init__(self, repository: Repository) -> None:
"""
default constructor
Args:
repository(Repository): repository instance
"""
self.repository = repository
def partition(self, packages: Iterable[Package]) -> list[list[Package]]:
"""
split packages into partitions to be processed by this worker
Args:
packages(Iterable[Package]): list of packages to partition
Returns:
list[list[Package]]: packages partitioned by this worker type
"""
return Tree.resolve(packages)
def update(self, updates: Iterable[Package], packagers: Packagers | None = None, *,
bump_pkgrel: bool = False) -> Result:
"""
run package updates
Args:
updates(Iterable[Package]): list of packages to update
packagers(Packagers | None, optional): optional override of username for build process
(Default value = None)
bump_pkgrel(bool, optional): bump pkgrel in case of local version conflict (Default value = False)
Returns:
Result: update result
"""
build_result = self.repository.process_build(updates, packagers, bump_pkgrel=bump_pkgrel)
packages = self.repository.packages_built()
update_result = self.repository.process_update(packages, packagers)
return build_result.merge(update_result)

View File

@ -0,0 +1,140 @@
#
# Copyright (c) 2021-2023 ahriman team.
#
# This file is part of ahriman
# (see https://github.com/arcan1s/ahriman).
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
from collections import deque
from collections.abc import Iterable
from ahriman.application.application.workers.updater import Updater
from ahriman.core.configuration import Configuration
from ahriman.core.http import SyncAhrimanClient
from ahriman.core.tree import Tree
from ahriman.models.package import Package
from ahriman.models.packagers import Packagers
from ahriman.models.repository_id import RepositoryId
from ahriman.models.result import Result
from ahriman.models.worker import Worker
class RemoteUpdater(Updater):
"""
remote update worker
Attributes:
configuration(Configuration): configuration instance
repository_id(RepositoryId): repository unique identifier
workers(list[Worker]): worker identifiers
"""
def __init__(self, workers: list[Worker], repository_id: RepositoryId, configuration: Configuration) -> None:
"""
default constructor
Args:
workers(list[Worker]): worker identifiers
repository_id(RepositoryId): repository unique identifier
configuration(Configuration): configuration instance
"""
self.workers = workers
self.repository_id = repository_id
self.configuration = configuration
self._clients: deque[tuple[Worker, SyncAhrimanClient]] = deque()
@property
def clients(self) -> dict[Worker, SyncAhrimanClient]:
"""
extract loaded clients. Note that this method yields only workers which have been already loaded
Returns:
dict[Worker, SyncAhrimanClient]: map of the worker to the related web client
"""
return dict(self._clients)
@staticmethod
def _update_url(worker: Worker) -> str:
"""
get url for updates
Args:
worker(Worker): worker identifier
Returns:
str: full url for web service to run update process
"""
return f"{worker.address}/api/v1/service/add"
def next_worker(self) -> tuple[Worker, SyncAhrimanClient]:
"""
generate next not-used web client. In case if all clients have been already used, it yields next not used client
Returns:
tuple[Worker, SyncAhrimanClient]: worker and constructed client instance for the web
"""
# check if there is not used yet worker
worker = next((worker for worker in self.workers if worker not in self.clients), None)
if worker is not None:
client = SyncAhrimanClient(self.configuration, "status")
client.address = worker.address
else:
worker, client = self._clients.popleft()
# register worker in the queue
self._clients.append((worker, client))
return worker, client
def partition(self, packages: Iterable[Package]) -> list[list[Package]]:
"""
split packages into partitions to be processed by this worker
Args:
packages(Iterable[Package]): list of packages to partition
Returns:
list[list[Package]]: packages partitioned by this worker type
"""
return Tree.partition(packages, count=len(self.workers))
def update(self, updates: Iterable[Package], packagers: Packagers | None = None, *,
bump_pkgrel: bool = False) -> Result:
"""
run package updates
Args:
updates(Iterable[Package]): list of packages to update
packagers(Packagers | None, optional): optional override of username for build process
(Default value = None)
bump_pkgrel(bool, optional): bump pkgrel in case of local version conflict (Default value = False)
Returns:
Result: update result
"""
payload = {
"increment": bump_pkgrel,
"packager": packagers.default if packagers is not None else None,
"packages": [package.base for package in updates],
"patches": [], # might be used later
"refresh": True,
}
worker, client = self.next_worker()
client.make_request("POST", self._update_url(worker), params=self.repository_id.query(), json=payload)
# we don't block here for process
return Result()

View File

@ -0,0 +1,102 @@
#
# Copyright (c) 2021-2023 ahriman team.
#
# This file is part of ahriman
# (see https://github.com/arcan1s/ahriman).
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
from __future__ import annotations
from collections.abc import Iterable
from ahriman.core.configuration import Configuration
from ahriman.core.log import LazyLogging
from ahriman.core.repository import Repository
from ahriman.models.package import Package
from ahriman.models.packagers import Packagers
from ahriman.models.repository_id import RepositoryId
from ahriman.models.result import Result
from ahriman.models.worker import Worker
class Updater(LazyLogging):
"""
updater handler interface
Attributes:
split_method(Callable[[Iterable[Package]], list[list[Package]]]): method to split packages into chunks
"""
@staticmethod
def load(repository_id: RepositoryId, configuration: Configuration,
repository: Repository, workers: list[Worker] | None = None) -> Updater:
"""
construct updaters from parameters
Args:
repository_id(RepositoryId): repository unique identifier
configuration(Configuration): configuration instance
repository(Repository): repository instance
workers(list[Worker] | None, optional): worker identifiers if any (Default value = None)
Returns:
Updater: constructed updater worker
"""
if workers is None:
# no workers set explicitly, try to guess from configuration
workers = [Worker(address) for address in configuration.getlist("build", "workers", fallback=[])]
if workers:
# there is something we could use as remote workers
from ahriman.application.application.workers.remote_updater import RemoteUpdater
return RemoteUpdater(workers, repository_id, configuration)
# and finally no workers available, just use local service
from ahriman.application.application.workers.local_updater import LocalUpdater
return LocalUpdater(repository)
def partition(self, packages: Iterable[Package]) -> list[list[Package]]:
"""
split packages into partitions to be processed by this worker
Args:
packages(Iterable[Package]): list of packages to partition
Returns:
list[list[Package]]: packages partitioned by this worker type
Raises:
NotImplementedError: not implemented method
"""
raise NotImplementedError
def update(self, updates: Iterable[Package], packagers: Packagers | None = None, *,
bump_pkgrel: bool = False) -> Result:
"""
run package updates
Args:
updates(Iterable[Package]): list of packages to update
packagers(Packagers | None, optional): optional override of username for build process
(Default value = None)
bump_pkgrel(bool, optional): bump pkgrel in case of local version conflict (Default value = False)
Returns:
Result: update result
Raises:
NotImplementedError: not implemented method
"""
raise NotImplementedError

View File

@ -21,6 +21,7 @@ from ahriman.application.handlers.handler import Handler
from ahriman.application.handlers.add import Add
from ahriman.application.handlers.backup import Backup
from ahriman.application.handlers.change import Change
from ahriman.application.handlers.clean import Clean
from ahriman.application.handlers.daemon import Daemon
from ahriman.application.handlers.dump import Dump

View File

@ -0,0 +1,59 @@
#
# Copyright (c) 2021-2023 ahriman team.
#
# This file is part of ahriman
# (see https://github.com/arcan1s/ahriman).
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
import argparse
from ahriman.application.application import Application
from ahriman.application.handlers import Handler
from ahriman.core.configuration import Configuration
from ahriman.core.formatters import ChangesPrinter
from ahriman.models.action import Action
from ahriman.models.changes import Changes
from ahriman.models.repository_id import RepositoryId
class Change(Handler):
"""
package changes handler
"""
ALLOW_MULTI_ARCHITECTURE_RUN = False # conflicting io
@classmethod
def run(cls, args: argparse.Namespace, repository_id: RepositoryId, configuration: Configuration, *,
report: bool) -> None:
"""
callback for command line
Args:
args(argparse.Namespace): command line args
repository_id(RepositoryId): repository unique identifier
configuration(Configuration): configuration instance
report(bool): force enable or disable reporting
"""
application = Application(repository_id, configuration, report=True)
client = application.repository.reporter
match args.action:
case Action.List:
changes = client.package_changes_get(args.package)
ChangesPrinter(changes)(verbose=True, separator="")
Change.check_if_empty(args.exit_code, changes.is_empty)
case Action.Remove:
client.package_changes_set(args.package, Changes())

View File

@ -43,8 +43,10 @@ class Daemon(Handler):
report(bool): force enable or disable reporting
"""
from ahriman.application.handlers import Update
Update.run(args, repository_id, configuration, report=report)
timer = threading.Timer(args.interval, Daemon.run, args=[args, repository_id, configuration],
kwargs={"report": report})
timer.start()
timer.join()
event = threading.Event()
try:
while not event.wait(args.interval):
Update.run(args, repository_id, configuration, report=report)
except KeyboardInterrupt:
pass # normal exit

View File

@ -79,7 +79,7 @@ class Patch(Handler):
Returns:
tuple[str, PkgbuildPatch]: package base and created PKGBUILD patch based on the diff from master HEAD
to current files
to current files
"""
package = Package.from_build(sources_dir, architecture, None)
patch = Sources.patch_create(sources_dir, *track)

View File

@ -47,9 +47,13 @@ class Update(Handler):
"""
application = Application(repository_id, configuration, report=report, refresh_pacman_database=args.refresh)
application.on_start()
packages = application.updates(args.package, aur=args.aur, local=args.local, manual=args.manual, vcs=args.vcs)
Update.check_if_empty(args.exit_code, not packages)
if args.dry_run:
if args.dry_run: # some check specific actions
if args.changes: # generate changes if requested
application.changes(packages)
Update.check_if_empty(args.exit_code, not packages) # status code check
return
packages = application.with_dependencies(packages, process_dependencies=args.dependencies)

View File

@ -125,7 +125,7 @@ class Validate(Handler):
Returns:
dict[str, Any]: schema with added elements from source schema if they were set before and not presented
in the new one. Note, that schema will be modified in-place
in the new one. Note, that schema will be modified in-place
"""
for key, value in source.items():
if key not in schema:

View File

@ -20,7 +20,7 @@
from typing import Any
try:
import aiohttp_security # type: ignore[import-untyped]
import aiohttp_security
_has_aiohttp_security = True
except ImportError:
_has_aiohttp_security = False

View File

@ -21,6 +21,7 @@ import shutil
from pathlib import Path
from ahriman.core.exceptions import CalledProcessError
from ahriman.core.log import LazyLogging
from ahriman.core.util import check_output, utcnow, walk
from ahriman.models.package import Package
@ -42,6 +43,25 @@ class Sources(LazyLogging):
DEFAULT_BRANCH = "master" # default fallback branch
DEFAULT_COMMIT_AUTHOR = ("ahriman", "ahriman@localhost")
@staticmethod
def changes(source_dir: Path, last_commit_sha: str | None) -> str | None:
"""
extract changes from the last known commit if available
Args:
source_dir(Path): local path to directory with source files
last_commit_sha(str | None): last known commit hash
Returns:
str | None: changes from the last commit if available or ``None`` otherwise
"""
if last_commit_sha is None:
return None # no previous reference found
instance = Sources()
instance.fetch_until(source_dir, commit_sha=last_commit_sha)
return instance.diff(source_dir, last_commit_sha)
@staticmethod
def extend_architectures(sources_dir: Path, architecture: str) -> list[PkgbuildPatch]:
"""
@ -61,13 +81,16 @@ class Sources(LazyLogging):
return [PkgbuildPatch("arch", list(architectures))]
@staticmethod
def fetch(sources_dir: Path, remote: RemoteSource) -> None:
def fetch(sources_dir: Path, remote: RemoteSource) -> str | None:
"""
either clone repository or update it to origin/``remote.branch``
Args:
sources_dir(Path): local path to fetch
remote(RemoteSource): remote target (from where to fetch)
Returns:
str | None: current commit sha if available
"""
instance = Sources()
# local directory exists and there is .git directory
@ -75,13 +98,12 @@ class Sources(LazyLogging):
if is_initialized_git and not instance.has_remotes(sources_dir):
# there is git repository, but no remote configured so far
instance.logger.info("skip update at %s because there are no branches configured", sources_dir)
return
return instance.head(sources_dir)
branch = remote.branch or instance.DEFAULT_BRANCH
if is_initialized_git:
instance.logger.info("update HEAD to remote at %s using branch %s", sources_dir, branch)
check_output("git", "fetch", "--quiet", "--depth", "1", "origin", branch,
cwd=sources_dir, logger=instance.logger)
instance.fetch_until(sources_dir, branch=branch)
elif remote.git_url is not None:
instance.logger.info("clone remote %s to %s using branch %s", remote.git_url, sources_dir, branch)
check_output("git", "clone", "--quiet", "--depth", "1", "--branch", branch, "--single-branch",
@ -100,6 +122,8 @@ class Sources(LazyLogging):
pkgbuild_dir = remote.pkgbuild_dir or sources_dir.resolve()
instance.move((sources_dir / pkgbuild_dir).resolve(), sources_dir)
return instance.head(sources_dir)
@staticmethod
def has_remotes(sources_dir: Path) -> bool:
"""
@ -136,7 +160,7 @@ class Sources(LazyLogging):
instance.commit(sources_dir)
@staticmethod
def load(sources_dir: Path, package: Package, patches: list[PkgbuildPatch], paths: RepositoryPaths) -> None:
def load(sources_dir: Path, package: Package, patches: list[PkgbuildPatch], paths: RepositoryPaths) -> str | None:
"""
fetch sources from remote and apply patches
@ -145,17 +169,22 @@ class Sources(LazyLogging):
package(Package): package definitions
patches(list[PkgbuildPatch]): optional patch to be applied
paths(RepositoryPaths): repository paths instance
Returns:
str | None: current commit sha if available
"""
instance = Sources()
if (cache_dir := paths.cache_for(package.base)).is_dir() and cache_dir != sources_dir:
# no need to clone whole repository, just copy from cache first
shutil.copytree(cache_dir, sources_dir, dirs_exist_ok=True)
instance.fetch(sources_dir, package.remote)
last_commit_sha = instance.fetch(sources_dir, package.remote)
patches.extend(instance.extend_architectures(sources_dir, paths.repository_id.architecture))
for patch in patches:
instance.patch_apply(sources_dir, patch)
return last_commit_sha
@staticmethod
def patch_create(sources_dir: Path, *pattern: str) -> str:
"""
@ -247,17 +276,47 @@ class Sources(LazyLogging):
return True
def diff(self, sources_dir: Path) -> str:
def diff(self, sources_dir: Path, sha: str | None = None) -> str:
"""
generate diff from the current version and write it to the output file
Args:
sources_dir(Path): local path to git repository
sha(str | None, optional): optional commit sha to calculate diff (Default value = None)
Returns:
str: patch as plain string
"""
return check_output("git", "diff", cwd=sources_dir, logger=self.logger)
args = []
if sha is not None:
args.append(sha)
return check_output("git", "diff", *args, cwd=sources_dir, logger=self.logger)
def fetch_until(self, sources_dir: Path, *, branch: str | None = None, commit_sha: str | None = None) -> None:
"""
fetch repository until commit sha
Args:
sources_dir(Path): local path to git repository
branch(str | None, optional): use specified branch (Default value = None)
commit_sha(str | None, optional): commit hash to fetch. If none set, only one will be fetched
(Default value = None)
"""
commit_sha = commit_sha or "HEAD" # if none set we just fetch the last commit
commits_count = 1
while commit_sha is not None:
command = ["git", "fetch", "--quiet", "--depth", str(commits_count)]
if branch is not None:
command += ["origin", branch]
check_output(*command, cwd=sources_dir, logger=self.logger) # fetch one more level
try:
# check if there is an object in current git directory
check_output("git", "cat-file", "-e", commit_sha, cwd=sources_dir, logger=self.logger)
commit_sha = None # reset search
except CalledProcessError:
commits_count += 1 # increase depth
def has_changes(self, sources_dir: Path) -> bool:
"""
@ -273,6 +332,20 @@ class Sources(LazyLogging):
changes = check_output("git", "diff", "--cached", "--name-only", cwd=sources_dir, logger=self.logger)
return bool(changes)
def head(self, sources_dir: Path, ref_name: str = "HEAD") -> str:
"""
extract HEAD reference for the current git repository
Args:
sources_dir(Path): local path to git repository
ref_name(str, optional): reference name (Default value = "HEAD")
Returns:
str: HEAD commit hash
"""
# we might want to parse git files instead though
return check_output("git", "rev-parse", ref_name, cwd=sources_dir)
def move(self, pkgbuild_dir: Path, sources_dir: Path) -> None:
"""
move content from pkgbuild_dir to sources_dir

View File

@ -109,7 +109,7 @@ class Task(LazyLogging):
).splitlines()
return [Path(package) for package in packages]
def init(self, sources_dir: Path, database: SQLite, local_version: str | None) -> None:
def init(self, sources_dir: Path, database: SQLite, local_version: str | None) -> str | None:
"""
fetch package from git
@ -118,10 +118,13 @@ class Task(LazyLogging):
database(SQLite): database instance
local_version(str | None): local version of the package. If set and equal to current version, it will
automatically bump pkgrel
Returns:
str | None: current commit sha if available
"""
Sources.load(sources_dir, self.package, database.patches_get(self.package.base), self.paths)
last_commit_sha = Sources.load(sources_dir, self.package, database.patches_get(self.package.base), self.paths)
if local_version is None:
return # there is no local package or pkgrel increment is disabled
return last_commit_sha # there is no local package or pkgrel increment is disabled
# load fresh package
loaded_package = Package.from_build(sources_dir, self.architecture, None)
@ -129,3 +132,5 @@ class Task(LazyLogging):
self.logger.info("package %s is the same as in repo, bumping pkgrel to %s", self.package.base, pkgrel)
patch = PkgbuildPatch("pkgrel", pkgrel)
patch.write(sources_dir / "PKGBUILD")
return last_commit_sha

View File

@ -213,6 +213,15 @@ CONFIGURATION_SCHEMA: ConfigurationSchema = {
"coerce": "integer",
"min": 0,
},
"workers": {
"type": "list",
"coerce": "list",
"schema": {
"type": "string",
"empty": False,
"is_url": [],
},
},
},
},
"repository": {
@ -333,6 +342,10 @@ CONFIGURATION_SCHEMA: ConfigurationSchema = {
"min": 0,
"max": 65535,
},
"service_only": {
"type": "boolean",
"coerce": "boolean",
},
"static_path": {
"type": "path",
"coerce": "absolute_path",

View File

@ -0,0 +1,33 @@
#
# Copyright (c) 2021-2023 ahriman team.
#
# This file is part of ahriman
# (see https://github.com/arcan1s/ahriman).
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
__all__ = ["steps"]
steps = [
"""
create table package_changes (
package_base text not null,
repository text not null,
last_commit_sha text not null,
changes text,
unique (package_base, repository)
)
""",
]

View File

@ -21,6 +21,7 @@ from ahriman.core.database.operations.operations import Operations
from ahriman.core.database.operations.auth_operations import AuthOperations
from ahriman.core.database.operations.build_operations import BuildOperations
from ahriman.core.database.operations.changes_operations import ChangesOperations
from ahriman.core.database.operations.logs_operations import LogsOperations
from ahriman.core.database.operations.package_operations import PackageOperations
from ahriman.core.database.operations.patch_operations import PatchOperations

View File

@ -0,0 +1,143 @@
#
# Copyright (c) 2021-2023 ahriman team.
#
# This file is part of ahriman
# (see https://github.com/arcan1s/ahriman).
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
from sqlite3 import Connection
from ahriman.core.database.operations import Operations
from ahriman.models.changes import Changes
from ahriman.models.repository_id import RepositoryId
class ChangesOperations(Operations):
"""
operations for source files changes
"""
def changes_get(self, package_base: str, repository_id: RepositoryId | None = None) -> Changes:
"""
get changes for the specific package base if available
Args:
package_base(str): package base to search
repository_id(RepositoryId, optional): repository unique identifier override (Default value = None)
Returns:
Changes: changes for the package base if available
"""
repository_id = repository_id or self._repository_id
def run(connection: Connection) -> Changes:
return next(
(
Changes(row["last_commit_sha"], row["changes"] or None)
for row in connection.execute(
"""
select last_commit_sha, changes from package_changes
where package_base = :package_base and repository = :repository
""",
{
"package_base": package_base,
"repository": repository_id.id,
}
)
),
Changes()
)
return self.with_connection(run)
def changes_insert(self, package_base: str, changes: Changes, repository_id: RepositoryId | None = None) -> None:
"""
insert packages to build queue
Args:
package_base(str): package base to insert
changes(Changes): package changes (as in patch format)
repository_id(RepositoryId, optional): repository unique identifier override (Default value = None)
"""
repository_id = repository_id or self._repository_id
def run(connection: Connection) -> None:
connection.execute(
"""
insert into package_changes
(package_base, last_commit_sha, changes, repository)
values
(:package_base, :last_commit_sha, :changes ,:repository)
on conflict (package_base, repository) do update set
last_commit_sha = :last_commit_sha, changes = :changes
""",
{
"package_base": package_base,
"last_commit_sha": changes.last_commit_sha,
"changes": changes.changes,
"repository": repository_id.id,
})
if changes.last_commit_sha is None:
return self.changes_remove(package_base, repository_id)
return self.with_connection(run, commit=True)
def changes_remove(self, package_base: str | None, repository_id: RepositoryId | None = None) -> None:
"""
remove packages changes
Args:
package_base(str | None): optional filter by package base
repository_id(RepositoryId, optional): repository unique identifier override (Default value = None)
"""
repository_id = repository_id or self._repository_id
def run(connection: Connection) -> None:
connection.execute(
"""
delete from package_changes
where (:package_base is null or package_base = :package_base)
and repository = :repository
""",
{
"package_base": package_base,
"repository": repository_id.id,
})
return self.with_connection(run, commit=True)
def hashes_get(self, repository_id: RepositoryId | None = None) -> dict[str, str]:
"""
extract last commit hashes if available
Args:
repository_id(RepositoryId, optional): repository unique identifier override (Default value = None)
Returns:
dict[str, str]: map of package base to its last commit hash
"""
repository_id = repository_id or self._repository_id
def run(connection: Connection) -> dict[str, str]:
return {
row["package_base"]: row["last_commit_sha"]
for row in connection.execute(
"""select package_base, last_commit_sha from package_changes where repository = :repository""",
{"repository": repository_id.id}
)
}
return self.with_connection(run)

View File

@ -246,6 +246,21 @@ class PackageOperations(Operations):
)
}
def package_base_update(self, package: Package, repository_id: RepositoryId | None = None) -> None:
"""
update package base only
Args:
package(Package): package properties
repository_id(RepositoryId, optional): repository unique identifier override (Default value = None)
"""
repository_id = repository_id or self._repository_id
def run(connection: Connection) -> None:
self._package_update_insert_base(connection, package, repository_id)
return self.with_connection(run, commit=True)
def package_remove(self, package_base: str, repository_id: RepositoryId | None = None) -> None:
"""
remove package from database
@ -302,21 +317,6 @@ class PackageOperations(Operations):
return self.with_connection(lambda connection: list(run(connection)))
def remote_update(self, package: Package, repository_id: RepositoryId | None = None) -> None:
"""
update package remote source
Args:
package(Package): package properties
repository_id(RepositoryId, optional): repository unique identifier override (Default value = None)
"""
repository_id = repository_id or self._repository_id
def run(connection: Connection) -> None:
self._package_update_insert_base(connection, package, repository_id)
return self.with_connection(run, commit=True)
def remotes_get(self, repository_id: RepositoryId | None = None) -> dict[str, RemoteSource]:
"""
get packages remotes based on current settings

View File

@ -25,11 +25,12 @@ from typing import Self
from ahriman.core.configuration import Configuration
from ahriman.core.database.migrations import Migrations
from ahriman.core.database.operations import AuthOperations, BuildOperations, LogsOperations, PackageOperations, \
PatchOperations
from ahriman.core.database.operations import AuthOperations, BuildOperations, ChangesOperations, LogsOperations, \
PackageOperations, PatchOperations
class SQLite(AuthOperations, BuildOperations, LogsOperations, PackageOperations, PatchOperations):
# pylint: disable=too-many-ancestors
class SQLite(AuthOperations, BuildOperations, ChangesOperations, LogsOperations, PackageOperations, PatchOperations):
"""
wrapper for sqlite3 database

View File

@ -18,16 +18,17 @@
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
from ahriman.core.formatters.printer import Printer
from ahriman.core.formatters.string_printer import StringPrinter
from ahriman.core.formatters.aur_printer import AurPrinter
from ahriman.core.formatters.build_printer import BuildPrinter
from ahriman.core.formatters.changes_printer import ChangesPrinter
from ahriman.core.formatters.configuration_paths_printer import ConfigurationPathsPrinter
from ahriman.core.formatters.configuration_printer import ConfigurationPrinter
from ahriman.core.formatters.package_printer import PackagePrinter
from ahriman.core.formatters.patch_printer import PatchPrinter
from ahriman.core.formatters.repository_printer import RepositoryPrinter
from ahriman.core.formatters.status_printer import StatusPrinter
from ahriman.core.formatters.string_printer import StringPrinter
from ahriman.core.formatters.tree_printer import TreePrinter
from ahriman.core.formatters.update_printer import UpdatePrinter
from ahriman.core.formatters.user_printer import UserPrinter

View File

@ -17,7 +17,7 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
from ahriman.core.formatters import StringPrinter
from ahriman.core.formatters.string_printer import StringPrinter
from ahriman.core.util import pretty_datetime
from ahriman.models.aur_package import AURPackage
from ahriman.models.property import Property

View File

@ -17,7 +17,7 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
from ahriman.core.formatters import StringPrinter
from ahriman.core.formatters.string_printer import StringPrinter
from ahriman.models.package import Package

View File

@ -0,0 +1,64 @@
#
# Copyright (c) 2021-2023 ahriman team.
#
# This file is part of ahriman
# (see https://github.com/arcan1s/ahriman).
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
from ahriman.core.formatters import Printer
from ahriman.models.changes import Changes
from ahriman.models.property import Property
class ChangesPrinter(Printer):
"""
print content of the changes object
Attributes:
changes(Changes): package changes
"""
def __init__(self, changes: Changes) -> None:
"""
default constructor
Args:
changes(Changes): package changes
"""
Printer.__init__(self)
self.changes = changes
def properties(self) -> list[Property]:
"""
convert content into printable data
Returns:
list[Property]: list of content properties
"""
if self.changes.is_empty:
return []
return [Property("", self.changes.changes, is_required=True, indent=0)]
# pylint: disable=redundant-returns-doc
def title(self) -> str | None:
"""
generate entry title from content
Returns:
str | None: content title if it can be generated and None otherwise
"""
if self.changes.is_empty:
return None
return self.changes.last_commit_sha

View File

@ -19,7 +19,7 @@
#
from pathlib import Path
from ahriman.core.formatters import StringPrinter
from ahriman.core.formatters.string_printer import StringPrinter
from ahriman.models.property import Property

View File

@ -17,7 +17,7 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
from ahriman.core.formatters import StringPrinter
from ahriman.core.formatters.string_printer import StringPrinter
from ahriman.models.property import Property

View File

@ -17,7 +17,7 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
from ahriman.core.formatters import StringPrinter
from ahriman.core.formatters.string_printer import StringPrinter
from ahriman.models.build_status import BuildStatus
from ahriman.models.package import Package
from ahriman.models.property import Property

View File

@ -17,7 +17,7 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
from ahriman.core.formatters import StringPrinter
from ahriman.core.formatters.string_printer import StringPrinter
from ahriman.models.pkgbuild_patch import PkgbuildPatch
from ahriman.models.property import Property

View File

@ -17,7 +17,7 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
from ahriman.core.formatters import StringPrinter
from ahriman.core.formatters.string_printer import StringPrinter
from ahriman.models.property import Property
from ahriman.models.repository_id import RepositoryId

View File

@ -17,7 +17,7 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
from ahriman.core.formatters import StringPrinter
from ahriman.core.formatters.string_printer import StringPrinter
from ahriman.models.build_status import BuildStatus

View File

@ -17,7 +17,7 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
from ahriman.core.formatters import StringPrinter
from ahriman.core.formatters.string_printer import StringPrinter
from ahriman.models.package import Package
from ahriman.models.property import Property

View File

@ -17,7 +17,7 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
from ahriman.core.formatters import StringPrinter
from ahriman.core.formatters.string_printer import StringPrinter
from ahriman.models.package import Package
from ahriman.models.property import Property

View File

@ -17,7 +17,7 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
from ahriman.core.formatters import StringPrinter
from ahriman.core.formatters.string_printer import StringPrinter
from ahriman.models.property import Property
from ahriman.models.user import User

View File

@ -20,7 +20,7 @@
from collections.abc import Generator
from typing import Any
from ahriman.core.formatters import StringPrinter
from ahriman.core.formatters.string_printer import StringPrinter
from ahriman.models.property import Property

View File

@ -17,7 +17,7 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
from ahriman.core.formatters import StringPrinter
from ahriman.core.formatters.string_printer import StringPrinter
from ahriman.models.property import Property

View File

@ -81,7 +81,7 @@ class RemoteCall(Report):
bool: True in case if remote process is alive and False otherwise
"""
try:
response = self.client.make_request("GET", f"/api/v1/service/process/{process_id}")
response = self.client.make_request("GET", f"{self.client.address}/api/v1/service/process/{process_id}")
except requests.HTTPError as ex:
status_code = ex.response.status_code if ex.response is not None else None
if status_code == 404:
@ -100,7 +100,7 @@ class RemoteCall(Report):
Returns:
str: remote process id
"""
response = self.client.make_request("POST", "/api/v1/service/update",
response = self.client.make_request("POST", f"{self.client.address}/api/v1/service/update",
params=self.repository_id.query(),
json={
"aur": self.update_aur,

View File

@ -25,45 +25,20 @@ from tempfile import TemporaryDirectory
from ahriman.core.build_tools.task import Task
from ahriman.core.repository.cleaner import Cleaner
from ahriman.core.repository.package_info import PackageInfo
from ahriman.core.util import safe_filename
from ahriman.models.changes import Changes
from ahriman.models.package import Package
from ahriman.models.package_description import PackageDescription
from ahriman.models.packagers import Packagers
from ahriman.models.result import Result
class Executor(Cleaner):
class Executor(PackageInfo, Cleaner):
"""
trait for common repository update processes
"""
def load_archives(self, packages: Iterable[Path]) -> list[Package]:
"""
load packages from list of archives
Args:
packages(Iterable[Path]): paths to package archives
Returns:
list[Package]: list of read packages
Raises:
NotImplementedError: not implemented method
"""
raise NotImplementedError
def packages(self) -> list[Package]:
"""
generate list of repository packages
Returns:
list[Package]: list of packages properties
Raises:
NotImplementedError: not implemented method
"""
raise NotImplementedError
def process_build(self, updates: Iterable[Package], packagers: Packagers | None = None, *,
bump_pkgrel: bool = False) -> Result:
"""
@ -78,16 +53,18 @@ class Executor(Cleaner):
Returns:
Result: build result
"""
def build_single(package: Package, local_path: Path, packager_id: str | None) -> None:
def build_single(package: Package, local_path: Path, packager_id: str | None) -> str | None:
self.reporter.set_building(package.base)
task = Task(package, self.configuration, self.architecture, self.paths)
local_version = local_versions.get(package.base) if bump_pkgrel else None
task.init(local_path, self.database, local_version)
commit_sha = task.init(local_path, self.database, local_version)
built = task.build(local_path, PACKAGER=packager_id)
for src in built:
dst = self.paths.packages / src.name
shutil.move(src, dst)
return commit_sha
packagers = packagers or Packagers()
local_versions = {package.base: package.version for package in self.packages()}
@ -97,7 +74,9 @@ class Executor(Cleaner):
TemporaryDirectory(ignore_cleanup_errors=True) as dir_name:
try:
packager = self.packager(packagers, single.base)
build_single(single, Path(dir_name), packager.packager_id)
last_commit_sha = build_single(single, Path(dir_name), packager.packager_id)
# clear changes and update commit hash
self.reporter.package_changes_set(single.base, Changes(last_commit_sha))
result.add_updated(single)
except Exception:
self.reporter.set_failed(single.base)
@ -122,6 +101,7 @@ class Executor(Cleaner):
self.database.build_queue_clear(package_base)
self.database.patches_remove(package_base, [])
self.database.logs_remove(package_base, None)
self.database.changes_remove(package_base)
self.reporter.package_remove(package_base) # we only update status page in case of base removal
except Exception:
self.logger.exception("could not remove base %s", package_base)

View File

@ -0,0 +1,126 @@
#
# Copyright (c) 2021-2023 ahriman team.
#
# This file is part of ahriman
# (see https://github.com/arcan1s/ahriman).
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
from collections.abc import Iterable
from pathlib import Path
from tempfile import TemporaryDirectory
from ahriman.core.build_tools.sources import Sources
from ahriman.core.repository.repository_properties import RepositoryProperties
from ahriman.core.util import package_like
from ahriman.models.changes import Changes
from ahriman.models.package import Package
class PackageInfo(RepositoryProperties):
"""
handler for the package information
"""
def load_archives(self, packages: Iterable[Path]) -> list[Package]:
"""
load packages from list of archives
Args:
packages(Iterable[Path]): paths to package archives
Returns:
list[Package]: list of read packages
"""
sources = self.database.remotes_get()
result: dict[str, Package] = {}
# we are iterating over bases, not single packages
for full_path in packages:
try:
local = Package.from_archive(full_path, self.pacman)
if (source := sources.get(local.base)) is not None:
local.remote = source
current = result.setdefault(local.base, local)
if current.version != local.version:
# force version to max of them
self.logger.warning("version of %s differs, found %s and %s",
current.base, current.version, local.version)
if current.is_outdated(local, self.paths, calculate_version=False):
current.version = local.version
current.packages.update(local.packages)
except Exception:
self.logger.exception("could not load package from %s", full_path)
return list(result.values())
def package_changes(self, package: Package, last_commit_sha: str | None) -> Changes:
"""
extract package change for the package since last commit if available
Args:
package(Package): package properties
last_commit_sha(str | None): last known commit hash
Returns:
Changes: changes if available
"""
with TemporaryDirectory(ignore_cleanup_errors=True) as dir_name:
dir_path = Path(dir_name)
current_commit_sha = Sources.load(dir_path, package, self.database.patches_get(package.base), self.paths)
changes: str | None = None
if current_commit_sha != last_commit_sha:
changes = Sources.changes(dir_path, last_commit_sha)
return Changes(last_commit_sha, changes)
def packages(self) -> list[Package]:
"""
generate list of repository packages
Returns:
list[Package]: list of packages properties
"""
return self.load_archives(filter(package_like, self.paths.repository.iterdir()))
def packages_built(self) -> list[Path]:
"""
get list of files in built packages directory
Returns:
list[Path]: list of filenames from the directory
"""
return list(filter(package_like, self.paths.packages.iterdir()))
def packages_depend_on(self, packages: list[Package], depends_on: Iterable[str] | None) -> list[Package]:
"""
extract list of packages which depends on specified package
Args:
packages(list[Package]): list of packages to be filtered
depends_on(Iterable[str] | None): dependencies of the packages
Returns:
list[Package]: list of repository packages which depend on specified packages
"""
if depends_on is None:
return packages # no list provided extract everything by default
depends_on = set(depends_on)
return [
package
for package in packages
if depends_on.intersection(package.full_depends(self.pacman, packages))
]

View File

@ -17,8 +17,6 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
from collections.abc import Iterable
from pathlib import Path
from typing import Self
from ahriman.core import _Context, context
@ -28,9 +26,7 @@ from ahriman.core.database import SQLite
from ahriman.core.repository.executor import Executor
from ahriman.core.repository.update_handler import UpdateHandler
from ahriman.core.sign.gpg import GPG
from ahriman.core.util import package_like
from ahriman.models.context_key import ContextKey
from ahriman.models.package import Package
from ahriman.models.pacman_synchronization import PacmanSynchronization
from ahriman.models.repository_id import RepositoryId
@ -101,74 +97,3 @@ class Repository(Executor, UpdateHandler):
ctx.set(ContextKey("repository", type(self)), self)
context.set(ctx)
def load_archives(self, packages: Iterable[Path]) -> list[Package]:
"""
load packages from list of archives
Args:
packages(Iterable[Path]): paths to package archives
Returns:
list[Package]: list of read packages
"""
sources = self.database.remotes_get()
result: dict[str, Package] = {}
# we are iterating over bases, not single packages
for full_path in packages:
try:
local = Package.from_archive(full_path, self.pacman)
if (source := sources.get(local.base)) is not None:
local.remote = source
current = result.setdefault(local.base, local)
if current.version != local.version:
# force version to max of them
self.logger.warning("version of %s differs, found %s and %s",
current.base, current.version, local.version)
if current.is_outdated(local, self.paths, calculate_version=False):
current.version = local.version
current.packages.update(local.packages)
except Exception:
self.logger.exception("could not load package from %s", full_path)
return list(result.values())
def packages(self) -> list[Package]:
"""
generate list of repository packages
Returns:
list[Package]: list of packages properties
"""
return self.load_archives(filter(package_like, self.paths.repository.iterdir()))
def packages_built(self) -> list[Path]:
"""
get list of files in built packages directory
Returns:
list[Path]: list of filenames from the directory
"""
return list(filter(package_like, self.paths.packages.iterdir()))
def packages_depend_on(self, packages: list[Package], depends_on: Iterable[str] | None) -> list[Package]:
"""
extract list of packages which depends on specified package
Args:
packages(list[Package]): list of packages to be filtered
depends_on(Iterable[str] | None): dependencies of the packages
Returns:
list[Package]: list of repository packages which depend on specified packages
"""
if depends_on is None:
return packages # no list provided extract everything by default
depends_on = set(depends_on)
return [
package
for package in packages
if depends_on.intersection(package.full_depends(self.pacman, packages))
]

View File

@ -22,28 +22,17 @@ from collections.abc import Iterable
from ahriman.core.build_tools.sources import Sources
from ahriman.core.exceptions import UnknownPackageError
from ahriman.core.repository.cleaner import Cleaner
from ahriman.core.repository.package_info import PackageInfo
from ahriman.models.package import Package
from ahriman.models.package_source import PackageSource
from ahriman.models.remote_source import RemoteSource
class UpdateHandler(Cleaner):
class UpdateHandler(PackageInfo, Cleaner):
"""
trait to get package update list
"""
def packages(self) -> list[Package]:
"""
generate list of repository packages
Returns:
list[Package]: list of packages properties
Raises:
NotImplementedError: not implemented method
"""
raise NotImplementedError
def updates_aur(self, filter_packages: Iterable[str], *, vcs: bool) -> list[Package]:
"""
check AUR for updates

View File

@ -174,7 +174,7 @@ class Spawn(Thread, LazyLogging):
return self._spawn_process(repository_id, "service-key-import", key, **kwargs)
def packages_add(self, repository_id: RepositoryId, packages: Iterable[str], username: str | None, *,
patches: list[PkgbuildPatch], now: bool) -> str:
patches: list[PkgbuildPatch], now: bool, increment: bool, refresh: bool) -> str:
"""
add packages
@ -184,19 +184,26 @@ class Spawn(Thread, LazyLogging):
username(str | None): optional override of username for build process
patches(list[PkgbuildPatch]): list of patches to be passed
now(bool): build packages now
increment(bool): increment pkgrel on conflict
refresh(bool): refresh pacman database before process
Returns:
str: spawned process identifier
"""
kwargs: dict[str, str | list[str] | None] = {"username": username}
kwargs: dict[str, str | list[str] | None] = {
"username": username,
"variable": [patch.serialize() for patch in patches],
self.boolean_action_argument("increment", increment): "",
}
if now:
kwargs["now"] = ""
if patches:
kwargs["variable"] = [patch.serialize() for patch in patches]
if refresh:
kwargs["refresh"] = ""
return self._spawn_process(repository_id, "package-add", *packages, **kwargs)
def packages_rebuild(self, repository_id: RepositoryId, depends_on: str, username: str | None) -> str:
def packages_rebuild(self, repository_id: RepositoryId, depends_on: str, username: str | None, *,
increment: bool) -> str:
"""
rebuild packages which depend on the specified package
@ -204,11 +211,16 @@ class Spawn(Thread, LazyLogging):
repository_id(RepositoryId): repository unique identifier
depends_on(str): packages dependency
username(str | None): optional override of username for build process
increment(bool): increment pkgrel on conflict
Returns:
str: spawned process identifier
"""
kwargs = {"depends-on": depends_on, "username": username}
kwargs = {
"depends-on": depends_on,
"username": username,
self.boolean_action_argument("increment", increment): "",
}
return self._spawn_process(repository_id, "repo-rebuild", **kwargs)
def packages_remove(self, repository_id: RepositoryId, packages: Iterable[str]) -> str:
@ -225,7 +237,7 @@ class Spawn(Thread, LazyLogging):
return self._spawn_process(repository_id, "package-remove", *packages)
def packages_update(self, repository_id: RepositoryId, username: str | None, *,
aur: bool, local: bool, manual: bool) -> str:
aur: bool, local: bool, manual: bool, increment: bool, refresh: bool) -> str:
"""
run full repository update
@ -235,6 +247,8 @@ class Spawn(Thread, LazyLogging):
aur(bool): check for aur updates
local(bool): check for local packages updates
manual(bool): check for manual packages
increment(bool): increment pkgrel on conflict
refresh(bool): refresh pacman database before process
Returns:
str: spawned process identifier
@ -244,7 +258,11 @@ class Spawn(Thread, LazyLogging):
self.boolean_action_argument("aur", aur): "",
self.boolean_action_argument("local", local): "",
self.boolean_action_argument("manual", manual): "",
self.boolean_action_argument("increment", increment): "",
}
if refresh:
kwargs["refresh"] = ""
return self._spawn_process(repository_id, "repo-update", **kwargs)
def run(self) -> None:

View File

@ -23,6 +23,7 @@ import logging
from ahriman.core.configuration import Configuration
from ahriman.models.build_status import BuildStatus, BuildStatusEnum
from ahriman.models.changes import Changes
from ahriman.models.internal_status import InternalStatus
from ahriman.models.log_record_id import LogRecordId
from ahriman.models.package import Package
@ -75,6 +76,28 @@ class Client:
status(BuildStatusEnum): current package build status
"""
def package_changes_get(self, package_base: str) -> Changes:
"""
get package changes
Args:
package_base(str): package base to retrieve
Returns:
Changes: package changes if available and empty object otherwise
"""
del package_base
return Changes()
def package_changes_set(self, package_base: str, changes: Changes) -> None:
"""
update package changes
Args:
package_base(str): package base to update
changes(Changes): changes descriptor
"""
def package_get(self, package_base: str | None) -> list[tuple[Package, BuildStatus]]:
"""
get package status

View File

@ -21,6 +21,7 @@ from ahriman.core.database import SQLite
from ahriman.core.exceptions import UnknownPackageError
from ahriman.core.log import LazyLogging
from ahriman.models.build_status import BuildStatus, BuildStatusEnum
from ahriman.models.changes import Changes
from ahriman.models.log_record_id import LogRecordId
from ahriman.models.package import Package
from ahriman.models.pkgbuild_patch import PkgbuildPatch
@ -113,6 +114,23 @@ class Watcher(LazyLogging):
self._last_log_record_id = log_record_id
self.database.logs_insert(log_record_id, created, record, self.repository_id)
def package_changes_get(self, package_base: str) -> Changes:
"""
retrieve package changes
Args:
package_base(str): package base
Returns:
Changes: package changes if available
Raises:
UnknownPackageError: if no package found
"""
if package_base not in self.known:
raise UnknownPackageError(package_base)
return self.database.changes_get(package_base, self.repository_id)
def package_get(self, package_base: str) -> tuple[Package, BuildStatus]:
"""
get current package base build status

View File

@ -26,6 +26,7 @@ from ahriman.core.configuration import Configuration
from ahriman.core.http import SyncAhrimanClient
from ahriman.core.status.client import Client
from ahriman.models.build_status import BuildStatus, BuildStatusEnum
from ahriman.models.changes import Changes
from ahriman.models.internal_status import InternalStatus
from ahriman.models.log_record_id import LogRecordId
from ahriman.models.package import Package
@ -83,6 +84,18 @@ class WebClient(Client, SyncAhrimanClient):
address = f"http://{host}:{port}"
return "web", address
def _changes_url(self, package_base: str) -> str:
"""
get url for the changes api
Args:
package_base(str): package base
Returns:
str: full url for web service for logs
"""
return f"{self.address}/api/v1/packages/{package_base}/changes"
def _logs_url(self, package_base: str) -> str:
"""
get url for the logs api
@ -134,6 +147,37 @@ class WebClient(Client, SyncAhrimanClient):
self.make_request("POST", self._package_url(package.base),
params=self.repository_id.query(), json=payload)
def package_changes_get(self, package_base: str) -> Changes:
"""
get package changes
Args:
package_base(str): package base to retrieve
Returns:
Changes: package changes if available and empty object otherwise
"""
with contextlib.suppress(Exception):
response = self.make_request("GET", self._changes_url(package_base),
params=self.repository_id.query())
response_json = response.json()
return Changes.from_json(response_json)
return Changes()
def package_changes_set(self, package_base: str, changes: Changes) -> None:
"""
update package changes
Args:
package_base(str): package base to update
changes(Changes): changes descriptor
"""
with contextlib.suppress(Exception):
self.make_request("POST", self._changes_url(package_base),
params=self.repository_id.query(), json=changes.view())
def package_get(self, package_base: str | None) -> list[tuple[Package, BuildStatus]]:
"""
get package status

View File

@ -148,6 +148,8 @@ class Tree:
sorted(part, key=lambda leaf: leaf.package.base)
for part in partitions if part
]
if not partitions: # nothing to balance
return partitions
while True:
min_part, max_part = minmax(partitions, key=len)
@ -182,7 +184,7 @@ class Tree:
Returns:
list[list[Package]]: list of packages lists based on their dependencies. The amount of elements in each
sublist is less or equal to ``count``
sublist is less or equal to ``count``
Raises:
PartitionError: in case if it is impossible to divide tree by specified amount of partitions

View File

@ -0,0 +1,71 @@
#
# Copyright (c) 2021-2023 ahriman team.
#
# This file is part of ahriman
# (see https://github.com/arcan1s/ahriman).
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
from dataclasses import dataclass, fields
from typing import Any, Self
from ahriman.core.util import dataclass_view, filter_json
@dataclass(frozen=True)
class Changes:
"""
package source files changes holder
Attributes:
last_commit_sha(str | None): last commit hash
changes(str | None): package change since the last commit if available
"""
last_commit_sha: str | None = None
changes: str | None = None
@property
def is_empty(self) -> bool:
"""
validate that changes are not empty
Returns:
bool: ``True`` in case if changes are not set and ``False`` otherwise
"""
return self.changes is None
@classmethod
def from_json(cls, dump: dict[str, Any]) -> Self:
"""
construct changes from the json dump
Args:
dump(dict[str, Any]): json dump body
Returns:
Self: changes object
"""
# filter to only known fields
known_fields = [pair.name for pair in fields(cls)]
return cls(**filter_json(dump, known_fields))
def view(self) -> dict[str, Any]:
"""
generate json change view
Returns:
dict[str, Any]: json-friendly dictionary
"""
return dataclass_view(self)

View File

@ -505,7 +505,7 @@ class Package(LazyLogging):
Returns:
bool: True in case if package was built after the specified date and False otherwise. In case if build date
is not set by any of packages, it returns False
is not set by any of packages, it returns False
"""
return any(
package.build_date > timestamp

View File

@ -75,7 +75,7 @@ class RepositoryPaths(LazyLogging):
Returns:
Path: relative path which contains only architecture segment in case if legacy tree is used and repository
name and architecture otherwise
name and architecture otherwise
"""
if not self._force_current_tree:
if (self._repository_root / self.repository_id.architecture).is_dir():
@ -181,7 +181,7 @@ class RepositoryPaths(LazyLogging):
Returns:
set[str]: list of repository names for which there is created tree. Returns empty set in case if repository
is loaded in legacy mode
is loaded in legacy mode
"""
# simply walk through the root. In case if there are subdirectories, emit the name
def walk(paths: RepositoryPaths) -> Generator[str, None, None]:

View File

@ -49,7 +49,7 @@ class Waiter:
Returns:
bool: True in case current monotonic time is more than :attr:`start_time` and :attr:`wait_timeout`
doesn't equal to 0
doesn't equal to 0
"""
since_start: float = time.monotonic() - self.start_time
return self.wait_timeout != 0 and since_start > self.wait_timeout

View File

@ -0,0 +1,41 @@
#
# Copyright (c) 2021-2023 ahriman team.
#
# This file is part of ahriman
# (see https://github.com/arcan1s/ahriman).
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
from dataclasses import dataclass, field
from urllib.parse import urlparse
@dataclass(frozen=True)
class Worker:
"""
worker descriptor
Attributes:
address(str): worker address to be reachable outside
identifier(str): worker unique identifier. If none set it will be automatically generated from the address
"""
address: str
identifier: str = field(default="", kw_only=True)
def __post_init__(self) -> None:
"""
update identifier based on settings
"""
object.__setattr__(self, "identifier", self.identifier or urlparse(self.address).netloc)

View File

@ -17,7 +17,7 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
import aiohttp_security # type: ignore[import-untyped]
import aiohttp_security
import socket
import types
@ -25,6 +25,7 @@ from aiohttp.web import Application, Request, StaticResource, StreamResponse, mi
from aiohttp_session import setup as setup_session
from aiohttp_session.cookie_storage import EncryptedCookieStorage
from cryptography import fernet
from enum import Enum
from ahriman.core.auth import Auth
from ahriman.core.configuration import Configuration
@ -50,6 +51,7 @@ class _AuthorizationPolicy(aiohttp_security.AbstractAuthorizationPolicy):
Args:
validator(Auth): authorization module instance
"""
aiohttp_security.AbstractAuthorizationPolicy.__init__(self)
self.validator = validator
async def authorized_userid(self, identity: str) -> str | None:
@ -64,18 +66,21 @@ class _AuthorizationPolicy(aiohttp_security.AbstractAuthorizationPolicy):
"""
return identity if await self.validator.known_username(identity) else None
async def permits(self, identity: str, permission: UserAccess, context: str | None = None) -> bool:
async def permits(self, identity: str | None, permission: str | Enum, context: str | None = None) -> bool:
"""
check user permissions
Args:
identity(str): username
permission(UserAccess): requested permission level
identity(str | None): username
permission(str | Enum): requested permission level
context(str | None, optional): URI request path (Default value = None)
Returns:
bool: True in case if user is allowed to perform this request and False otherwise
"""
# some methods for type checking and parent class compatibility
if identity is None or not isinstance(permission, UserAccess):
return False # no identity provided or invalid access rights requested
return await self.validator.verify_access(identity, permission, context)

View File

@ -25,18 +25,20 @@ from pkgutil import ModuleInfo, iter_modules
from types import ModuleType
from typing import Any, Type, TypeGuard
from ahriman.core.configuration import Configuration
from ahriman.web.views.base import BaseView
__all__ = ["setup_routes"]
def _dynamic_routes(module_root: Path) -> dict[str, Type[View]]:
def _dynamic_routes(module_root: Path, configuration: Configuration) -> dict[str, Type[View]]:
"""
extract dynamic routes based on views
Args:
module_root(Path): root module path with views
configuration(Configuration): configuration instance
Returns:
dict[str, Type[View]]: map of the route to its view
@ -52,7 +54,9 @@ def _dynamic_routes(module_root: Path) -> dict[str, Type[View]]:
view = getattr(module, attribute_name)
if not is_base_view(view):
continue
routes.update([(route, view) for route in view.ROUTES])
view_routes = view.routes(configuration)
routes.update([(route, view) for route in view_routes])
return routes
@ -101,16 +105,16 @@ def _modules(module_root: Path) -> Generator[ModuleInfo, None, None]:
yield module_info
def setup_routes(application: Application, static_path: Path) -> None:
def setup_routes(application: Application, configuration: Configuration) -> None:
"""
setup all defined routes
Args:
application(Application): web application instance
static_path(Path): path to static files directory
configuration(Configuration): configuration instance
"""
application.router.add_static("/static", static_path, follow_symlinks=True)
application.router.add_static("/static", configuration.getpath("web", "static_path"), follow_symlinks=True)
views = Path(__file__).parent / "views"
for route, view in _dynamic_routes(views).items():
views_root = Path(__file__).parent / "views"
for route, view in _dynamic_routes(views_root, configuration).items():
application.router.add_view(route, view)

View File

@ -19,6 +19,8 @@
#
from ahriman.web.schemas.aur_package_schema import AURPackageSchema
from ahriman.web.schemas.auth_schema import AuthSchema
from ahriman.web.schemas.build_options_schema import BuildOptionsSchema
from ahriman.web.schemas.changes_schema import ChangesSchema
from ahriman.web.schemas.counters_schema import CountersSchema
from ahriman.web.schemas.error_schema import ErrorSchema
from ahriman.web.schemas.file_schema import FileSchema

View File

@ -0,0 +1,36 @@
#
# Copyright (c) 2021-2023 ahriman team.
#
# This file is part of ahriman
# (see https://github.com/arcan1s/ahriman).
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
from marshmallow import Schema, fields
class BuildOptionsSchema(Schema):
"""
request build options schema
"""
increment = fields.Boolean(dump_default=True, metadata={
"description": "Increment pkgrel on conflicts",
})
packager = fields.String(metadata={
"description": "Packager identity if applicable",
})
refresh = fields.Boolean(dump_default=True, metadata={
"description": "Refresh pacman database"
})

View File

@ -0,0 +1,34 @@
#
# Copyright (c) 2021-2023 ahriman team.
#
# This file is part of ahriman
# (see https://github.com/arcan1s/ahriman).
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
from marshmallow import Schema, fields
class ChangesSchema(Schema):
"""
response package changes schema
"""
last_commit_sha = fields.String(metadata={
"description": "Last recorded commit hash",
"example": "f1875edca1eb8fc0e55c41d1cae5fa05b6b7c6",
})
changes = fields.String(metadata={
"description": "Package changes in patch format",
})

View File

@ -17,10 +17,12 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
from marshmallow import Schema, fields
from marshmallow import fields
from ahriman.web.schemas.build_options_schema import BuildOptionsSchema
class PackageNamesSchema(Schema):
class PackageNamesSchema(BuildOptionsSchema):
"""
request package names schema
"""

View File

@ -17,20 +17,22 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
from marshmallow import Schema, fields
from marshmallow import fields
from ahriman.web.schemas.build_options_schema import BuildOptionsSchema
class UpdateFlagsSchema(Schema):
class UpdateFlagsSchema(BuildOptionsSchema):
"""
update flags request schema
"""
aur = fields.Bool(dump_default=True, metadata={
aur = fields.Boolean(dump_default=True, metadata={
"description": "Check AUR for updates",
})
local = fields.Bool(dump_default=True, metadata={
local = fields.Boolean(dump_default=True, metadata={
"description": "Check local packages for updates",
})
manual = fields.Bool(dump_default=True, metadata={
manual = fields.Boolean(dump_default=True, metadata={
"description": "Check manually built packages",
})

View File

@ -20,7 +20,7 @@
from aiohttp_cors import CorsViewMixin # type: ignore[import-untyped]
from aiohttp.web import HTTPBadRequest, HTTPNotFound, Request, StreamResponse, View
from collections.abc import Awaitable, Callable
from typing import Any, TypeVar
from typing import TypeVar
from ahriman.core.auth import Auth
from ahriman.core.configuration import Configuration
@ -115,6 +115,21 @@ class BaseView(View, CorsViewMixin):
permission: UserAccess = getattr(cls, f"{method}_PERMISSION", UserAccess.Full)
return permission
@classmethod
def routes(cls, configuration: Configuration) -> list[str]:
"""
extract routes list for the view
Args:
configuration(Configuration): configuration instance
Returns:
list[str]: list of routes defined for the view. By default, it tries to read :attr:`ROUTES` option if set
and returns empty list otherwise
"""
del configuration
return cls.ROUTES
@staticmethod
def get_non_empty(extractor: Callable[[str], T | None], key: str) -> T:
"""
@ -138,49 +153,8 @@ class BaseView(View, CorsViewMixin):
raise KeyError(f"Key {key} is missing or empty") from None
return value
async def data_as_json(self, list_keys: list[str]) -> dict[str, Any]:
"""
extract form data and convert it to json object
Args:
list_keys(list[str]): list of keys which must be forced to list from form data
Returns:
dict[str, Any]: form data converted to json. In case if a key is found multiple times
it will be returned as list
"""
raw = await self.request.post()
json: dict[str, Any] = {}
for key, value in raw.items():
if key in json and isinstance(json[key], list):
json[key].append(value)
elif key in json:
json[key] = [json[key], value]
elif key in list_keys:
json[key] = [value]
else:
json[key] = value
return json
async def extract_data(self, list_keys: list[str] | None = None) -> dict[str, Any]:
"""
extract json data from either json or form data
Args:
list_keys(list[str] | None, optional): optional list of keys which must be forced to list from form data
(Default value = None)
Returns:
dict[str, Any]: raw json object or form data converted to json
"""
try:
json: dict[str, Any] = await self.request.json()
return json
except ValueError:
return await self.data_as_json(list_keys or [])
# pylint: disable=not-callable,protected-access
async def head(self) -> StreamResponse: # type: ignore[return]
async def head(self) -> StreamResponse:
"""
HEAD method implementation based on the result of GET method
@ -263,6 +237,13 @@ class BaseView(View, CorsViewMixin):
Returns:
str | None: authorized username if any and None otherwise (e.g. if authorization is disabled)
"""
try: # try to read from payload
data: dict[str, str] = await self.request.json() # technically it is not, but we only need str here
if (packager := data.get("packager")) is not None:
return packager
except Exception:
self.request.app.logger.exception("could not extract json data for packager")
policy = self.request.app.get("identity")
if policy is not None:
identity: str = await policy.identify(self.request)

View File

@ -0,0 +1,44 @@
#
# Copyright (c) 2021-2023 ahriman team.
#
# This file is part of ahriman
# (see https://github.com/arcan1s/ahriman).
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
from ahriman.core.configuration import Configuration
class StatusViewGuard:
"""
helper for check if status routes are enabled
"""
ROUTES: list[str]
@classmethod
def routes(cls, configuration: Configuration) -> list[str]:
"""
extract routes list for the view
Args:
configuration(Configuration): configuration instance
Returns:
list[str]: list of routes defined for the view. By default, it tries to read :attr:`ROUTES` option if set
and returns empty list otherwise
"""
if configuration.getboolean("web", "service_only", fallback=False):
return []
return cls.ROUTES

View File

@ -66,7 +66,7 @@ class AddView(BaseView):
HTTPBadRequest: if bad data is supplied
"""
try:
data = await self.extract_data(["packages", "patches"])
data = await self.request.json()
packages = self.get_non_empty(lambda key: [package for package in data[key] if package], "packages")
patches = [PkgbuildPatch(patch["key"], patch.get("value", "")) for patch in data.get("patches", [])]
except Exception as ex:
@ -74,6 +74,14 @@ class AddView(BaseView):
repository_id = self.repository_id()
username = await self.username()
process_id = self.spawner.packages_add(repository_id, packages, username, patches=patches, now=True)
process_id = self.spawner.packages_add(
repository_id,
packages,
username,
patches=patches,
now=True,
increment=data.get("increment", True),
refresh=data.get("refresh", False),
)
return json_response({"process_id": process_id})

View File

@ -104,9 +104,8 @@ class PGPView(BaseView):
Raises:
HTTPBadRequest: if bad data is supplied
"""
data = await self.extract_data()
try:
data = await self.request.json()
key = self.get_non_empty(data.get, "key")
except Exception as ex:
raise HTTPBadRequest(reason=str(ex))

View File

@ -65,7 +65,7 @@ class RebuildView(BaseView):
HTTPBadRequest: if bad data is supplied
"""
try:
data = await self.extract_data(["packages"])
data = await self.request.json()
packages = self.get_non_empty(lambda key: [package for package in data[key] if package], "packages")
depends_on = next(iter(packages))
except Exception as ex:
@ -73,6 +73,11 @@ class RebuildView(BaseView):
repository_id = self.repository_id()
username = await self.username()
process_id = self.spawner.packages_rebuild(repository_id, depends_on, username)
process_id = self.spawner.packages_rebuild(
repository_id,
depends_on,
username,
increment=data.get("increment", True),
)
return json_response({"process_id": process_id})

View File

@ -65,7 +65,7 @@ class RemoveView(BaseView):
HTTPBadRequest: if bad data is supplied
"""
try:
data = await self.extract_data(["packages"])
data = await self.request.json()
packages = self.get_non_empty(lambda key: [package for package in data[key] if package], "packages")
except Exception as ex:
raise HTTPBadRequest(reason=str(ex))

View File

@ -66,7 +66,7 @@ class RequestView(BaseView):
HTTPBadRequest: if bad data is supplied
"""
try:
data = await self.extract_data(["packages", "patches"])
data = await self.request.json()
packages = self.get_non_empty(lambda key: [package for package in data[key] if package], "packages")
patches = [PkgbuildPatch(patch["key"], patch.get("value", "")) for patch in data.get("patches", [])]
except Exception as ex:
@ -74,6 +74,14 @@ class RequestView(BaseView):
username = await self.username()
repository_id = self.repository_id()
process_id = self.spawner.packages_add(repository_id, packages, username, patches=patches, now=False)
process_id = self.spawner.packages_add(
repository_id,
packages,
username,
patches=patches,
now=False,
increment=False, # no-increment doesn't work here
refresh=False, # refresh doesn't work here
)
return json_response({"process_id": process_id})

View File

@ -65,7 +65,7 @@ class UpdateView(BaseView):
HTTPBadRequest: if bad data is supplied
"""
try:
data = await self.extract_data()
data = await self.request.json()
except Exception as ex:
raise HTTPBadRequest(reason=str(ex))
@ -77,6 +77,8 @@ class UpdateView(BaseView):
aur=data.get("aur", True),
local=data.get("local", True),
manual=data.get("manual", True),
increment=data.get("increment", True),
refresh=data.get("refresh", False),
)
return json_response({"process_id": process_id})

View File

@ -0,0 +1,119 @@
#
# Copyright (c) 2021-2023 ahriman team.
#
# This file is part of ahriman
# (see https://github.com/arcan1s/ahriman).
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
import aiohttp_apispec # type: ignore[import-untyped]
from aiohttp.web import HTTPBadRequest, HTTPNoContent, HTTPNotFound, Response, json_response
from ahriman.core.exceptions import UnknownPackageError
from ahriman.models.changes import Changes
from ahriman.models.user_access import UserAccess
from ahriman.web.schemas import AuthSchema, ChangesSchema, ErrorSchema, PackageNameSchema, RepositoryIdSchema
from ahriman.web.views.base import BaseView
from ahriman.web.views.status_view_guard import StatusViewGuard
class ChangesView(StatusViewGuard, BaseView):
"""
package changes web view
Attributes:
GET_PERMISSION(UserAccess): (class attribute) get permissions of self
POST_PERMISSION(UserAccess): (class attribute) post permissions of self
"""
GET_PERMISSION = UserAccess.Reporter
POST_PERMISSION = UserAccess.Full
ROUTES = ["/api/v1/packages/{package}/changes"]
@aiohttp_apispec.docs(
tags=["Packages"],
summary="Get package changes",
description="Retrieve package changes since the last build",
responses={
200: {"description": "Success response", "schema": ChangesSchema},
401: {"description": "Authorization required", "schema": ErrorSchema},
403: {"description": "Access is forbidden", "schema": ErrorSchema},
404: {"description": "Package base and/or repository are unknown", "schema": ErrorSchema},
500: {"description": "Internal server error", "schema": ErrorSchema},
},
security=[{"token": [GET_PERMISSION]}],
)
@aiohttp_apispec.cookies_schema(AuthSchema)
@aiohttp_apispec.match_info_schema(PackageNameSchema)
@aiohttp_apispec.querystring_schema(RepositoryIdSchema)
async def get(self) -> Response:
"""
get package changes
Returns:
Response: 200 with package change on success
Raises:
HTTPNotFound: if package base is unknown
"""
package_base = self.request.match_info["package"]
try:
changes = self.service().package_changes_get(package_base)
except UnknownPackageError:
raise HTTPNotFound(reason=f"Package {package_base} is unknown")
return json_response(changes.view())
@aiohttp_apispec.docs(
tags=["Packages"],
summary="Update package changes",
description="Update package changes to the new ones",
responses={
204: {"description": "Success response"},
400: {"description": "Bad data is supplied", "schema": ErrorSchema},
401: {"description": "Authorization required", "schema": ErrorSchema},
403: {"description": "Access is forbidden", "schema": ErrorSchema},
404: {"description": "Repository is unknown", "schema": ErrorSchema},
500: {"description": "Internal server error", "schema": ErrorSchema},
},
security=[{"token": [POST_PERMISSION]}],
)
@aiohttp_apispec.cookies_schema(AuthSchema)
@aiohttp_apispec.match_info_schema(PackageNameSchema)
@aiohttp_apispec.querystring_schema(RepositoryIdSchema)
@aiohttp_apispec.json_schema(ChangesSchema)
async def post(self) -> None:
"""
insert new package changes
Raises:
HTTPBadRequest: if bad data is supplied
HTTPNoContent: in case of success response
"""
package_base = self.request.match_info["package"]
try:
data = await self.request.json()
last_commit_sha = data.get("last_commit_sha") # empty/null meant removal
change = data.get("changes")
except Exception as ex:
raise HTTPBadRequest(reason=str(ex))
changes = Changes(last_commit_sha, change)
repository_id = self.repository_id()
self.service(repository_id).database.changes_insert(package_base, changes, repository_id)
raise HTTPNoContent

View File

@ -28,9 +28,10 @@ from ahriman.models.user_access import UserAccess
from ahriman.web.schemas import AuthSchema, ErrorSchema, LogsSchema, PackageNameSchema, RepositoryIdSchema, \
VersionedLogSchema
from ahriman.web.views.base import BaseView
from ahriman.web.views.status_view_guard import StatusViewGuard
class LogsView(BaseView):
class LogsView(StatusViewGuard, BaseView):
"""
package logs web view
@ -139,9 +140,9 @@ class LogsView(BaseView):
HTTPNoContent: in case of success response
"""
package_base = self.request.match_info["package"]
data = await self.extract_data()
try:
data = await self.request.json()
created = data["created"]
record = data["message"]
version = data["version"]

View File

@ -28,9 +28,10 @@ from ahriman.models.user_access import UserAccess
from ahriman.web.schemas import AuthSchema, ErrorSchema, PackageNameSchema, PackageStatusSchema, \
PackageStatusSimplifiedSchema, RepositoryIdSchema
from ahriman.web.views.base import BaseView
from ahriman.web.views.status_view_guard import StatusViewGuard
class PackageView(BaseView):
class PackageView(StatusViewGuard, BaseView):
"""
package base specific web view
@ -142,9 +143,9 @@ class PackageView(BaseView):
HTTPNoContent: in case of success response
"""
package_base = self.request.match_info["package"]
data = await self.extract_data()
try:
data = await self.request.json()
package = Package.from_json(data["package"]) if "package" in data else None
status = BuildStatusEnum(data["status"])
except Exception as ex:

View File

@ -28,9 +28,10 @@ from ahriman.models.package import Package
from ahriman.models.user_access import UserAccess
from ahriman.web.schemas import AuthSchema, ErrorSchema, PackageStatusSchema, PaginationSchema, RepositoryIdSchema
from ahriman.web.views.base import BaseView
from ahriman.web.views.status_view_guard import StatusViewGuard
class PackagesView(BaseView):
class PackagesView(StatusViewGuard, BaseView):
"""
global watcher view

View File

@ -24,9 +24,10 @@ from aiohttp.web import HTTPNoContent, HTTPNotFound, Response, json_response
from ahriman.models.user_access import UserAccess
from ahriman.web.schemas import AuthSchema, ErrorSchema, PatchNameSchema, PatchSchema
from ahriman.web.views.base import BaseView
from ahriman.web.views.status_view_guard import StatusViewGuard
class PatchView(BaseView):
class PatchView(StatusViewGuard, BaseView):
"""
package patch web view

View File

@ -25,9 +25,10 @@ from ahriman.models.pkgbuild_patch import PkgbuildPatch
from ahriman.models.user_access import UserAccess
from ahriman.web.schemas import AuthSchema, ErrorSchema, PackageNameSchema, PatchSchema
from ahriman.web.views.base import BaseView
from ahriman.web.views.status_view_guard import StatusViewGuard
class PatchesView(BaseView):
class PatchesView(StatusViewGuard, BaseView):
"""
package patches web view
@ -95,9 +96,9 @@ class PatchesView(BaseView):
HTTPNoContent: on success response
"""
package_base = self.request.match_info["package"]
data = await self.extract_data()
try:
data = await self.request.json()
key = data["key"]
value = data["value"]
except Exception as ex:

View File

@ -28,9 +28,10 @@ from ahriman.models.internal_status import InternalStatus
from ahriman.models.user_access import UserAccess
from ahriman.web.schemas import AuthSchema, ErrorSchema, InternalStatusSchema, StatusSchema, RepositoryIdSchema
from ahriman.web.views.base import BaseView
from ahriman.web.views.status_view_guard import StatusViewGuard
class StatusView(BaseView):
class StatusView(StatusViewGuard, BaseView):
"""
web service status web view
@ -102,7 +103,7 @@ class StatusView(BaseView):
HTTPNoContent: in case of success response
"""
try:
data = await self.extract_data()
data = await self.request.json()
status = BuildStatusEnum(data["status"])
except Exception as ex:
raise HTTPBadRequest(reason=str(ex))

View File

@ -19,7 +19,7 @@
#
import aiohttp_apispec # type: ignore[import-untyped]
from aiohttp.web import HTTPFound, HTTPMethodNotAllowed, HTTPUnauthorized
from aiohttp.web import HTTPBadRequest, HTTPFound, HTTPMethodNotAllowed, HTTPUnauthorized
from ahriman.core.auth.helpers import remember
from ahriman.models.user_access import UserAccess
@ -93,6 +93,7 @@ class LoginView(BaseView):
description="Login by using username and password",
responses={
302: {"description": "Success response"},
400: {"description": "Bad data is supplied", "schema": ErrorSchema},
401: {"description": "Authorization required", "schema": ErrorSchema},
500: {"description": "Internal server error", "schema": ErrorSchema},
},
@ -107,11 +108,15 @@ class LoginView(BaseView):
HTTPFound: on success response
HTTPUnauthorized: if case of authorization error
"""
data = await self.extract_data()
identity = data.get("username")
try:
data = await self.request.json()
identity = data["username"]
password = data["password"]
except Exception as ex:
raise HTTPBadRequest(reason=str(ex))
response = HTTPFound("/")
if identity is not None and await self.validator.check_credentials(identity, data.get("password")):
if await self.validator.check_credentials(identity, password):
await remember(self.request, response, identity)
raise response

View File

@ -24,9 +24,10 @@ from aiohttp.web import Response, json_response
from ahriman.models.user_access import UserAccess
from ahriman.web.schemas import AuthSchema, ErrorSchema, LogSchema, PackageNameSchema, PaginationSchema
from ahriman.web.views.base import BaseView
from ahriman.web.views.status_view_guard import StatusViewGuard
class LogsView(BaseView):
class LogsView(StatusViewGuard, BaseView):
"""
package logs web view

View File

@ -146,7 +146,7 @@ def setup_server(configuration: Configuration, spawner: Spawn, repositories: lis
application.middlewares.append(exception_handler(application.logger))
application.logger.info("setup routes")
setup_routes(application, configuration.getpath("web", "static_path"))
setup_routes(application, configuration)
application.logger.info("setup CORS")
setup_cors(application)

Some files were not shown because too many files have changed in this diff Show More