Compare commits

...

28 Commits

Author SHA1 Message Date
7769a4a6e0 Release 2.18.3 2025-06-20 17:20:19 +03:00
066d1b1dde refactor: rework few tests and build system
This commit includes the following changes
* Bump github actions
* Update tests github action to check documentation and streamline
  process
* Update test cases to use temporary directories as roots
* Simplify tox.ini
2025-06-20 17:04:57 +03:00
1f22a27360 build: remove defaults from pylint config 2025-06-18 16:20:14 +03:00
75682bc7be feat: add support of openmetrics (#144)
* feat: add openmetrics support & endpoint

* add support of named resources

* update docstrings

* generate docs

* add another test for http api
2025-06-18 14:42:09 +03:00
e5d824b03f build: add regress weekly pipeline
This commit also adds manual dispatch for tests and setup
2025-06-16 21:15:52 +03:00
8d0d597473 Release 2.18.2 2025-06-16 19:03:05 +03:00
995b396360 bug: fix invalid logs rotation 2025-06-16 16:36:34 +03:00
7f813cf0c3 Release 2.18.1 2025-06-16 15:33:24 +03:00
d4eb55ef95 bug: correctly close sqlite3 connection
After the last updates, tests produce warnings that the connection to
database is leaked, which appears to be correct. This commit changes
behaviour to closing connection explicitly via contextlib
2025-06-16 15:24:57 +03:00
09350e88ab style: fix few typos 2025-06-14 23:34:53 +03:00
2feaa14f46 Release 2.18.0 2025-06-13 16:37:58 +03:00
9653fc4f4a type: support new configparser signatures 2025-05-31 02:16:07 +03:00
bcd46c66e8 test: use new-style fixtures instead of event_loop for asyncio 2025-05-12 15:57:05 +03:00
6ea56faede build: fix tox environment creation with the latest updates 2025-05-09 15:00:40 +03:00
9e346530f2 refactor: use backslashreplace error handling instead of guessing encoding 2025-05-08 14:03:47 +03:00
d283dccc1e type: update for new cors release 2025-03-17 13:56:59 +02:00
8a4e900ab9 docs: update docs
This commit includes following changes
* add newly added option to configuration referenec
* remove few legacy options from configuration schemas used for
  validation, which might lead to errors during validation.
  Note, however, that settings will be still read by the service
* add link to aurcache
* hide service-setup command description under spoiler
2025-03-17 13:43:04 +02:00
fa6cf8ce36 website: use date instead of version for listing logs
website: make dropdown from logs versions to add some space
2025-03-13 15:45:31 +02:00
a706fbb751 bug: handle dependencies iteratively (fix #141)
It has been found that if there are missing dependencies than whole
process will break instead of just skipping packages. During package
addition it is fine-ish, but it will break updates run
2025-03-13 15:45:27 +02:00
9a23f5c79d refactor: streamline migrations 2025-03-09 23:22:24 +02:00
aaab9069bf docs: rebuild indices 2025-03-09 15:43:41 +02:00
f00b575641 type: use ClassVar decorator for class attributes 2025-03-09 15:43:27 +02:00
6f57ed550b feat: refine log system (#142)
* refine package logging

* add interface

* revert version selection

* replace tuple with model

* rename column in logs table, add coverters

* generate process identifier for child proocesses
2025-03-09 14:46:33 +02:00
08640d9108 feat: add dashboard (#139) 2025-02-24 00:10:15 +02:00
65324633b4 feat: add counters to repository stats overview 2025-02-24 00:10:15 +02:00
ed67898012 fix: parse non-utf pkgbuilds as well (#140)
it has been reported that duriing reading pkgbuilds with latin-1 charset
the execption will be raised. Well, it is one more point to rewrite
parser to use own impl instead of shlex and parse raw byte array instead
2025-02-24 00:10:15 +02:00
a1a8dd68e8 type: remove unused ignore directive 2025-02-24 00:10:15 +02:00
a9505386c2 fix: force dry run build on task initialization for VCS packages
Previously if package is VCS and version in PKGBUILD doesn't match to
AUR one, then makepkg will update pkgbuild ignoring all previous pkgrel
patches

With this change during task init dry ryn process is always run for vcs
packages
2025-02-24 00:10:15 +02:00
195 changed files with 5034 additions and 3325 deletions

View File

@ -1,6 +0,0 @@
skips:
- B101
- B104
- B105
- B106
- B404

View File

@ -8,6 +8,10 @@ on:
- '*' - '*'
- '!*rc*' - '!*rc*'
permissions:
contents: read
packages: write
jobs: jobs:
docker-image: docker-image:
@ -17,18 +21,18 @@ jobs:
packages: write packages: write
steps: steps:
- uses: docker/setup-qemu-action@v2 - uses: docker/setup-qemu-action@v3
- uses: docker/setup-buildx-action@v2 - uses: docker/setup-buildx-action@v3
- name: Login to docker hub - name: Login to docker hub
uses: docker/login-action@v2 uses: docker/login-action@v3
with: with:
username: ${{ secrets.DOCKERHUB_USERNAME }} username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }} password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Login to github container registry - name: Login to github container registry
uses: docker/login-action@v2 uses: docker/login-action@v3
with: with:
registry: ghcr.io registry: ghcr.io
username: ${{ github.repository_owner }} username: ${{ github.repository_owner }}
@ -36,7 +40,7 @@ jobs:
- name: Extract docker metadata - name: Extract docker metadata
id: meta id: meta
uses: docker/metadata-action@v3 uses: docker/metadata-action@v5
with: with:
images: | images: |
arcan1s/ahriman arcan1s/ahriman
@ -46,7 +50,7 @@ jobs:
type=edge type=edge
- name: Build an image and push - name: Build an image and push
uses: docker/build-push-action@v4 uses: docker/build-push-action@v6
with: with:
file: docker/Dockerfile file: docker/Dockerfile
push: true push: true

View File

@ -1,6 +1,12 @@
name: Regress name: Regress
on: workflow_dispatch on:
schedule:
- cron: 1 0 * * 0
workflow_dispatch:
permissions:
contents: read
jobs: jobs:
run-regress-tests: run-regress-tests:
@ -31,8 +37,6 @@ jobs:
- repo:/var/lib/ahriman - repo:/var/lib/ahriman
steps: steps:
- uses: actions/checkout@v3
- run: pacman -Sy - run: pacman -Sy
- name: Init repository - name: Init repository

View File

@ -5,13 +5,16 @@ on:
tags: tags:
- '*' - '*'
permissions:
contents: write
jobs: jobs:
make-release: make-release:
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- uses: actions/checkout@v3 - uses: actions/checkout@v4
- name: Extract version - name: Extract version
id: version id: version
@ -24,8 +27,7 @@ jobs:
token: ${{ secrets.GITHUB_TOKEN }} token: ${{ secrets.GITHUB_TOKEN }}
filter: 'Release \d+\.\d+\.\d+' filter: 'Release \d+\.\d+\.\d+'
- name: Install dependencies - uses: ConorMacBride/install-package@v1.1.0
uses: ConorMacBride/install-package@v1.1.0
with: with:
apt: tox apt: tox
@ -35,7 +37,7 @@ jobs:
VERSION: ${{ steps.version.outputs.VERSION }} VERSION: ${{ steps.version.outputs.VERSION }}
- name: Publish release - name: Publish release
uses: softprops/action-gh-release@v1 uses: softprops/action-gh-release@v2
with: with:
body: | body: |
${{ steps.changelog.outputs.compareurl }} ${{ steps.changelog.outputs.compareurl }}

View File

@ -7,6 +7,10 @@ on:
pull_request: pull_request:
branches: branches:
- master - master
workflow_dispatch:
permissions:
contents: read
jobs: jobs:
run-setup-minimal: run-setup-minimal:
@ -20,7 +24,7 @@ jobs:
- ${{ github.workspace }}:/build - ${{ github.workspace }}:/build
steps: steps:
- uses: actions/checkout@v3 - uses: actions/checkout@v4
- name: Setup the minimal service in arch linux container - name: Setup the minimal service in arch linux container
run: .github/workflows/setup.sh minimal run: .github/workflows/setup.sh minimal
@ -36,7 +40,7 @@ jobs:
options: --privileged -w /build options: --privileged -w /build
steps: steps:
- uses: actions/checkout@v3 - uses: actions/checkout@v4
- name: Setup the service in arch linux container - name: Setup the service in arch linux container
run: .github/workflows/setup.sh run: .github/workflows/setup.sh

View File

@ -1,10 +0,0 @@
#!/bin/bash
# Install dependencies and run test in container
set -ex
# install dependencies
pacman --noconfirm -Syyu base-devel python-tox
# run test and check targets
tox

View File

@ -9,6 +9,10 @@ on:
- master - master
schedule: schedule:
- cron: 1 0 * * * - cron: 1 0 * * *
workflow_dispatch:
permissions:
contents: read
jobs: jobs:
run-tests: run-tests:
@ -22,7 +26,16 @@ jobs:
- ${{ github.workspace }}:/build - ${{ github.workspace }}:/build
steps: steps:
- uses: actions/checkout@v3 - run: pacman --noconfirm -Syu base-devel git python-tox
- name: Run check and tests in arch linux container - run: git config --global --add safe.directory *
run: .github/workflows/tests.sh
- uses: actions/checkout@v4
- name: Run check and tests
run: tox
- name: Generate documentation and check if there are untracked changes
run: |
tox -e docs
[ -z "$(git status --porcelain docs/*.rst)" ]

45
.pylint.toml Normal file
View File

@ -0,0 +1,45 @@
[tool.pylint.main]
init-hook = "sys.path.append('pylint_plugins')"
load-plugins = [
"pylint.extensions.docparams",
"pylint.extensions.bad_builtin",
"definition_order",
"import_order",
]
[tool.pylint.classes]
bad-functions = [
"print",
]
[tool.pylint.design]
max-parents = 15
[tool.pylint."messages control"]
disable = [
"raw-checker-failed",
"bad-inline-option",
"locally-disabled",
"file-ignored",
"suppressed-message",
"useless-suppression",
"deprecated-pragma",
"use-symbolic-message-instead",
"use-implicit-booleaness-not-comparison-to-string",
"use-implicit-booleaness-not-comparison-to-zero",
"missing-module-docstring",
"line-too-long",
"no-name-in-module",
"import-outside-toplevel",
"invalid-name",
"raise-missing-from",
"wrong-import-order",
"too-few-public-methods",
"too-many-instance-attributes",
"broad-exception-caught",
"fixme",
"too-many-arguments",
"duplicate-code",
"cyclic-import",
"too-many-positional-arguments",
]

651
.pylintrc
View File

@ -1,651 +0,0 @@
[MAIN]
# Analyse import fallback blocks. This can be used to support both Python 2 and
# 3 compatible code, which means that the block might have code that exists
# only in one or another interpreter, leading to false positives when analysed.
analyse-fallback-blocks=no
# Clear in-memory caches upon conclusion of linting. Useful if running pylint
# in a server-like mode.
clear-cache-post-run=no
# Load and enable all available extensions. Use --list-extensions to see a list
# all available extensions.
#enable-all-extensions=
# In error mode, messages with a category besides ERROR or FATAL are
# suppressed, and no reports are done by default. Error mode is compatible with
# disabling specific errors.
#errors-only=
# Always return a 0 (non-error) status code, even if lint errors are found.
# This is primarily useful in continuous integration scripts.
#exit-zero=
# A comma-separated list of package or module names from where C extensions may
# be loaded. Extensions are loading into the active Python interpreter and may
# run arbitrary code.
extension-pkg-allow-list=
# A comma-separated list of package or module names from where C extensions may
# be loaded. Extensions are loading into the active Python interpreter and may
# run arbitrary code. (This is an alternative name to extension-pkg-allow-list
# for backward compatibility.)
extension-pkg-whitelist=
# Return non-zero exit code if any of these messages/categories are detected,
# even if score is above --fail-under value. Syntax same as enable. Messages
# specified are enabled, while categories only check already-enabled messages.
fail-on=
# Specify a score threshold under which the program will exit with error.
fail-under=10
# Interpret the stdin as a python script, whose filename needs to be passed as
# the module_or_package argument.
#from-stdin=
# Files or directories to be skipped. They should be base names, not paths.
ignore=CVS
# Add files or directories matching the regular expressions patterns to the
# ignore-list. The regex matches against paths and can be in Posix or Windows
# format. Because '\\' represents the directory delimiter on Windows systems,
# it can't be used as an escape character.
ignore-paths=
# Files or directories matching the regular expression patterns are skipped.
# The regex matches against base names, not paths. The default value ignores
# Emacs file locks
ignore-patterns=^\.#
# List of module names for which member attributes should not be checked
# (useful for modules/projects where namespaces are manipulated during runtime
# and thus existing member attributes cannot be deduced by static analysis). It
# supports qualified module names, as well as Unix pattern matching.
ignored-modules=
# Python code to execute, usually for sys.path manipulation such as
# pygtk.require().
init-hook='import sys; sys.path.append("pylint_plugins")'
# Use multiple processes to speed up Pylint. Specifying 0 will auto-detect the
# number of processors available to use, and will cap the count on Windows to
# avoid hangs.
jobs=1
# Control the amount of potential inferred values when inferring a single
# object. This can help the performance when dealing with large functions or
# complex, nested conditions.
limit-inference-results=100
# List of plugins (as comma separated values of python module names) to load,
# usually to register additional checkers.
load-plugins=pylint.extensions.docparams,
pylint.extensions.bad_builtin,
definition_order,
import_order,
# Pickle collected data for later comparisons.
persistent=yes
# Minimum Python version to use for version dependent checks. Will default to
# the version used to run pylint.
py-version=3.11
# Discover python modules and packages in the file system subtree.
recursive=no
# Add paths to the list of the source roots. Supports globbing patterns. The
# source root is an absolute path or a path relative to the current working
# directory used to determine a package namespace for modules located under the
# source root.
source-roots=
# When enabled, pylint would attempt to guess common misconfiguration and emit
# user-friendly hints instead of false-positive error messages.
suggestion-mode=yes
# Allow loading of arbitrary C extensions. Extensions are imported into the
# active Python interpreter and may run arbitrary code.
unsafe-load-any-extension=no
# In verbose mode, extra non-checker-related info will be displayed.
#verbose=
[BASIC]
# Naming style matching correct argument names.
argument-naming-style=snake_case
# Regular expression matching correct argument names. Overrides argument-
# naming-style. If left empty, argument names will be checked with the set
# naming style.
#argument-rgx=
# Naming style matching correct attribute names.
attr-naming-style=snake_case
# Regular expression matching correct attribute names. Overrides attr-naming-
# style. If left empty, attribute names will be checked with the set naming
# style.
#attr-rgx=
bad-functions=print,
# Bad variable names which should always be refused, separated by a comma.
bad-names=foo,
bar,
baz,
toto,
tutu,
tata
# Bad variable names regexes, separated by a comma. If names match any regex,
# they will always be refused
bad-names-rgxs=
# Naming style matching correct class attribute names.
class-attribute-naming-style=any
# Regular expression matching correct class attribute names. Overrides class-
# attribute-naming-style. If left empty, class attribute names will be checked
# with the set naming style.
#class-attribute-rgx=
# Naming style matching correct class constant names.
class-const-naming-style=UPPER_CASE
# Regular expression matching correct class constant names. Overrides class-
# const-naming-style. If left empty, class constant names will be checked with
# the set naming style.
#class-const-rgx=
# Naming style matching correct class names.
class-naming-style=PascalCase
# Regular expression matching correct class names. Overrides class-naming-
# style. If left empty, class names will be checked with the set naming style.
#class-rgx=
# Naming style matching correct constant names.
const-naming-style=UPPER_CASE
# Regular expression matching correct constant names. Overrides const-naming-
# style. If left empty, constant names will be checked with the set naming
# style.
#const-rgx=
# Minimum line length for functions/classes that require docstrings, shorter
# ones are exempt.
docstring-min-length=-1
# Naming style matching correct function names.
function-naming-style=snake_case
# Regular expression matching correct function names. Overrides function-
# naming-style. If left empty, function names will be checked with the set
# naming style.
#function-rgx=
# Good variable names which should always be accepted, separated by a comma.
good-names=i,
j,
k,
ex,
Run,
_
# Good variable names regexes, separated by a comma. If names match any regex,
# they will always be accepted
good-names-rgxs=
# Include a hint for the correct naming format with invalid-name.
include-naming-hint=no
# Naming style matching correct inline iteration names.
inlinevar-naming-style=any
# Regular expression matching correct inline iteration names. Overrides
# inlinevar-naming-style. If left empty, inline iteration names will be checked
# with the set naming style.
#inlinevar-rgx=
# Naming style matching correct method names.
method-naming-style=snake_case
# Regular expression matching correct method names. Overrides method-naming-
# style. If left empty, method names will be checked with the set naming style.
#method-rgx=
# Naming style matching correct module names.
module-naming-style=snake_case
# Regular expression matching correct module names. Overrides module-naming-
# style. If left empty, module names will be checked with the set naming style.
#module-rgx=
# Colon-delimited sets of names that determine each other's naming style when
# the name regexes allow several styles.
name-group=
# Regular expression which should only match function or class names that do
# not require a docstring.
no-docstring-rgx=
# List of decorators that produce properties, such as abc.abstractproperty. Add
# to this list to register other decorators that produce valid properties.
# These decorators are taken in consideration only for invalid-name.
property-classes=abc.abstractproperty
# Regular expression matching correct type alias names. If left empty, type
# alias names will be checked with the set naming style.
#typealias-rgx=
# Regular expression matching correct type variable names. If left empty, type
# variable names will be checked with the set naming style.
#typevar-rgx=
# Naming style matching correct variable names.
variable-naming-style=snake_case
# Regular expression matching correct variable names. Overrides variable-
# naming-style. If left empty, variable names will be checked with the set
# naming style.
#variable-rgx=
[CLASSES]
# Warn about protected attribute access inside special methods
check-protected-access-in-special-methods=no
# List of method names used to declare (i.e. assign) instance attributes.
defining-attr-methods=__init__,
__new__,
setUp,
asyncSetUp,
__post_init__
# List of member names, which should be excluded from the protected access
# warning.
exclude-protected=_asdict,_fields,_replace,_source,_make,os._exit
# List of valid names for the first argument in a class method.
valid-classmethod-first-arg=cls
# List of valid names for the first argument in a metaclass class method.
valid-metaclass-classmethod-first-arg=mcs
[DESIGN]
# List of regular expressions of class ancestor names to ignore when counting
# public methods (see R0903)
exclude-too-few-public-methods=
# List of qualified class names to ignore when counting class parents (see
# R0901)
ignored-parents=
# Maximum number of arguments for function / method.
max-args=5
# Maximum number of attributes for a class (see R0902).
max-attributes=7
# Maximum number of boolean expressions in an if statement (see R0916).
max-bool-expr=5
# Maximum number of branch for function / method body.
max-branches=12
# Maximum number of locals for function / method body.
max-locals=15
# Maximum number of parents for a class (see R0901).
max-parents=15
# Maximum number of public methods for a class (see R0904).
max-public-methods=20
# Maximum number of return / yield for function / method body.
max-returns=6
# Maximum number of statements in function / method body.
max-statements=50
# Minimum number of public methods for a class (see R0903).
min-public-methods=2
[EXCEPTIONS]
# Exceptions that will emit a warning when caught.
overgeneral-exceptions=builtins.BaseException,builtins.Exception
[FORMAT]
# Expected format of line ending, e.g. empty (any line ending), LF or CRLF.
expected-line-ending-format=
# Regexp for a line that is allowed to be longer than the limit.
ignore-long-lines=^\s*(# )?<?https?://\S+>?$
# Number of spaces of indent required inside a hanging or continued line.
indent-after-paren=4
# String used as indentation unit. This is usually " " (4 spaces) or "\t" (1
# tab).
indent-string=' '
# Maximum number of characters on a single line.
max-line-length=100
# Maximum number of lines in a module.
max-module-lines=1000
# Allow the body of a class to be on the same line as the declaration if body
# contains single statement.
single-line-class-stmt=no
# Allow the body of an if to be on the same line as the test if there is no
# else.
single-line-if-stmt=no
[IMPORTS]
# List of modules that can be imported at any level, not just the top level
# one.
allow-any-import-level=
# Allow explicit reexports by alias from a package __init__.
allow-reexport-from-package=no
# Allow wildcard imports from modules that define __all__.
allow-wildcard-with-all=no
# Deprecated modules which should not be used, separated by a comma.
deprecated-modules=
# Output a graph (.gv or any supported image format) of external dependencies
# to the given file (report RP0402 must not be disabled).
ext-import-graph=
# Output a graph (.gv or any supported image format) of all (i.e. internal and
# external) dependencies to the given file (report RP0402 must not be
# disabled).
import-graph=
# Output a graph (.gv or any supported image format) of internal dependencies
# to the given file (report RP0402 must not be disabled).
int-import-graph=
# Force import order to recognize a module as part of the standard
# compatibility libraries.
known-standard-library=
# Force import order to recognize a module as part of a third party library.
known-third-party=enchant
# Couples of modules and preferred modules, separated by a comma.
preferred-modules=
[LOGGING]
# The type of string formatting that logging methods do. `old` means using %
# formatting, `new` is for `{}` formatting.
logging-format-style=old
# Logging modules to check that the string format arguments are in logging
# function parameter format.
logging-modules=logging
[MESSAGES CONTROL]
# Only show warnings with the listed confidence levels. Leave empty to show
# all. Valid levels: HIGH, CONTROL_FLOW, INFERENCE, INFERENCE_FAILURE,
# UNDEFINED.
confidence=HIGH,
CONTROL_FLOW,
INFERENCE,
INFERENCE_FAILURE,
UNDEFINED
# Disable the message, report, category or checker with the given id(s). You
# can either give multiple identifiers separated by comma (,) or put this
# option multiple times (only on the command line, not in the configuration
# file where it should appear only once). You can also use "--disable=all" to
# disable everything first and then re-enable specific checks. For example, if
# you want to run only the similarities checker, you can use "--disable=all
# --enable=similarities". If you want to run only the classes checker, but have
# no Warning level messages displayed, use "--disable=all --enable=classes
# --disable=W".
disable=raw-checker-failed,
bad-inline-option,
locally-disabled,
file-ignored,
suppressed-message,
useless-suppression,
deprecated-pragma,
use-symbolic-message-instead,
missing-module-docstring,
line-too-long,
no-name-in-module,
import-outside-toplevel,
invalid-name,
raise-missing-from,
wrong-import-order,
too-few-public-methods,
too-many-instance-attributes,
broad-except,
fixme,
too-many-arguments,
duplicate-code,
cyclic-import,
too-many-positional-arguments,
# Enable the message, report, category or checker with the given id(s). You can
# either give multiple identifier separated by comma (,) or put this option
# multiple time (only on the command line, not in the configuration file where
# it should appear only once). See also the "--disable" option for examples.
enable=c-extension-no-member
[METHOD_ARGS]
# List of qualified names (i.e., library.method) which require a timeout
# parameter e.g. 'requests.api.get,requests.api.post'
timeout-methods=requests.api.delete,requests.api.get,requests.api.head,requests.api.options,requests.api.patch,requests.api.post,requests.api.put,requests.api.request
[MISCELLANEOUS]
# List of note tags to take in consideration, separated by a comma.
notes=FIXME,
XXX,
TODO
# Regular expression of note tags to take in consideration.
notes-rgx=
[REFACTORING]
# Maximum number of nested blocks for function / method body
max-nested-blocks=5
# Complete name of functions that never returns. When checking for
# inconsistent-return-statements if a never returning function is called then
# it will be considered as an explicit return statement and no message will be
# printed.
never-returning-functions=sys.exit,argparse.parse_error
[REPORTS]
# Python expression which should return a score less than or equal to 10. You
# have access to the variables 'fatal', 'error', 'warning', 'refactor',
# 'convention', and 'info' which contain the number of messages in each
# category, as well as 'statement' which is the total number of statements
# analyzed. This score is used by the global evaluation report (RP0004).
evaluation=max(0, 0 if fatal else 10.0 - ((float(5 * error + warning + refactor + convention) / statement) * 10))
# Template used to display messages. This is a python new-style format string
# used to format the message information. See doc for all details.
msg-template=
# Set the output format. Available formats are text, parseable, colorized, json
# and msvs (visual studio). You can also give a reporter class, e.g.
# mypackage.mymodule.MyReporterClass.
#output-format=
# Tells whether to display a full report or only the messages.
reports=no
# Activate the evaluation score.
score=yes
[SIMILARITIES]
# Comments are removed from the similarity computation
ignore-comments=yes
# Docstrings are removed from the similarity computation
ignore-docstrings=yes
# Imports are removed from the similarity computation
ignore-imports=yes
# Signatures are removed from the similarity computation
ignore-signatures=yes
# Minimum lines number of a similarity.
min-similarity-lines=4
[SPELLING]
# Limits count of emitted suggestions for spelling mistakes.
max-spelling-suggestions=4
# Spelling dictionary name. No available dictionaries : You need to install
# both the python package and the system dependency for enchant to work..
spelling-dict=
# List of comma separated words that should be considered directives if they
# appear at the beginning of a comment and should not be checked.
spelling-ignore-comment-directives=fmt: on,fmt: off,noqa:,noqa,nosec,isort:skip,mypy:
# List of comma separated words that should not be checked.
spelling-ignore-words=
# A path to a file that contains the private dictionary; one word per line.
spelling-private-dict-file=
# Tells whether to store unknown words to the private dictionary (see the
# --spelling-private-dict-file option) instead of raising a message.
spelling-store-unknown-words=no
[STRING]
# This flag controls whether inconsistent-quotes generates a warning when the
# character used as a quote delimiter is used inconsistently within a module.
check-quote-consistency=no
# This flag controls whether the implicit-str-concat should generate a warning
# on implicit string concatenation in sequences defined over several lines.
check-str-concat-over-line-jumps=no
[TYPECHECK]
# List of decorators that produce context managers, such as
# contextlib.contextmanager. Add to this list to register other decorators that
# produce valid context managers.
contextmanager-decorators=contextlib.contextmanager
# List of members which are set dynamically and missed by pylint inference
# system, and so shouldn't trigger E1101 when accessed. Python regular
# expressions are accepted.
generated-members=
# Tells whether to warn about missing members when the owner of the attribute
# is inferred to be None.
ignore-none=yes
# This flag controls whether pylint should warn about no-member and similar
# checks whenever an opaque object is returned when inferring. The inference
# can return multiple potential results while evaluating a Python object, but
# some branches might not be evaluated, which results in partial inference. In
# that case, it might be useful to still emit no-member and other checks for
# the rest of the inferred objects.
ignore-on-opaque-inference=yes
# List of symbolic message names to ignore for Mixin members.
ignored-checks-for-mixins=no-member,
not-async-context-manager,
not-context-manager,
attribute-defined-outside-init
# List of class names for which member attributes should not be checked (useful
# for classes with dynamically set attributes). This supports the use of
# qualified names.
ignored-classes=optparse.Values,thread._local,_thread._local,argparse.Namespace
# Show a hint with possible names when a member name was not found. The aspect
# of finding the hint is based on edit distance.
missing-member-hint=yes
# The minimum edit distance a name should have in order to be considered a
# similar match for a missing member name.
missing-member-hint-distance=1
# The total number of similar names that should be taken in consideration when
# showing a hint for a missing member.
missing-member-max-choices=1
# Regex pattern to define which classes are considered mixins.
mixin-class-rgx=.*[Mm]ixin
# List of decorators that change the signature of a decorated function.
signature-mutators=
[VARIABLES]
# List of additional names supposed to be defined in builtins. Remember that
# you should avoid defining new builtins when possible.
additional-builtins=
# Tells whether unused global variables should be treated as a violation.
allow-global-unused-variables=yes
# List of names allowed to shadow builtins
allowed-redefined-builtins=
# List of strings which can identify a callback function by name. A callback
# name must start or end with one of those strings.
callbacks=cb_,
_cb
# A regular expression matching the name of dummy variables (i.e. expected to
# not be used).
dummy-variables-rgx=_+$|(_[a-zA-Z0-9_]*[a-zA-Z0-9]+?$)|dummy|^ignored_|^unused_
# Argument names that match this expression will be ignored.
ignored-argument-names=_.*|^ignored_|^unused_
# Tells whether we should check for unused import in __init__ files.
init-import=no
# List of qualified module names which can have objects that can redefine
# builtins.
redefining-builtins-modules=six.moves,past.builtins,future.builtins,builtins,io

View File

@ -9,13 +9,7 @@ build:
python: python:
install: install:
- method: pip - requirements: docs/requirements.txt
path: .
extra_requirements:
- docs
- s3
- validator
- web
formats: formats:
- pdf - pdf

View File

@ -80,7 +80,7 @@ Again, the most checks can be performed by `tox` command, though some additional
>>> clazz = Clazz() >>> clazz = Clazz()
""" """
CLAZZ_ATTRIBUTE = 42 CLAZZ_ATTRIBUTE: ClassVar[int] = 42
def __init__(self, *args: Any, **kwargs: Any) -> None: def __init__(self, *args: Any, **kwargs: Any) -> None:
""" """
@ -96,6 +96,7 @@ Again, the most checks can be performed by `tox` command, though some additional
* Type annotations are the must, even for local functions. For the function argument `self` (for instance methods) and `cls` (for class methods) should not be annotated. * Type annotations are the must, even for local functions. For the function argument `self` (for instance methods) and `cls` (for class methods) should not be annotated.
* For collection types built-in classes must be used if possible (e.g. `dict` instead of `typing.Dict`, `tuple` instead of `typing.Tuple`). In case if built-in type is not available, but `collections.abc` provides interface, it must be used (e.g. `collections.abc.Awaitable` instead of `typing.Awaitable`, `collections.abc.Iterable` instead of `typing.Iterable`). For union classes, the bar operator (`|`) must be used (e.g. `float | int` instead of `typing.Union[float, int]`), which also includes `typing.Optional` (e.g. `str | None` instead of `Optional[str]`). * For collection types built-in classes must be used if possible (e.g. `dict` instead of `typing.Dict`, `tuple` instead of `typing.Tuple`). In case if built-in type is not available, but `collections.abc` provides interface, it must be used (e.g. `collections.abc.Awaitable` instead of `typing.Awaitable`, `collections.abc.Iterable` instead of `typing.Iterable`). For union classes, the bar operator (`|`) must be used (e.g. `float | int` instead of `typing.Union[float, int]`), which also includes `typing.Optional` (e.g. `str | None` instead of `Optional[str]`).
* `classmethod` should (almost) always return `Self`. In case of mypy warning (e.g. if there is a branch in which function doesn't return the instance of `cls`) consider using `staticmethod` instead. * `classmethod` should (almost) always return `Self`. In case of mypy warning (e.g. if there is a branch in which function doesn't return the instance of `cls`) consider using `staticmethod` instead.
* Class attributes must be decorated as `ClassVar[...]`.
* Recommended order of function definitions in class: * Recommended order of function definitions in class:
```python ```python

View File

@ -40,6 +40,7 @@ RUN pacman -S --noconfirm --asdeps \
pacman -S --noconfirm --asdeps \ pacman -S --noconfirm --asdeps \
git \ git \
python-aiohttp \ python-aiohttp \
python-aiohttp-openmetrics \
python-boto3 \ python-boto3 \
python-cerberus \ python-cerberus \
python-cryptography \ python-cryptography \
@ -112,6 +113,7 @@ RUN pacman -S --noconfirm ahriman
RUN pacman -S --noconfirm --asdeps \ RUN pacman -S --noconfirm --asdeps \
python-aioauth-client \ python-aioauth-client \
python-aiohttp-apispec-git \ python-aiohttp-apispec-git \
python-aiohttp-openmetrics \
python-aiohttp-security \ python-aiohttp-security \
python-aiohttp-session \ python-aiohttp-session \
python-boto3 \ python-boto3 \

File diff suppressed because it is too large Load Diff

View File

@ -124,6 +124,14 @@ ahriman.core.database.migrations.m014\_auditlog module
:no-undoc-members: :no-undoc-members:
:show-inheritance: :show-inheritance:
ahriman.core.database.migrations.m015\_logs\_process\_id module
---------------------------------------------------------------
.. automodule:: ahriman.core.database.migrations.m015_logs_process_id
:members:
:no-undoc-members:
:show-inheritance:
Module contents Module contents
--------------- ---------------

View File

@ -92,6 +92,14 @@ ahriman.core.formatters.repository\_printer module
:no-undoc-members: :no-undoc-members:
:show-inheritance: :show-inheritance:
ahriman.core.formatters.repository\_stats\_printer module
---------------------------------------------------------
.. automodule:: ahriman.core.formatters.repository_stats_printer
:members:
:no-undoc-members:
:show-inheritance:
ahriman.core.formatters.status\_printer module ahriman.core.formatters.status\_printer module
---------------------------------------------- ----------------------------------------------

View File

@ -100,6 +100,14 @@ ahriman.models.log\_handler module
:no-undoc-members: :no-undoc-members:
:show-inheritance: :show-inheritance:
ahriman.models.log\_record module
---------------------------------
.. automodule:: ahriman.models.log_record
:members:
:no-undoc-members:
:show-inheritance:
ahriman.models.log\_record\_id module ahriman.models.log\_record\_id module
------------------------------------- -------------------------------------
@ -236,6 +244,14 @@ ahriman.models.repository\_paths module
:no-undoc-members: :no-undoc-members:
:show-inheritance: :show-inheritance:
ahriman.models.repository\_stats module
---------------------------------------
.. automodule:: ahriman.models.repository_stats
:members:
:no-undoc-members:
:show-inheritance:
ahriman.models.result module ahriman.models.result module
---------------------------- ----------------------------
@ -252,6 +268,14 @@ ahriman.models.scan\_paths module
:no-undoc-members: :no-undoc-members:
:show-inheritance: :show-inheritance:
ahriman.models.series\_statistics module
----------------------------------------
.. automodule:: ahriman.models.series_statistics
:members:
:no-undoc-members:
:show-inheritance:
ahriman.models.sign\_settings module ahriman.models.sign\_settings module
------------------------------------ ------------------------------------

View File

@ -20,6 +20,14 @@ ahriman.web.middlewares.exception\_handler module
:no-undoc-members: :no-undoc-members:
:show-inheritance: :show-inheritance:
ahriman.web.middlewares.metrics\_handler module
-----------------------------------------------
.. automodule:: ahriman.web.middlewares.metrics_handler
:members:
:no-undoc-members:
:show-inheritance:
Module contents Module contents
--------------- ---------------

View File

@ -4,6 +4,14 @@ ahriman.web.schemas package
Submodules Submodules
---------- ----------
ahriman.web.schemas.any\_schema module
--------------------------------------
.. automodule:: ahriman.web.schemas.any_schema
:members:
:no-undoc-members:
:show-inheritance:
ahriman.web.schemas.aur\_package\_schema module ahriman.web.schemas.aur\_package\_schema module
----------------------------------------------- -----------------------------------------------
@ -116,6 +124,14 @@ ahriman.web.schemas.login\_schema module
:no-undoc-members: :no-undoc-members:
:show-inheritance: :show-inheritance:
ahriman.web.schemas.logs\_rotate\_schema module
-----------------------------------------------
.. automodule:: ahriman.web.schemas.logs_rotate_schema
:members:
:no-undoc-members:
:show-inheritance:
ahriman.web.schemas.logs\_schema module ahriman.web.schemas.logs\_schema module
--------------------------------------- ---------------------------------------
@ -260,6 +276,14 @@ ahriman.web.schemas.repository\_id\_schema module
:no-undoc-members: :no-undoc-members:
:show-inheritance: :show-inheritance:
ahriman.web.schemas.repository\_stats\_schema module
----------------------------------------------------
.. automodule:: ahriman.web.schemas.repository_stats_schema
:members:
:no-undoc-members:
:show-inheritance:
ahriman.web.schemas.search\_schema module ahriman.web.schemas.search\_schema module
----------------------------------------- -----------------------------------------
@ -284,14 +308,6 @@ ahriman.web.schemas.update\_flags\_schema module
:no-undoc-members: :no-undoc-members:
:show-inheritance: :show-inheritance:
ahriman.web.schemas.versioned\_log\_schema module
-------------------------------------------------
.. automodule:: ahriman.web.schemas.versioned_log_schema
:members:
:no-undoc-members:
:show-inheritance:
ahriman.web.schemas.worker\_schema module ahriman.web.schemas.worker\_schema module
----------------------------------------- -----------------------------------------

View File

@ -12,6 +12,14 @@ ahriman.web.views.v1.service.add module
:no-undoc-members: :no-undoc-members:
:show-inheritance: :show-inheritance:
ahriman.web.views.v1.service.logs module
----------------------------------------
.. automodule:: ahriman.web.views.v1.service.logs
:members:
:no-undoc-members:
:show-inheritance:
ahriman.web.views.v1.service.pgp module ahriman.web.views.v1.service.pgp module
--------------------------------------- ---------------------------------------

View File

@ -12,6 +12,14 @@ ahriman.web.views.v1.status.info module
:no-undoc-members: :no-undoc-members:
:show-inheritance: :show-inheritance:
ahriman.web.views.v1.status.metrics module
------------------------------------------
.. automodule:: ahriman.web.views.v1.status.metrics
:members:
:no-undoc-members:
:show-inheritance:
ahriman.web.views.v1.status.repositories module ahriman.web.views.v1.status.repositories module
----------------------------------------------- -----------------------------------------------

View File

@ -413,10 +413,11 @@ Web application
Web application requires the following python packages to be installed: Web application requires the following python packages to be installed:
* Core part requires ``aiohttp`` (application itself), ``aiohttp_jinja2`` and ``Jinja2`` (HTML generation from templates). * Core part requires ``aiohttp`` (application itself), ``aiohttp_jinja2`` and ``Jinja2`` (HTML generation from templates).
* Additional web features also require ``aiohttp-apispec`` (autogenerated documentation), ``aiohttp_cors`` (CORS support, required by documentation). * Additional web features also require ``aiohttp-apispec`` (autogenerated documentation, optional), ``aiohttp_cors`` (CORS support, required by documentation).
* In addition, authorization feature requires ``aiohttp_security``, ``aiohttp_session`` and ``cryptography``. * In addition, authorization feature requires ``aiohttp_security``, ``aiohttp_session`` and ``cryptography``.
* In addition to base authorization dependencies, OAuth2 also requires ``aioauth-client`` library. * In addition to base authorization dependencies, OAuth2 also requires ``aioauth-client`` library.
* In addition if you would like to disable authorization for local access (recommended way in order to run the application itself with reporting support), the ``requests-unixsocket2`` library is required. * In addition if you would like to disable authorization for local access (recommended way in order to run the application itself with reporting support), the ``requests-unixsocket2`` library is required.
* Application metrics will be automatically enabled after installing ``aiohttp-openmetrics`` package.
Middlewares Middlewares
^^^^^^^^^^^ ^^^^^^^^^^^

View File

@ -15,9 +15,8 @@ import sys
from pathlib import Path from pathlib import Path
from ahriman import __version__
# support package imports
basedir = Path(__file__).resolve().parent.parent / "src" basedir = Path(__file__).resolve().parent.parent / "src"
sys.path.insert(0, str(basedir)) sys.path.insert(0, str(basedir))
@ -29,6 +28,7 @@ copyright = f"2021-{datetime.date.today().year}, ahriman team"
author = "ahriman team" author = "ahriman team"
# The full version, including alpha/beta/rc tags # The full version, including alpha/beta/rc tags
from ahriman import __version__
release = __version__ release = __version__
@ -91,7 +91,13 @@ autoclass_content = "both"
autodoc_member_order = "groupwise" autodoc_member_order = "groupwise"
autodoc_mock_imports = ["cryptography", "pyalpm"] autodoc_mock_imports = [
"aioauth_client",
"aiohttp_security",
"aiohttp_session",
"cryptography",
"pyalpm",
]
autodoc_default_options = { autodoc_default_options = {
"no-undoc-members": True, "no-undoc-members": True,

View File

@ -81,6 +81,7 @@ Base configuration settings.
* ``apply_migrations`` - perform database migrations on the application start, boolean, optional, default ``yes``. Useful if you are using git version. Note, however, that this option must be changed only if you know what to do and going to handle migrations manually. * ``apply_migrations`` - perform database migrations on the application start, boolean, optional, default ``yes``. Useful if you are using git version. Note, however, that this option must be changed only if you know what to do and going to handle migrations manually.
* ``database`` - path to the application SQLite database, string, required. * ``database`` - path to the application SQLite database, string, required.
* ``include`` - path to directory with configuration files overrides, string, optional. Files will be read in alphabetical order. * ``include`` - path to directory with configuration files overrides, string, optional. Files will be read in alphabetical order.
* ``keep_last_logs`` - amount of build logs to be kept for each package, integer, optional ,default ``0``. Logs will be cleared at the end of each process.
* ``logging`` - path to logging configuration, string, required. Check ``logging.ini`` for reference. * ``logging`` - path to logging configuration, string, required. Check ``logging.ini`` for reference.
``alpm:*`` groups ``alpm:*`` groups
@ -217,7 +218,7 @@ Mirrorlist generator plugin
``remote-pull`` group ``remote-pull`` group
--------------------- ---------------------
Remote git source synchronization settings. Unlike ``Upload`` triggers those triggers are used for PKGBUILD synchronization - fetch from remote repository PKGBUILDs before updating process. Remote git source synchronization settings. Unlike ``upload`` triggers those triggers are used for PKGBUILD synchronization - fetch from remote repository PKGBUILDs before updating process.
It supports authorization; to do so you'd need to prefix the URL with authorization part, e.g. ``https://key:token@github.com/arcan1s/ahriman.git``. It is highly recommended to use application tokens instead of your user authorization details. Alternatively, you can use any other option supported by git, e.g.: It supports authorization; to do so you'd need to prefix the URL with authorization part, e.g. ``https://key:token@github.com/arcan1s/ahriman.git``. It is highly recommended to use application tokens instead of your user authorization details. Alternatively, you can use any other option supported by git, e.g.:

View File

@ -56,6 +56,13 @@ Though originally I've created ahriman by trying to improve the project, it stil
It is automation tools for ``repoctl`` mentioned above. Except for using shell it looks pretty cool and also offers some additional features like patches, remote synchronization (isn't it?) and reporting. It is automation tools for ``repoctl`` mentioned above. Except for using shell it looks pretty cool and also offers some additional features like patches, remote synchronization (isn't it?) and reporting.
`AURCache <https://github.com/Lukas-Heiligenbrunner/AURCache>`__
""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
That's really cool project if you are looking for simple service to build AUR packages. It provides very informative dashboard and easy to configure and use. However, it doesn't provide direct way to control build process (e.g. it is neither trivial to build packages for architectures which are not supported by default nor to change build flags).
Also this application relies on docker setup (e.g. builders are only available as special docker containers). In addition, it uses ``paru`` to build packages instead of ``devtools``.
How to check service logs How to check service logs
^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^

128
docs/requirements.txt Normal file
View File

@ -0,0 +1,128 @@
# This file was autogenerated by uv via the following command:
# uv pip compile --group pyproject.toml:docs --extra s3 --extra validator --extra web --output-file docs/requirements.txt pyproject.toml
aiohappyeyeballs==2.6.1
# via aiohttp
aiohttp==3.11.18
# via
# ahriman (pyproject.toml)
# aiohttp-cors
# aiohttp-jinja2
aiohttp-cors==0.8.1
# via ahriman (pyproject.toml)
aiohttp-jinja2==1.6
# via ahriman (pyproject.toml)
aiosignal==1.3.2
# via aiohttp
alabaster==1.0.0
# via sphinx
argparse-manpage==4.6
# via ahriman (pyproject.toml:docs)
attrs==25.3.0
# via aiohttp
babel==2.17.0
# via sphinx
bcrypt==4.3.0
# via ahriman (pyproject.toml)
boto3==1.38.11
# via ahriman (pyproject.toml)
botocore==1.38.11
# via
# boto3
# s3transfer
cerberus==1.3.7
# via ahriman (pyproject.toml)
certifi==2025.4.26
# via requests
charset-normalizer==3.4.2
# via requests
docutils==0.21.2
# via
# sphinx
# sphinx-argparse
# sphinx-rtd-theme
frozenlist==1.6.0
# via
# aiohttp
# aiosignal
idna==3.10
# via
# requests
# yarl
imagesize==1.4.1
# via sphinx
inflection==0.5.1
# via ahriman (pyproject.toml)
jinja2==3.1.6
# via
# aiohttp-jinja2
# sphinx
jmespath==1.0.1
# via
# boto3
# botocore
markupsafe==3.0.2
# via jinja2
multidict==6.4.3
# via
# aiohttp
# yarl
packaging==25.0
# via sphinx
propcache==0.3.1
# via
# aiohttp
# yarl
pydeps==3.0.1
# via ahriman (pyproject.toml:docs)
pyelftools==0.32
# via ahriman (pyproject.toml)
pygments==2.19.1
# via sphinx
python-dateutil==2.9.0.post0
# via botocore
requests==2.32.3
# via
# ahriman (pyproject.toml)
# sphinx
roman-numerals-py==3.1.0
# via sphinx
s3transfer==0.12.0
# via boto3
shtab==1.7.2
# via ahriman (pyproject.toml:docs)
six==1.17.0
# via python-dateutil
snowballstemmer==3.0.0.1
# via sphinx
sphinx==8.2.3
# via
# ahriman (pyproject.toml:docs)
# sphinx-argparse
# sphinx-rtd-theme
# sphinxcontrib-jquery
sphinx-argparse==0.5.2
# via ahriman (pyproject.toml:docs)
sphinx-rtd-theme==3.0.2
# via ahriman (pyproject.toml:docs)
sphinxcontrib-applehelp==2.0.0
# via sphinx
sphinxcontrib-devhelp==2.0.0
# via sphinx
sphinxcontrib-htmlhelp==2.1.0
# via sphinx
sphinxcontrib-jquery==4.1
# via sphinx-rtd-theme
sphinxcontrib-jsmath==1.0.1
# via sphinx
sphinxcontrib-qthelp==2.0.0
# via sphinx
sphinxcontrib-serializinghtml==2.0.0
# via sphinx
stdlib-list==0.11.1
# via pydeps
urllib3==2.4.0
# via
# botocore
# requests
yarl==1.20.0
# via aiohttp

View File

@ -12,19 +12,22 @@ Initial setup
sudo ahriman -a x86_64 -r aur service-setup ... sudo ahriman -a x86_64 -r aur service-setup ...
``service-setup`` literally does the following steps: .. admonition:: Details
:collapsible: closed
#. ``service-setup`` literally does the following steps:
Create ``/var/lib/ahriman/.makepkg.conf`` with ``makepkg.conf`` overrides if required (at least you might want to set ``PACKAGER``):
.. code-block:: shell #.
Create ``/var/lib/ahriman/.makepkg.conf`` with ``makepkg.conf`` overrides if required (at least you might want to set ``PACKAGER``):
echo 'PACKAGER="ahriman bot <ahriman@example.com>"' | sudo -u ahriman tee -a /var/lib/ahriman/.makepkg.conf .. code-block:: shell
#. echo 'PACKAGER="ahriman bot <ahriman@example.com>"' | sudo -u ahriman tee -a /var/lib/ahriman/.makepkg.conf
Configure build tools (it is required for correct dependency management system):
#. #.
Configure build tools (it is required for correct dependency management system):
#.
Create build command (you can choose any name for command, basically it should be ``{name}-{arch}-build``): Create build command (you can choose any name for command, basically it should be ``{name}-{arch}-build``):
.. code-block:: shell .. code-block:: shell
@ -67,7 +70,7 @@ Initial setup
echo 'ahriman ALL=(ALL) NOPASSWD:SETENV: CARCHBUILD_CMD' | tee -a /etc/sudoers.d/ahriman echo 'ahriman ALL=(ALL) NOPASSWD:SETENV: CARCHBUILD_CMD' | tee -a /etc/sudoers.d/ahriman
chmod 400 /etc/sudoers.d/ahriman chmod 400 /etc/sudoers.d/ahriman
This command supports several arguments, kindly refer to its help message. This command supports several arguments, kindly refer to its help message.
#. #.
Start and enable ``ahriman@.timer`` via ``systemctl``: Start and enable ``ahriman@.timer`` via ``systemctl``:

View File

@ -2,7 +2,7 @@
pkgbase='ahriman' pkgbase='ahriman'
pkgname=('ahriman' 'ahriman-core' 'ahriman-triggers' 'ahriman-web') pkgname=('ahriman' 'ahriman-core' 'ahriman-triggers' 'ahriman-web')
pkgver=2.17.1 pkgver=2.18.3
pkgrel=1 pkgrel=1
pkgdesc="ArcH linux ReposItory MANager" pkgdesc="ArcH linux ReposItory MANager"
arch=('any') arch=('any')
@ -75,6 +75,7 @@ package_ahriman-web() {
depends=("$pkgbase-core=$pkgver" 'python-aiohttp-cors' 'python-aiohttp-jinja2') depends=("$pkgbase-core=$pkgver" 'python-aiohttp-cors' 'python-aiohttp-jinja2')
optdepends=('python-aioauth-client: OAuth2 authorization support' optdepends=('python-aioauth-client: OAuth2 authorization support'
'python-aiohttp-apispec>=3.0.0: autogenerated API documentation' 'python-aiohttp-apispec>=3.0.0: autogenerated API documentation'
'python-aiohttp-openmetrics: HTTP metrics support'
'python-aiohttp-security: authorization support' 'python-aiohttp-security: authorization support'
'python-aiohttp-session: authorization support' 'python-aiohttp-session: authorization support'
'python-cryptography: authorization support') 'python-cryptography: authorization support')

View File

@ -7,6 +7,8 @@ logging = ahriman.ini.d/logging.ini
;apply_migrations = yes ;apply_migrations = yes
; Path to the application SQLite database. ; Path to the application SQLite database.
database = ${repository:root}/ahriman.db database = ${repository:root}/ahriman.db
; Keep last build logs for each package
keep_last_logs = 5
[alpm] [alpm]
; Path to pacman system database cache. ; Path to pacman system database cache.

View File

@ -16,11 +16,11 @@
<div class="container"> <div class="container">
<nav class="navbar navbar-expand-lg"> <nav class="navbar navbar-expand-lg">
<div class="navbar-brand"><a href="https://github.com/arcan1s/ahriman" title="logo"><img src="/static/logo.svg" width="30" height="30" alt=""></a></div> <div class="navbar-brand"><a href="https://github.com/arcan1s/ahriman" title="logo"><img src="/static/logo.svg" width="30" height="30" alt=""></a></div>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#repositories-navbar-supported-content" aria-controls="repositories-navbar-supported-content" aria-expanded="false" aria-label="Toggle navigation"> <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#repositories-navbar" aria-controls="repositories-navbar" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span> <span class="navbar-toggler-icon"></span>
</button> </button>
<div id="repositories-navbar-supported-content" class="collapse navbar-collapse"> <div id="repositories-navbar" class="collapse navbar-collapse">
<ul id="repositories" class="nav nav-tabs"> <ul id="repositories" class="nav nav-tabs">
{% for repository in repositories %} {% for repository in repositories %}
<li class="nav-item"> <li class="nav-item">
@ -36,7 +36,9 @@
<div class="container"> <div class="container">
<div id="toolbar" class="dropdown"> <div id="toolbar" class="dropdown">
<a id="badge-status" tabindex="0" role="button" class="btn btn-outline-secondary" data-bs-toggle="popover" data-bs-trigger="focus" data-bs-content="no run data"><i class="bi bi-info-circle"></i></a> <button id="dashboard-button" type="button" class="btn btn-outline-secondary" data-bs-toggle="modal" data-bs-target="#dashboard-modal">
<i class="bi bi-info-circle"></i>
</button>
{% if not auth.enabled or auth.username is not none %} {% if not auth.enabled or auth.username is not none %}
<button type="button" class="btn btn-primary dropdown-toggle" data-bs-toggle="dropdown" aria-expanded="false"> <button type="button" class="btn btn-primary dropdown-toggle" data-bs-toggle="dropdown" aria-expanded="false">
@ -152,6 +154,7 @@
{% include "build-status/alerts.jinja2" %} {% include "build-status/alerts.jinja2" %}
{% include "build-status/dashboard.jinja2" %}
{% include "build-status/package-add-modal.jinja2" %} {% include "build-status/package-add-modal.jinja2" %}
{% include "build-status/package-rebuild-modal.jinja2" %} {% include "build-status/package-rebuild-modal.jinja2" %}
{% include "build-status/key-import-modal.jinja2" %} {% include "build-status/key-import-modal.jinja2" %}

View File

@ -0,0 +1,76 @@
<div id="dashboard-modal" tabindex="-1" role="dialog" class="modal fade">
<div class="modal-dialog modal-xl" role="document">
<div class="modal-content">
<div id="dashboard-modal-header" class="modal-header">
<h4 class="modal-title">System health</h4>
<button type="button" class="btn-close" data-bs-dismiss="modal" aria-label="close"></button>
</div>
<div class="modal-body">
<div class="form-group row mt-2">
<div class="col-4 col-lg-2" style="text-align: right">Repository name</div>
<div id="dashboard-name" class="col-8 col-lg-3"></div>
<div class="col-4 col-lg-2" style="text-align: right">Repository architecture</div>
<div id="dashboard-architecture" class="col-8 col-lg-3"></div>
</div>
<div class="form-group row mt-2">
<div class="col-4 col-lg-2" style="text-align: right">Current status</div>
<div id="dashboard-status" class="col-8 col-lg-3"></div>
<div class="col-4 col-lg-2" style="text-align: right">Updated at</div>
<div id="dashboard-status-timestamp" class="col-8 col-lg-3"></div>
</div>
<div id="dashboard-canvas" class="form-group row mt-2">
<div class="col-8 col-lg-6">
<canvas id="dashboard-packages-count-chart"></canvas>
</div>
<div class="col-8 col-lg-6">
<canvas id="dashboard-packages-statuses-chart"></canvas>
</div>
</div>
</div>
<div class="modal-footer">
<button type="button" class="btn btn-primary" data-bs-dismiss="modal"><i class="bi bi-x"></i><span class="d-none d-sm-inline"> close</span></button>
</div>
</div>
</div>
</div>
<script>
const dashboardModal = document.getElementById("dashboard-modal");
const dashboardModalHeader = document.getElementById("dashboard-modal-header");
const dashboardName = document.getElementById("dashboard-name");
const dashboardArchitecture = document.getElementById("dashboard-architecture");
const dashboardStatus = document.getElementById("dashboard-status");
const dashboardStatusTimestamp = document.getElementById("dashboard-status-timestamp");
const dashboardCanvas = document.getElementById("dashboard-canvas");
const dashboardPackagesStatusesChartCanvas = document.getElementById("dashboard-packages-statuses-chart");
let dashboardPackagesStatusesChart = null;
const dashboardPackagesCountChartCanvas = document.getElementById("dashboard-packages-count-chart");
let dashboardPackagesCountChart = null;
ready(_ => {
dashboardPackagesStatusesChart = new Chart(dashboardPackagesStatusesChartCanvas, {
type: "pie",
data: {},
options: {
responsive: true,
},
});
dashboardPackagesCountChart = new Chart(dashboardPackagesCountChartCanvas, {
type: "bar",
data: {},
options: {
maintainAspectRatio: false,
responsive: true,
scales: {
x: {
stacked: true,
},
},
},
});
});
</script>

View File

@ -59,7 +59,17 @@
</nav> </nav>
<div class="tab-content" id="nav-tabContent"> <div class="tab-content" id="nav-tabContent">
<div id="package-info-logs" class="tab-pane fade show active" role="tabpanel" aria-labelledby="package-info-logs-button" tabindex="0"> <div id="package-info-logs" class="tab-pane fade show active" role="tabpanel" aria-labelledby="package-info-logs-button" tabindex="0">
<pre class="language-console"><code id="package-info-logs-input" class="pre-scrollable language-console"></code><button id="package-info-logs-copy-button" type="button" class="btn language-console" onclick="copyLogs()"><i class="bi bi-clipboard"></i> copy</button></pre> <div class="row">
<div class="col-1 dropend">
<button id="package-info-logs-dropdown" class="btn dropdown-toggle" type="button" data-bs-toggle="dropdown" aria-expanded="false">
<i class="bi bi-list"></i>
</button>
<nav id="package-info-logs-versions" class="dropdown-menu" aria-labelledby="package-info-logs-dropdown"></nav>
</div>
<div class="col-11">
<pre class="language-console"><code id="package-info-logs-input" class="pre-scrollable language-console"></code><button id="package-info-logs-copy-button" type="button" class="btn language-console" onclick="copyLogs()"><i class="bi bi-clipboard"></i> copy</button></pre>
</div>
</div>
</div> </div>
<div id="package-info-changes" class="tab-pane fade" role="tabpanel" aria-labelledby="package-info-changes-button" tabindex="0"> <div id="package-info-changes" class="tab-pane fade" role="tabpanel" aria-labelledby="package-info-changes-button" tabindex="0">
<pre class="language-diff"><code id="package-info-changes-input" class="pre-scrollable language-diff"></code><button id="package-info-changes-copy-button" type="button" class="btn language-diff" onclick="copyChanges()"><i class="bi bi-clipboard"></i> copy</button></pre> <pre class="language-diff"><code id="package-info-changes-input" class="pre-scrollable language-diff"></code><button id="package-info-changes-copy-button" type="button" class="btn language-diff" onclick="copyChanges()"><i class="bi bi-clipboard"></i> copy</button></pre>
@ -100,6 +110,7 @@
const packageInfoModalHeader = document.getElementById("package-info-modal-header"); const packageInfoModalHeader = document.getElementById("package-info-modal-header");
const packageInfo = document.getElementById("package-info"); const packageInfo = document.getElementById("package-info");
const packageInfoLogsVersions = document.getElementById("package-info-logs-versions");
const packageInfoLogsInput = document.getElementById("package-info-logs-input"); const packageInfoLogsInput = document.getElementById("package-info-logs-input");
const packageInfoLogsCopyButton = document.getElementById("package-info-logs-copy-button"); const packageInfoLogsCopyButton = document.getElementById("package-info-logs-copy-button");
@ -285,25 +296,51 @@
convert: response => response.json(), convert: response => response.json(),
}, },
data => { data => {
const logs = data.map(log_record => { const selectors = Object
return `[${new Date(1000 * log_record.created).toISOString()}] ${log_record.message}`; .values(
}); data.reduce((acc, log_record) => {
packageInfoLogsInput.textContent = logs.join("\n"); const id = `${log_record.version}-${log_record.process_id}`;
highlight(packageInfoLogsInput); if (acc[id])
acc[id].created = Math.min(log_record.created, acc[id].created);
else
acc[id] = log_record;
return acc;
}, {})
)
.sort(({created: left}, {created: right}) =>
right - left
)
.map(version => {
const link = document.createElement("a");
link.classList.add("dropdown-item");
link.textContent = new Date(1000 * version.created).toISOStringShort();
link.href = "#";
link.onclick = _ => {
const logs = data
.filter(log_record => log_record.version === version.version && log_record.process_id === version.process_id)
.map(log_record => `[${new Date(1000 * log_record.created).toISOString()}] ${log_record.message}`);
packageInfoLogsInput.textContent = logs.join("\n");
highlight(packageInfoLogsInput);
Array.from(packageInfoLogsVersions.children).forEach(el => el.classList.remove("active"));
link.classList.add("active");
return false;
};
return link;
});
packageInfoLogsVersions.replaceChildren(...selectors);
selectors.find(Boolean)?.click();
}, },
onFailure, onFailure,
); );
} }
function loadPackage(packageBase, onFailure) { function loadPackage(packageBase, onFailure) {
const headerClass = status => {
if (status === "pending") return ["bg-warning"];
if (status === "building") return ["bg-warning"];
if (status === "failed") return ["bg-danger", "text-white"];
if (status === "success") return ["bg-success", "text-white"];
return ["bg-secondary", "text-white"];
};
makeRequest( makeRequest(
`/api/v1/packages/${packageBase}`, `/api/v1/packages/${packageBase}`,
{ {

View File

@ -7,7 +7,7 @@
// so far bootstrap-table only operates with jquery elements // so far bootstrap-table only operates with jquery elements
const table = $(document.getElementById("packages")); const table = $(document.getElementById("packages"));
const statusBadge = document.getElementById("badge-status"); const dashboardButton = document.getElementById("dashboard-button");
const versionBadge = document.getElementById("badge-version"); const versionBadge = document.getElementById("badge-version");
function doPackageAction(uri, packages, repository, successText, failureText, data) { function doPackageAction(uri, packages, repository, successText, failureText, data) {
@ -141,14 +141,62 @@
data => { data => {
versionBadge.innerHTML = `<i class="bi bi-github"></i> ahriman ${safe(data.version)}`; versionBadge.innerHTML = `<i class="bi bi-github"></i> ahriman ${safe(data.version)}`;
statusBadge.classList.remove(...statusBadge.classList); dashboardButton.classList.remove(...dashboardButton.classList);
statusBadge.classList.add("btn"); dashboardButton.classList.add("btn");
statusBadge.classList.add(badgeClass(data.status.status)); dashboardButton.classList.add(badgeClass(data.status.status));
const popover = bootstrap.Popover.getOrCreateInstance(statusBadge); dashboardModalHeader.classList.remove(...dashboardModalHeader.classList);
popover.dispose(); dashboardModalHeader.classList.add("modal-header");
statusBadge.dataset.bsContent = `${data.status.status} at ${new Date(1000 * data.status.timestamp).toISOStringShort()}`; headerClass(data.status.status).forEach(clz => dashboardModalHeader.classList.add(clz));
bootstrap.Popover.getOrCreateInstance(statusBadge);
dashboardName.textContent = data.repository;
dashboardArchitecture.textContent = data.architecture;
dashboardStatus.textContent = data.status.status;
dashboardStatusTimestamp.textContent = new Date(1000 * data.status.timestamp).toISOStringShort();
if (dashboardPackagesStatusesChart) {
const labels = [
"unknown",
"pending",
"building",
"failed",
"success",
];
dashboardPackagesStatusesChart.config.data = {
labels: labels,
datasets: [{
label: "packages in status",
data: labels.map(label => data.packages[label]),
backgroundColor: [
"rgb(55, 58, 60)",
"rgb(255, 117, 24)",
"rgb(255, 117, 24)",
"rgb(255, 0, 57)",
"rgb(63, 182, 24)", // copy-paste from current style
],
}],
};
dashboardPackagesStatusesChart.update();
}
if (dashboardPackagesCountChart) {
dashboardPackagesCountChart.config.data = {
labels: ["packages"],
datasets: [
{
label: "archives",
data: [data.stats.packages],
},
{
label: "bases",
data: [data.stats.bases],
},
],
};
dashboardPackagesCountChart.update();
}
dashboardCanvas.hidden = data.status.total > 0;
}, },
); );
} }
@ -227,7 +275,6 @@
}); });
}); });
bootstrap.Popover.getOrCreateInstance(statusBadge);
selectRepository(); selectRepository();
}); });
</script> </script>

View File

@ -58,6 +58,14 @@
return value.includes(dataList[index].toLowerCase()); return value.includes(dataList[index].toLowerCase());
} }
function headerClass(status) {
if (status === "pending") return ["bg-warning"];
if (status === "building") return ["bg-warning"];
if (status === "failed") return ["bg-danger", "text-white"];
if (status === "success") return ["bg-success", "text-white"];
return ["bg-secondary", "text-white"];
}
function listToTable(data) { function listToTable(data) {
return Array.from(new Set(data)) return Array.from(new Set(data))
.sort() .sort()

View File

@ -635,6 +635,7 @@ _set_new_action() {
# ${!x} -> ${hello} -> "world" # ${!x} -> ${hello} -> "world"
_shtab_ahriman() { _shtab_ahriman() {
local completing_word="${COMP_WORDS[COMP_CWORD]}" local completing_word="${COMP_WORDS[COMP_CWORD]}"
local previous_word="${COMP_WORDS[COMP_CWORD-1]}"
local completed_positional_actions local completed_positional_actions
local current_action local current_action
local current_action_args_start_index local current_action_args_start_index
@ -691,6 +692,10 @@ _shtab_ahriman() {
if [[ $pos_only = 0 && "${completing_word}" == -* ]]; then if [[ $pos_only = 0 && "${completing_word}" == -* ]]; then
# optional argument started: use option strings # optional argument started: use option strings
COMPREPLY=( $(compgen -W "${current_option_strings[*]}" -- "${completing_word}") ) COMPREPLY=( $(compgen -W "${current_option_strings[*]}" -- "${completing_word}") )
elif [[ "${previous_word}" == ">" || "${previous_word}" == ">>" ||
"${previous_word}" =~ ^[12]">" || "${previous_word}" =~ ^[12]">>" ]]; then
# handle redirection operators
COMPREPLY=( $(compgen -f -- "${completing_word}") )
else else
# use choices & compgen # use choices & compgen
local IFS=$'\n' # items may contain spaces, so delimit using newline local IFS=$'\n' # items may contain spaces, so delimit using newline

View File

@ -1,4 +1,4 @@
.TH AHRIMAN "1" "2025\-01\-05" "ahriman" "Generated Python Manual" .TH AHRIMAN "1" "2025\-06\-16" "ahriman" "Generated Python Manual"
.SH NAME .SH NAME
ahriman ahriman
.SH SYNOPSIS .SH SYNOPSIS

View File

@ -25,15 +25,68 @@ dependencies = [
dynamic = ["version"] dynamic = ["version"]
[project.optional-dependencies]
journald = [
"systemd-python",
]
# FIXME technically this dependency is required, but in some cases we do not have access to
# the libalpm which is required in order to install the package. Thus in case if we do not
# really need to run the application we can move it to "optional" dependencies
pacman = [
"pyalpm",
]
reports = [
"Jinja2",
]
s3 = [
"boto3",
]
shell = [
"IPython"
]
stats = [
"matplotlib",
]
unixsocket = [
"requests-unixsocket2", # required by unix socket support
]
validator = [
"cerberus",
]
web = [
"aiohttp",
"aiohttp_cors",
"aiohttp_jinja2",
]
web_api-docs = [
"ahriman[web]",
"aiohttp-apispec",
"setuptools", # required by aiohttp-apispec
]
web_auth = [
"ahriman[web]",
"aiohttp_session",
"aiohttp_security",
"cryptography",
]
web_metrics = [
"ahriman[web]",
"aiohttp-openmetrics",
]
web_oauth2 = [
"ahriman[web_auth]",
"aioauth-client",
]
[project.scripts]
ahriman = "ahriman.application.ahriman:run"
[project.urls] [project.urls]
Documentation = "https://ahriman.readthedocs.io/" Documentation = "https://ahriman.readthedocs.io/"
Repository = "https://github.com/arcan1s/ahriman" Repository = "https://github.com/arcan1s/ahriman"
Changelog = "https://github.com/arcan1s/ahriman/releases" Changelog = "https://github.com/arcan1s/ahriman/releases"
[project.scripts] [dependency-groups]
ahriman = "ahriman.application.ahriman:run"
[project.optional-dependencies]
check = [ check = [
"autopep8", "autopep8",
"bandit", "bandit",
@ -47,24 +100,6 @@ docs = [
"shtab", "shtab",
"sphinx-argparse", "sphinx-argparse",
"sphinx-rtd-theme>=1.1.1", # https://stackoverflow.com/a/74355734 "sphinx-rtd-theme>=1.1.1", # https://stackoverflow.com/a/74355734
]
journald = [
"systemd-python",
]
# FIXME technically this dependency is required, but in some cases we do not have access to
# the libalpm which is required in order to install the package. Thus in case if we do not
# really need to run the application we can move it to "optional" dependencies
pacman = [
"pyalpm",
]
s3 = [
"boto3",
]
shell = [
"IPython"
]
stats = [
"matplotlib",
] ]
tests = [ tests = [
"pytest", "pytest",
@ -75,22 +110,6 @@ tests = [
"pytest-resource-path", "pytest-resource-path",
"pytest-spec", "pytest-spec",
] ]
validator = [
"cerberus",
]
web = [
"Jinja2",
"aioauth-client",
"aiohttp",
"aiohttp-apispec",
"aiohttp_cors",
"aiohttp_jinja2",
"aiohttp_session",
"aiohttp_security",
"cryptography",
"requests-unixsocket2", # required by unix socket support
"setuptools", # required by aiohttp-apispec
]
[tool.flit.sdist] [tool.flit.sdist]
include = [ include = [

View File

@ -17,4 +17,4 @@
# You should have received a copy of the GNU General Public License # You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>. # along with this program. If not, see <http://www.gnu.org/licenses/>.
# #
__version__ = "2.17.1" __version__ = "2.18.3"

View File

@ -117,7 +117,7 @@ class Application(ApplicationPackages, ApplicationRepository):
Args: Args:
packages(list[Package]): list of source packages of which dependencies have to be processed packages(list[Package]): list of source packages of which dependencies have to be processed
process_dependencies(bool): if no set, dependencies will not be processed process_dependencies(bool): if set to ``False``, dependencies will not be processed
Returns: Returns:
list[Package]: updated packages list. Packager for dependencies will be copied from the original package list[Package]: updated packages list. Packager for dependencies will be copied from the original package
@ -130,6 +130,9 @@ class Application(ApplicationPackages, ApplicationRepository):
>>> packages = application.with_dependencies(packages, process_dependencies=True) >>> packages = application.with_dependencies(packages, process_dependencies=True)
>>> application.print_updates(packages, log_fn=print) >>> application.print_updates(packages, log_fn=print)
""" """
if not process_dependencies or not packages:
return packages
def missing_dependencies(source: Iterable[Package]) -> dict[str, str | None]: def missing_dependencies(source: Iterable[Package]) -> dict[str, str | None]:
# append list of known packages with packages which are in current sources # append list of known packages with packages which are in current sources
satisfied_packages = known_packages | { satisfied_packages = known_packages | {
@ -145,22 +148,29 @@ class Application(ApplicationPackages, ApplicationRepository):
if dependency not in satisfied_packages if dependency not in satisfied_packages
} }
if not process_dependencies or not packages: def new_packages(root: Package) -> dict[str, Package]:
return packages portion = {root.base: root}
while missing := missing_dependencies(portion.values()):
for package_name, packager in missing.items():
if (source_dir := self.repository.paths.cache_for(package_name)).is_dir():
# there is local cache, load package from it
leaf = Package.from_build(source_dir, self.repository.architecture, packager)
else:
leaf = Package.from_aur(package_name, packager)
portion[leaf.base] = leaf
# register package in the database
self.repository.reporter.set_unknown(leaf)
return portion
known_packages = self._known_packages() known_packages = self._known_packages()
with_dependencies = {package.base: package for package in packages} with_dependencies: dict[str, Package] = {}
for package in packages:
while missing := missing_dependencies(with_dependencies.values()): with self.in_package_context(package.base, package.version): # use the same context for the logger
for package_name, username in missing.items(): try:
if (source_dir := self.repository.paths.cache_for(package_name)).is_dir(): with_dependencies |= new_packages(package)
# there is local cache, load package from it except Exception:
package = Package.from_build(source_dir, self.repository.architecture, username) self.logger.exception("could not process dependencies of %s, skip the package", package.base)
else:
package = Package.from_aur(package_name, username)
with_dependencies[package.base] = package
# register package in the database
self.repository.reporter.set_unknown(package)
return list(with_dependencies.values()) return list(with_dependencies.values())

View File

@ -22,7 +22,7 @@ import logging
from collections.abc import Callable, Iterable from collections.abc import Callable, Iterable
from multiprocessing import Pool from multiprocessing import Pool
from typing import TypeVar from typing import ClassVar, TypeVar
from ahriman.application.lock import Lock from ahriman.application.lock import Lock
from ahriman.core.configuration import Configuration from ahriman.core.configuration import Configuration
@ -53,13 +53,13 @@ class Handler:
Wrapper for all command line actions, though each derived class implements :func:`run()` method, it usually Wrapper for all command line actions, though each derived class implements :func:`run()` method, it usually
must not be called directly. The recommended way is to call :func:`execute()` class method, e.g.:: must not be called directly. The recommended way is to call :func:`execute()` class method, e.g.::
>>> from ahriman.application.handlers import Add >>> from ahriman.application.handlers.add import Add
>>> >>>
>>> Add.execute(args) >>> Add.execute(args)
""" """
ALLOW_MULTI_ARCHITECTURE_RUN = True ALLOW_MULTI_ARCHITECTURE_RUN: ClassVar[bool] = True
arguments: list[Callable[[SubParserAction], argparse.ArgumentParser]] arguments: ClassVar[list[Callable[[SubParserAction], argparse.ArgumentParser]]]
@classmethod @classmethod
def call(cls, args: argparse.Namespace, repository_id: RepositoryId) -> bool: def call(cls, args: argparse.Namespace, repository_id: RepositoryId) -> bool:

View File

@ -21,6 +21,7 @@ import argparse
from collections.abc import Callable, Iterable from collections.abc import Callable, Iterable
from dataclasses import fields from dataclasses import fields
from typing import ClassVar
from ahriman.application.handlers.handler import Handler, SubParserAction from ahriman.application.handlers.handler import Handler, SubParserAction
from ahriman.core.alpm.remote import AUR, Official from ahriman.core.alpm.remote import AUR, Official
@ -40,7 +41,7 @@ class Search(Handler):
""" """
ALLOW_MULTI_ARCHITECTURE_RUN = False # system-wide action ALLOW_MULTI_ARCHITECTURE_RUN = False # system-wide action
SORT_FIELDS = { SORT_FIELDS: ClassVar[set[str]] = {
field.name field.name
for field in fields(AURPackage) for field in fields(AURPackage)
if field.default_factory is not list if field.default_factory is not list

View File

@ -21,6 +21,7 @@ import argparse
from pathlib import Path from pathlib import Path
from pwd import getpwuid from pwd import getpwuid
from typing import ClassVar
from urllib.parse import quote_plus as url_encode from urllib.parse import quote_plus as url_encode
from ahriman.application.application import Application from ahriman.application.application import Application
@ -46,9 +47,9 @@ class Setup(Handler):
ALLOW_MULTI_ARCHITECTURE_RUN = False # conflicting io ALLOW_MULTI_ARCHITECTURE_RUN = False # conflicting io
ARCHBUILD_COMMAND_PATH = Path("/") / "usr" / "bin" / "archbuild" ARCHBUILD_COMMAND_PATH: ClassVar[Path] = Path("/") / "usr" / "bin" / "archbuild"
MIRRORLIST_PATH = Path("/") / "etc" / "pacman.d" / "mirrorlist" MIRRORLIST_PATH: ClassVar[Path] = Path("/") / "etc" / "pacman.d" / "mirrorlist"
SUDOERS_DIR_PATH = Path("/") / "etc" / "sudoers.d" SUDOERS_DIR_PATH: ClassVar[Path] = Path("/") / "etc" / "sudoers.d"
@classmethod @classmethod
def run(cls, args: argparse.Namespace, repository_id: RepositoryId, configuration: Configuration, *, def run(cls, args: argparse.Namespace, repository_id: RepositoryId, configuration: Configuration, *,

View File

@ -27,7 +27,7 @@ from pathlib import Path
from ahriman.application.application import Application from ahriman.application.application import Application
from ahriman.application.handlers.handler import Handler, SubParserAction from ahriman.application.handlers.handler import Handler, SubParserAction
from ahriman.core.configuration import Configuration from ahriman.core.configuration import Configuration
from ahriman.core.formatters import EventStatsPrinter, PackageStatsPrinter from ahriman.core.formatters import EventStatsPrinter, PackageStatsPrinter, RepositoryStatsPrinter
from ahriman.core.utils import enum_values, pretty_datetime from ahriman.core.utils import enum_values, pretty_datetime
from ahriman.models.event import Event, EventType from ahriman.models.event import Event, EventType
from ahriman.models.repository_id import RepositoryId from ahriman.models.repository_id import RepositoryId
@ -64,6 +64,7 @@ class Statistics(Handler):
match args.package: match args.package:
case None: case None:
RepositoryStatsPrinter(repository_id, application.reporter.statistics())(verbose=True)
Statistics.stats_per_package(args.event, events, args.chart) Statistics.stats_per_package(args.event, events, args.chart)
case _: case _:
Statistics.stats_for_package(args.event, events, args.chart) Statistics.stats_for_package(args.event, events, args.chart)

View File

@ -23,6 +23,7 @@ import sys
from collections.abc import Generator from collections.abc import Generator
from importlib import metadata from importlib import metadata
from typing import ClassVar
from ahriman import __version__ from ahriman import __version__
from ahriman.application.handlers.handler import Handler, SubParserAction from ahriman.application.handlers.handler import Handler, SubParserAction
@ -36,11 +37,11 @@ class Versions(Handler):
version handler version handler
Attributes: Attributes:
PEP423_PACKAGE_NAME(str): (class attribute) special regex for valid PEP423 package name PEP423_PACKAGE_NAME(re.Pattern[str]): (class attribute) special regex for valid PEP423 package name
""" """
ALLOW_MULTI_ARCHITECTURE_RUN = False # system-wide action ALLOW_MULTI_ARCHITECTURE_RUN = False # system-wide action
PEP423_PACKAGE_NAME = re.compile(r"^[A-Za-z0-9._-]+") PEP423_PACKAGE_NAME: ClassVar[re.Pattern[str]] = re.compile(r"^[A-Za-z0-9._-]+")
@classmethod @classmethod
def run(cls, args: argparse.Namespace, repository_id: RepositoryId, configuration: Configuration, *, def run(cls, args: argparse.Namespace, repository_id: RepositoryId, configuration: Configuration, *,

View File

@ -23,6 +23,7 @@ import shutil
from email.utils import parsedate_to_datetime from email.utils import parsedate_to_datetime
from pathlib import Path from pathlib import Path
from pyalpm import DB # type: ignore[import-not-found] from pyalpm import DB # type: ignore[import-not-found]
from typing import ClassVar
from urllib.parse import urlparse from urllib.parse import urlparse
from ahriman.core.configuration import Configuration from ahriman.core.configuration import Configuration
@ -41,7 +42,7 @@ class PacmanDatabase(SyncHttpClient):
sync_files_database(bool): sync files database sync_files_database(bool): sync files database
""" """
LAST_MODIFIED_HEADER = "Last-Modified" LAST_MODIFIED_HEADER: ClassVar[str] = "Last-Modified"
def __init__(self, database: DB, configuration: Configuration) -> None: def __init__(self, database: DB, configuration: Configuration) -> None:
""" """

View File

@ -34,14 +34,14 @@ class PkgbuildToken(StrEnum):
well-known tokens dictionary well-known tokens dictionary
Attributes: Attributes:
ArrayEnds(PkgbuildToken): (class attribute) array ends token ArrayEnds(PkgbuildToken): array ends token
ArrayStarts(PkgbuildToken): (class attribute) array starts token ArrayStarts(PkgbuildToken): array starts token
Comma(PkgbuildToken): (class attribute) comma token Comma(PkgbuildToken): comma token
Comment(PkgbuildToken): (class attribute) comment token Comment(PkgbuildToken): comment token
FunctionDeclaration(PkgbuildToken): (class attribute) function declaration token FunctionDeclaration(PkgbuildToken): function declaration token
FunctionEnds(PkgbuildToken): (class attribute) function ends token FunctionEnds(PkgbuildToken): function ends token
FunctionStarts(PkgbuildToken): (class attribute) function starts token FunctionStarts(PkgbuildToken): function starts token
NewLine(PkgbuildToken): (class attribute) new line token NewLine(PkgbuildToken): new line token
""" """
ArrayStarts = "(" ArrayStarts = "("

View File

@ -17,7 +17,7 @@
# You should have received a copy of the GNU General Public License # You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>. # along with this program. If not, see <http://www.gnu.org/licenses/>.
# #
from typing import Any from typing import Any, ClassVar
from ahriman.core.alpm.pacman import Pacman from ahriman.core.alpm.pacman import Pacman
from ahriman.core.alpm.remote.remote import Remote from ahriman.core.alpm.remote.remote import Remote
@ -35,9 +35,9 @@ class AUR(Remote):
DEFAULT_RPC_VERSION(str): (class attribute) default AUR RPC version DEFAULT_RPC_VERSION(str): (class attribute) default AUR RPC version
""" """
DEFAULT_AUR_URL = "https://aur.archlinux.org" DEFAULT_AUR_URL: ClassVar[str] = "https://aur.archlinux.org"
DEFAULT_RPC_URL = f"{DEFAULT_AUR_URL}/rpc" DEFAULT_RPC_URL: ClassVar[str] = f"{DEFAULT_AUR_URL}/rpc"
DEFAULT_RPC_VERSION = "5" DEFAULT_RPC_VERSION: ClassVar[str] = "5"
@classmethod @classmethod
def remote_git_url(cls, package_base: str, repository: str) -> str: def remote_git_url(cls, package_base: str, repository: str) -> str:

View File

@ -17,7 +17,7 @@
# You should have received a copy of the GNU General Public License # You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>. # along with this program. If not, see <http://www.gnu.org/licenses/>.
# #
from typing import Any from typing import Any, ClassVar
from ahriman.core.alpm.pacman import Pacman from ahriman.core.alpm.pacman import Pacman
from ahriman.core.alpm.remote.remote import Remote from ahriman.core.alpm.remote.remote import Remote
@ -36,10 +36,10 @@ class Official(Remote):
DEFAULT_RPC_URL(str): (class attribute) default archlinux repositories RPC url DEFAULT_RPC_URL(str): (class attribute) default archlinux repositories RPC url
""" """
DEFAULT_ARCHLINUX_GIT_URL = "https://gitlab.archlinux.org" DEFAULT_ARCHLINUX_GIT_URL: ClassVar[str] = "https://gitlab.archlinux.org"
DEFAULT_ARCHLINUX_URL = "https://archlinux.org" DEFAULT_ARCHLINUX_URL: ClassVar[str] = "https://archlinux.org"
DEFAULT_SEARCH_REPOSITORIES = ["Core", "Extra", "Multilib"] DEFAULT_SEARCH_REPOSITORIES: ClassVar[list[str]] = ["Core", "Extra", "Multilib"]
DEFAULT_RPC_URL = "https://archlinux.org/packages/search/json" DEFAULT_RPC_URL: ClassVar[str] = "https://archlinux.org/packages/search/json"
@classmethod @classmethod
def remote_git_url(cls, package_base: str, repository: str) -> str: def remote_git_url(cls, package_base: str, repository: str) -> str:

View File

@ -35,7 +35,7 @@ class Remote(SyncHttpClient):
>>> package = AUR.info("ahriman") >>> package = AUR.info("ahriman")
>>> search_result = Official.multisearch("pacman", "manager", pacman=pacman) >>> search_result = Official.multisearch("pacman", "manager", pacman=pacman)
Differnece between :func:`search()` and :func:`multisearch()` is that :func:`search()` passes all arguments to Difference between :func:`search()` and :func:`multisearch()` is that :func:`search()` passes all arguments to
underlying wrapper directly, whereas :func:`multisearch()` splits search one by one and finds intersection underlying wrapper directly, whereas :func:`multisearch()` splits search one by one and finds intersection
between search results. between search results.
""" """

View File

@ -21,6 +21,7 @@ import shutil
from collections.abc import Generator from collections.abc import Generator
from pathlib import Path from pathlib import Path
from typing import ClassVar
from ahriman.core.exceptions import CalledProcessError from ahriman.core.exceptions import CalledProcessError
from ahriman.core.log import LazyLogging from ahriman.core.log import LazyLogging
@ -42,9 +43,9 @@ class Sources(LazyLogging):
GITCONFIG(dict[str, str]): (class attribute) git config options to suppress annoying hints GITCONFIG(dict[str, str]): (class attribute) git config options to suppress annoying hints
""" """
DEFAULT_BRANCH = "master" # default fallback branch DEFAULT_BRANCH: ClassVar[str] = "master" # default fallback branch
DEFAULT_COMMIT_AUTHOR = ("ahriman", "ahriman@localhost") DEFAULT_COMMIT_AUTHOR: ClassVar[tuple[str, str]] = ("ahriman", "ahriman@localhost")
GITCONFIG = { GITCONFIG: ClassVar[dict[str, str]] = {
"init.defaultBranch": DEFAULT_BRANCH, "init.defaultBranch": DEFAULT_BRANCH,
} }

View File

@ -149,8 +149,11 @@ class Task(LazyLogging):
str | None: current commit sha if available str | None: current commit sha if available
""" """
last_commit_sha = Sources.load(sources_dir, self.package, patches, self.paths) last_commit_sha = Sources.load(sources_dir, self.package, patches, self.paths)
if local_version is None: if self.package.is_vcs: # if package is VCS, then make sure to update PKGBUILD to the latest version
return last_commit_sha # there is no local package or pkgrel increment is disabled self.build(sources_dir, dry_run=True)
if local_version is None: # there is no local package or pkgrel increment is disabled
return last_commit_sha
# load fresh package # load fresh package
loaded_package = Package.from_build(sources_dir, self.architecture, None) loaded_package = Package.from_build(sources_dir, self.architecture, None)

View File

@ -22,7 +22,7 @@ import shlex
import sys import sys
from pathlib import Path from pathlib import Path
from typing import Any, Self from typing import Any, ClassVar, Self
from ahriman.core.configuration.configuration_multi_dict import ConfigurationMultiDict from ahriman.core.configuration.configuration_multi_dict import ConfigurationMultiDict
from ahriman.core.configuration.shell_interpolator import ShellInterpolator from ahriman.core.configuration.shell_interpolator import ShellInterpolator
@ -65,8 +65,8 @@ class Configuration(configparser.RawConfigParser):
""" """
_LEGACY_ARCHITECTURE_SPECIFIC_SECTIONS = ["web"] _LEGACY_ARCHITECTURE_SPECIFIC_SECTIONS = ["web"]
ARCHITECTURE_SPECIFIC_SECTIONS = ["alpm", "build", "sign"] ARCHITECTURE_SPECIFIC_SECTIONS: ClassVar[list[str]] = ["alpm", "build", "sign"]
SYSTEM_CONFIGURATION_PATH = Path(sys.prefix) / "share" / "ahriman" / "settings" / "ahriman.ini" SYSTEM_CONFIGURATION_PATH: ClassVar[Path] = Path(sys.prefix) / "share" / "ahriman" / "settings" / "ahriman.ini"
def __init__(self, allow_no_value: bool = False, allow_multi_key: bool = True) -> None: def __init__(self, allow_no_value: bool = False, allow_multi_key: bool = True) -> None:
""" """
@ -210,6 +210,17 @@ class Configuration(configparser.RawConfigParser):
raise InitializeError("Configuration path and/or repository id are not set") raise InitializeError("Configuration path and/or repository id are not set")
return self.path, self.repository_id return self.path, self.repository_id
def copy_from(self, configuration: Self) -> None:
"""
copy values from another instance overriding existing
Args:
configuration(Self): configuration instance to merge from
"""
for section in configuration.sections():
for key, value in configuration.items(section):
self.set_option(section, key, value)
def dump(self) -> dict[str, dict[str, str]]: def dump(self) -> dict[str, dict[str, str]]:
""" """
dump configuration to dictionary dump configuration to dictionary
@ -220,6 +231,7 @@ class Configuration(configparser.RawConfigParser):
return { return {
section: dict(self.items(section)) section: dict(self.items(section))
for section in self.sections() for section in self.sections()
if self[section]
} }
# pylint and mypy are too stupid to find these methods # pylint and mypy are too stupid to find these methods

View File

@ -57,7 +57,7 @@ class ConfigurationMultiDict(dict[str, Any]):
OptionError: if the key already exists in the dictionary, but not a single value list or a string OptionError: if the key already exists in the dictionary, but not a single value list or a string
""" """
match self.get(key): match self.get(key):
case [current_value] | str(current_value): # type: ignore[misc] case [current_value] | str(current_value):
value = f"{current_value} {value}" value = f"{current_value} {value}"
case None: case None:
pass pass

View File

@ -45,6 +45,11 @@ CONFIGURATION_SCHEMA: ConfigurationSchema = {
"path_exists": True, "path_exists": True,
"path_type": "dir", "path_type": "dir",
}, },
"keep_last_logs": {
"type": "integer",
"coerce": "integer",
"min": 0,
},
"logging": { "logging": {
"type": "path", "type": "path",
"coerce": "absolute_path", "coerce": "absolute_path",
@ -52,10 +57,6 @@ CONFIGURATION_SCHEMA: ConfigurationSchema = {
"path_exists": True, "path_exists": True,
"path_type": "file", "path_type": "file",
}, },
"suppress_http_log_errors": {
"type": "boolean",
"coerce": "boolean",
}
}, },
}, },
"alpm": { "alpm": {
@ -342,10 +343,6 @@ CONFIGURATION_SCHEMA: ConfigurationSchema = {
"coerce": "integer", "coerce": "integer",
"min": 0, "min": 0,
}, },
"password": {
"type": "string",
"empty": False,
},
"port": { "port": {
"type": "integer", "type": "integer",
"coerce": "integer", "coerce": "integer",
@ -374,11 +371,6 @@ CONFIGURATION_SCHEMA: ConfigurationSchema = {
}, },
"empty": False, "empty": False,
}, },
"timeout": {
"type": "integer",
"coerce": "integer",
"min": 0,
},
"unix_socket": { "unix_socket": {
"type": "path", "type": "path",
"coerce": "absolute_path", "coerce": "absolute_path",
@ -387,10 +379,6 @@ CONFIGURATION_SCHEMA: ConfigurationSchema = {
"type": "boolean", "type": "boolean",
"coerce": "boolean", "coerce": "boolean",
}, },
"username": {
"type": "string",
"empty": False,
},
"wait_timeout": { "wait_timeout": {
"type": "integer", "type": "integer",
"coerce": "integer", "coerce": "integer",

View File

@ -23,6 +23,7 @@ import sys
from collections.abc import Generator, Mapping, MutableMapping from collections.abc import Generator, Mapping, MutableMapping
from string import Template from string import Template
from typing import Any, ClassVar
from ahriman.core.configuration.shell_template import ShellTemplate from ahriman.core.configuration.shell_template import ShellTemplate
@ -32,7 +33,7 @@ class ShellInterpolator(configparser.Interpolation):
custom string interpolator, because we cannot use defaults argument due to config validation custom string interpolator, because we cannot use defaults argument due to config validation
""" """
DATA_LINK_ESCAPE = "\x10" DATA_LINK_ESCAPE: ClassVar[str] = "\x10"
@staticmethod @staticmethod
def _extract_variables(parser: MutableMapping[str, Mapping[str, str]], value: str, def _extract_variables(parser: MutableMapping[str, Mapping[str, str]], value: str,
@ -84,7 +85,7 @@ class ShellInterpolator(configparser.Interpolation):
"prefix": sys.prefix, "prefix": sys.prefix,
} }
def before_get(self, parser: MutableMapping[str, Mapping[str, str]], section: str, option: str, value: str, def before_get(self, parser: MutableMapping[str, Mapping[str, str]], section: Any, option: Any, value: str,
defaults: Mapping[str, str]) -> str: defaults: Mapping[str, str]) -> str:
""" """
interpolate option value interpolate option value
@ -99,8 +100,8 @@ class ShellInterpolator(configparser.Interpolation):
Args: Args:
parser(MutableMapping[str, Mapping[str, str]]): option parser parser(MutableMapping[str, Mapping[str, str]]): option parser
section(str): section name section(Any): section name
option(str): option name option(Any): option name
value(str): source (not-converted) value value(str): source (not-converted) value
defaults(Mapping[str, str]): default values defaults(Mapping[str, str]): default values

View File

@ -28,9 +28,6 @@ class ShellTemplate(Template):
""" """
extension to the default :class:`Template` class, which also adds additional tokens to braced regex and enables extension to the default :class:`Template` class, which also adds additional tokens to braced regex and enables
bash expansion bash expansion
Attributes:
braceidpattern(str): regular expression to match every character except for closing bracket
""" """
braceidpattern = r"(?a:[_a-z0-9][^}]*)" braceidpattern = r"(?a:[_a-z0-9][^}]*)"

View File

@ -62,24 +62,31 @@ class Migrations(LazyLogging):
""" """
return Migrations(connection, configuration).run() return Migrations(connection, configuration).run()
def migration(self, cursor: Cursor, migration: Migration) -> None: def apply_migrations(self, migrations: list[Migration]) -> None:
""" """
perform single migration perform migrations explicitly
Args: Args:
cursor(Cursor): connection cursor migrations(list[Migration]): list of migrations to perform
migration(Migration): single migration to perform
""" """
self.logger.info("applying table migration %s at index %s", migration.name, migration.index) previous_isolation = self.connection.isolation_level
for statement in migration.steps: try:
cursor.execute(statement) self.connection.isolation_level = None
self.logger.info("table migration %s at index %s has been applied", migration.name, migration.index) cursor = self.connection.cursor()
try:
self.logger.info("perform data migration %s at index %s", migration.name, migration.index) cursor.execute("begin exclusive")
migration.migrate_data(self.connection, self.configuration) for migration in migrations:
self.logger.info( self.perform_migration(cursor, migration)
"data migration %s at index %s has been performed", except Exception:
migration.name, migration.index) self.logger.exception("migration failed with exception")
cursor.execute("rollback")
raise
else:
cursor.execute("commit")
finally:
cursor.close()
finally:
self.connection.isolation_level = previous_isolation
def migrations(self) -> list[Migration]: def migrations(self) -> list[Migration]:
""" """
@ -114,6 +121,25 @@ class Migrations(LazyLogging):
return migrations return migrations
def perform_migration(self, cursor: Cursor, migration: Migration) -> None:
"""
perform single migration
Args:
cursor(Cursor): connection cursor
migration(Migration): single migration to perform
"""
self.logger.info("applying table migration %s at index %s", migration.name, migration.index)
for statement in migration.steps:
cursor.execute(statement)
self.logger.info("table migration %s at index %s has been applied", migration.name, migration.index)
self.logger.info("perform data migration %s at index %s", migration.name, migration.index)
migration.migrate_data(self.connection, self.configuration)
self.logger.info(
"data migration %s at index %s has been performed",
migration.name, migration.index)
def run(self) -> MigrationResult: def run(self) -> MigrationResult:
""" """
perform migrations perform migrations
@ -122,6 +148,7 @@ class Migrations(LazyLogging):
MigrationResult: current schema version MigrationResult: current schema version
""" """
migrations = self.migrations() migrations = self.migrations()
current_version = self.user_version() current_version = self.user_version()
expected_version = len(migrations) expected_version = len(migrations)
result = MigrationResult(old_version=current_version, new_version=expected_version) result = MigrationResult(old_version=current_version, new_version=expected_version)
@ -130,25 +157,8 @@ class Migrations(LazyLogging):
self.logger.info("no migrations required") self.logger.info("no migrations required")
return result return result
previous_isolation = self.connection.isolation_level self.apply_migrations(migrations[current_version:])
try: self.connection.execute(f"pragma user_version = {expected_version}") # no support for ? placeholders
self.connection.isolation_level = None
cursor = self.connection.cursor()
try:
cursor.execute("begin exclusive")
for migration in migrations[current_version:]:
self.migration(cursor, migration)
cursor.execute(f"pragma user_version = {expected_version}") # no support for ? placeholders
except Exception:
self.logger.exception("migration failed with exception")
cursor.execute("rollback")
raise
else:
cursor.execute("commit")
finally:
cursor.close()
finally:
self.connection.isolation_level = previous_isolation
self.logger.info("migrations have been performed from version %s to %s", result.old_version, result.new_version) self.logger.info("migrations have been performed from version %s to %s", result.old_version, result.new_version)
return result return result

View File

@ -0,0 +1,30 @@
#
# Copyright (c) 2021-2025 ahriman team.
#
# This file is part of ahriman
# (see https://github.com/arcan1s/ahriman).
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
__all__ = ["steps"]
steps = [
"""
alter table logs add column process_id text not null default ''
""",
"""
alter table logs rename column record to message
""",
]

View File

@ -20,7 +20,7 @@
from sqlite3 import Connection from sqlite3 import Connection
from ahriman.core.database.operations.operations import Operations from ahriman.core.database.operations.operations import Operations
from ahriman.models.log_record_id import LogRecordId from ahriman.models.log_record import LogRecord
from ahriman.models.repository_id import RepositoryId from ahriman.models.repository_id import RepositoryId
@ -30,7 +30,7 @@ class LogsOperations(Operations):
""" """
def logs_get(self, package_base: str, limit: int = -1, offset: int = 0, def logs_get(self, package_base: str, limit: int = -1, offset: int = 0,
repository_id: RepositoryId | None = None) -> list[tuple[float, str]]: repository_id: RepositoryId | None = None) -> list[LogRecord]:
""" """
extract logs for specified package base extract logs for specified package base
@ -41,16 +41,16 @@ class LogsOperations(Operations):
repository_id(RepositoryId, optional): repository unique identifier override (Default value = None) repository_id(RepositoryId, optional): repository unique identifier override (Default value = None)
Return: Return:
list[tuple[float, str]]: sorted package log records and their timestamps list[LogRecord]: sorted package log records
""" """
repository_id = repository_id or self._repository_id repository_id = repository_id or self._repository_id
def run(connection: Connection) -> list[tuple[float, str]]: def run(connection: Connection) -> list[LogRecord]:
return [ return [
(row["created"], row["record"]) LogRecord.from_json(package_base, row)
for row in connection.execute( for row in connection.execute(
""" """
select created, record from ( select created, message, version, process_id from (
select * from logs select * from logs
where package_base = :package_base and repository = :repository where package_base = :package_base and repository = :repository
order by created desc limit :limit offset :offset order by created desc limit :limit offset :offset
@ -66,15 +66,12 @@ class LogsOperations(Operations):
return self.with_connection(run) return self.with_connection(run)
def logs_insert(self, log_record_id: LogRecordId, created: float, record: str, def logs_insert(self, log_record: LogRecord, repository_id: RepositoryId | None = None) -> None:
repository_id: RepositoryId | None = None) -> None:
""" """
write new log record to database write new log record to database
Args: Args:
log_record_id(LogRecordId): current log record id log_record(LogRecord): log record object
created(float): log created timestamp from log record attribute
record(str): log record
repository_id(RepositoryId, optional): repository unique identifier override (Default value = None) repository_id(RepositoryId, optional): repository unique identifier override (Default value = None)
""" """
repository_id = repository_id or self._repository_id repository_id = repository_id or self._repository_id
@ -83,17 +80,14 @@ class LogsOperations(Operations):
connection.execute( connection.execute(
""" """
insert into logs insert into logs
(package_base, version, created, record, repository) (package_base, version, created, message, repository, process_id)
values values
(:package_base, :version, :created, :record, :repository) (:package_base, :version, :created, :message, :repository, :process_id)
""", """,
{ {
"package_base": log_record_id.package_base, "package_base": log_record.log_record_id.package_base,
"version": log_record_id.version,
"created": created,
"record": record,
"repository": repository_id.id, "repository": repository_id.id,
} } | log_record.view()
) )
return self.with_connection(run, commit=True) return self.with_connection(run, commit=True)
@ -125,3 +119,57 @@ class LogsOperations(Operations):
) )
return self.with_connection(run, commit=True) return self.with_connection(run, commit=True)
def logs_rotate(self, keep_last_records: int, repository_id: RepositoryId | None = None) -> None:
"""
rotate logs in storage. This method will remove old logs, keeping only the last N records for each package
Args:
keep_last_records(int): number of last records to keep
repository_id(RepositoryId, optional): repository unique identifier override (Default value = None)
"""
repository_id = repository_id or self._repository_id
def remove_duplicates(connection: Connection) -> None:
connection.execute(
"""
delete from logs
where (package_base, version, repository, process_id) not in (
select package_base, version, repository, process_id from logs
where (package_base, version, repository, created) in (
select package_base, version, repository, max(created) from logs
where repository = :repository
group by package_base, version, repository
)
)
""",
{
"repository": repository_id.id,
}
)
def remove_older(connection: Connection) -> None:
connection.execute(
"""
delete from logs
where (package_base, repository, process_id) in (
select package_base, repository, process_id from (
select package_base, repository, process_id, row_number() over (partition by package_base order by max(created) desc) as rn
from logs
where repository = :repository
group by package_base, repository, process_id
)
where rn > :offset
)
""",
{
"offset": keep_last_records,
"repository": repository_id.id,
}
)
def run(connection: Connection) -> None:
remove_duplicates(connection)
remove_older(connection)
return self.with_connection(run, commit=True)

View File

@ -17,6 +17,7 @@
# You should have received a copy of the GNU General Public License # You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>. # along with this program. If not, see <http://www.gnu.org/licenses/>.
# #
import contextlib
import sqlite3 import sqlite3
from collections.abc import Callable from collections.abc import Callable
@ -87,10 +88,12 @@ class Operations(LazyLogging):
Returns: Returns:
T: result of the ``query`` call T: result of the ``query`` call
""" """
with sqlite3.connect(self.path, detect_types=sqlite3.PARSE_DECLTYPES) as connection: with contextlib.closing(sqlite3.connect(self.path, detect_types=sqlite3.PARSE_DECLTYPES)) as connection:
connection.set_trace_callback(self.logger.debug) connection.set_trace_callback(self.logger.debug)
connection.row_factory = self.factory connection.row_factory = self.factory
result = query(connection) result = query(connection)
if commit: if commit:
connection.commit() connection.commit()
return result return result

View File

@ -22,7 +22,6 @@ import contextlib
from functools import cached_property from functools import cached_property
from ahriman.core.configuration import Configuration from ahriman.core.configuration import Configuration
from ahriman.core.configuration.schema import ConfigurationSchema
from ahriman.core.status.web_client import WebClient from ahriman.core.status.web_client import WebClient
from ahriman.core.triggers import Trigger from ahriman.core.triggers import Trigger
from ahriman.models.repository_id import RepositoryId from ahriman.models.repository_id import RepositoryId
@ -34,7 +33,7 @@ class DistributedSystem(Trigger, WebClient):
simple class to (un)register itself as a distributed worker simple class to (un)register itself as a distributed worker
""" """
CONFIGURATION_SCHEMA: ConfigurationSchema = { CONFIGURATION_SCHEMA = {
"worker": { "worker": {
"type": "dict", "type": "dict",
"schema": { "schema": {

View File

@ -28,6 +28,7 @@ from ahriman.core.formatters.package_stats_printer import PackageStatsPrinter
from ahriman.core.formatters.patch_printer import PatchPrinter from ahriman.core.formatters.patch_printer import PatchPrinter
from ahriman.core.formatters.printer import Printer from ahriman.core.formatters.printer import Printer
from ahriman.core.formatters.repository_printer import RepositoryPrinter from ahriman.core.formatters.repository_printer import RepositoryPrinter
from ahriman.core.formatters.repository_stats_printer import RepositoryStatsPrinter
from ahriman.core.formatters.status_printer import StatusPrinter from ahriman.core.formatters.status_printer import StatusPrinter
from ahriman.core.formatters.string_printer import StringPrinter from ahriman.core.formatters.string_printer import StringPrinter
from ahriman.core.formatters.tree_printer import TreePrinter from ahriman.core.formatters.tree_printer import TreePrinter

View File

@ -17,6 +17,8 @@
# You should have received a copy of the GNU General Public License # You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>. # along with this program. If not, see <http://www.gnu.org/licenses/>.
# #
from typing import ClassVar
from ahriman.core.formatters.string_printer import StringPrinter from ahriman.core.formatters.string_printer import StringPrinter
from ahriman.models.property import Property from ahriman.models.property import Property
@ -31,7 +33,7 @@ class ConfigurationPrinter(StringPrinter):
values(dict[str, str]): configuration values dictionary values(dict[str, str]): configuration values dictionary
""" """
HIDE_KEYS = [ HIDE_KEYS: ClassVar[list[str]] = [
"api_key", # telegram key "api_key", # telegram key
"client_secret", # oauth secret "client_secret", # oauth secret
"cookie_secret_key", # cookie secret key "cookie_secret_key", # cookie secret key

View File

@ -17,11 +17,9 @@
# You should have received a copy of the GNU General Public License # You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>. # along with this program. If not, see <http://www.gnu.org/licenses/>.
# #
import statistics
from ahriman.core.formatters.string_printer import StringPrinter from ahriman.core.formatters.string_printer import StringPrinter
from ahriman.core.utils import minmax
from ahriman.models.property import Property from ahriman.models.property import Property
from ahriman.models.series_statistics import SeriesStatistics
class EventStatsPrinter(StringPrinter): class EventStatsPrinter(StringPrinter):
@ -29,7 +27,7 @@ class EventStatsPrinter(StringPrinter):
print event statistics print event statistics
Attributes: Attributes:
events(list[float | int]): event values to build statistics statistics(SeriesStatistics): statistics object
""" """
def __init__(self, event_type: str, events: list[float | int]) -> None: def __init__(self, event_type: str, events: list[float | int]) -> None:
@ -39,7 +37,7 @@ class EventStatsPrinter(StringPrinter):
events(list[float | int]): event values to build statistics events(list[float | int]): event values to build statistics
""" """
StringPrinter.__init__(self, event_type) StringPrinter.__init__(self, event_type)
self.events = events self.statistics = SeriesStatistics(events)
def properties(self) -> list[Property]: def properties(self) -> list[Property]:
""" """
@ -49,24 +47,17 @@ class EventStatsPrinter(StringPrinter):
list[Property]: list of content properties list[Property]: list of content properties
""" """
properties = [ properties = [
Property("total", len(self.events)), Property("total", self.statistics.total),
] ]
# time statistics # time statistics
if self.events: if self.statistics:
min_time, max_time = minmax(self.events) mean = self.statistics.mean
mean = statistics.mean(self.events)
if len(self.events) > 1:
st_dev = statistics.stdev(self.events)
average = f"{mean:.3f} ± {st_dev:.3f}"
else:
average = f"{mean:.3f}"
properties.extend([ properties.extend([
Property("min", min_time), Property("min", self.statistics.min),
Property("average", average), Property("average", f"{mean:.3f} ± {self.statistics.st_dev:.3f}"),
Property("max", max_time), Property("max", self.statistics.max),
]) ])
return properties return properties

View File

@ -17,6 +17,8 @@
# You should have received a copy of the GNU General Public License # You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>. # along with this program. If not, see <http://www.gnu.org/licenses/>.
# #
from typing import ClassVar
from ahriman.core.formatters.string_printer import StringPrinter from ahriman.core.formatters.string_printer import StringPrinter
from ahriman.models.property import Property from ahriman.models.property import Property
@ -26,10 +28,11 @@ class PackageStatsPrinter(StringPrinter):
print packages statistics print packages statistics
Attributes: Attributes:
MAX_COUNT(int): (class attribute) maximum number of packages to print
events(dict[str, int]): map of package to its event frequency events(dict[str, int]): map of package to its event frequency
""" """
MAX_COUNT = 10 MAX_COUNT: ClassVar[int] = 10
def __init__(self, events: dict[str, int]) -> None: def __init__(self, events: dict[str, int]) -> None:
""" """

View File

@ -0,0 +1,53 @@
#
# Copyright (c) 2021-2025 ahriman team.
#
# This file is part of ahriman
# (see https://github.com/arcan1s/ahriman).
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
from ahriman.core.formatters.string_printer import StringPrinter
from ahriman.core.utils import pretty_size
from ahriman.models.property import Property
from ahriman.models.repository_id import RepositoryId
from ahriman.models.repository_stats import RepositoryStats
class RepositoryStatsPrinter(StringPrinter):
"""
print repository statistics
Attributes:
statistics(RepositoryStats): repository statistics
"""
def __init__(self, repository_id: RepositoryId, statistics: RepositoryStats) -> None:
"""
Args:
statistics(RepositoryStats): repository statistics
"""
StringPrinter.__init__(self, str(repository_id))
self.statistics = statistics
def properties(self) -> list[Property]:
"""
convert content into printable data
Returns:
list[Property]: list of content properties
"""
return [
Property("Packages", self.statistics.bases),
Property("Repository size", pretty_size(self.statistics.archive_size)),
]

View File

@ -17,12 +17,16 @@
# You should have received a copy of the GNU General Public License # You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>. # along with this program. If not, see <http://www.gnu.org/licenses/>.
# #
import atexit
import logging import logging
import uuid
from typing import Self from typing import Self
from ahriman.core.configuration import Configuration from ahriman.core.configuration import Configuration
from ahriman.core.status import Client from ahriman.core.status import Client
from ahriman.models.log_record import LogRecord
from ahriman.models.log_record_id import LogRecordId
from ahriman.models.repository_id import RepositoryId from ahriman.models.repository_id import RepositoryId
@ -33,6 +37,7 @@ class HttpLogHandler(logging.Handler):
method method
Attributes: Attributes:
keep_last_records(int): number of last records to keep
reporter(Client): build status reporter instance reporter(Client): build status reporter instance
suppress_errors(bool): suppress logging errors (e.g. if no web server available) suppress_errors(bool): suppress logging errors (e.g. if no web server available)
""" """
@ -51,6 +56,7 @@ class HttpLogHandler(logging.Handler):
self.reporter = Client.load(repository_id, configuration, report=report) self.reporter = Client.load(repository_id, configuration, report=report)
self.suppress_errors = suppress_errors self.suppress_errors = suppress_errors
self.keep_last_records = configuration.getint("settings", "keep_last_logs", fallback=0)
@classmethod @classmethod
def load(cls, repository_id: RepositoryId, configuration: Configuration, *, report: bool) -> Self: def load(cls, repository_id: RepositoryId, configuration: Configuration, *, report: bool) -> Self:
@ -76,6 +82,9 @@ class HttpLogHandler(logging.Handler):
handler = cls(repository_id, configuration, report=report, suppress_errors=suppress_errors) handler = cls(repository_id, configuration, report=report, suppress_errors=suppress_errors)
root.addHandler(handler) root.addHandler(handler)
LogRecordId.DEFAULT_PROCESS_ID = str(uuid.uuid4()) # assign default process identifier for log records
atexit.register(handler.rotate)
return handler return handler
def emit(self, record: logging.LogRecord) -> None: def emit(self, record: logging.LogRecord) -> None:
@ -90,8 +99,14 @@ class HttpLogHandler(logging.Handler):
return # in case if no package base supplied we need just skip log message return # in case if no package base supplied we need just skip log message
try: try:
self.reporter.package_logs_add(log_record_id, record.created, record.getMessage()) self.reporter.package_logs_add(LogRecord(log_record_id, record.created, record.getMessage()))
except Exception: except Exception:
if self.suppress_errors: if self.suppress_errors:
return return
self.handleError(record) self.handleError(record)
def rotate(self) -> None:
"""
rotate log records, removing older ones
"""
self.reporter.logs_rotate(self.keep_last_records)

View File

@ -74,7 +74,7 @@ class LazyLogging:
def package_record_factory(*args: Any, **kwargs: Any) -> logging.LogRecord: def package_record_factory(*args: Any, **kwargs: Any) -> logging.LogRecord:
record = current_factory(*args, **kwargs) record = current_factory(*args, **kwargs)
record.package_id = LogRecordId(package_base, version or "") record.package_id = LogRecordId(package_base, version or "<unknown>")
return record return record
logging.setLogRecordFactory(package_record_factory) logging.setLogRecordFactory(package_record_factory)
@ -99,24 +99,3 @@ class LazyLogging:
yield yield
finally: finally:
self._package_logger_reset() self._package_logger_reset()
@contextlib.contextmanager
def suppress_logging(self, log_level: int = logging.WARNING) -> Generator[None, None, None]:
"""
silence log messages in context
Args:
log_level(int, optional): the highest log level to keep (Default value = logging.WARNING)
Examples:
This function is designed to be used to suppress all log messages in context, e.g.:
>>> with self.suppress_logging():
>>> do_some_noisy_actions()
"""
current_level = self.logger.manager.disable
try:
logging.disable(log_level)
yield
finally:
logging.disable(current_level)

View File

@ -21,6 +21,7 @@ import logging
from logging.config import fileConfig from logging.config import fileConfig
from pathlib import Path from pathlib import Path
from typing import ClassVar
from ahriman.core.configuration import Configuration from ahriman.core.configuration import Configuration
from ahriman.core.log.http_log_handler import HttpLogHandler from ahriman.core.log.http_log_handler import HttpLogHandler
@ -38,9 +39,9 @@ class LogLoader:
DEFAULT_SYSLOG_DEVICE(Path): (class attribute) default path to syslog device DEFAULT_SYSLOG_DEVICE(Path): (class attribute) default path to syslog device
""" """
DEFAULT_LOG_FORMAT = "[%(levelname)s %(asctime)s] [%(filename)s:%(lineno)d %(funcName)s]: %(message)s" DEFAULT_LOG_FORMAT: ClassVar[str] = "[%(levelname)s %(asctime)s] [%(name)s]: %(message)s"
DEFAULT_LOG_LEVEL = logging.DEBUG DEFAULT_LOG_LEVEL: ClassVar[int] = logging.DEBUG
DEFAULT_SYSLOG_DEVICE = Path("/") / "dev" / "log" DEFAULT_SYSLOG_DEVICE: ClassVar[Path] = Path("/") / "dev" / "log"
@staticmethod @staticmethod
def handler(selected: LogHandler | None) -> LogHandler: def handler(selected: LogHandler | None) -> LogHandler:

View File

@ -40,7 +40,7 @@ class JinjaTemplate:
* homepage - link to homepage, string, optional * homepage - link to homepage, string, optional
* last_update - report generation time, pretty printed datetime, required * last_update - report generation time, pretty printed datetime, required
* link_path - prefix fo packages to download, string, required * link_path - prefix of packages to download, string, required
* has_package_signed - ``True`` in case if package sign enabled, ``False`` otherwise, required * has_package_signed - ``True`` in case if package sign enabled, ``False`` otherwise, required
* has_repo_signed - ``True`` in case if repository database sign enabled, ``False`` otherwise, required * has_repo_signed - ``True`` in case if repository database sign enabled, ``False`` otherwise, required
* packages - sorted list of packages properties, required * packages - sorted list of packages properties, required
@ -64,7 +64,7 @@ class JinjaTemplate:
Attributes: Attributes:
default_pgp_key(str | None): default PGP key default_pgp_key(str | None): default PGP key
homepage(str | None): homepage link if any (for footer) homepage(str | None): homepage link if any (for footer)
link_path(str): prefix fo packages to download link_path(str): prefix of packages to download
name(str): repository name name(str): repository name
rss_url(str | None): link to the RSS feed rss_url(str | None): link to the RSS feed
sign_targets(set[SignSettings]): targets to sign enabled in configuration sign_targets(set[SignSettings]): targets to sign enabled in configuration

View File

@ -67,14 +67,6 @@ class ReportTrigger(Trigger):
"type": "string", "type": "string",
"allowed": ["email"], "allowed": ["email"],
}, },
"full_template_path": {
"type": "path",
"coerce": "absolute_path",
"excludes": ["template_full"],
"required": True,
"path_exists": True,
"path_type": "file",
},
"homepage": { "homepage": {
"type": "string", "type": "string",
"empty": False, "empty": False,
@ -132,26 +124,16 @@ class ReportTrigger(Trigger):
}, },
"template": { "template": {
"type": "string", "type": "string",
"excludes": ["template_path"],
"dependencies": ["templates"], "dependencies": ["templates"],
"required": True, "required": True,
"empty": False, "empty": False,
}, },
"template_full": { "template_full": {
"type": "string", "type": "string",
"excludes": ["template_path"],
"dependencies": ["templates"], "dependencies": ["templates"],
"required": True, "required": True,
"empty": False, "empty": False,
}, },
"template_path": {
"type": "path",
"coerce": "absolute_path",
"excludes": ["template"],
"required": True,
"path_exists": True,
"path_type": "file",
},
"templates": { "templates": {
"type": "list", "type": "list",
"coerce": "list", "coerce": "list",
@ -199,19 +181,10 @@ class ReportTrigger(Trigger):
}, },
"template": { "template": {
"type": "string", "type": "string",
"excludes": ["template_path"],
"dependencies": ["templates"], "dependencies": ["templates"],
"required": True, "required": True,
"empty": False, "empty": False,
}, },
"template_path": {
"type": "path",
"coerce": "absolute_path",
"excludes": ["template"],
"required": True,
"path_exists": True,
"path_type": "file",
},
"templates": { "templates": {
"type": "list", "type": "list",
"coerce": "list", "coerce": "list",
@ -225,76 +198,6 @@ class ReportTrigger(Trigger):
}, },
}, },
}, },
"telegram": {
"type": "dict",
"schema": {
"type": {
"type": "string",
"allowed": ["telegram"],
},
"api_key": {
"type": "string",
"required": True,
"empty": False,
},
"chat_id": {
"type": "string",
"required": True,
"empty": False,
},
"homepage": {
"type": "string",
"empty": False,
"is_url": ["http", "https"],
},
"link_path": {
"type": "string",
"required": True,
"empty": False,
"is_url": [],
},
"rss_url": {
"type": "string",
"empty": False,
"is_url": ["http", "https"],
},
"template": {
"type": "string",
"excludes": ["template_path"],
"dependencies": ["templates"],
"required": True,
"empty": False,
},
"template_path": {
"type": "path",
"coerce": "absolute_path",
"excludes": ["template"],
"required": True,
"path_exists": True,
"path_type": "file",
},
"template_type": {
"type": "string",
"allowed": ["MarkdownV2", "HTML", "Markdown"],
},
"templates": {
"type": "list",
"coerce": "list",
"schema": {
"type": "path",
"coerce": "absolute_path",
"path_exists": True,
"path_type": "dir",
},
"empty": False,
},
"timeout": {
"type": "integer",
"coerce": "integer",
"min": 0,
},
},
},
"remote-call": { "remote-call": {
"type": "dict", "type": "dict",
"schema": { "schema": {
@ -354,19 +257,10 @@ class ReportTrigger(Trigger):
}, },
"template": { "template": {
"type": "string", "type": "string",
"excludes": ["template_path"],
"dependencies": ["templates"], "dependencies": ["templates"],
"required": True, "required": True,
"empty": False, "empty": False,
}, },
"template_path": {
"type": "path",
"coerce": "absolute_path",
"excludes": ["template"],
"required": True,
"path_exists": True,
"path_type": "file",
},
"templates": { "templates": {
"type": "list", "type": "list",
"coerce": "list", "coerce": "list",
@ -380,6 +274,67 @@ class ReportTrigger(Trigger):
}, },
}, },
}, },
"telegram": {
"type": "dict",
"schema": {
"type": {
"type": "string",
"allowed": ["telegram"],
},
"api_key": {
"type": "string",
"required": True,
"empty": False,
},
"chat_id": {
"type": "string",
"required": True,
"empty": False,
},
"homepage": {
"type": "string",
"empty": False,
"is_url": ["http", "https"],
},
"link_path": {
"type": "string",
"required": True,
"empty": False,
"is_url": [],
},
"rss_url": {
"type": "string",
"empty": False,
"is_url": ["http", "https"],
},
"template": {
"type": "string",
"dependencies": ["templates"],
"required": True,
"empty": False,
},
"template_type": {
"type": "string",
"allowed": ["MarkdownV2", "HTML", "Markdown"],
},
"templates": {
"type": "list",
"coerce": "list",
"schema": {
"type": "path",
"coerce": "absolute_path",
"path_exists": True,
"path_type": "dir",
},
"empty": False,
},
"timeout": {
"type": "integer",
"coerce": "integer",
"min": 0,
},
},
},
} }
def __init__(self, repository_id: RepositoryId, configuration: Configuration) -> None: def __init__(self, repository_id: RepositoryId, configuration: Configuration) -> None:

View File

@ -17,6 +17,8 @@
# You should have received a copy of the GNU General Public License # You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>. # along with this program. If not, see <http://www.gnu.org/licenses/>.
# #
from typing import ClassVar
from ahriman.core.configuration import Configuration from ahriman.core.configuration import Configuration
from ahriman.core.http import SyncHttpClient from ahriman.core.http import SyncHttpClient
from ahriman.core.report.jinja_template import JinjaTemplate from ahriman.core.report.jinja_template import JinjaTemplate
@ -39,8 +41,8 @@ class Telegram(Report, JinjaTemplate, SyncHttpClient):
template_type(str): template message type to be used in parse mode, one of MarkdownV2, HTML, Markdown template_type(str): template message type to be used in parse mode, one of MarkdownV2, HTML, Markdown
""" """
TELEGRAM_API_URL = "https://api.telegram.org" TELEGRAM_API_URL: ClassVar[str] = "https://api.telegram.org"
TELEGRAM_MAX_CONTENT_LENGTH = 4096 TELEGRAM_MAX_CONTENT_LENGTH: ClassVar[int] = 4096
def __init__(self, repository_id: RepositoryId, configuration: Configuration, section: str) -> None: def __init__(self, repository_id: RepositoryId, configuration: Configuration, section: str) -> None:
""" """

View File

@ -71,7 +71,7 @@ class EventLogger:
>>> with self.in_event(package_base, EventType.PackageUpdated): >>> with self.in_event(package_base, EventType.PackageUpdated):
>>> do_something() >>> do_something()
Additional parameter ``failure`` can be set in order to emit an event on exception occured. If none set Additional parameter ``failure`` can be set in order to emit an event on exception occurred. If none set
(default), then no event will be recorded on exception (default), then no event will be recorded on exception
""" """
with MetricsTimer() as timer: with MetricsTimer() as timer:

View File

@ -144,8 +144,7 @@ class UpdateHandler(PackageInfo, Cleaner):
branch="master", branch="master",
) )
with self.suppress_logging(): Sources.fetch(cache_dir, source)
Sources.fetch(cache_dir, source)
remote = Package.from_build(cache_dir, self.architecture, None) remote = Package.from_build(cache_dir, self.architecture, None)
local = packages.get(remote.base) local = packages.get(remote.base)

View File

@ -27,10 +27,11 @@ from ahriman.models.changes import Changes
from ahriman.models.dependencies import Dependencies from ahriman.models.dependencies import Dependencies
from ahriman.models.event import Event, EventType from ahriman.models.event import Event, EventType
from ahriman.models.internal_status import InternalStatus from ahriman.models.internal_status import InternalStatus
from ahriman.models.log_record_id import LogRecordId from ahriman.models.log_record import LogRecord
from ahriman.models.package import Package from ahriman.models.package import Package
from ahriman.models.pkgbuild_patch import PkgbuildPatch from ahriman.models.pkgbuild_patch import PkgbuildPatch
from ahriman.models.repository_id import RepositoryId from ahriman.models.repository_id import RepositoryId
from ahriman.models.repository_stats import RepositoryStats
class Client: class Client:
@ -114,6 +115,14 @@ class Client:
""" """
raise NotImplementedError raise NotImplementedError
def logs_rotate(self, keep_last_records: int) -> None:
"""
remove older logs from storage
Args:
keep_last_records(int): number of last records to keep
"""
def package_changes_get(self, package_base: str) -> Changes: def package_changes_get(self, package_base: str) -> Changes:
""" """
get package changes get package changes
@ -185,18 +194,16 @@ class Client:
""" """
raise NotImplementedError raise NotImplementedError
def package_logs_add(self, log_record_id: LogRecordId, created: float, message: str) -> None: def package_logs_add(self, log_record: LogRecord) -> None:
""" """
post log record post log record
Args: Args:
log_record_id(LogRecordId): log record id log_record(LogRecord): log record
created(float): log created timestamp
message(str): log message
""" """
# this method does not raise NotImplementedError because it is actively used as dummy client for http log # this method does not raise NotImplementedError because it is actively used as dummy client for http log
def package_logs_get(self, package_base: str, limit: int = -1, offset: int = 0) -> list[tuple[float, str]]: def package_logs_get(self, package_base: str, limit: int = -1, offset: int = 0) -> list[LogRecord]:
""" """
get package logs get package logs
@ -206,7 +213,7 @@ class Client:
offset(int, optional): records offset (Default value = 0) offset(int, optional): records offset (Default value = 0)
Returns: Returns:
list[tuple[float, str]]: package logs list[LogRecord]: package logs
Raises: Raises:
NotImplementedError: not implemented method NotImplementedError: not implemented method
@ -354,6 +361,16 @@ class Client:
return # skip update in case if package is already known return # skip update in case if package is already known
self.package_update(package, BuildStatusEnum.Unknown) self.package_update(package, BuildStatusEnum.Unknown)
def statistics(self) -> RepositoryStats:
"""
get repository statistics
Returns:
RepositoryStats: repository statistics object
"""
packages = [package for package, _ in self.package_get(None)]
return RepositoryStats.from_packages(packages)
def status_get(self) -> InternalStatus: def status_get(self) -> InternalStatus:
""" """
get internal service status get internal service status

View File

@ -23,7 +23,7 @@ from ahriman.models.build_status import BuildStatus, BuildStatusEnum
from ahriman.models.changes import Changes from ahriman.models.changes import Changes
from ahriman.models.dependencies import Dependencies from ahriman.models.dependencies import Dependencies
from ahriman.models.event import Event, EventType from ahriman.models.event import Event, EventType
from ahriman.models.log_record_id import LogRecordId from ahriman.models.log_record import LogRecord
from ahriman.models.package import Package from ahriman.models.package import Package
from ahriman.models.pkgbuild_patch import PkgbuildPatch from ahriman.models.pkgbuild_patch import PkgbuildPatch
from ahriman.models.repository_id import RepositoryId from ahriman.models.repository_id import RepositoryId
@ -75,6 +75,15 @@ class LocalClient(Client):
""" """
return self.database.event_get(event, object_id, from_date, to_date, limit, offset, self.repository_id) return self.database.event_get(event, object_id, from_date, to_date, limit, offset, self.repository_id)
def logs_rotate(self, keep_last_records: int) -> None:
"""
remove older logs from storage
Args:
keep_last_records(int): number of last records to keep
"""
self.database.logs_rotate(keep_last_records, self.repository_id)
def package_changes_get(self, package_base: str) -> Changes: def package_changes_get(self, package_base: str) -> Changes:
""" """
get package changes get package changes
@ -134,18 +143,16 @@ class LocalClient(Client):
return packages return packages
return [(package, status) for package, status in packages if package.base == package_base] return [(package, status) for package, status in packages if package.base == package_base]
def package_logs_add(self, log_record_id: LogRecordId, created: float, message: str) -> None: def package_logs_add(self, log_record: LogRecord) -> None:
""" """
post log record post log record
Args: Args:
log_record_id(LogRecordId): log record id log_record(LogRecord): log record
created(float): log created timestamp
message(str): log message
""" """
self.database.logs_insert(log_record_id, created, message, self.repository_id) self.database.logs_insert(log_record, self.repository_id)
def package_logs_get(self, package_base: str, limit: int = -1, offset: int = 0) -> list[tuple[float, str]]: def package_logs_get(self, package_base: str, limit: int = -1, offset: int = 0) -> list[LogRecord]:
""" """
get package logs get package logs
@ -155,7 +162,7 @@ class LocalClient(Client):
offset(int, optional): records offset (Default value = 0) offset(int, optional): records offset (Default value = 0)
Returns: Returns:
list[tuple[float, str]]: package logs list[LogRecord]: package logs
""" """
return self.database.logs_get(package_base, limit, offset, self.repository_id) return self.database.logs_get(package_base, limit, offset, self.repository_id)

View File

@ -28,7 +28,7 @@ from ahriman.models.build_status import BuildStatus, BuildStatusEnum
from ahriman.models.changes import Changes from ahriman.models.changes import Changes
from ahriman.models.dependencies import Dependencies from ahriman.models.dependencies import Dependencies
from ahriman.models.event import Event, EventType from ahriman.models.event import Event, EventType
from ahriman.models.log_record_id import LogRecordId from ahriman.models.log_record import LogRecord
from ahriman.models.package import Package from ahriman.models.package import Package
from ahriman.models.pkgbuild_patch import PkgbuildPatch from ahriman.models.pkgbuild_patch import PkgbuildPatch
@ -53,9 +53,6 @@ class Watcher(LazyLogging):
self._known: dict[str, tuple[Package, BuildStatus]] = {} self._known: dict[str, tuple[Package, BuildStatus]] = {}
self.status = BuildStatus() self.status = BuildStatus()
# special variables for updating logs
self._last_log_record_id = LogRecordId("", "")
@property @property
def packages(self) -> list[tuple[Package, BuildStatus]]: def packages(self) -> list[tuple[Package, BuildStatus]]:
""" """
@ -81,6 +78,8 @@ class Watcher(LazyLogging):
for package, status in self.client.package_get(None) for package, status in self.client.package_get(None)
} }
logs_rotate: Callable[[int], None]
package_changes_get: Callable[[str], Changes] package_changes_get: Callable[[str], Changes]
package_changes_update: Callable[[str, Changes], None] package_changes_update: Callable[[str, Changes], None]
@ -108,22 +107,9 @@ class Watcher(LazyLogging):
except KeyError: except KeyError:
raise UnknownPackageError(package_base) from None raise UnknownPackageError(package_base) from None
def package_logs_add(self, log_record_id: LogRecordId, created: float, message: str) -> None: package_logs_add: Callable[[LogRecord], None]
"""
make new log record into database
Args: package_logs_get: Callable[[str, int, int], list[LogRecord]]
log_record_id(LogRecordId): log record id
created(float): log created timestamp
message(str): log message
"""
if self._last_log_record_id != log_record_id:
# there is new log record, so we remove old ones
self.package_logs_remove(log_record_id.package_base, log_record_id.version)
self._last_log_record_id = log_record_id
self.client.package_logs_add(log_record_id, created, message)
package_logs_get: Callable[[str, int, int], list[tuple[float, str]]]
package_logs_remove: Callable[[str, str | None], None] package_logs_remove: Callable[[str, str | None], None]

View File

@ -29,7 +29,7 @@ from ahriman.models.changes import Changes
from ahriman.models.dependencies import Dependencies from ahriman.models.dependencies import Dependencies
from ahriman.models.event import Event, EventType from ahriman.models.event import Event, EventType
from ahriman.models.internal_status import InternalStatus from ahriman.models.internal_status import InternalStatus
from ahriman.models.log_record_id import LogRecordId from ahriman.models.log_record import LogRecord
from ahriman.models.package import Package from ahriman.models.package import Package
from ahriman.models.pkgbuild_patch import PkgbuildPatch from ahriman.models.pkgbuild_patch import PkgbuildPatch
from ahriman.models.repository_id import RepositoryId from ahriman.models.repository_id import RepositoryId
@ -210,6 +210,18 @@ class WebClient(Client, SyncAhrimanClient):
return [] return []
def logs_rotate(self, keep_last_records: int) -> None:
"""
remove older logs from storage
Args:
keep_last_records(int): number of last records to keep
"""
query = self.repository_id.query() + [("keep_last_records", str(keep_last_records))]
with contextlib.suppress(Exception):
self.make_request("DELETE", f"{self.address}/api/v1/service/logs", params=query)
def package_changes_get(self, package_base: str) -> Changes: def package_changes_get(self, package_base: str) -> Changes:
""" """
get package changes get package changes
@ -294,28 +306,27 @@ class WebClient(Client, SyncAhrimanClient):
return [] return []
def package_logs_add(self, log_record_id: LogRecordId, created: float, message: str) -> None: def package_logs_add(self, log_record: LogRecord) -> None:
""" """
post log record post log record
Args: Args:
log_record_id(LogRecordId): log record id log_record(LogRecord): log record
created(float): log created timestamp
message(str): log message
""" """
payload = { payload = {
"created": created, "created": log_record.created,
"message": message, "message": log_record.message,
"version": log_record_id.version, "process_id": log_record.log_record_id.process_id,
"version": log_record.log_record_id.version,
} }
# this is special case, because we would like to do not suppress exception here # this is special case, because we would like to do not suppress exception here
# in case of exception raised it will be handled by upstream HttpLogHandler # in case of exception raised it will be handled by upstream HttpLogHandler
# In the other hand, we force to suppress all http logs here to avoid cyclic reporting # In the other hand, we force to suppress all http logs here to avoid cyclic reporting
self.make_request("POST", self._logs_url(log_record_id.package_base), self.make_request("POST", self._logs_url(log_record.log_record_id.package_base),
params=self.repository_id.query(), json=payload, suppress_errors=True) params=self.repository_id.query(), json=payload, suppress_errors=True)
def package_logs_get(self, package_base: str, limit: int = -1, offset: int = 0) -> list[tuple[float, str]]: def package_logs_get(self, package_base: str, limit: int = -1, offset: int = 0) -> list[LogRecord]:
""" """
get package logs get package logs
@ -325,7 +336,7 @@ class WebClient(Client, SyncAhrimanClient):
offset(int, optional): records offset (Default value = 0) offset(int, optional): records offset (Default value = 0)
Returns: Returns:
list[tuple[float, str]]: package logs list[LogRecord]: package logs
""" """
query = self.repository_id.query() + [("limit", str(limit)), ("offset", str(offset))] query = self.repository_id.query() + [("limit", str(limit)), ("offset", str(offset))]
@ -333,7 +344,7 @@ class WebClient(Client, SyncAhrimanClient):
response = self.make_request("GET", self._logs_url(package_base), params=query) response = self.make_request("GET", self._logs_url(package_base), params=query)
response_json = response.json() response_json = response.json()
return [(record["created"], record["message"]) for record in response_json] return [LogRecord.from_json(package_base, record) for record in response_json]
return [] return []

View File

@ -22,6 +22,7 @@ import itertools
from collections.abc import Callable, Generator from collections.abc import Callable, Generator
from pathlib import Path from pathlib import Path
from typing import ClassVar
from ahriman.core.utils import utcnow from ahriman.core.utils import utcnow
from ahriman.models.pkgbuild_patch import PkgbuildPatch from ahriman.models.pkgbuild_patch import PkgbuildPatch
@ -35,7 +36,7 @@ class PkgbuildGenerator:
PKGBUILD_STATIC_PROPERTIES(list[PkgbuildPatch]): (class attribute) list of default pkgbuild static properties PKGBUILD_STATIC_PROPERTIES(list[PkgbuildPatch]): (class attribute) list of default pkgbuild static properties
""" """
PKGBUILD_STATIC_PROPERTIES = [ PKGBUILD_STATIC_PROPERTIES: ClassVar[list[PkgbuildPatch]] = [
PkgbuildPatch("pkgrel", "1"), PkgbuildPatch("pkgrel", "1"),
PkgbuildPatch("arch", ["any"]), PkgbuildPatch("arch", ["any"]),
] ]

View File

@ -18,6 +18,7 @@
# along with this program. If not, see <http://www.gnu.org/licenses/>. # along with this program. If not, see <http://www.gnu.org/licenses/>.
# #
from collections.abc import Callable from collections.abc import Callable
from typing import ClassVar
from ahriman.core.configuration import Configuration from ahriman.core.configuration import Configuration
from ahriman.core.configuration.schema import ConfigurationSchema from ahriman.core.configuration.schema import ConfigurationSchema
@ -56,8 +57,8 @@ class Trigger(LazyLogging):
>>> loader.on_result(Result(), []) >>> loader.on_result(Result(), [])
""" """
CONFIGURATION_SCHEMA: ConfigurationSchema = {} CONFIGURATION_SCHEMA: ClassVar[ConfigurationSchema] = {}
CONFIGURATION_SCHEMA_FALLBACK: str | None = None CONFIGURATION_SCHEMA_FALLBACK: ClassVar[str | None] = None
def __init__(self, repository_id: RepositoryId, configuration: Configuration) -> None: def __init__(self, repository_id: RepositoryId, configuration: Configuration) -> None:
""" """

View File

@ -83,6 +83,20 @@ class UploadTrigger(Trigger):
}, },
}, },
}, },
"remote-service": {
"type": "dict",
"schema": {
"type": {
"type": "string",
"allowed": ["ahriman", "remote-service"],
},
"timeout": {
"type": "integer",
"coerce": "integer",
"min": 0,
},
},
},
"rsync": { "rsync": {
"type": "dict", "type": "dict",
"schema": { "schema": {
@ -107,20 +121,6 @@ class UploadTrigger(Trigger):
}, },
}, },
}, },
"remote-service": {
"type": "dict",
"schema": {
"type": {
"type": "string",
"allowed": ["ahriman", "remote-service"],
},
"timeout": {
"type": "integer",
"coerce": "integer",
"min": 0,
},
},
},
"s3": { "s3": {
"type": "dict", "type": "dict",
"schema": { "schema": {

View File

@ -25,9 +25,9 @@ class Action(StrEnum):
base action enumeration base action enumeration
Attributes: Attributes:
List(Action): (class attribute) list available values List(Action): list available values
Remove(Action): (class attribute) remove everything from local storage Remove(Action): remove everything from local storage
Update(Action): (class attribute) update local storage or add to Update(Action): update local storage or add to
""" """
List = "list" List = "list"

View File

@ -27,10 +27,10 @@ class AuthSettings(StrEnum):
web authorization type web authorization type
Attributes: Attributes:
Disabled(AuthSettings): (class attribute) authorization is disabled Disabled(AuthSettings): authorization is disabled
Configuration(AuthSettings): (class attribute) configuration based authorization Configuration(AuthSettings): configuration based authorization
OAuth(AuthSettings): (class attribute) OAuth based provider OAuth(AuthSettings): OAuth based provider
PAM(AuthSettings): (class attribute) PAM based provider PAM(AuthSettings): PAM based provider
""" """
Disabled = "disabled" Disabled = "disabled"

View File

@ -29,11 +29,11 @@ class BuildStatusEnum(StrEnum):
build status enumeration build status enumeration
Attributes: Attributes:
Unknown(BuildStatusEnum): (class attribute) build status is unknown Unknown(BuildStatusEnum): build status is unknown
Pending(BuildStatusEnum): (class attribute) package is out-of-dated and will be built soon Pending(BuildStatusEnum): package is out-of-dated and will be built soon
Building(BuildStatusEnum): (class attribute) package is building right now Building(BuildStatusEnum): package is building right now
Failed(BuildStatusEnum): (class attribute) package build failed Failed(BuildStatusEnum): package build failed
Success(BuildStatusEnum): (class attribute) package has been built without errors Success(BuildStatusEnum): package has been built without errors
""" """
Unknown = "unknown" Unknown = "unknown"

View File

@ -28,10 +28,10 @@ class EventType(StrEnum):
predefined event types predefined event types
Attributes: Attributes:
PackageOutdated(EventType): (class attribute) package has been marked as out-of-date PackageOutdated(EventType): package has been marked as out-of-date
PackageRemoved(EventType): (class attribute) package has been removed PackageRemoved(EventType): package has been removed
PackageUpdateFailed(EventType): (class attribute) package update has been failed PackageUpdateFailed(EventType): package update has been failed
PackageUpdated(EventType): (class attribute) package has been updated PackageUpdated(EventType): package has been updated
""" """
PackageOutdated = "package-outdated" PackageOutdated = "package-outdated"
@ -78,7 +78,7 @@ class Event:
dump(dict[str, Any]): json dump body dump(dict[str, Any]): json dump body
Returns: Returns:
Self: dependencies object Self: event object
""" """
return cls( return cls(
event=dump["event"], event=dump["event"],

View File

@ -23,6 +23,7 @@ from typing import Any, Self
from ahriman.core.utils import dataclass_view from ahriman.core.utils import dataclass_view
from ahriman.models.build_status import BuildStatus from ahriman.models.build_status import BuildStatus
from ahriman.models.counters import Counters from ahriman.models.counters import Counters
from ahriman.models.repository_stats import RepositoryStats
@dataclass(frozen=True, kw_only=True) @dataclass(frozen=True, kw_only=True)
@ -35,6 +36,7 @@ class InternalStatus:
architecture(str | None): repository architecture architecture(str | None): repository architecture
packages(Counters): packages statuses counter object packages(Counters): packages statuses counter object
repository(str | None): repository name repository(str | None): repository name
stats(RepositoryStats | None): repository stats
version(str | None): service version version(str | None): service version
""" """
@ -42,6 +44,7 @@ class InternalStatus:
architecture: str | None = None architecture: str | None = None
packages: Counters = field(default=Counters(total=0)) packages: Counters = field(default=Counters(total=0))
repository: str | None = None repository: str | None = None
stats: RepositoryStats | None = None
version: str | None = None version: str | None = None
@classmethod @classmethod
@ -56,11 +59,13 @@ class InternalStatus:
Self: internal status Self: internal status
""" """
counters = Counters.from_json(dump["packages"]) if "packages" in dump else Counters(total=0) counters = Counters.from_json(dump["packages"]) if "packages" in dump else Counters(total=0)
stats = RepositoryStats.from_json(dump["stats"]) if "stats" in dump else None
build_status = dump.get("status") or {} build_status = dump.get("status") or {}
return cls(status=BuildStatus.from_json(build_status), return cls(status=BuildStatus.from_json(build_status),
architecture=dump.get("architecture"), architecture=dump.get("architecture"),
packages=counters, packages=counters,
repository=dump.get("repository"), repository=dump.get("repository"),
stats=stats,
version=dump.get("version")) version=dump.get("version"))
def view(self) -> dict[str, Any]: def view(self) -> dict[str, Any]:

View File

@ -25,9 +25,9 @@ class LogHandler(StrEnum):
log handler as described by default configuration log handler as described by default configuration
Attributes: Attributes:
Console(LogHandler): (class attribute) write logs to console Console(LogHandler): write logs to console
Syslog(LogHandler): (class attribute) write logs to syslog device /dev/null Syslog(LogHandler): write logs to syslog device /dev/null
Journald(LogHandler): (class attribute) write logs to journald directly Journald(LogHandler): write logs to journald directly
""" """
Console = "console" Console = "console"

View File

@ -0,0 +1,76 @@
#
# Copyright (c) 2021-2025 ahriman team.
#
# This file is part of ahriman
# (see https://github.com/arcan1s/ahriman).
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
from dataclasses import dataclass
from typing import Any, Self
from ahriman.models.log_record_id import LogRecordId
@dataclass(frozen=True)
class LogRecord:
"""
log record
Attributes:
log_record_id(LogRecordId): log record identifier
created(float): log record creation timestamp
message(str): log record message
"""
log_record_id: LogRecordId
created: float
message: str
@classmethod
def from_json(cls, package_base: str, dump: dict[str, Any]) -> Self:
"""
construct log record from the json dump
Args:
package_base(str): package base for which log record belongs
dump(dict[str, Any]): json dump body
Returns:
Self: log record object
"""
if "process_id" in dump:
log_record_id = LogRecordId(package_base, dump["version"], dump["process_id"])
else:
log_record_id = LogRecordId(package_base, dump["version"])
return cls(
log_record_id=log_record_id,
created=dump["created"],
message=dump["message"],
)
def view(self) -> dict[str, Any]:
"""
generate json log record view
Returns:
dict[str, Any]: json-friendly dictionary
"""
return {
"created": self.created,
"message": self.message,
"version": self.log_record_id.version,
"process_id": self.log_record_id.process_id,
}

View File

@ -17,7 +17,10 @@
# You should have received a copy of the GNU General Public License # You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>. # along with this program. If not, see <http://www.gnu.org/licenses/>.
# #
import uuid
from dataclasses import dataclass from dataclasses import dataclass
from typing import ClassVar
@dataclass(frozen=True) @dataclass(frozen=True)
@ -26,9 +29,21 @@ class LogRecordId:
log record process identifier log record process identifier
Attributes: Attributes:
DEFAULT_PROCESS_ID(str): (class attribute) default process identifier
package_base(str): package base for which log record belongs package_base(str): package base for which log record belongs
version(str): package version for which log record belongs version(str): package version for which log record belongs
process_id(str, optional): unique process identifier
""" """
package_base: str package_base: str
version: str version: str
process_id: str = ""
DEFAULT_PROCESS_ID: ClassVar[str] = str(uuid.uuid4())
def __post_init__(self) -> None:
"""
assign process identifier from default if not set
"""
if not self.process_id:
object.__setattr__(self, "process_id", self.DEFAULT_PROCESS_ID)

View File

@ -69,7 +69,7 @@ class Package(LazyLogging):
:attr:`ahriman.models.package_source.PackageSource.Archive`, :attr:`ahriman.models.package_source.PackageSource.Archive`,
:attr:`ahriman.models.package_source.PackageSource.AUR`, :attr:`ahriman.models.package_source.PackageSource.AUR`,
:attr:`ahriman.models.package_source.PackageSource.Local` and :attr:`ahriman.models.package_source.PackageSource.Local` and
:attr:`ahriman.models.package_source.PackageSource.Repository` repsectively: :attr:`ahriman.models.package_source.PackageSource.Repository` respectively:
>>> ahriman_package = Package.from_aur("ahriman") >>> ahriman_package = Package.from_aur("ahriman")
>>> pacman_package = Package.from_official("pacman", pacman) >>> pacman_package = Package.from_official("pacman", pacman)
@ -429,14 +429,11 @@ class Package(LazyLogging):
task = Task(self, configuration, repository_id.architecture, paths) task = Task(self, configuration, repository_id.architecture, paths)
try: try:
with self.suppress_logging(): # create fresh chroot environment, fetch sources and - automagically - update PKGBUILD
# create fresh chroot environment, fetch sources and - automagically - update PKGBUILD task.init(paths.cache_for(self.base), [], None)
task.init(paths.cache_for(self.base), [], None) pkgbuild = Pkgbuild.from_file(paths.cache_for(self.base) / "PKGBUILD")
task.build(paths.cache_for(self.base), dry_run=True)
pkgbuild = Pkgbuild.from_file(paths.cache_for(self.base) / "PKGBUILD") return full_version(pkgbuild.get("epoch"), pkgbuild["pkgver"], pkgbuild["pkgrel"])
return full_version(pkgbuild.get("epoch"), pkgbuild["pkgver"], pkgbuild["pkgrel"])
except Exception: except Exception:
self.logger.exception("cannot determine version of VCS package") self.logger.exception("cannot determine version of VCS package")
finally: finally:

View File

@ -32,13 +32,13 @@ class PackageSource(StrEnum):
package source for addition enumeration package source for addition enumeration
Attributes: Attributes:
Auto(PackageSource): (class attribute) automatically determine type of the source Auto(PackageSource): automatically determine type of the source
Archive(PackageSource): (class attribute) source is a package archive Archive(PackageSource): source is a package archive
AUR(PackageSource): (class attribute) source is an AUR package for which it should search AUR(PackageSource): source is an AUR package for which it should search
Directory(PackageSource): (class attribute) source is a directory which contains packages Directory(PackageSource): source is a directory which contains packages
Local(PackageSource): (class attribute) source is locally stored PKGBUILD Local(PackageSource): source is locally stored PKGBUILD
Remote(PackageSource): (class attribute) source is remote (http, ftp etc...) link Remote(PackageSource): source is remote (http, ftp etc...) link
Repository(PackageSource): (class attribute) source is official repository Repository(PackageSource): source is official repository
Examples: Examples:
In case if source is unknown the :func:`resolve()` and the source In case if source is unknown the :func:`resolve()` and the source

View File

@ -25,9 +25,9 @@ class PacmanSynchronization(IntEnum):
pacman database synchronization flag pacman database synchronization flag
Attributes: Attributes:
Disabled(PacmanSynchronization): (class attribute) do not synchronize local database Disabled(PacmanSynchronization): do not synchronize local database
Enabled(PacmanSynchronization): (class attribute) synchronize local database (same as pacman -Sy) Enabled(PacmanSynchronization): synchronize local database (same as pacman -Sy)
Force(PacmanSynchronization): (class attribute) force synchronize local database (same as pacman -Syy) Force(PacmanSynchronization): force synchronize local database (same as pacman -Syy)
""" """
Disabled = 0 Disabled = 0

View File

@ -21,7 +21,7 @@ from collections.abc import Iterator, Mapping
from dataclasses import dataclass from dataclasses import dataclass
from io import StringIO from io import StringIO
from pathlib import Path from pathlib import Path
from typing import Any, IO, Self from typing import Any, ClassVar, IO, Self
from ahriman.core.alpm.pkgbuild_parser import PkgbuildParser, PkgbuildToken from ahriman.core.alpm.pkgbuild_parser import PkgbuildParser, PkgbuildToken
from ahriman.models.pkgbuild_patch import PkgbuildPatch from ahriman.models.pkgbuild_patch import PkgbuildPatch
@ -33,11 +33,14 @@ class Pkgbuild(Mapping[str, Any]):
model and proxy for PKGBUILD properties model and proxy for PKGBUILD properties
Attributes: Attributes:
DEFAULT_ENCODINGS(str): (class attribute) default encoding to be applied on the file content
fields(dict[str, PkgbuildPatch]): PKGBUILD fields fields(dict[str, PkgbuildPatch]): PKGBUILD fields
""" """
fields: dict[str, PkgbuildPatch] fields: dict[str, PkgbuildPatch]
DEFAULT_ENCODINGS: ClassVar[str] = "utf8"
@property @property
def variables(self) -> dict[str, str]: def variables(self) -> dict[str, str]:
""" """
@ -54,18 +57,29 @@ class Pkgbuild(Mapping[str, Any]):
} }
@classmethod @classmethod
def from_file(cls, path: Path) -> Self: def from_file(cls, path: Path, encoding: str | None = None) -> Self:
""" """
parse PKGBUILD from the file parse PKGBUILD from the file
Args: Args:
path(Path): path to the PKGBUILD file path(Path): path to the PKGBUILD file
encoding(str | None, optional): the encoding of the file (Default value = None)
Returns: Returns:
Self: constructed instance of self Self: constructed instance of self
Raises:
EncodeError: if encoding is unknown
""" """
with path.open(encoding="utf8") as input_file: # read raw bytes from file
return cls.from_io(input_file) with path.open("rb") as input_file:
content = input_file.read()
# decode bytes content based on either
encoding = encoding or cls.DEFAULT_ENCODINGS
io = StringIO(content.decode(encoding, errors="backslashreplace"))
return cls.from_io(io)
@classmethod @classmethod
def from_io(cls, stream: IO[str]) -> Self: def from_io(cls, stream: IO[str]) -> Self:

View File

@ -27,13 +27,13 @@ class ReportSettings(StrEnum):
report targets enumeration report targets enumeration
Attributes: Attributes:
Disabled(ReportSettings): (class attribute) option which generates no report for testing purpose Disabled(ReportSettings): option which generates no report for testing purpose
HTML(ReportSettings): (class attribute) html report generation HTML(ReportSettings): html report generation
Email(ReportSettings): (class attribute) email report generation Email(ReportSettings): email report generation
Console(ReportSettings): (class attribute) print result to console Console(ReportSettings): print result to console
Telegram(ReportSettings): (class attribute) markdown report to telegram channel Telegram(ReportSettings): markdown report to telegram channel
RSS(ReportSettings): (class attribute) RSS report generation RSS(ReportSettings): RSS report generation
RemoteCall(ReportSettings): (class attribute) remote ahriman server call RemoteCall(ReportSettings): remote ahriman server call
""" """
Disabled = "disabled" # for testing purpose Disabled = "disabled" # for testing purpose

View File

@ -97,3 +97,12 @@ class RepositoryId:
raise ValueError(f"'<' not supported between instances of '{type(self)}' and '{type(other)}'") raise ValueError(f"'<' not supported between instances of '{type(self)}' and '{type(other)}'")
return (self.name, self.architecture) < (other.name, other.architecture) return (self.name, self.architecture) < (other.name, other.architecture)
def __str__(self) -> str:
"""
string representation of the repository identifier
Returns:
str: string view of the repository identifier
"""
return f"{self.name} ({self.architecture})"

View File

@ -0,0 +1,77 @@
#
# Copyright (c) 2021-2025 ahriman team.
#
# This file is part of ahriman
# (see https://github.com/arcan1s/ahriman).
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
from dataclasses import dataclass, fields
from typing import Any, Self
from ahriman.core.utils import filter_json
from ahriman.models.package import Package
@dataclass(frozen=True, kw_only=True)
class RepositoryStats:
"""
repository stats representation
"""
bases: int
packages: int
archive_size: int
installed_size: int
@classmethod
def from_json(cls, dump: dict[str, Any]) -> Self:
"""
construct counters from json dump
Args:
dump(dict[str, Any]): json dump body
Returns:
Self: status counters
"""
# filter to only known fields
known_fields = [pair.name for pair in fields(cls)]
return cls(**filter_json(dump, known_fields))
@classmethod
def from_packages(cls, packages: list[Package]) -> Self:
"""
construct statistics from list of repository packages
Args:
packages(list[Packages]): list of repository packages
Returns:
Self: constructed statistics object
"""
return cls(
bases=len(packages),
packages=sum(len(package.packages) for package in packages),
archive_size=sum(
archive.archive_size or 0
for package in packages
for archive in package.packages.values()
),
installed_size=sum(
archive.installed_size or 0
for package in packages
for archive in package.packages.values()
),
)

View File

@ -20,7 +20,7 @@
from __future__ import annotations from __future__ import annotations
from collections.abc import Callable, Iterable from collections.abc import Callable, Iterable
from typing import Any, Self from typing import Any, ClassVar, Self
from ahriman.models.package import Package from ahriman.models.package import Package
@ -33,7 +33,7 @@ class Result:
STATUS_PRIORITIES(list[str]): (class attribute) list of statues according to their priorities STATUS_PRIORITIES(list[str]): (class attribute) list of statues according to their priorities
""" """
STATUS_PRIORITIES = [ STATUS_PRIORITIES: ClassVar[list[str]] = [
"failed", "failed",
"removed", "removed",
"updated", "updated",

View File

@ -0,0 +1,104 @@
#
# Copyright (c) 2021-2025 ahriman team.
#
# This file is part of ahriman
# (see https://github.com/arcan1s/ahriman).
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
import statistics
from dataclasses import dataclass
@dataclass(frozen=True)
class SeriesStatistics:
"""
series statistics helper
Attributes:
series(list[float | int]): list of values to be processed
"""
series: list[float | int]
@property
def max(self) -> float | int | None:
"""
get max value in series
Returns:
float | int | None: ``None`` if series is empty and maximal value otherwise``
"""
if self:
return max(self.series)
return None
@property
def mean(self) -> float | int | None:
"""
get mean value in series
Returns:
float | int | None: ``None`` if series is empty and mean value otherwise
"""
if self:
return statistics.mean(self.series)
return None
@property
def min(self) -> float | int | None:
"""
get min value in series
Returns:
float | int | None: ``None`` if series is empty and minimal value otherwise
"""
if self:
return min(self.series)
return None
@property
def st_dev(self) -> float | None:
"""
get standard deviation in series
Returns:
float | None: ``None`` if series size is less than 1, 0 if series contains single element and standard
deviation otherwise
"""
if not self:
return None
if len(self.series) > 1:
return statistics.stdev(self.series)
return 0.0
@property
def total(self) -> int:
"""
retrieve amount of elements
Returns:
int: the series collection size
"""
return len(self.series)
def __bool__(self) -> bool:
"""
check if series is empty or not
Returns:
bool: ``True`` if series contains elements and ``False`` otherwise
"""
return bool(self.total)

View File

@ -27,9 +27,9 @@ class SignSettings(StrEnum):
sign targets enumeration sign targets enumeration
Attributes: Attributes:
Disabled(SignSettings): (class attribute) option which generates no report for testing purpose Disabled(SignSettings): option which generates no report for testing purpose
Packages(SignSettings): (class attribute) sign each package Packages(SignSettings): sign each package
Repository(SignSettings): (class attribute) sign repository database file Repository(SignSettings): sign repository database file
""" """
Disabled = "disabled" Disabled = "disabled"

View File

@ -27,9 +27,9 @@ class SmtpSSLSettings(StrEnum):
SMTP SSL mode enumeration SMTP SSL mode enumeration
Attributes: Attributes:
Disabled(SmtpSSLSettings): (class attribute) no SSL enabled Disabled(SmtpSSLSettings): no SSL enabled
SSL(SmtpSSLSettings): (class attribute) use SMTP_SSL instead of normal SMTP client SSL(SmtpSSLSettings): use SMTP_SSL instead of normal SMTP client
STARTTLS(SmtpSSLSettings): (class attribute) use STARTTLS in normal SMTP client STARTTLS(SmtpSSLSettings): use STARTTLS in normal SMTP client
""" """
Disabled = "disabled" Disabled = "disabled"

View File

@ -27,11 +27,11 @@ class UploadSettings(StrEnum):
remote synchronization targets enumeration remote synchronization targets enumeration
Attributes: Attributes:
Disabled(UploadSettings): (class attribute) no sync will be performed, required for testing purpose Disabled(UploadSettings): no sync will be performed, required for testing purpose
Rsync(UploadSettings): (class attribute) sync via rsync Rsync(UploadSettings): sync via rsync
S3(UploadSettings): (class attribute) sync to Amazon S3 S3(UploadSettings): sync to Amazon S3
GitHub(UploadSettings): (class attribute) sync to GitHub releases page GitHub(UploadSettings): sync to GitHub releases page
RemoteService(UploadSettings): (class attribute) sync to another ahriman instance RemoteService(UploadSettings): sync to another ahriman instance
""" """
Disabled = "disabled" # for testing purpose Disabled = "disabled" # for testing purpose

Some files were not shown because too many files have changed in this diff Show More