arcanis/ahriman
1
0
Fork 0
mirror of https://github.com/arcan1s/ahriman.git synced 2025-04-24 07:17:17 +00:00
Code Issues Projects Releases Wiki Activity

allow cross reference in the configuration

Browse Source
This commit is contained in:
Evgenii Alekseev 2024-08-29 18:25:53 +03:00
parent a23a1bc613
commit 1669d60026
14 changed files with 226 additions and 36 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
CONTRIBUTING.md
View File

@ -36,6 +36,7 @@ Again, the most checks can be performed by `tox` command, though some additional
Notes:
Very important note about this function
Probably multi-line
Args:
argument(str): an argument. This argument has
@ -70,6 +71,7 @@ Again, the most checks can be performed by `tox` command, though some additional
Attributes:
CLAZZ_ATTRIBUTE(int): (class attribute) a brand-new class attribute
instance_attribute(str): an instance attribute
with the long description
Examples:
Very informative class usage example, e.g.::

3
docker/entrypoint.sh
View File

@ -8,9 +8,6 @@ cat <<EOF > "/etc/ahriman.ini.d/00-docker.ini"
[repository]
root = $AHRIMAN_REPOSITORY_ROOT
[settings]
database = $AHRIMAN_REPOSITORY_ROOT/ahriman.db
[web]
host = $AHRIMAN_HOST

8
docs/ahriman.core.report.rst
View File

@ -60,6 +60,14 @@ ahriman.core.report.report\_trigger module
:no-undoc-members:
:show-inheritance:
ahriman.core.report.rss module
------------------------------
.. automodule:: ahriman.core.report.rss
:members:
:no-undoc-members:
:show-inheritance:
ahriman.core.report.telegram module
-----------------------------------

23
docs/configuration.rst
View File

@ -17,14 +17,33 @@ There are two variable types which have been added to default ones, they are pat
Path values, except for casting to ``pathlib.Path`` type, will be also expanded to absolute paths relative to the configuration path. E.g. if path is set to ``ahriman.ini.d/logging.ini`` and root configuration path is ``/etc/ahriman.ini``, the value will be expanded to ``/etc/ahriman.ini.d/logging.ini``. In order to disable path expand, use the full path, e.g. ``/etc/ahriman.ini.d/logging.ini``.
Configuration allows string interpolation from environment variables, e.g.:
Configuration allows string interpolation from the same configuration file, e.g.:
.. code-block:: ini
[section]
key = ${anoher_key}
another_key = value
will read value for the ``section.key`` option from ``section.another_key``. In case if the cross-section reference is required, the ``${section:another_key}`` notation must be used. It also allows string interpolation from environment variables, e.g.:
.. code-block:: ini
[section]
key = $SECRET
will try to read value from ``SECRET`` environment variable. In case if the required environment variable wasn't found, it will keep original value (i.e. ``$SECRET`` in the example). Dollar sign can be set as ``$$``.
will try to read value from ``SECRET`` environment variable. In case if the required environment variable wasn't found, it will keep original value (i.e. ``$SECRET`` in the example). Dollar sign can be set as ``$$``. All those interpolations will be applied in succession and - expected to be - recursively, e.g. the following configuration:
.. code-block:: ini
[section1]
key = ${section2:key}
[section2]
key = ${home}
home = $HOME
will eventually lead ``section1.key`` option to be set to the value of ``HOME`` environment variable (if available).
There is also additional subcommand which will allow to validate configuration and print found errors. In order to do so, run ``service-config-validate`` subcommand, e.g.:

2
docs/faq/distributed.rst
View File

@ -292,7 +292,7 @@ Worker nodes (applicable for all workers) config (``worker.ini``) as:
Command to run worker nodes (considering there will be two workers, one is on ``8081`` port and other is on ``8082``):
.. code-block:: ini
.. code-block:: shell
docker run --privileged -p 8081:8081 -e AHRIMAN_PORT=8081 -v worker.ini:/etc/ahriman.ini.d/overrides.ini arcan1s/ahriman:latest web
docker run --privileged -p 8082:8082 -e AHRIMAN_PORT=8082 -v worker.ini:/etc/ahriman.ini.d/overrides.ini arcan1s/ahriman:latest web

2
docs/faq/reporting.rst
View File

@ -47,7 +47,7 @@ How to generate index page for S3
target = html
[html]
path = /var/lib/ahriman/repository/aur-clone/x86_64/index.html
path = ${repository:root}/repository/aur-clone/x86_64/index.html
link_path = http://example.com/aur-clone/x86_64
After these steps ``index.html`` file will be automatically synced to S3.

14
package/share/ahriman/settings/ahriman.ini
View File

@ -6,7 +6,7 @@ logging = ahriman.ini.d/logging.ini
; Perform database migrations on the application start. Do not touch this option unless you know what are you doing.
;apply_migrations = yes
; Path to the application SQLite database.
database = /var/lib/ahriman/ahriman.db
database = ${repository:root}/ahriman.db
[alpm]
; Path to pacman system database cache.
@ -119,9 +119,9 @@ host = 127.0.0.1
; Disable status (e.g. package status, logs, etc) endpoints. Useful for build only modes.
;service_only = no
; Path to directory with static files.
static_path = /usr/share/ahriman/templates/static
static_path = ${templates}/static
; List of directories with templates.
templates = /usr/share/ahriman/templates
templates = ${prefix}/share/ahriman/templates
; Path to unix socket. If none set, unix socket will be disabled.
;unix_socket =
; Allow unix socket to be world readable.
@ -246,7 +246,7 @@ template = email-index.jinja2
; Template name to be used for full packages list generation (same as HTML report).
;template_full =
; List of directories with templates.
templates = /usr/share/ahriman/templates
templates = ${prefix}/share/ahriman/templates
; SMTP user.
;user =
@ -265,7 +265,7 @@ templates = /usr/share/ahriman/templates
; Template name to be used.
template = repo-index.jinja2
; List of directories with templates.
templates = /usr/share/ahriman/templates
templates = ${prefix}/share/ahriman/templates
; Remote service callback trigger configuration sample.
[remote-call]
@ -295,7 +295,7 @@ templates = /usr/share/ahriman/templates
; Template name to be used.
template = rss.jinja2
; List of directories with templates.
templates = /usr/share/ahriman/templates
templates = ${prefix}/share/ahriman/templates
; Telegram reporting trigger configuration sample.
[telegram]
@ -316,7 +316,7 @@ template = telegram-index.jinja2
; Telegram specific template mode, one of MarkdownV2, HTML or Markdown.
;template_type = HTML
; List of directories with templates.
templates = /usr/share/ahriman/templates
templates = ${prefix}/share/ahriman/templates
; HTTP request timeout in seconds.
;timeout = 30

2
recipes/index/service.ini
View File

@ -2,5 +2,5 @@
target = html
[html]
path = /var/lib/ahriman/ahriman/repository/ahriman-demo/x86_64/index.html
path = ${repository:root}/repository/ahriman-demo/x86_64/index.html
link_path = http://localhost:8080/repo/ahriman-demo/x86_64

3
src/ahriman/application/application/application.py
View File

@ -120,8 +120,7 @@ class Application(ApplicationPackages, ApplicationRepository):
process_dependencies(bool): if no set, dependencies will not be processed
Returns:
list[Package]: updated packages list. Packager for dependencies will be copied from
original package
list[Package]: updated packages list. Packager for dependencies will be copied from the original package
Examples:
In the most cases, in order to avoid build failure, it is required to add missing packages, which can be

2
src/ahriman/core/build_tools/package_archive.py
View File

@ -236,7 +236,7 @@ class PackageArchive:
extract list of the installed packages and their content
Returns:
dict[str, FilesystemPackage]; map of package name to list of directories and files contained
dict[str, FilesystemPackage]: map of package name to list of directories and files contained
by this package
"""
result = {}

91
src/ahriman/core/configuration/shell_interpolator.py
View File

@ -19,21 +19,93 @@
#
import configparser
import os
import sys
from collections.abc import Mapping, MutableMapping
from collections.abc import Generator, Mapping, MutableMapping
from string import Template
class ExtendedTemplate(Template):
"""
extension to the default :class:`Template` class, which also enabled braces regex to lookup in sections
Attributes:
braceidpattern(str): regular expression to match a colon inside braces
"""
braceidpattern = r"(?a:[_a-z0-9][_a-z0-9:]*)"
class ShellInterpolator(configparser.Interpolation):
"""
custom string interpolator, because we cannot use defaults argument due to config validation
"""
DATA_LINK_ESCAPE = "\x10"
@staticmethod
def _extract_variables(parser: MutableMapping[str, Mapping[str, str]], value: str,
defaults: Mapping[str, str]) -> Generator[tuple[str, str], None, None]:
"""
extract keys and values (if available) from the configuration. In case if a key is not available, it will be
silently skipped from the result
Args:
parser(MutableMapping[str, Mapping[str, str]]): option parser
value(str): source (not-converted) value
defaults(Mapping[str, str]): default values
Yields:
tuple[str, str]: variable name used for substitution and its value
"""
def identifiers() -> Generator[tuple[str | None, str], None, None]:
# extract all found identifiers and parse them
for identifier in ExtendedTemplate(value).get_identifiers():
match identifier.split(":"):
case [lookup_option]: # single option from the same section
yield None, lookup_option
case [lookup_section, lookup_option]: # reference to another section
yield lookup_section, lookup_option
for section, option in identifiers():
# key to be substituted
key = f"{section}:{option}" if section else option
if section is not None: # foreign section case
if section not in parser:
continue # section was not found, silently skip it
values = parser[section]
else: # same section
values = defaults
if (raw := values.get(option)) is not None:
yield key, raw
@staticmethod
def environment() -> dict[str, str]:
"""
extract environment variables
Returns:
dict[str, str]: environment variables and some custom variables
"""
return os.environ | {
"prefix": sys.prefix,
}
def before_get(self, parser: MutableMapping[str, Mapping[str, str]], section: str, option: str, value: str,
defaults: Mapping[str, str]) -> str:
"""
interpolate option value
Notes:
This method is using :class:`string.Template` class in order to render both cross-references and
environment variables, because it seems that it is the most legit way to handle it. In addition,
we are using shell-like variables in some cases (see :attr:`alpm.mirror` option), thus we would like
to keep them alive.
First this method resolves substitution from the configuration and then renders environment variables
Args:
parser(MutableMapping[str, Mapping[str, str]]): option parser
section(str): section name
@ -44,8 +116,15 @@ class ShellInterpolator(configparser.Interpolation):
Returns:
str: substituted value
"""
# At the moment it seems that it is the most legit way to handle environment variables
# Template behaviour is literally the same as shell
# In addition, we are using shell-like variables in some cases (see :attr:`alpm.mirror` option),
# thus we would like to keep them alive
return Template(value).safe_substitute(os.environ)
# because any substitution effectively replace escaped $ ($$) in result, we have to escape it manually
escaped = value.replace("$$", self.DATA_LINK_ESCAPE)
# resolve internal references
variables = dict(self._extract_variables(parser, value, defaults))
internal = ExtendedTemplate(escaped).safe_substitute(variables)
# resolve enriched environment variables by using default Template class
environment = Template(internal).safe_substitute(self.environment())
# replace escaped values back
return environment.replace(self.DATA_LINK_ESCAPE, "$")

18
src/ahriman/models/package.py
View File

@ -161,10 +161,10 @@ class Package(LazyLogging):
bool: ``True`` in case if package base looks like VCS package and ``False`` otherwise
"""
return self.base.endswith("-bzr") \
or self.base.endswith("-csv")\
or self.base.endswith("-darcs")\
or self.base.endswith("-git")\
or self.base.endswith("-hg")\
or self.base.endswith("-csv") \
or self.base.endswith("-darcs") \
or self.base.endswith("-git") \
or self.base.endswith("-hg") \
or self.base.endswith("-svn")
@property
@ -358,7 +358,7 @@ class Package(LazyLogging):
Yields:
Path: list of paths of files which belong to the package and distributed together with this tarball.
All paths are relative to the ``path``
All paths are relative to the ``path``
Raises:
PackageInfoError: if there are parsing errors
@ -504,8 +504,8 @@ class Package(LazyLogging):
timestamp(float | int): timestamp to check build date against
Returns:
bool: ``True`` in case if package was built after the specified date and ``False`` otherwise. In case if build date
is not set by any of packages, it returns False
bool: ``True`` in case if package was built after the specified date and ``False`` otherwise.
In case if build date is not set by any of packages, it returns False
"""
return any(
package.build_date > timestamp
@ -550,8 +550,8 @@ class Package(LazyLogging):
Returns:
str | None: new generated package release version if any. In case if the release contains dot (e.g. 1.2),
the minor part will be incremented by 1. If the release does not contain major.minor notation, the minor
version equals to 1 will be appended
the minor part will be incremented by 1. If the release does not contain major.minor notation, the minor
version equals to 1 will be appended
"""
if local_version is None:
return None # local version not found, keep upstream pkgrel

2
src/ahriman/models/waiter.py
View File

@ -97,7 +97,7 @@ class Waiter:
Attributes:
interval(float): interval in seconds between checks
wait_timeout(float): timeout in seconds to wait for. Negative value will result in immediate exit. Zero value
means infinite timeout
means infinite timeout
"""
wait_timeout: float

90
tests/ahriman/core/configuration/test_shell_interpolator.py
View File

@ -1,6 +1,65 @@
import os
from ahriman.core.configuration.shell_interpolator import ShellInterpolator
from ahriman.core.configuration import Configuration
from ahriman.core.configuration.shell_interpolator import ExtendedTemplate, ShellInterpolator
def _parser() -> dict[str, dict[str, str]]:
"""
parser mock
Returns:
dict[str, dict[str, str]]: options to be used as configparser mock
"""
return {
"section1": {
"home": "$HOME",
"key1": "value1",
"key4": "${home}",
},
"section2": {
"key2": "value2",
},
"section3": {
"key3": "${section1:home}",
"key5": "${section1:key4}",
},
}
def test_extended_template() -> None:
"""
must match colons in braces
"""
assert ExtendedTemplate("$key:value").get_identifiers() == ["key"]
assert ExtendedTemplate("${key:value}").get_identifiers() == ["key:value"]
def test_extract_variables() -> None:
"""
must extract variables list
"""
parser = _parser()
assert dict(ShellInterpolator._extract_variables(parser, "${key1}", parser["section1"])) == {
"key1": "value1",
}
assert not dict(ShellInterpolator._extract_variables(parser, "${key2}", parser["section1"]))
assert dict(ShellInterpolator._extract_variables(parser, "${section2:key2}", parser["section1"])) == {
"section2:key2": "value2",
}
assert not dict(ShellInterpolator._extract_variables(parser, "${section2:key1}", parser["section1"]))
assert not dict(ShellInterpolator._extract_variables(parser, "${section4:key1}", parser["section1"]))
def test_environment() -> None:
"""
must extend environment variables
"""
assert "HOME" in ShellInterpolator.environment()
assert "prefix" in ShellInterpolator.environment()
def test_before_get() -> None:
@ -8,8 +67,35 @@ def test_before_get() -> None:
must correctly extract environment variables
"""
interpolator = ShellInterpolator()
assert interpolator.before_get({}, "", "", "value", {}) == "value"
assert interpolator.before_get({}, "", "", "$value", {}) == "$value"
assert interpolator.before_get({}, "", "", "$HOME", {}) == os.environ["HOME"]
def test_before_get_escaped() -> None:
"""
must correctly read escaped variables
"""
interpolator = ShellInterpolator()
assert interpolator.before_get({}, "", "", "$$HOME", {}) == "$HOME"
def test_before_get_reference() -> None:
"""
must correctly extract environment variables after resolving cross-reference
"""
interpolator = ShellInterpolator()
assert interpolator.before_get(_parser(), "", "", "${section1:home}", {}) == os.environ["HOME"]
def test_before_get_reference_recursive(configuration: Configuration) -> None:
"""
must correctly extract environment variables after resolving cross-reference recursively
"""
interpolator = ShellInterpolator()
for section, items in _parser().items():
for option, value in items.items():
configuration.set_option(section, option, value)
assert interpolator.before_get(configuration, "", "", "${section1:home}", {}) == os.environ["HOME"]
assert interpolator.before_get(configuration, "", "", "${section3:key5}", {}) == os.environ["HOME"]