passlib uses deprecated crypt module which is deprecated and scheduled
for removal in 3.13. Unfortunately, this module seems to be
unmaintained, so this commit replaces passlib with bcrypt, unfortunately
breaking current passwords
It has been found that there are two cases in which pkgbuild was not
parsed correctly
1. Major case in which there is quotation mark inside comment line,
which would cause ValueError: No closing quotation error
2. Minor case, if there are utf symbols in pkgbuild file (e.g.
hieroglyphs, see ttf-google-fonts-git), it will case incorrect
reading in `_is_escaped` method
In case of VCS packages, if PKGBUILD contains older version, the pkgrel
remains the same during the rebuild process. This fix bumps pkgrel in
any case if the local version is newer than the remote
ordering
So basically initial implementation, with limit=1, would emit the oldest
record in series. New implementation will return the most recent one
instead
The response is still sorted by ascension
It has been found that previous system didn't allow to configure
specific cases (e.g. a whitelisted directory inside /usr/lib/cmake). The
current solution replaces two options to single one, which also allows a
regular expressions
Also PackageArchive class has been moved to core package, because it is
more about service rather than model
It has been broken since reporter improvements, because it effectivelly
1) didn't call remove functions in database
2) used empty repository identifier for web service
With those changes it also raises exception when you try to call id on
empty identifier
It has been found that in some cases additional packages have been added
as dependencies, like usr/share/applications, usr/lib/cmake, etc
This commit adds an ability to blacklist specific paths from processing
Add naive implementation of user password check by calling su command.
Also change some authentication method to require username to be string
instead of optional string
The issue appears in case if versions ar the same (e.g. rebuild); in
this case printer doesn't increment version as builder does.
Also util has been renamed to utils, keeping backward compatibiltiy
Instead of trying to load every database and look for files, this commit
introduces the optimization in which, the service loads packages first,
groups them by database and load files later.
In some cases it significantly descreases times for loading files
This mr improves implicit dependencies processing by reducing tree leaves by using the following algorithm:
* remove paths which belong to any base package
* remove packages which are (opt)dependencies of one of the package which provides same path. It also tries to handle circular dependencies by excluding them from being "satisfied"
* remove packages which are already satisfied by any children path
* implement local reporter mode
* simplify watcher class
* review changes
* do not update unknown status
* allow empty key patches via api
* fix some pylint warnings in tests
Initial implementation requires explicit context key name to be set.
Though it is still useful sometimes (e.g. if there should be two
variables with the same type), in the most used scenarios internally
only type is required. This commit extends set and get methods to allow
to construct ContextKey from type directly
Also it breaks old keys, since - in order to reduce amount of possible
mistakes - internal classes uses this generation method
In some cases (probably slow internet) in place initialization can cause
exception, because elements are not available yet. This commit moves
events initialization to $()
The feature is implemented as supplying !debug option to makepkg when
generating package list. In this case debug packages still will be
built, however, they will not be added to the repository
Previous improvements raise 404 error in case if no packages were found
for patches endpoints. However, in case of multirepo setup this feature
doesn't work properly because package can be located in any other
repository different from default
String catenation used for url generators didn't encode package names
which could lead to missing data in case if e.g. there is slash (/) in
package name
It is fine when application is able to log request, however, normally it
produces a lot noise, which has been handled by adding special logger.
However, nowadays it requires a lot endpoints to be filtered and doesn't
provide any choice.
Instead of it lets disable access logger by default and let users decide
do they need or not to see access log messages
In the most cases it was enough to just add lock. In case of worker
trigger, since there is atomic operation on timer, it was also required
to add queue (coz python doesn't have atomics)
Some parsers are shared between different subcommands. It causes errors
when wwe add new arguments to one of them. This commit adds some tests
to cover those cases (except for well-known differencies)
Add support of changes generation. Changes will be generated (unless explicitly asked not to) automatically during check process (i.e. `repo-update --dry-run` and aliases) and uploaded to the remote server. Changes can be reviewed either by web interface or by special subcommands.
Changes will be automatically cleared during next successful build
Old solution causes amount of thread to be growing as well as stack is
increased during each iteration. Instead of cycle-free implementation,
this commit just uses while cycle
It has been a while since all pages have moved to json instead of form
data, except for login page. This commit changes login to json data
instead of form one
Instead of automatic package addition now it is required to add package
manually after clone. Less magic, plus would allow to use caches for
multi-repo setup (see #109)
* Allow to use single web instance for any repository
* some improvements
* drop includes from user home directory, introduce new variables to docker
The old solution didn't actually work as expected, because devtools
configuration belongs to filesystem (as well as sudo one), so it was
still required to run setup command.
In order to handle additional repositories, the POSTSETUP and PRESETUP
commands variables have been introduced. FAQ has been updated as well
* raise 404 in case if repository is unknown
Earlier applied fix bc9682373d introduced
errors with interaction, because (in docker container) HOME variable was
passed also to subprocesses. This fix limits variables to be passed to
the whitelisted ones
* allow to use one application for multiple repositories
* update tests
* handle None append argument everywhere
* rewrite repository definition logic
* drop optional flags from docs
* support of new schema in systemd units
* add migration docs and ability to migrate tree automatically
* use repostory id instead
* verbose multiarchitectureerror
* object path support for s3 sync
* fix tests after rebase
In some cases for better readability of logs, exceptions are now raised
without parent exception stacktrace. Also updated docs and contributing
guidelines
* add support of remote task tracking
* add remote call trigger implementation
* docs update
* add cross-service upload
* add notes about user
* add more ability to control upload
* multipart upload with signatures as well as safe file save
* configuration reference update
* rename watcher methods
* erase logs based on current package version
Old implementation has used process id instead, but it leads to log
removal in case of remote process trigger
* add --server flag for setup command
* restore behavior of the httploghandler
This field is required in order to pass config validation in case if
section name differs from default one. Also by default keyring_generator
and mirrorlist_generator have been renamed to keyring-generator and
mirrorlist-generator respectively for consistence
It makes sense to read some values from environment. In particular this
feature is useful in case of running application in containers in ci/cd
See #108 for more details
Used implementation of the hasher includes salt itself, thus additional
salt is optional and can be safely (in terms of security) treat as empty
string
Since llast upgrade build is broken. Lets fully migrate to
pyproject.toml. Note for maintaners: because data_files option is
deprectated (see https://github.com/pypa/setuptools/discussions/2648)
you will have to install files manually inside your packaging process
The new --(no-)increment flag has been added to add, update and rebuild
subcommands. In case if it is true and package version is the same as in
repository, it will automatically bump pkgrel appending (increasing)
minor part of it (e.g. 1.0.0-1 -> 1.0.0-1.1).
Inn order to implement this, the shadow (e.g. it will not store it in
database) patch for pkgrel will be created
* The issue appears when repository contains PKGBUILD in root. In this
case it will copy tree with loosing package information, because
the repository will be cloned to temporary path with random generated
name
* The issue appears when branch which is different from master is used
for any reposittory with git files (e.g. single-pkgbuild repo or repo
with submodules)
The main reason for having shell handler is to be able to fix if
something (e.g. migrations) goes wrong. In this way we need to reduce
actions inside this wrapper
This feature sometimes causes the main process termination. Since the
child process has been already terminated (or going to) there is no need
to terminate it manually
This migration includes
* removal of community repository as it is no more
* fixed paths for devtools configurations
* migration of archlinux packaging git url
Original implementation sends requests to httpbin which sometimes might
not be available. With proposed changes we are blocking redirects and
just check request itself
In previous revisions server was terminated by itself, thus no lock or
socket was removed. In new version, graceful termination of the queue
has been added as well as server now handles singals
Old implementation has used add step in order to fetch dependencies,
which could lead to build errors in case if dependency list was updated.
New solution uses dependencies which are declared at current version and
fetch them (if required and if enabled) before update process.
Closes#90
The newest mypy produces the following warning:
src/ahriman/application/handlers/search.py:43: error: Non-overlapping identity check (left operand type: "Union[_DefaultFactory[Any], Literal[_MISSING_TYPE.MISSING]]", right operand type: "Type[List[Any]]") [comparison-overlap]
which is more likely caused by updated dataclass models to protoocol (however decorators are still calllable). This commit masks problematic line from checking
This change requires srcinfo at least 0.1.2 version. Unfortunatelly aur
api don't support architecture specific arrays for now, so we just leave
it as is
Closes#82
In some cases (e.g. during addition of the package to build queue) we don't have
full information about package inself; in these cases we produce lines
with empty architecture, which duplicates normal ones.
This commit changes architecture column type to required and also
filters packages which don't have architecture set yet.
Closes#83
This commit also extends configuration of the multilib option, adding
the ability to exlcude multilib repository from repositories list
Note, that in order to support repository list and mirror correctly,
alpm configuration section is now architectture specific
Some commands have been moved to another group and thus having another
default name (old subcommands are still available...for now):
* daemon -> repo-daemon
* key-import -> service-key-import
* repo-clean -> service-clean
* repo-config -> service-config
* repo-config-validate -> service-config-validate
* repo-setup -> service-setup
* repo-shell -> service-shell
* version -> help-version
Note that this commit contains the following breaking changes:
* remote pull and remote push triggers are now enabled by default (with
empty target list)
* remote pull and remote push triggers now require target option to be
set (old behaviour had fallback on `gitremote`)
* validation is now considered to be stable, so it is enabled by default
in docker image (can be disabled however)
Some commands were made unsafe in old versions, but nowadays they can be
run without having special privileges.
There was also a bug in which status commands were not available if you
are not ahriman user and unix socket is used. It has been fixed by
switching to manual socket creation (see also
https://github.com/aio-libs/aiohttp/issues/4155)
This reverts commit 11732a8609.
Original solution has introduced special workaround (strict flag) which
contradicts the concept of immutable context. Moreover, it introduces
possible side-effects, because child process will use the one set by
parent instead of having own one.
The correct solution is to re-create context in process entry point
Sorry, it was Jan 1 and I was drunk :(
In case of immediate handle load it would try to sync databases (or at
least to create database files), which is not possible in case if
command is run as non-ahriman user. This commit makes handle load lazy
and allows to run some commands as non-ahriman user
* Unlike older version, currently service will always try to pull AUR
package to check version. Previously if no-vcs flag is set, it would
ignore VCS packages completelly
* Introduce build.vcs_allowed_age option. If set, it will skip version
calculation if package age (now - build_date) is less than this value
* Improve some wording (again)
* Change default type for refresh option to False (does not affect
behavior)
* Update docstrings to reflect last changes
* Configuration.__convert_path has been replaced by shlex
* aiosecurity functions support kwargs now
This feature can be used for unauthorized access to apis - e.g. for
reporting service if it is run on the same machine. Since now it becomes
recommended way for the interprocess communication, thus some options
(e.g. creating user with as-service flag) are no longer available now
In case if password is asked via getpass, it is possible to make typo
and user will not see the mistake. In order to avoid it, additional
confirmation has been added
* implement log storage at backend
* handle process id during removal. During one process we can write logs from different packages in different times (e.g. check and update later) and we would like to store all logs belong to the same process
* set package context in main functions
* implement logs support in interface
* filter out logs posting http logs
* add timestamp to log records
* hide getting logs under reporter permission
List of breaking changes:
* `ahriman.core.lazy_logging.LazyLogging` has been renamed to `ahriman.core.log.LazyLogging`
* `ahriman.core.configuration.Configuration.from_path` does not have `quiet` attribute now
* `ahriman.core.configuration.Configuration` class does not have `load_logging` method now
* `ahriman.core.status.client.Client.load` requires `report` argument now
The issue appears together with --intent-to-add flag for adding new
files. Original testing has been performed by having already added new
files, thus it passed all checks.
This commit also adds `commit_author` option which will allow to
overwrite the author.
Old logic used OR condition, i.e. if set from-database, it would ignore
the --depends-on flag. In new logic it calculates dependencies based on
the package list, which can be retrieved from database
By default this feature is enabled. On the first run it will copy (if
exists) databases from filesystem to local cache (one per each
architecture). Later it will use this cache for all alpm operations. In
order to update this cache, some commands (mainly package building)
provide `-y`/`--refresh` option which has same semantics as pacman -Sy
does.
Note however that due to extending `Pacman` class some methods were
renamed in order to be more descriptive:
* `Pacman.all_packages` -> `Pacman.packages`
* `Pacman.get` -> `Pacman.package_get`
This commit also adds multilib repository to the default docker image
which was missed.
* Move devtools executable to ahriman home, because we don't really
need to use executable inside root
* Use named sudoers file instead of single file. It will allow easily to
remove file as well as use setup command for several
repositories/architectures
In order to force new triggers to use on_result method, the old method
has been removed. However, default on_result method still checks if the
old method exists and tries to run it
This option could lead to missing warnings about missing or invalid
configuration values because code usually expects that values are exists
and not empty unless it is explicitly specified.
However, pacman configuration still requires this option in order to be
able to deal with boolean values
The issue appears when there is no boto, jinja and some other libraries
are not installed because the classes which use these libraries are
still being imported inside the package file. The fix removes those
imports from package root, because they should not be here, in fact,
content of report and upload packages must be imported only inside the
trigger class and only if they are actually required
This commit also adds setuptools as required dependency since it is used
for some parsers (previously it was provided dependency)
(without special settings)
The issue appears if file or its version contains one of special URI
characters, e.g. +. Theu will be interpreted as query parameters by
(some) servers (e.g. S3 works in this way). In this commit we rename
archive to the one with safe name.
* migrate docstrings from reST to google format
* add raises note
Also change behaviour of the `from_option` method to fallback to
disabled instead of raising exception on unknown option
* fix part of warnings for sphinx
* make identation a bit more readable
* review fixes
* add verbose description for properties to make them parsed by sphinx extenstion
* add demo sphinx generator
Old versions cached full output to memory and only after that printed it
into log. This behaviour causes issues in case if operation stucks and
you would need to find the step at which it does. New check_output
method uses Popen directly and iterates over stdout lines
Also changed behaviour from merging stderr into stdout to separate
stderr logging.
Any other behaviour of the function must be the same.
Also changed GPG.key_import method to disable local signing since it
seems it is useless (and may break process in case if there is no
private key)
this also requires to move default configuration files to share/ahriman.
Thus the following features have been added
* default configuration is not stored in /usr/share/ahriman/settings
* package installed via PKGBUILD now copies files from /usr
* configuration class now fallbacks to default in /usr
In current workflow you need to run setup to run init (because of
repository name), but you need to run init before setup (because of
repository tree rights).
New solution just add `Repo.init()` method call to setup subcommand
after config reload to make sure that repository name has been applied.
In addition chown method as well as setuid method for check_output have
been added.
* Allow spaces in lists. This feature has been done in the way as shell
interprets arguments by using quotation marks
* Clear current content on reload
This feature also introduces the followiing changes
* aur-search command now works as expected with multiterms
* printer classes for managing of data print
* --sort-by argument for aur-search subcommand instead of using package
name
* --quiet argument now has also --no-quite option
* if --quite is supplied, the log level will be set to warn instead of
critical to be able to see error messages
* pretty_datetime function now also supports datetime objects
* BuildStatus is now pure dataclass
In this feature target option must allways point to section name instead
of type. Type will be read from type option. In case if type option is
not presented it will try to check if section with architecture exists
(e.g. target = email, section = email:x86_64); if it does, the correct
section name and type will be used. Otherwise it will check if the
specified section exists; if it does, seection name and type will be
returned.
in order to sort method correctly we are going to use the following
namiing schema:
{subject}_{action}_{details}
This schema still have some exceptions, e.g. single word methods, bool
methods (is_) and getters in case if they are singular (i.e. there is
no any other method with this subject)
* add ability to add manually stored packages
* update tests
* handle manual packages in remove-unknown method
* live fixes
also rename branches to has_remotes method and change return type
move logic to separated shell scripts and also create shell script for
repository setup
Also force create directory according to systemd recommendations
according to the source code defaults always updates the values
dictionary. This in this specific case it is impossible to override the
value it will be always empty list.
In order to handle it we are adding another property to the Handler
class which allows to run with None architecture list.
This particular set_defaults behaviour is still useful for other cases
when we have to run command without any specific architecture
* add external process spawner and update test cases
* pass no_report to handlers
* provide service api endpoints
* do not spawn process for single architecture run
* pass no report to handlers
* make _call method of handlers public and also simplify process spawn
* move update under add
* implement actions from web page
* clear logging & improve l&f
* initial auth implementation
* add create user parser
* add tests
* update dependencies list
* add login annd logout to index also improve auth
* realworld fixes
* add method set_option to Configuration and also use it everywhere
* split CreateUser handler to additional read method
* check user duplicate on auth mapping read
* generate salt by using passlib instead of random.choice
* case-insensetive usernames
* update dependencies
* update configuration reference
* improve tests
* fix codefactor errors
* hide fields if authorization is enabled, but no auth supplied
* add settings object for auth provider
* readme update
* add init subcommand
* add also init command to repository object
* add ability to generate list of architectures
* check if architecture list is not empty
* import pgp key implementation
* do not ask confirmation for local sign. Also add argparser test
* superseed requests by python-aur package
* ...and drop --skippgpcheck makgepkg flag by default
General idea is to use classmethod for every constructor and
statismethod otherwise.
Also use self and cls whenever it's possible to call static and class
methods
* add models tests (#1)
also replace single quote to double one to confort PEP docstring
+ move _check_output to class properties to make it available for
mocking
* alpm tests implementation
* try to replace os with pathlib
* update tests for pathlib
* fix includes glob and trim version from dependencies
* build_tools package tests
* repository component tests
* add sign tests
* complete status tests
* handle exceptions in actual_version calls
* complete core tests
* move configuration to root conftest
* application tests
* complete application tests
* change copyright to more generic one
* base web tests
* complete web tests
* complete testkit
also add argument parsers test
2021-03-28 15:30:51 +03:00
29 changed files with 2247 additions and 11981 deletions
@ -12,8 +12,7 @@ Packages have strict rules of importing:
Full dependency diagram:
..image:: _static/architecture.svg
:target:_static/architecture.svg
..graphviz:: _static/architecture.dot
:alt:architecture
``ahriman.application`` package
@ -148,7 +147,7 @@ There are multiple subdirectories, some of them are commons for any repository,
*``pacman/{repository}/{architecture}`` is the repository and architecture specific caches for pacman's databases.
*``repository/{repository}/{architecture}`` is a repository packages directory.
Normally you should avoid direct interaction with the application tree. For tree migration process refer to the :doc:`migration notes <migration>`.
Normally you should avoid direct interaction with the application tree. For tree migration process refer to the :doc:`migration notes <migrations/index>`.
@ -148,13 +148,11 @@ Before using this command you will need to create local directory and put ``PKGB
How to copy package from another repository
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
As simple as add package from archive. Considering case when you would like to copy package ``package`` with version ``ver-rel`` from repository ``source-repository`` to ``target-respository`` (same architecture), the command will be following:
It is possible to copy package and its metadata between local repositories, optionally removing the source archive, e.g.:
Normally the most of migrations are handled automatically after application start, however, some upgrades require manual interventions; this document describes them.
Upgrades to breakpoints
-----------------------
To 2.9.0
^^^^^^^^
This release includes major upgrade for the newest devtools and archlinux repository structure. In order to upgrade package need to:
#. Upgrade to the latest major release of python (3.11) (required by other changes).
#. Upgrade devtools to the latest release.
#. Backup local settings, ``/etc/ahriman.ini.d/00-setup-overrides.ini`` by default.
#. Run setup command (i.e. ``ahriman service-setup``) again with the same arguments as used before. This step can be done manually by moving ``devtools`` configuration (something like ``/usr/share/devtools/pacman-ahriman*.conf``) to new location ``/usr/share/devtools/pacman.conf.d/`` under name ``ahriman.conf``. After that make sure to remove any ``community`` mentions from configurations (e.g. ``/usr/share/devtools/pacman.conf.d/ahriman.conf``, ``/etc/ahriman.ini``) if there were any. The only thing which will change is ``devtools`` configuration.
#. Remove build chroot as it is incompatible, e.g. ``sudo ahriman service-clean --chroot``.
#. Run ``sudo -u ahriman ahriman update --no-aur --no-local --no-manual -yy`` in order to update local databases.
To 2.12.0
^^^^^^^^^
---------
This release includes paths migration. Unlike usual case, no automatic migration is performed because it might break user configuration. The following noticeable changes have been made:
This release replaces ``passlib`` dependency with ``bcrypt``.
The reason behind this change is that python developers have deprecated and scheduled for removal ``crypt`` module, which is used by ``passlib``. (By the way, they recommend to use ``passlib`` as a replacement.) Unfortunately, it appears that ``passlib`` is unmaintained (see `the issue <https://foss.heptapod.net/python-libs/passlib/-/issues/187>`__), so the only solution is to migrate to anoher library.
Because passwords are stored as hashes, it is near to impossible to shadow change passwords in database, the manual intervention is required if:
#. Authentication is used.
#. Notification provider is ``configuration`` or a user with explicitly set password exists.
Manual steps might look as:
#. Get list of users with their roles ``ahriman user-list``.
#. For each user run update command, i.e. ``ahriman user-add <username> -R <role>``. Type password when it will be requested.
This release includes major upgrade for the newest devtools and archlinux repository structure. In order to upgrade package need to:
#. Upgrade to the latest major release of python (3.11) (required by other changes).
#. Upgrade devtools to the latest release.
#. Backup local settings, ``/etc/ahriman.ini.d/00-setup-overrides.ini`` by default.
#. Run setup command (i.e. ``ahriman service-setup``) again with the same arguments as used before. This step can be done manually by moving ``devtools`` configuration (something like ``/usr/share/devtools/pacman-ahriman*.conf``) to new location ``/usr/share/devtools/pacman.conf.d/`` under name ``ahriman.conf``. After that make sure to remove any ``community`` mentions from configurations (e.g. ``/usr/share/devtools/pacman.conf.d/ahriman.conf``, ``/etc/ahriman.ini``) if there were any. The only thing which will change is ``devtools`` configuration.
#. Remove build chroot as it is incompatible, e.g. ``sudo ahriman service-clean --chroot``.
#. Run ``sudo -u ahriman ahriman update --no-aur --no-local --no-manual -yy`` in order to update local databases.
Normally the most of migrations are handled automatically after application start, however, some upgrades require manual interventions; this document describes them.
find ../docs -type f -name "{[tox]project_name}*.rst" -delete
sphinx-apidoc -o ../docs .
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
