Commit Graph

361 Commits

Author SHA1 Message Date
fd3c6343f1 fix: explicitly process list of packages
Small workaround to remove debug packages from being processed
2024-08-14 17:07:10 +03:00
434057ec49 feat: remove excess dependencies leaves (#128)
This mr improves implicit dependencies processing by reducing tree leaves by using the following algorithm:

* remove paths which belong to any base package
* remove packages which are (opt)dependencies of one of the package which provides same path. It also tries to handle circular dependencies by excluding them from being "satisfied"
* remove packages which are already satisfied by any children path
2024-08-14 17:07:10 +03:00
a01b090c2b feat: improve lock mechanisms
* improve lock mechanisms

* use /run/ahriman for sockett

* better water
2024-08-14 17:07:10 +03:00
8ffc1299f0 feat: implement local reporter mode (#126)
* implement local reporter mode

* simplify watcher class

* review changes

* do not update unknown status

* allow empty key patches via api

* fix some pylint warnings in tests
2024-08-14 17:07:10 +03:00
02b13de7f4 feat: allow to use simplified keys for context
Initial implementation requires explicit context key name to be set.
Though it is still useful sometimes (e.g. if there should be two
variables with the same type), in the most used scenarios internally
only type is required. This commit extends set and get methods to allow
to construct ContextKey from type directly

Also it breaks old keys, since - in order to reduce amount of possible
mistakes - internal classes uses this generation method
2024-08-14 17:07:10 +03:00
50a045434d feat: add abillity to check broken dependencies (#122)
* implement elf dynamic linking check

* load local database too in pacman wrapper
2024-08-14 17:07:10 +03:00
f8b725d175 fix: parse array variable from command 2024-05-09 13:21:42 +03:00
3cd9cff4ee refactor: update code to the latest python (3.12+) 2024-05-05 21:17:30 +03:00
9e02d7a38f fix: remove debug packages together with normal ones (#124) 2024-05-05 21:17:30 +03:00
99ca0cb2fd fix: update Repo.init to the latest pacman release 2024-04-04 13:16:05 +03:00
855b55237b feat: add ability to disable debug packages distribution
The feature is implemented as supplying !debug option to makepkg when
generating package list. In this case debug packages still will be
built, however, they will not be added to the repository
2024-04-04 13:14:17 +03:00
f68e48fb47 refactor: rename packages http methods to own package
docs: update docs import
2024-01-22 02:20:11 +02:00
91598951c5 fix: do not raise 404 in case of unknown package on patches endpoints
Previous improvements raise 404 error in case if no packages were found
for patches endpoints. However, in case of multirepo setup this feature
doesn't work properly because package can be located in any other
repository different from default
2024-01-08 14:32:40 +02:00
29bb6284b0 test: add pytlint imports plugin and fix errors 2024-01-05 19:52:51 +02:00
70b7fcf47a refactor: fix pylint warnings in tests 2024-01-05 16:40:38 +02:00
706808fc97 refactor: simplify lock processing in worker trigger 2024-01-05 16:11:32 +02:00
56e97040d6 feat: read username if email is not available for oauth provider
Also add recipe for OAuth with GitHub setup
2024-01-05 12:28:31 +02:00
6eeef39fe5 fix: safe urls for packages
String catenation used for url generators didn't encode package names
which could lead to missing data in case if e.g. there is slash (/) in
package name
2024-01-03 14:28:20 +02:00
0e6434faad refactor: remove custom access logger
It is fine when application is able to log request, however, normally it
produces a lot noise, which has been handled by adding special logger.
However, nowadays it requires a lot endpoints to be filtered and doesn't
provide any choice.

Instead of it lets disable access logger by default and let users decide
do they need or not to see access log messages
2024-01-03 12:18:50 +02:00
f1095fe007 feat: raise 404 in case if package is unknown for logs and patches 2024-01-03 12:09:10 +02:00
1af04448c9 feat: threadsafe services
In the most cases it was enough to just add lock. In case of worker
trigger, since there is atomic operation on timer, it was also required
to add queue (coz python doesn't have atomics)
2024-01-03 12:09:10 +02:00
aad607eaef feat: add workers autodicsovery feature (#121)
* add workers autodicsovery feature

* suppress erros while retrieving worker list

* update recipes

* fix tests and update docs

* filter health checks

* ping based workers
2024-01-03 02:25:24 +02:00
f3e9fbf9d0 refactor: fix warnings and typos, cleanup code 2023-12-28 16:53:45 +02:00
215fe7a592 test: add tests to check args list
Some parsers are shared between different subcommands. It causes errors
when wwe add new arguments to one of them. This commit adds some tests
to cover those cases (except for well-known differencies)
2023-12-27 15:01:07 +02:00
dd8d5d130b refactor: use AppKey's instead of string identifiers for web application 2023-12-27 13:53:23 +02:00
b4fa10781b feat: allow to run daemon mode with split packages check (#120) 2023-12-27 03:05:44 +02:00
f6cdd806b2 feat: add healh-check like endpoint, support of healthcheck in docker compose 2023-12-21 17:49:35 +02:00
1b93f4f5e0 fix: drop debug toolbar 2023-12-20 16:26:14 +02:00
a872ecfc23 docs: docs review 2023-12-20 10:00:12 +02:00
e784032bc6 feat: add ability to disable specific routes (#119) 2023-12-15 14:34:03 +02:00
c54b14b833 feat: add ability to run build process to remote instances (#118) 2023-12-13 15:38:51 +02:00
e61b246216 fix: correct url for update requests in remote-call trigger 2023-12-11 15:43:28 +02:00
2a9eab5f1a feat: changes screen implementation (#117)
Add support of changes generation. Changes will be generated (unless explicitly asked not to) automatically during check process (i.e. `repo-update --dry-run` and aliases) and uploaded to the remote server. Changes can be reviewed either by web interface or by special subcommands.

Changes will be automatically cleared during next successful build
2023-11-30 14:56:41 +02:00
acc204de6d fix: use event instead of chained timer for daemon
Old solution causes amount of thread to be growing as well as stack is
increased during each iteration. Instead of cycle-free implementation,
this commit just uses while cycle
2023-11-30 13:40:59 +02:00
0991dbb59c type: update to the typed aiohttp release 2023-11-23 15:35:38 +02:00
18d17d4d52 feat: forbid form data in html
It has been a while since all pages have moved to json instead of form
data, except for login page. This commit changes login to json data
instead of form one
2023-11-16 16:42:27 +02:00
de7184fc3a feat: add separated web client for ahriman web services 2023-11-14 16:41:33 +02:00
e823fd3f27 fix: fix some security issues 2023-11-12 22:22:05 +02:00
e4a857dad0 feat: add separated switch for status reporting 2023-11-11 15:18:17 +02:00
fc8f6c2985 feat: extend result class 2023-11-10 17:09:01 +02:00
bb6414f9d4 fix: pass packagers object insteaed of string during rebuild 2023-11-08 16:05:27 +02:00
deab8ddae6 feat: serve favicon in root 2023-11-06 15:46:32 +02:00
a73341e885 feat: allow to pass repository identifier to all database methods 2023-11-05 13:26:11 +02:00
df787657aa chore: add metthod definition order plugin to pylint
Also reorder some methods to fix errors
2023-11-05 13:26:11 +02:00
eec94521a7 feat: read old web sections 2023-11-04 13:34:53 +02:00
8f047f9a96 feat: add patch controls to web, review web, enrich info tab (#115)
* add ability to specify one-time patch on package addition

* support vars in interface
2023-10-29 23:41:20 +02:00
54bd016c17 refactor: drop _check_output class attribute 2023-10-23 02:24:53 +03:00
42f77742f8 refactor: add ability to pass anything as environment variable to build task 2023-10-23 02:24:41 +03:00
45eba6a6e3 fix: do not automatically add unknown local packages
Instead of automatic package addition now it is required to add package
manually after clone. Less magic, plus would allow to use caches for
multi-repo setup (see #109)
2023-10-21 01:33:12 +03:00
4c1d0abb85 feat: add ability to run multiple commands on success 2023-10-19 04:34:49 +03:00
1e00bf9398 feat: allow to use single web instance for all repositories (#114)
* Allow to use single web instance for any repository

* some improvements

* drop includes from user home directory, introduce new variables to docker

The old solution didn't actually work as expected, because devtools
configuration belongs to filesystem (as well as sudo one), so it was
still required to run setup command.

In order to handle additional repositories, the POSTSETUP and PRESETUP
commands variables have been introduced. FAQ has been updated as well

* raise 404 in case if repository is unknown
2023-10-17 03:58:50 +03:00
8ff567cac3 feat: add __call__ method to printers 2023-10-01 15:14:18 +03:00
9fe760efdf feat: load http views dynamically (#113) 2023-09-30 01:24:04 +03:00
26b991bf47 feat: add pagination to packages list 2023-09-11 18:08:26 +03:00
8bf422b02a docs: update documentation to new api calls 2023-09-11 15:57:00 +03:00
bef8d33570 feat: improve template processing (#112)
* Improve template processing

* docs update, config validation rules update
2023-09-08 23:38:07 +03:00
b540ced77f perf: limit clone and fetch by the latest commit 2023-09-08 15:52:59 +03:00
59356e905a feat: allow to use one application for multiple repositories (#111)
* allow to use one application for multiple repositories

* update tests

* handle None append argument everywhere

* rewrite repository definition logic

* drop optional flags from docs

* support of new schema in systemd units

* add migration docs and ability to migrate tree automatically

* use repostory id instead

* verbose multiarchitectureerror

* object path support for s3 sync

* fix tests after rebase
2023-09-08 14:01:07 +03:00
99eecdebf3 feat: pagination support for logs request 2023-09-08 01:41:36 +03:00
9a521af078 feat: show help message if no command specified 2023-09-05 18:26:22 +03:00
d1132b49fc feat: add ability to skip migrations
This feature can be used in order to handle conflicting migrations
2023-09-02 14:30:50 +03:00
3cac53ac11 add ability to partition tree before calculationn 2023-08-27 01:55:57 +03:00
9fa6722eaa small improvements on code smell
* fix some grammar/typo errors
* change some statements to be more clear
* use pattern matching for enum processing
2023-08-25 04:15:10 +03:00
2cda4a2469 use http client class for all http requests 2023-08-23 03:27:42 +03:00
598af7d9db verbose subprocess exception handle annd quite git 2023-08-21 01:35:43 +03:00
ad1c0051c4 Remote call trigger support (#105)
* add support of remote task tracking
* add remote call trigger implementation
* docs update
* add cross-service upload
* add notes about user
* add more ability to control upload
* multipart upload with signatures as well as safe file save
* configuration reference update
* rename watcher methods
* erase logs based on current package version

Old implementation has used process id instead, but it leads to log
removal in case of remote process trigger

* add --server flag for setup command
* restore behavior of the httploghandler
2023-08-20 03:44:31 +03:00
42fccb76ef add dummy type fields for pkgbuild generators
This field is required in order to pass config validation in case if
section name differs from default one. Also by default keyring_generator
and mirrorlist_generator have been renamed to keyring-generator and
mirrorlist-generator respectively for consistence
2023-08-17 16:02:15 +03:00
c37f67c558 close descriptor after uploading archive on github 2023-08-15 02:31:46 +03:00
116dc0b329 resoolve dependencies by using local cache too (#107) 2023-08-14 02:31:24 +03:00
e3ed36fafd add ability to read values from environment variables
It makes sense to read some values from environment. In particular this
feature is useful in case of running application in containers in ci/cd

See #108 for more details
2023-08-14 01:48:08 +03:00
573ade4ba5 remove unused absolute path validator (#106)
Extracted path is always absolute, so there is no need to check it
2023-08-13 20:48:07 +03:00
9259d9c727 Local packages support improvements (#104)
* handle git author correctly
* make remote source required argument
2023-08-13 15:45:53 +03:00
c863ee063c make auth.salt parameter optional
Used implementation of the hasher includes salt itself, thus additional
salt is optional and can be safely (in terms of security) treat as empty
string
2023-08-11 16:31:47 +03:00
480c3a54b8 full support of pep517
Since llast upgrade build is broken. Lets fully migrate to
pyproject.toml. Note for maintaners: because data_files option is
deprectated (see https://github.com/pypa/setuptools/discussions/2648)
you will have to install files manually inside your packaging process
2023-08-11 03:55:31 +03:00
3b3ef43863 automatically bump pkgrel on version duplicates
The new --(no-)increment flag has been added to add, update and rebuild
subcommands. In case if it is true and package version is the same as in
repository, it will automatically bump pkgrel appending (increasing)
minor part of it (e.g. 1.0.0-1 -> 1.0.0-1.1).

Inn order to implement this, the shadow (e.g. it will not store it in
database) patch for pkgrel will be created
2023-08-08 03:14:47 +03:00
adf6a0a952 remove napoleon contrib dependency 2023-08-08 02:53:08 +03:00
575a618096 fix issues with remote pull triggers (see #103)
* The issue appears when repository contains PKGBUILD in root. In this
  case it will copy tree with loosing package information, because
  the repository will be cloned to temporary path with random generated
  name
* The issue appears when branch which is different from master is used
  for any reposittory with git files (e.g. single-pkgbuild repo or repo
  with submodules)
2023-08-08 02:34:44 +03:00
9cb39f6767 rase 405 error in case if GET login method is used whereas no aioauth
library installed
2023-07-23 03:01:25 +03:00
c7de182f3d optimize schemas import 2023-07-23 02:24:39 +03:00
ad570aae0f remove unsafe flag from handlers
This flag became reduntant there and tree creation has been moved to
lock
2023-07-07 03:25:05 +03:00
5e5171cf3e add salt generator to setup command instead 2023-07-06 19:16:49 +03:00
af3360e84a register dependency package before build
If package has been added as dependency, the service miss remote as well
as causes some 400 errors in reporter
2023-07-06 03:11:19 +03:00
95f52f7ebe handle packages load from aur by package name also
In general package names array may not contain package base, thus it
leads to inability to load packages from aur by its base during update
process
2023-07-01 15:55:04 +03:00
f5fe200247 explicitly pass user agent for the arch linux sites requests 2023-06-26 02:52:08 +03:00
a1c12200b8 print configuration paths in dump command 2023-06-11 15:11:34 +03:00
09839f755a remove salt generation from users handler
It causes issues, because users handler is operating with service user,
but writtinng salt requires root privileges
2023-06-05 05:25:10 +03:00
4dd5a1e82e packagers support (#100) 2023-06-05 02:37:19 +03:00
e3347aec2d use cached property instead of custom __getattr__ implementation 2023-05-31 19:17:16 +03:00
3ad6cd27c6 support check dependencies 2023-05-31 19:17:16 +03:00
54a68279be mimic parent arguments during spawn process (#99) 2023-05-30 18:09:35 +03:00
d99091a3b4 do not terminate child process explicitly
This feature sometimes causes the main process termination. Since the
child process has been already terminated (or going to) there is no need
to terminate it manually
2023-05-29 17:27:04 +03:00
be6cf5b8af fix case where it was impossible to create session if user and password
are required

Issue appears if auth is enabled and unix socket is disabled and caused
by d81e91e117
2023-05-29 17:23:57 +03:00
91e548569d runtime logger handler selector 2023-05-29 03:41:11 +03:00
785607c70b allow to filter packages for rebuild by their statuses 2023-05-25 17:41:54 +03:00
adb5b58a1d Support type triggers (#96)
* implement mirrorlist package generator

* implement keyring package generator

* docs update

* do not skip empty lines

* fill remote source for local packages

* faq update
2023-05-25 16:27:16 +03:00
6bc6bdaeae migrate to the latest devtools & archlinux git changes
This migration includes
* removal of community repository as it is no more
* fixed paths for devtools configurations
* migration of archlinux packaging git url
2023-05-22 16:51:27 +03:00
7458c73c7d disable debugpanel for now 2023-05-04 14:30:27 +03:00
c73a6c7bae PEP-673 use Self as return type for classmethods (#94)
* PEP-673 use Self as return type for classmethods

* add dummy test file

* remove python3.10 compat
2023-05-04 03:28:08 +03:00
0b70b5af45 fix flapping tests for oauth
Original implementation sends requests to httpbin which sometimes might
not be available. With proposed changes we are blocking redirects and
just check request itself
2023-05-03 14:53:35 +03:00
1e0d1a5a32 fix some pycharm warnings 2023-04-17 02:23:50 +03:00