Commit Graph

600 Commits

Author SHA1 Message Date
29bb6284b0 test: add pytlint imports plugin and fix errors 2024-01-05 19:52:51 +02:00
706808fc97 refactor: simplify lock processing in worker trigger 2024-01-05 16:11:32 +02:00
7e5f21c26d refactor: split Path elements to / and first directory 2024-01-05 15:22:46 +02:00
56e97040d6 feat: read username if email is not available for oauth provider
Also add recipe for OAuth with GitHub setup
2024-01-05 12:28:31 +02:00
9510a14b90 refactor: use root RequestException instead of the one from subpackage 2024-01-05 11:48:38 +02:00
6eeef39fe5 fix: safe urls for packages
String catenation used for url generators didn't encode package names
which could lead to missing data in case if e.g. there is slash (/) in
package name
2024-01-03 14:28:20 +02:00
0e6434faad refactor: remove custom access logger
It is fine when application is able to log request, however, normally it
produces a lot noise, which has been handled by adding special logger.
However, nowadays it requires a lot endpoints to be filtered and doesn't
provide any choice.

Instead of it lets disable access logger by default and let users decide
do they need or not to see access log messages
2024-01-03 12:18:50 +02:00
768132bb6c chore: copyright update 2024-01-03 12:09:10 +02:00
f1095fe007 feat: raise 404 in case if package is unknown for logs and patches 2024-01-03 12:09:10 +02:00
1af04448c9 feat: threadsafe services
In the most cases it was enough to just add lock. In case of worker
trigger, since there is atomic operation on timer, it was also required
to add queue (coz python doesn't have atomics)
2024-01-03 12:09:10 +02:00
aad607eaef feat: add workers autodicsovery feature (#121)
* add workers autodicsovery feature

* suppress erros while retrieving worker list

* update recipes

* fix tests and update docs

* filter health checks

* ping based workers
2024-01-03 02:25:24 +02:00
f3e9fbf9d0 refactor: fix warnings and typos, cleanup code 2023-12-28 16:53:45 +02:00
215fe7a592 test: add tests to check args list
Some parsers are shared between different subcommands. It causes errors
when wwe add new arguments to one of them. This commit adds some tests
to cover those cases (except for well-known differencies)
2023-12-27 15:01:07 +02:00
dd8d5d130b refactor: use AppKey's instead of string identifiers for web application 2023-12-27 13:53:23 +02:00
b4fa10781b feat: allow to run daemon mode with split packages check (#120) 2023-12-27 03:05:44 +02:00
f6cdd806b2 feat: add healh-check like endpoint, support of healthcheck in docker compose 2023-12-21 17:49:35 +02:00
1b93f4f5e0 fix: drop debug toolbar 2023-12-20 16:26:14 +02:00
a872ecfc23 docs: docs review 2023-12-20 10:00:12 +02:00
58e702213d docs: restore docs for the view 2023-12-15 16:17:19 +02:00
e784032bc6 feat: add ability to disable specific routes (#119) 2023-12-15 14:34:03 +02:00
c54b14b833 feat: add ability to run build process to remote instances (#118) 2023-12-13 15:38:51 +02:00
e61b246216 fix: correct url for update requests in remote-call trigger 2023-12-11 15:43:28 +02:00
2a9eab5f1a feat: changes screen implementation (#117)
Add support of changes generation. Changes will be generated (unless explicitly asked not to) automatically during check process (i.e. `repo-update --dry-run` and aliases) and uploaded to the remote server. Changes can be reviewed either by web interface or by special subcommands.

Changes will be automatically cleared during next successful build
2023-11-30 14:56:41 +02:00
acc204de6d fix: use event instead of chained timer for daemon
Old solution causes amount of thread to be growing as well as stack is
increased during each iteration. Instead of cycle-free implementation,
this commit just uses while cycle
2023-11-30 13:40:59 +02:00
0991dbb59c type: update to the typed aiohttp release 2023-11-23 15:35:38 +02:00
18d17d4d52 feat: forbid form data in html
It has been a while since all pages have moved to json instead of form
data, except for login page. This commit changes login to json data
instead of form one
2023-11-16 16:42:27 +02:00
de7184fc3a feat: add separated web client for ahriman web services 2023-11-14 16:41:33 +02:00
2d21c999d1 docs: move timeout settings from web to status group 2023-11-13 16:53:16 +02:00
ff78577d61 Release 2.12.2 2023-11-13 12:26:00 +02:00
eb6eda578c style: highligh __del__ methods in formatting 2023-11-13 12:16:30 +02:00
e823fd3f27 fix: fix some security issues 2023-11-12 22:22:05 +02:00
e4a857dad0 feat: add separated switch for status reporting 2023-11-11 15:18:17 +02:00
fc8f6c2985 feat: extend result class 2023-11-10 17:09:01 +02:00
bb6414f9d4 fix: pass packagers object insteaed of string during rebuild 2023-11-08 16:05:27 +02:00
7d16142b3b Release 2.12.1 2023-11-06 16:31:56 +02:00
5b4bff9c93 Release 2.12.0 2023-11-06 15:50:11 +02:00
deab8ddae6 feat: serve favicon in root 2023-11-06 15:46:32 +02:00
dc2eebce39 docs: correct crossreference in docs 2023-11-06 14:04:15 +02:00
6b755b4828 refactor: replace enum with intenum and strenum 2023-11-05 13:26:11 +02:00
a73341e885 feat: allow to pass repository identifier to all database methods 2023-11-05 13:26:11 +02:00
df787657aa chore: add metthod definition order plugin to pylint
Also reorder some methods to fix errors
2023-11-05 13:26:11 +02:00
eec94521a7 feat: read old web sections 2023-11-04 13:34:53 +02:00
8f047f9a96 feat: add patch controls to web, review web, enrich info tab (#115)
* add ability to specify one-time patch on package addition

* support vars in interface
2023-10-29 23:41:20 +02:00
54bd016c17 refactor: drop _check_output class attribute 2023-10-23 02:24:53 +03:00
42f77742f8 refactor: add ability to pass anything as environment variable to build task 2023-10-23 02:24:41 +03:00
45eba6a6e3 fix: do not automatically add unknown local packages
Instead of automatic package addition now it is required to add package
manually after clone. Less magic, plus would allow to use caches for
multi-repo setup (see #109)
2023-10-21 01:33:12 +03:00
4c1d0abb85 feat: add ability to run multiple commands on success 2023-10-19 04:34:49 +03:00
1e00bf9398 feat: allow to use single web instance for all repositories (#114)
* Allow to use single web instance for any repository

* some improvements

* drop includes from user home directory, introduce new variables to docker

The old solution didn't actually work as expected, because devtools
configuration belongs to filesystem (as well as sudo one), so it was
still required to run setup command.

In order to handle additional repositories, the POSTSETUP and PRESETUP
commands variables have been introduced. FAQ has been updated as well

* raise 404 in case if repository is unknown
2023-10-17 03:58:50 +03:00
bf9a46936c chore: update comments to the latest mypy release 2023-10-12 20:31:00 +03:00
a0b17981e3 feat: add ability to override oauth2 icon 2023-10-07 18:12:17 +03:00
c440f5bbd5 fix: whitelist environment variables instead of passing all of them
Earlier applied fix bc9682373d introduced
errors with interaction, because (in docker container) HOME variable was
passed also to subprocesses. This fix limits variables to be passed to
the whitelisted ones
2023-10-02 16:37:20 +03:00
8ff567cac3 feat: add __call__ method to printers 2023-10-01 15:14:18 +03:00
9fe760efdf feat: load http views dynamically (#113) 2023-09-30 01:24:04 +03:00
bc9682373d fix: processes in docker images and pass full environment to
subprocesses

Since 1.0.4 release devtools require systemd to be run in order to
create slice for unit
2023-09-29 03:57:26 +03:00
26b991bf47 feat: add pagination to packages list 2023-09-11 18:08:26 +03:00
8bf422b02a docs: update documentation to new api calls 2023-09-11 15:57:00 +03:00
bef8d33570 feat: improve template processing (#112)
* Improve template processing

* docs update, config validation rules update
2023-09-08 23:38:07 +03:00
b540ced77f perf: limit clone and fetch by the latest commit 2023-09-08 15:52:59 +03:00
59356e905a feat: allow to use one application for multiple repositories (#111)
* allow to use one application for multiple repositories

* update tests

* handle None append argument everywhere

* rewrite repository definition logic

* drop optional flags from docs

* support of new schema in systemd units

* add migration docs and ability to migrate tree automatically

* use repostory id instead

* verbose multiarchitectureerror

* object path support for s3 sync

* fix tests after rebase
2023-09-08 14:01:07 +03:00
99eecdebf3 feat: pagination support for logs request 2023-09-08 01:41:36 +03:00
5e42dd4e70 fix: mark failed packages as failed in console handler 2023-09-06 18:22:05 +03:00
9a521af078 feat: show help message if no command specified 2023-09-05 18:26:22 +03:00
d1132b49fc feat: add ability to skip migrations
This feature can be used in order to handle conflicting migrations
2023-09-02 14:30:50 +03:00
bba90b3eb8 Release 2.11.0 2023-08-27 02:12:21 +03:00
3cac53ac11 add ability to partition tree before calculationn 2023-08-27 01:55:57 +03:00
9fa6722eaa small improvements on code smell
* fix some grammar/typo errors
* change some statements to be more clear
* use pattern matching for enum processing
2023-08-25 04:15:10 +03:00
2cda4a2469 use http client class for all http requests 2023-08-23 03:27:42 +03:00
598af7d9db verbose subprocess exception handle annd quite git 2023-08-21 01:35:43 +03:00
6530afbfc7 review exception raise
In some cases for better readability of logs, exceptions are now raised
without parent exception stacktrace. Also updated docs and contributing
guidelines
2023-08-20 17:03:46 +03:00
ad1c0051c4 Remote call trigger support (#105)
* add support of remote task tracking
* add remote call trigger implementation
* docs update
* add cross-service upload
* add notes about user
* add more ability to control upload
* multipart upload with signatures as well as safe file save
* configuration reference update
* rename watcher methods
* erase logs based on current package version

Old implementation has used process id instead, but it leads to log
removal in case of remote process trigger

* add --server flag for setup command
* restore behavior of the httploghandler
2023-08-20 03:44:31 +03:00
42fccb76ef add dummy type fields for pkgbuild generators
This field is required in order to pass config validation in case if
section name differs from default one. Also by default keyring_generator
and mirrorlist_generator have been renamed to keyring-generator and
mirrorlist-generator respectively for consistence
2023-08-17 16:02:15 +03:00
c37f67c558 close descriptor after uploading archive on github 2023-08-15 02:31:46 +03:00
116dc0b329 resoolve dependencies by using local cache too (#107) 2023-08-14 02:31:24 +03:00
e3ed36fafd add ability to read values from environment variables
It makes sense to read some values from environment. In particular this
feature is useful in case of running application in containers in ci/cd

See #108 for more details
2023-08-14 01:48:08 +03:00
573ade4ba5 remove unused absolute path validator (#106)
Extracted path is always absolute, so there is no need to check it
2023-08-13 20:48:07 +03:00
9259d9c727 Local packages support improvements (#104)
* handle git author correctly
* make remote source required argument
2023-08-13 15:45:53 +03:00
c863ee063c make auth.salt parameter optional
Used implementation of the hasher includes salt itself, thus additional
salt is optional and can be safely (in terms of security) treat as empty
string
2023-08-11 16:31:47 +03:00
480c3a54b8 full support of pep517
Since llast upgrade build is broken. Lets fully migrate to
pyproject.toml. Note for maintaners: because data_files option is
deprectated (see https://github.com/pypa/setuptools/discussions/2648)
you will have to install files manually inside your packaging process
2023-08-11 03:55:31 +03:00
3b3ef43863 automatically bump pkgrel on version duplicates
The new --(no-)increment flag has been added to add, update and rebuild
subcommands. In case if it is true and package version is the same as in
repository, it will automatically bump pkgrel appending (increasing)
minor part of it (e.g. 1.0.0-1 -> 1.0.0-1.1).

Inn order to implement this, the shadow (e.g. it will not store it in
database) patch for pkgrel will be created
2023-08-08 03:14:47 +03:00
368db86dde Release 2.10.2 2023-08-08 02:54:23 +03:00
575a618096 fix issues with remote pull triggers (see #103)
* The issue appears when repository contains PKGBUILD in root. In this
  case it will copy tree with loosing package information, because
  the repository will be cloned to temporary path with random generated
  name
* The issue appears when branch which is different from master is used
  for any reposittory with git files (e.g. single-pkgbuild repo or repo
  with submodules)
2023-08-08 02:34:44 +03:00
2c03354d4a Release 2.10.1 2023-07-28 03:21:19 +03:00
4008aa2e0a do not create application in shell handler
The main reason for having shell handler is to be able to fix if
something (e.g. migrations) goes wrong. In this way we need to reduce
actions inside this wrapper
2023-07-28 03:06:28 +03:00
9cb39f6767 rase 405 error in case if GET login method is used whereas no aioauth
library installed
2023-07-23 03:01:25 +03:00
c7de182f3d optimize schemas import 2023-07-23 02:24:39 +03:00
d21c3fdf9f Release 2.10.0 2023-07-22 05:28:57 +03:00
ad570aae0f remove unsafe flag from handlers
This flag became reduntant there and tree creation has been moved to
lock
2023-07-07 03:25:05 +03:00
5e5171cf3e add salt generator to setup command instead 2023-07-06 19:16:49 +03:00
af3360e84a register dependency package before build
If package has been added as dependency, the service miss remote as well
as causes some 400 errors in reporter
2023-07-06 03:11:19 +03:00
95f52f7ebe handle packages load from aur by package name also
In general package names array may not contain package base, thus it
leads to inability to load packages from aur by its base during update
process
2023-07-01 15:55:04 +03:00
f5fe200247 explicitly pass user agent for the arch linux sites requests 2023-06-26 02:52:08 +03:00
a1c12200b8 print configuration paths in dump command 2023-06-11 15:11:34 +03:00
09839f755a remove salt generation from users handler
It causes issues, because users handler is operating with service user,
but writtinng salt requires root privileges
2023-06-05 05:25:10 +03:00
fcd4d60cad complitely disable signature check for local repository in devtools
It appears that with optional level pacman still tries to validate the
key, which can lead to errors whille processing in docker container
2023-06-05 04:41:03 +03:00
3681473942 packager documentation update 2023-06-05 03:40:02 +03:00
4dd5a1e82e packagers support (#100) 2023-06-05 02:37:19 +03:00
e3347aec2d use cached property instead of custom __getattr__ implementation 2023-05-31 19:17:16 +03:00
3ad6cd27c6 support check dependencies 2023-05-31 19:17:16 +03:00
54a68279be mimic parent arguments during spawn process (#99) 2023-05-30 18:09:35 +03:00
d99091a3b4 do not terminate child process explicitly
This feature sometimes causes the main process termination. Since the
child process has been already terminated (or going to) there is no need
to terminate it manually
2023-05-29 17:27:04 +03:00
be6cf5b8af fix case where it was impossible to create session if user and password
are required

Issue appears if auth is enabled and unix socket is disabled and caused
by d81e91e117
2023-05-29 17:23:57 +03:00
91e548569d runtime logger handler selector 2023-05-29 03:41:11 +03:00
785607c70b allow to filter packages for rebuild by their statuses 2023-05-25 17:41:54 +03:00
adb5b58a1d Support type triggers (#96)
* implement mirrorlist package generator

* implement keyring package generator

* docs update

* do not skip empty lines

* fill remote source for local packages

* faq update
2023-05-25 16:27:16 +03:00
6834dd442c Release 2.9.0 2023-05-22 17:04:32 +03:00
6bc6bdaeae migrate to the latest devtools & archlinux git changes
This migration includes
* removal of community repository as it is no more
* fixed paths for devtools configurations
* migration of archlinux packaging git url
2023-05-22 16:51:27 +03:00
7458c73c7d disable debugpanel for now 2023-05-04 14:30:27 +03:00
2cecbb3d53 do not assign path in context manager as it is deprectated 2023-05-04 14:29:39 +03:00
c73a6c7bae PEP-673 use Self as return type for classmethods (#94)
* PEP-673 use Self as return type for classmethods

* add dummy test file

* remove python3.10 compat
2023-05-04 03:28:08 +03:00
0b70b5af45 fix flapping tests for oauth
Original implementation sends requests to httpbin which sometimes might
not be available. With proposed changes we are blocking redirects and
just check request itself
2023-05-03 14:53:35 +03:00
fde75a14c2 PEP-561 complaint: add py.typed marker 2023-04-17 18:56:49 +03:00
1e0d1a5a32 fix some pycharm warnings 2023-04-17 02:23:50 +03:00
75919637e8 suppress status errors also if option is set 2023-04-15 05:05:55 +03:00
10fbc31d2a PEP-585 complaint: remove type aliases (#93) 2023-04-12 04:09:28 +03:00
7ab00ee579 Release 2.8.0 2023-04-09 13:06:54 +03:00
2d62148ee6 try to remove unknown packages from api 2023-04-06 18:14:36 +03:00
c1f2d5ff4a hide cookie key and salt from config output 2023-04-06 00:31:50 +03:00
2888649ced optimize imports 2023-04-06 00:24:39 +03:00
63855f5964 argument annootation update 2023-04-06 00:24:39 +03:00
8bf665d7d3 fix license url 2023-04-06 00:24:39 +03:00
a7ac77ce4d extract schemas automatically from views 2023-04-06 00:24:39 +03:00
fc01bf3d1c use api generated docs instead of comments (#92) 2023-04-06 00:24:39 +03:00
d81e91e117 execute request in context methods instead of handling them each time
manually
2023-03-23 12:43:04 +02:00
57f62ba163 Release 2.7.1 2023-03-06 01:15:47 +02:00
ded896ee1b gracefully terminate web server
In previous revisions server was terminated by itself, thus no lock or
socket was removed. In new version, graceful termination of the queue
has been added as well as server now handles singals
2023-03-06 01:13:41 +02:00
1fdcea0524 replace InitializeException with InitializeError in docs 2023-03-02 11:07:59 +02:00
a93f43dcd0 simplify login ttl processing 2023-02-24 16:52:55 +02:00
20974dae6f add more validation rules 2023-02-23 15:18:56 +02:00
cbcfff27b8 add ability to read cookie secret from config 2023-02-22 18:47:56 +02:00
96f394bab0 Release 2.7.0 2023-02-20 03:05:08 +02:00
aae3c82f3d update dependencies before build (#91)
Old implementation has used add step in order to fetch dependencies,
which could lead to build errors in case if dependency list was updated.

New solution uses dependencies which are declared at current version and
fetch them (if required and if enabled) before update process.

Closes #90
2023-02-12 06:02:30 +03:00
ab9e1fb1bd handle .gitignore file correctly in remote push trigger 2023-02-11 04:41:24 +02:00
82322f7a6c mask mypy warning
The newest mypy produces the following warning:

src/ahriman/application/handlers/search.py:43: error: Non-overlapping identity check (left operand type: "Union[_DefaultFactory[Any], Literal[_MISSING_TYPE.MISSING]]", right operand type: "Type[List[Any]]")  [comparison-overlap]

which is more likely caused by updated dataclass models to protoocol (however decorators are still calllable). This commit masks problematic line from checking
2023-02-09 22:46:08 +02:00
125da217d3 hide passwords and secrets from repo-config subcommand by default 2023-02-05 16:44:48 +02:00
050041b2de handle architecture specific fields for dependencies
This change requires srcinfo at least 0.1.2 version. Unfortunatelly aur
api don't support architecture specific arrays for now, so we just leave
it as is

Closes #82
2023-01-31 14:34:09 +02:00
e0ed636d7e filter empty packages from database
In some cases (e.g. during addition of the package to build queue) we don't have
full information about package inself; in these cases we produce lines
with empty architecture, which duplicates normal ones.

This commit changes architecture column type to required and also
filters packages which don't have architecture set yet.

Closes #83
2023-01-30 17:57:13 +02:00
61feb2ebc7 trim version from provides list
Closes #87
2023-01-30 17:27:34 +02:00
da1c6b0101 add ability to suppress http logging errors (#86) 2023-01-30 17:19:01 +02:00
c1718b3862 calculate dependencies based on package information (#89) 2023-01-30 17:28:05 +03:00
34fe8128aa remove debug line 2023-01-27 16:41:41 +02:00
a78c48ed4d Release 2.6.1 2023-01-25 15:28:27 +02:00
e368396653 make oauth client trully optional (#84)
Same old song, after migraiton to packages some optional modules are
being imported globally which lead to making hard dependency
2023-01-25 15:25:42 +02:00
f4abd83568 enable lock for web service 2023-01-18 01:39:55 +02:00
e6888ba788 Release 2.6.0 2023-01-16 01:35:12 +02:00
93876307f8 add ability to override pacman mirror in devtools configuration
This commit also extends configuration of the multilib option, adding
the ability to exlcude multilib repository from repositories list

Note, that in order to support repository list and mirror correctly,
alpm configuration section is now architectture specific
2023-01-15 18:05:15 +02:00
461fbd40ce consider vcs flag and vcs_allowed_age during local packages update 2023-01-12 15:37:05 +02:00
419da3db89 add ability to check for service updates 2023-01-12 15:37:05 +02:00
00ca06ffab subcommand review
Some commands have been moved to another group and thus having another
default name (old subcommands are still available...for now):

* daemon -> repo-daemon
* key-import -> service-key-import
* repo-clean -> service-clean
* repo-config -> service-config
* repo-config-validate -> service-config-validate
* repo-setup -> service-setup
* repo-shell -> service-shell
* version -> help-version
2023-01-12 15:37:05 +02:00
5a05c8ce91 expose trigger configuration schema
Note that this commit contains the following breaking changes:

* remote pull and remote push triggers are now enabled by default (with
  empty target list)
* remote pull and remote push triggers now require target option to be
  set (old behaviour had fallback on `gitremote`)
* validation is now considered to be stable, so it is enabled by default
  in docker image (can be disabled however)
2023-01-10 03:33:52 +02:00
b09aea13af add config validator subcommand (#80)
* add config validator subcommand

* add --exit-code flag

* docs & faq update
2023-01-09 18:22:29 +03:00