Commit Graph

318 Commits

Author SHA1 Message Date
8f0a6cf6b5 allow to filter packages for rebuild by their statuses 2023-05-25 17:41:54 +03:00
1cf027cb55
Support type triggers (#96)
* implement mirrorlist package generator

* implement keyring package generator

* docs update

* do not skip empty lines

* fill remote source for local packages

* faq update
2023-05-25 16:27:16 +03:00
70e9e3e822 migrate to the latest devtools & archlinux git changes
This migration includes
* removal of community repository as it is no more
* fixed paths for devtools configurations
* migration of archlinux packaging git url
2023-05-22 16:51:27 +03:00
8c55438140 disable debugpanel for now 2023-05-04 14:30:27 +03:00
2ff56965d9
PEP-673 use Self as return type for classmethods (#94)
* PEP-673 use Self as return type for classmethods

* add dummy test file

* remove python3.10 compat
2023-05-04 03:28:08 +03:00
9dc6d56a8d fix flapping tests for oauth
Original implementation sends requests to httpbin which sometimes might
not be available. With proposed changes we are blocking redirects and
just check request itself
2023-05-03 14:53:35 +03:00
31f47b8069 fix some pycharm warnings 2023-04-17 02:23:50 +03:00
e42ca95789 suppress status errors also if option is set 2023-04-15 05:05:55 +03:00
abf29083c4
PEP-585 complaint: remove type aliases (#93) 2023-04-12 04:09:28 +03:00
9503a9f2ae try to remove unknown packages from api 2023-04-06 18:14:36 +03:00
58379e7bf3 optimize imports 2023-04-06 00:24:39 +03:00
e08ab2db10 extract schemas automatically from views 2023-04-06 00:24:39 +03:00
8f4a2547e8 use api generated docs instead of comments (#92) 2023-04-06 00:24:39 +03:00
7f5e541120 execute request in context methods instead of handling them each time
manually
2023-03-23 12:43:04 +02:00
df23be9269 gracefully terminate web server
In previous revisions server was terminated by itself, thus no lock or
socket was removed. In new version, graceful termination of the queue
has been added as well as server now handles singals
2023-03-06 01:13:41 +02:00
a274f91677 simplify login ttl processing 2023-02-24 16:52:55 +02:00
13faf66bdb add more validation rules 2023-02-23 15:18:56 +02:00
4fb9335df9 add ability to read cookie secret from config 2023-02-22 18:47:56 +02:00
37e57c13c8
update dependencies before build (#91)
Old implementation has used add step in order to fetch dependencies,
which could lead to build errors in case if dependency list was updated.

New solution uses dependencies which are declared at current version and
fetch them (if required and if enabled) before update process.

Closes #90
2023-02-12 06:02:30 +03:00
19bb19e9f5 handle .gitignore file correctly in remote push trigger 2023-02-11 04:41:24 +02:00
3a4e8f4d97 mask mypy warning
The newest mypy produces the following warning:

src/ahriman/application/handlers/search.py:43: error: Non-overlapping identity check (left operand type: "Union[_DefaultFactory[Any], Literal[_MISSING_TYPE.MISSING]]", right operand type: "Type[List[Any]]")  [comparison-overlap]

which is more likely caused by updated dataclass models to protoocol (however decorators are still calllable). This commit masks problematic line from checking
2023-02-09 22:46:08 +02:00
4db8ad8e8d hide passwords and secrets from repo-config subcommand by default 2023-02-05 16:44:48 +02:00
917ec48be5 handle architecture specific fields for dependencies
This change requires srcinfo at least 0.1.2 version. Unfortunatelly aur
api don't support architecture specific arrays for now, so we just leave
it as is

Closes #82
2023-01-31 14:34:09 +02:00
0a2ba4ae07 filter empty packages from database
In some cases (e.g. during addition of the package to build queue) we don't have
full information about package inself; in these cases we produce lines
with empty architecture, which duplicates normal ones.

This commit changes architecture column type to required and also
filters packages which don't have architecture set yet.

Closes #83
2023-01-30 17:57:13 +02:00
9d7f63e549 trim version from provides list
Closes #87
2023-01-30 17:27:34 +02:00
25eee9ca5e add ability to suppress http logging errors (#86) 2023-01-30 17:19:01 +02:00
5af84955ac
calculate dependencies based on package information (#89) 2023-01-30 17:28:05 +03:00
17466d8d37 make oauth client trully optional (#84)
Same old song, after migraiton to packages some optional modules are
being imported globally which lead to making hard dependency
2023-01-25 15:25:42 +02:00
9e4e3b701b enable lock for web service 2023-01-18 01:39:55 +02:00
616a1950ae add ability to override pacman mirror in devtools configuration
This commit also extends configuration of the multilib option, adding
the ability to exlcude multilib repository from repositories list

Note, that in order to support repository list and mirror correctly,
alpm configuration section is now architectture specific
2023-01-15 18:05:15 +02:00
d7356926c4 consider vcs flag and vcs_allowed_age during local packages update 2023-01-12 15:37:05 +02:00
43a7d09cab add ability to check for service updates 2023-01-12 15:37:05 +02:00
77954b988b subcommand review
Some commands have been moved to another group and thus having another
default name (old subcommands are still available...for now):

* daemon -> repo-daemon
* key-import -> service-key-import
* repo-clean -> service-clean
* repo-config -> service-config
* repo-config-validate -> service-config-validate
* repo-setup -> service-setup
* repo-shell -> service-shell
* version -> help-version
2023-01-12 15:37:05 +02:00
0239fb50b6 expose trigger configuration schema
Note that this commit contains the following breaking changes:

* remote pull and remote push triggers are now enabled by default (with
  empty target list)
* remote pull and remote push triggers now require target option to be
  set (old behaviour had fallback on `gitremote`)
* validation is now considered to be stable, so it is enabled by default
  in docker image (can be disabled however)
2023-01-10 03:33:52 +02:00
d942a70272
add config validator subcommand (#80)
* add config validator subcommand

* add --exit-code flag

* docs & faq update
2023-01-09 18:22:29 +03:00
014007ade3 review unsafe commands access
Some commands were made unsafe in old versions, but nowadays they can be
run without having special privileges.

There was also a bug in which status commands were not available if you
are not ahriman user and unix socket is used. It has been fixed by
switching to manual socket creation (see also
https://github.com/aio-libs/aiohttp/issues/4155)
2023-01-04 17:45:44 +02:00
04e5a263b7 add notes about documentation and methods inside class
Because I always forget which way I used before
2023-01-03 01:53:10 +02:00
caca1576c8 Correct way to allow setting context with existing
This reverts commit 5c4d3eeffd.

Original solution has introduced special workaround (strict flag) which
contradicts the concept of immutable context. Moreover, it introduces
possible side-effects, because child process will use the one set by
parent instead of having own one.

The correct solution is to re-create context in process entry point

Sorry, it was Jan 1 and I was drunk :(
2023-01-03 00:48:14 +02:00
5c4d3eeffd allow setting context with existing
In case of running command from web interface, it will raise exception
because context has been copied with subprocesses
2023-01-02 03:21:15 +02:00
2c2eae2334 remote all gitfiles in git remote trigger
In case if there is .gitignore file with asterics, the pkgbuild upload
would not appear
2023-01-02 01:45:50 +02:00
f984ea75d0 fully lazy handle load
In case of immediate handle load it would try to sync databases (or at
least to create database files), which is not possible in case if
command is run as non-ahriman user. This commit makes handle load lazy
and allows to run some commands as non-ahriman user
2022-12-31 14:48:21 +02:00
81d9526054 improve VCS packages checks
* Unlike older version, currently service will always try to pull AUR
  package to check version. Previously if no-vcs flag is set, it would
  ignore VCS packages completelly
* Introduce build.vcs_allowed_age option. If set, it will skip version
  calculation if package age (now - build_date) is less than this value
2022-12-30 21:19:23 +02:00
7de94a8daa add repo-structure subcommand
This commit also changes Tree class, replacing load method by resolve
2022-12-27 10:35:03 +02:00
8864855c14
write patches via gitremote push trigger (#79)
* write patches via gitremote push trigger

* implement context variables intead of custom database class
2022-12-25 02:10:38 +03:00
7df4adfc9d replace alert bodals with toasts 2022-12-09 11:33:19 +02:00
262462d3c3 improve wording in documentation 2022-12-02 15:45:01 +02:00
7aa91f9e2e do not trigger update on sign command 2022-12-02 01:41:23 +02:00
01eda513cf improve setup command by --makeflags-jobs argument and fix repository sign on creation 2022-12-02 01:41:23 +02:00
0161617e36 implement support of unix socket for server
This feature can be used for unauthorized access to apis - e.g. for
reporting service if it is run on the same machine. Since now it becomes
recommended way for the interprocess communication, thus some options
(e.g. creating user with as-service flag) are no longer available now
2022-12-02 01:41:23 +02:00
4811dec759 do not invoke configuration write in case if no salt or user was written 2022-12-02 01:41:23 +02:00
09623c20d5 add rebuild implementation to interface 2022-12-02 01:41:23 +02:00
5073c80af1 add key-import button to interface 2022-12-02 01:41:23 +02:00
d36e851a29 render httpunauthorized as html in plain http requests 2022-11-22 23:40:27 +02:00
3bca780bdd check log record in handler instead of client 2022-11-22 02:30:37 +02:00
78e6b48c24 ask users to repeat password
In case if password is asked via getpass, it is possible to make typo
and user will not see the mistake. In order to avoid it, additional
confirmation has been added
2022-11-22 02:19:37 +02:00
137d62e2f8
Extended package status page (#76)
* implement log storage at backend
* handle process id during removal. During one process we can write logs from different packages in different times (e.g. check and update later) and we would like to store all logs belong to the same process
* set package context in main functions
* implement logs support in interface
* filter out logs posting http logs
* add timestamp to log records
* hide getting logs under reporter permission

List of breaking changes:

* `ahriman.core.lazy_logging.LazyLogging` has been renamed to `ahriman.core.log.LazyLogging`
* `ahriman.core.configuration.Configuration.from_path` does not have `quiet` attribute now
* `ahriman.core.configuration.Configuration` class does not have `load_logging` method now
* `ahriman.core.status.client.Client.load` requires `report` argument now
2022-11-22 02:58:22 +03:00
cdd66ee780 fix case when no files were commited in remote push trigger
The issue appears together with --intent-to-add flag for adding new
files. Original testing has been performed by having already added new
files, thus it passed all checks.

This commit also adds `commit_author` option which will allow to
overwrite the author.
2022-11-14 00:59:43 +02:00
551ee670bf rollback cwd parameter for clone 2022-11-11 21:19:27 +02:00
4f21eb6fe6 Fix issue when there is no cached source directory yet (closes #75) 2022-11-11 17:23:15 +02:00
0cd07afa0f use intersection of from_database and depends_on filters for the rebuild subcommand
Old logic used OR condition, i.e. if set from-database, it would ignore
the --depends-on flag. In new logic it calculates dependencies based on
the package list, which can be retrieved from database
2022-11-11 16:02:19 +02:00
9556779ee2 add exec to shell subcommand 2022-11-10 18:52:55 +02:00
8b32bd43db make exception optional argument for check_output method 2022-11-10 18:05:32 +02:00
0eadef597a
replace several store_true keys to booleanoptionalaction alternative (#74) 2022-11-10 18:34:01 +03:00
e58ccdc8ad write .makepkg.conf to home dir instead of repository root (#72) 2022-11-08 17:02:13 +02:00
2a07356d24
add ability to use ahriman pacman database instead of system one (#71)
By default this feature is enabled. On the first run it will copy (if
exists) databases from filesystem to local cache (one per each
architecture). Later it will use this cache for all alpm operations. In
order to update this cache, some commands (mainly package building)
provide `-y`/`--refresh` option which has same semantics as pacman -Sy
does.

Note however that due to extending `Pacman` class some methods were
renamed in order to be more descriptive:
* `Pacman.all_packages` -> `Pacman.packages`
* `Pacman.get` -> `Pacman.package_get`

This commit also adds multilib repository to the default docker image
which was missed.
2022-11-08 17:26:51 +03:00
43c553a3db improve repo-setup command
* Move devtools executable to ahriman home, because we don't really
  need to use executable inside root
* Use named sudoers file instead of single file. It will allow easily to
  remove file as well as use setup command for several
  repositories/architectures
2022-11-06 14:25:44 +02:00
0d1a890bd0 fix case in which if no source PKGBUILDs were updated, an exception will occur 2022-11-03 03:39:54 +02:00
fb6b22cdd7 change gitremote api to be same as report and upload
These changes are keeping fallback to old settings, but will allow to
run multiple git targets with different settings
2022-11-02 04:09:42 +02:00
4f35ec6542 fix update handler tests
If daemon tests are run before the update handler it causes test
failure, because there are other calls
2022-10-31 11:51:12 +02:00
d98211e5e5 add possibility to run full update
In case if packages are not set from web, the spawner will run full
repository update
2022-10-31 02:41:24 +02:00
b97c8928e1 add daemon subcommand
This command emulates default systemd timer and can be useful in docker
container in order to run 24/7
2022-10-31 01:38:01 +02:00
649df81aa5
implement single-function patches (#69) 2022-10-30 03:11:03 +03:00
ad7cdb7d95 drop ahriman.core.triggers.Trigger.run method
In order to force new triggers to use on_result method, the old method
has been removed. However, default on_result method still checks if the
old method exists and tries to run it
2022-10-19 20:07:31 +03:00
342b3cb652
Add gitremote triggers (#68)
* add gitremote pull trigger

* add push gitremote trigger

* docs update
2022-10-18 01:46:27 +03:00
fc0d8387df extend triggers to on_start and on_stop methods
This commit also replaces old run method to new on_result
2022-09-26 01:22:54 +03:00
664b6369bb skip architecture list patching in case if any architecture is set 2022-09-17 04:04:28 +03:00
1912790ccc Make optional dependencies trully optional (#67)
The issue appears when there is no boto, jinja and some other libraries
are not installed because the classes which use these libraries are
still being imported inside the package file. The fix removes those
imports from package root, because they should not be here, in fact,
content of report and upload packages must be imported only inside the
trigger class and only if they are actually required

This commit also adds setuptools as required dependency since it is used
for some parsers (previously it was provided dependency)
2022-09-11 01:44:06 +03:00
cf3c48ffeb
patch architecture list in runtime (#66) 2022-08-09 15:18:20 +03:00
6633766cc3 frozen dataclasses 2022-07-26 14:40:28 +03:00
9c2f73af8c simplify tmpdir method 2022-06-28 19:11:38 +03:00
cbbe2b01e4 fix case with package name which cannot be downloaded
(without special settings)

The issue appears if file or its version contains one of special URI
characters, e.g. +. Theu will be interpreted as query parameters by
(some) servers (e.g. S3 works in this way). In this commit we rename
archive to the one with safe name.
2022-06-27 18:53:48 +03:00
cd361a483d review loggers once more
This commit makes loggers like java.util.logging with fully qualified
logger name which is created by LazyLogging trait
2022-06-27 01:41:49 +03:00
18daecaac7 review loggers 2022-06-21 11:48:42 +03:00
03c298c762 replace if with while for telelgram reporting 2022-06-10 12:02:00 +03:00
a0cfb7561d rewrite api under single /api endpoint 2022-05-29 17:15:49 +03:00
f4131b8cd7 add shell and version parser 2022-05-27 18:34:07 +03:00
9f134e37b6 update readme 2022-05-22 16:00:24 +03:00
07b7689701 do not bump timestamp on empty updates 2022-05-21 04:19:04 +03:00
47de715d7d
dynamic html load (#63)
* dynamic html load
* split by classes
2022-05-20 22:29:36 +03:00
5674b7b388 add ability to run only speicifed triggers from command line
This commit also restores repo-report and repo-sync subcommands
2022-05-12 18:43:36 +03:00
f6e57a411c fix case when provided trigger path is directory
If trigger is directory and we have permissions to read it, the loaded
will try to load this from it and will fail with IsADirectoryError
2022-05-12 07:22:50 +03:00
99874845b5 triggers implementation (#62) 2022-05-09 20:00:20 +03:00
b6db2a8035 fix error with missing sources
In case if package has local cache it will fail to load because no
remote source set. Particially this case can be observed during tree
load
2022-05-08 03:56:54 +03:00
98910240dd shorten public imports 2022-05-06 04:08:05 +03:00
33e9fea47c Docs update (#61)
* Improve sphinx documentation

* update faq formatting

* fix setup doc

* fix docs according to the generated htmls
2022-05-06 02:54:37 +03:00
b7debddaea
Complete official repository support (#59) 2022-05-03 00:49:32 +03:00
1cfc751d21 rename classes in order to make documentation reference work 2022-04-18 05:30:02 +03:00
a2610504e5 apply data migration in the same transaction block with schema migration 2022-04-18 01:19:38 +03:00
d90f417cae
Docstring update (#58)
* migrate docstrings from reST to google format

* add raises note

Also change behaviour of the `from_option` method to fallback to
disabled instead of raising exception on unknown option

* fix part of warnings for sphinx

* make identation a bit more readable

* review fixes

* add verbose description for properties to make them parsed by sphinx extenstion

* add demo sphinx generator
2022-04-17 20:25:28 +03:00
cb63bc08ff add backup and restore subcommands 2022-04-10 21:34:34 +03:00
6551c8d983 merge restore to rebuild commannd 2022-04-10 01:51:12 +03:00
5ff2f43506 change telegram default index to telegram-index 2022-04-08 04:32:34 +03:00
86af13f09e add telegram integraion 2022-04-08 03:41:07 +03:00
783c16b2ed trim versions before dependency list calculation
When dependencies list contains same package with version it tries to
find packages which don't exists
2022-04-07 20:32:55 +03:00
2536b8dc1f add support of repository restoration 2022-04-07 04:49:07 +03:00
e200ac9776 add support of officiall repositories api 2022-04-07 04:19:37 +03:00
6946745153 fix descriptions 2022-04-06 01:48:03 +03:00
2d2cd73010 better processing for subcommands
Old versions cached full output to memory and only after that printed it
into log. This behaviour causes issues in case if operation stucks and
you would need to find the step at which it does. New check_output
method uses Popen directly and iterates over stdout lines

Also changed behaviour from merging stderr into stdout to separate
stderr logging.

Any other behaviour of the function must be the same.

Also changed GPG.key_import method to disable local signing since it
seems it is useless (and may break process in case if there is no
private key)
2022-04-04 00:03:22 +03:00
e909e4f570
add configurable exit codes to some commands (#55) 2022-04-01 18:30:11 +03:00
06ec16ac77 create user in docker container
also fix user reading and add nocolor flag by default
2022-03-31 03:25:53 +03:00
2bc2b63a8a add help command 2022-03-31 02:18:39 +03:00
28cc38aaa5
port part of settings to database (#54) 2022-03-31 01:48:06 +03:00
fb02e676af migrate to tox
this also requires to move default configuration files to share/ahriman.
Thus the following features have been added
* default configuration is not stored in /usr/share/ahriman/settings
* package installed via PKGBUILD now copies files from /usr
* configuration class now fallbacks to default in /usr
2022-03-21 04:38:42 +03:00
13121298f5 deprecate init/repo-init command
In current workflow you need to run setup to run init (because of
repository name), but you need to run init before setup (because of
repository tree rights).

New solution just add `Repo.init()` method call to setup subcommand
after config reload to make sure that repository name has been applied.
In addition chown method as well as setuid method for check_output have
been added.
2022-03-21 01:32:11 +03:00
b83bc5a732 fallback to AUR source in case of PermissionError exception 2022-03-20 03:46:56 +03:00
57f25c309a add console printer
also add python-requests as explicit dependency and escape symbols in
repository name for badges in default tempate
2022-03-20 02:31:50 +03:00
d33844e7b2 ALLOW_AUTO_ARCHITECTURE_RUN flag for unsafe commands 2022-03-19 03:08:45 +03:00
61efbb71a2
add docker support (#52)
* add docker support

* make shellcheck happy
2022-03-13 23:43:25 +03:00
9f4acacada strict test checks 2022-03-04 02:04:52 +03:00
026f74121a set success status on up to date packages 2022-02-06 06:22:58 +03:00
60a2e25b9a update for new aiohttp api 2022-02-06 04:05:33 +03:00
683abca9e5
use own aur wrapper (#49) 2022-02-06 03:44:57 +03:00
bfca7e41ab handle dependencies recursively 2021-12-22 19:35:09 +03:00
603c5449a8
initial implementation of the local git clones (#48) 2021-12-22 15:56:24 +03:00
042638d40e
handle packages which have been removed from the repository (#45)
* handle packages which have been removed from the repository

* manually remove packages which have been removed from the base
2021-11-10 01:37:25 +03:00
fa4244d21e take python laziness into account 2021-11-04 21:30:34 +03:00
32a4a82603 improve configuration extension
* Allow spaces in lists. This feature has been done in the way as shell
  interprets arguments by using quotation marks
* Clear current content on reload
2021-10-28 03:19:50 +03:00
a2639f8dbb add update printer which will print current version if any 2021-10-27 03:11:43 +03:00
65ba590ace use PackageSource enum for Package.load method
When using add function it sill tries to load data with invalid source
2021-10-27 02:49:23 +03:00
d3ea81d234 unify aur.search method
due to specific of the AUR API in order to reduce the code we are using
own wrapper and work with it instead of direct library calls
2021-10-26 04:49:55 +03:00
09b0f2914d Add ability to show more info in search and status subcommands
This feature also introduces the followiing changes
* aur-search command now works as expected with multiterms
* printer classes for managing of data print
* --sort-by argument for aur-search subcommand instead of using package
  name
* --quiet argument now has also --no-quite option
* if --quite is supplied, the log level will be set to warn instead of
  critical to be able to see error messages
* pretty_datetime function now also supports datetime objects
* BuildStatus is now pure dataclass
2021-10-26 04:27:36 +03:00
dfd87c502f split application class into traits 2021-10-23 13:44:57 +03:00
0b9ab09879 add patches to clean command 2021-10-20 03:22:16 +03:00
47c54f0b40 add ability to download package from external links (e.g. HTTP) 2021-10-20 03:09:58 +03:00
eb16ef12f3 always return json in responses 2021-10-20 02:12:39 +03:00
22d92e3b4e add repo-status-update subcommand 2021-10-17 06:38:49 +03:00
56b77a84a6 allow to use multiple upload and report targets with the same name
In this feature target option must allways point to section name instead
of type. Type will be read from type option. In case if type option is
not presented it will try to check if section with architecture exists
(e.g. target = email, section = email:x86_64); if it does, the correct
section name and type will be used. Otherwise it will check if the
specified section exists; if it does, seection name and type will be
returned.
2021-10-17 06:06:08 +03:00
a5a99ec0b8 split github upload into generic http method and github specific
We might use some features from the http upload for another parser
2021-10-15 23:36:26 +03:00
4521c2adde disallow to create tree in case of unsafe run 2021-10-14 04:53:09 +03:00
5c5e54228f use generic removal method 2021-10-14 04:08:21 +03:00
6514924b2d change method spelling
in order to sort method correctly we are going to use the following
namiing schema:

{subject}_{action}_{details}

This schema still have some exceptions, e.g. single word methods, bool
methods (is_) and getters in case if they are singular (i.e. there is
no any other method with this subject)
2021-10-14 04:01:54 +03:00
16aa977fa8 add test for every file 2021-10-14 03:34:12 +03:00
4502931c39 exactly one called with instead of last call check 2021-10-14 03:12:45 +03:00
fcb167b1a3
github upload support (#41) 2021-10-14 02:30:13 +03:00
72b26603bf
add ability to add manually stored packages (#40)
* add ability to add manually stored packages

* update tests

* handle manual packages in remove-unknown method

* live fixes

also rename branches to has_remotes method and change return type
2021-10-12 21:15:35 +03:00
7c4f84fbc7 more verbose help messages 2021-10-07 01:56:22 +03:00
3b6b2efcb1 patch control subcommands 2021-10-05 08:57:42 +03:00
9f99dd3ff2
patch support (#35) 2021-10-03 15:20:36 +03:00
bee97df87f do not write anything on httpexceptions in log 2021-10-03 02:05:28 +03:00
6becd01803 replace no-log with quiet
Also behavior of the flag has been changed: now it disables logs at all
2021-10-03 01:59:33 +03:00