Commit Graph

352 Commits

Author SHA1 Message Date
4a5b9d9fa1 fix case in which if no source PKGBUILDs were updated, an exception will occur 2022-11-03 03:39:54 +02:00
48da4646cf change gitremote api to be same as report and upload
These changes are keeping fallback to old settings, but will allow to
run multiple git targets with different settings
2022-11-02 04:09:42 +02:00
3d98dd267a fix update handler tests
If daemon tests are run before the update handler it causes test
failure, because there are other calls
2022-10-31 11:51:12 +02:00
09678a3ae1 add possibility to run full update
In case if packages are not set from web, the spawner will run full
repository update
2022-10-31 02:41:24 +02:00
8e2732f6fe add daemon subcommand
This command emulates default systemd timer and can be useful in docker
container in order to run 24/7
2022-10-31 01:38:01 +02:00
73e311a41c implement single-function patches (#69) 2022-10-30 03:11:03 +03:00
1e8388af5d drop ahriman.core.triggers.Trigger.run method
In order to force new triggers to use on_result method, the old method
has been removed. However, default on_result method still checks if the
old method exists and tries to run it
2022-10-19 20:07:31 +03:00
a5ce6b78dd Add gitremote triggers (#68)
* add gitremote pull trigger

* add push gitremote trigger

* docs update
2022-10-18 01:46:27 +03:00
1a83dd6f5a extend triggers to on_start and on_stop methods
This commit also replaces old run method to new on_result
2022-09-26 01:22:54 +03:00
75c6bf6119 skip architecture list patching in case if any architecture is set 2022-09-17 04:04:28 +03:00
43696b9920 Make optional dependencies trully optional (#67)
The issue appears when there is no boto, jinja and some other libraries
are not installed because the classes which use these libraries are
still being imported inside the package file. The fix removes those
imports from package root, because they should not be here, in fact,
content of report and upload packages must be imported only inside the
trigger class and only if they are actually required

This commit also adds setuptools as required dependency since it is used
for some parsers (previously it was provided dependency)
2022-09-11 01:44:06 +03:00
8befee58fe patch architecture list in runtime (#66) 2022-08-09 15:18:20 +03:00
9d016f51b5 frozen dataclasses 2022-07-26 14:40:28 +03:00
695ef708b6 simplify tmpdir method 2022-06-28 19:11:38 +03:00
7b647a9b5a fix case with package name which cannot be downloaded
(without special settings)

The issue appears if file or its version contains one of special URI
characters, e.g. +. Theu will be interpreted as query parameters by
(some) servers (e.g. S3 works in this way). In this commit we rename
archive to the one with safe name.
2022-06-27 18:53:48 +03:00
fac228d6c6 review loggers once more
This commit makes loggers like java.util.logging with fully qualified
logger name which is created by LazyLogging trait
2022-06-27 01:41:49 +03:00
047925dcfe review loggers 2022-06-21 11:48:42 +03:00
b8d2775050 replace if with while for telelgram reporting 2022-06-10 12:02:00 +03:00
1ea3911531 rewrite api under single /api endpoint 2022-05-29 17:15:49 +03:00
d7966e419d add shell and version parser 2022-05-27 18:34:07 +03:00
3a5268993e update readme 2022-05-22 16:00:24 +03:00
ad84895e18 do not bump timestamp on empty updates 2022-05-21 04:19:04 +03:00
b1dfafe275 dynamic html load (#63)
* dynamic html load
* split by classes
2022-05-20 22:29:36 +03:00
375f9fcfb7 add ability to run only speicifed triggers from command line
This commit also restores repo-report and repo-sync subcommands
2022-05-12 18:43:36 +03:00
04dcaa93ab fix case when provided trigger path is directory
If trigger is directory and we have permissions to read it, the loaded
will try to load this from it and will fail with IsADirectoryError
2022-05-12 07:22:50 +03:00
b9cd98235e triggers implementation (#62) 2022-05-09 20:00:20 +03:00
a7018d9f7d fix error with missing sources
In case if package has local cache it will fail to load because no
remote source set. Particially this case can be observed during tree
load
2022-05-08 03:56:54 +03:00
7f87863734 shorten public imports 2022-05-06 04:08:05 +03:00
2dac5a2520 Docs update (#61)
* Improve sphinx documentation

* update faq formatting

* fix setup doc

* fix docs according to the generated htmls
2022-05-06 02:54:37 +03:00
571f720ae2 Complete official repository support (#59) 2022-05-03 00:49:32 +03:00
5030395025 rename classes in order to make documentation reference work 2022-04-18 05:30:02 +03:00
cdc018ad07 apply data migration in the same transaction block with schema migration 2022-04-18 01:19:38 +03:00
4daff81d5f Docstring update (#58)
* migrate docstrings from reST to google format

* add raises note

Also change behaviour of the `from_option` method to fallback to
disabled instead of raising exception on unknown option

* fix part of warnings for sphinx

* make identation a bit more readable

* review fixes

* add verbose description for properties to make them parsed by sphinx extenstion

* add demo sphinx generator
2022-04-17 20:25:28 +03:00
c094d4ee79 add backup and restore subcommands 2022-04-10 21:34:34 +03:00
3cb479ba4b merge restore to rebuild commannd 2022-04-10 01:51:12 +03:00
fb7275f9dd change telegram default index to telegram-index 2022-04-08 04:32:34 +03:00
1a83e55d64 add telegram integraion 2022-04-08 03:41:07 +03:00
bee41dcc55 trim versions before dependency list calculation
When dependencies list contains same package with version it tries to
find packages which don't exists
2022-04-07 20:32:55 +03:00
9f7995fb2f add support of repository restoration 2022-04-07 04:49:07 +03:00
9ce1c36f35 add support of officiall repositories api 2022-04-07 04:19:37 +03:00
4990ce4198 fix descriptions 2022-04-06 01:48:03 +03:00
432ca0cc48 better processing for subcommands
Old versions cached full output to memory and only after that printed it
into log. This behaviour causes issues in case if operation stucks and
you would need to find the step at which it does. New check_output
method uses Popen directly and iterates over stdout lines

Also changed behaviour from merging stderr into stdout to separate
stderr logging.

Any other behaviour of the function must be the same.

Also changed GPG.key_import method to disable local signing since it
seems it is useless (and may break process in case if there is no
private key)
2022-04-04 00:03:22 +03:00
d5503b22ba add configurable exit codes to some commands (#55) 2022-04-01 18:30:11 +03:00
a132b1544a create user in docker container
also fix user reading and add nocolor flag by default
2022-03-31 03:25:53 +03:00
8e6473d2a0 add help command 2022-03-31 02:18:39 +03:00
83931f5cf4 port part of settings to database (#54) 2022-03-31 01:48:06 +03:00
d4eadf0013 migrate to tox
this also requires to move default configuration files to share/ahriman.
Thus the following features have been added
* default configuration is not stored in /usr/share/ahriman/settings
* package installed via PKGBUILD now copies files from /usr
* configuration class now fallbacks to default in /usr
2022-03-21 04:38:42 +03:00
04174a3e6d deprecate init/repo-init command
In current workflow you need to run setup to run init (because of
repository name), but you need to run init before setup (because of
repository tree rights).

New solution just add `Repo.init()` method call to setup subcommand
after config reload to make sure that repository name has been applied.
In addition chown method as well as setuid method for check_output have
been added.
2022-03-21 01:32:11 +03:00
a11cce43ff fallback to AUR source in case of PermissionError exception 2022-03-20 03:46:56 +03:00
a7c6d95b34 add console printer
also add python-requests as explicit dependency and escape symbols in
repository name for badges in default tempate
2022-03-20 02:31:50 +03:00
060c7412b1 ALLOW_AUTO_ARCHITECTURE_RUN flag for unsafe commands 2022-03-19 03:08:45 +03:00
046febc440 add docker support (#52)
* add docker support

* make shellcheck happy
2022-03-13 23:43:25 +03:00
9964a96296 strict test checks 2022-03-04 02:04:52 +03:00
9529525cb5 set success status on up to date packages 2022-02-06 06:22:58 +03:00
f9b1d89756 update for new aiohttp api 2022-02-06 04:05:33 +03:00
9197b416e6 use own aur wrapper (#49) 2022-02-06 03:44:57 +03:00
1e85c1db70 handle dependencies recursively 2021-12-22 19:35:09 +03:00
32aaa4a1f8 initial implementation of the local git clones (#48) 2021-12-22 15:56:24 +03:00
2752e68ade handle packages which have been removed from the repository (#45)
* handle packages which have been removed from the repository

* manually remove packages which have been removed from the base
2021-11-10 01:37:25 +03:00
79e1cff421 take python laziness into account 2021-11-04 21:30:34 +03:00
7e3ef889e8 improve configuration extension
* Allow spaces in lists. This feature has been done in the way as shell
  interprets arguments by using quotation marks
* Clear current content on reload
2021-10-28 03:19:50 +03:00
62496478ae add update printer which will print current version if any 2021-10-27 03:11:43 +03:00
ce48169aad use PackageSource enum for Package.load method
When using add function it sill tries to load data with invalid source
2021-10-27 02:49:23 +03:00
0a8b172e9c unify aur.search method
due to specific of the AUR API in order to reduce the code we are using
own wrapper and work with it instead of direct library calls
2021-10-26 04:49:55 +03:00
7d782f120d Add ability to show more info in search and status subcommands
This feature also introduces the followiing changes
* aur-search command now works as expected with multiterms
* printer classes for managing of data print
* --sort-by argument for aur-search subcommand instead of using package
  name
* --quiet argument now has also --no-quite option
* if --quite is supplied, the log level will be set to warn instead of
  critical to be able to see error messages
* pretty_datetime function now also supports datetime objects
* BuildStatus is now pure dataclass
2021-10-26 04:27:36 +03:00
35df354c14 split application class into traits 2021-10-23 13:44:57 +03:00
4d4e5e9dde add patches to clean command 2021-10-20 03:22:16 +03:00
ff24188ca1 add ability to download package from external links (e.g. HTTP) 2021-10-20 03:09:58 +03:00
d8523bd83b always return json in responses 2021-10-20 02:12:39 +03:00
0d806e3471 add repo-status-update subcommand 2021-10-17 06:38:49 +03:00
20962f0385 allow to use multiple upload and report targets with the same name
In this feature target option must allways point to section name instead
of type. Type will be read from type option. In case if type option is
not presented it will try to check if section with architecture exists
(e.g. target = email, section = email:x86_64); if it does, the correct
section name and type will be used. Otherwise it will check if the
specified section exists; if it does, seection name and type will be
returned.
2021-10-17 06:06:08 +03:00
fd38dfd176 split github upload into generic http method and github specific
We might use some features from the http upload for another parser
2021-10-15 23:36:26 +03:00
233b1f7f39 disallow to create tree in case of unsafe run 2021-10-14 04:53:09 +03:00
a83f1d5aa5 use generic removal method 2021-10-14 04:08:21 +03:00
be09c04e78 change method spelling
in order to sort method correctly we are going to use the following
namiing schema:

{subject}_{action}_{details}

This schema still have some exceptions, e.g. single word methods, bool
methods (is_) and getters in case if they are singular (i.e. there is
no any other method with this subject)
2021-10-14 04:01:54 +03:00
bfbbb987ad add test for every file 2021-10-14 03:34:12 +03:00
86bd49ed1b exactly one called with instead of last call check 2021-10-14 03:12:45 +03:00
f6ad609616 github upload support (#41) 2021-10-14 02:30:13 +03:00
2f5790f69f add ability to add manually stored packages (#40)
* add ability to add manually stored packages

* update tests

* handle manual packages in remove-unknown method

* live fixes

also rename branches to has_remotes method and change return type
2021-10-12 21:15:35 +03:00
ef44895b0d more verbose help messages 2021-10-07 01:56:22 +03:00
5cfffbcd46 patch control subcommands 2021-10-05 08:57:42 +03:00
3e0058620b patch support (#35) 2021-10-03 15:20:36 +03:00
e897e2cde2 do not write anything on httpexceptions in log 2021-10-03 02:05:28 +03:00
edef4944f6 replace no-log with quiet
Also behavior of the flag has been changed: now it disables logs at all
2021-10-03 01:59:33 +03:00
1192f12f91 do not use set_defaults for architecture arguments
according to the source code defaults always updates the values
dictionary. This in this specific case it is impossible to override the
value it will be always empty list.

In order to handle it we are adding another property to the Handler
class which allows to run with None architecture list.

This particular set_defaults behaviour is still useful for other cases
when we have to run command without any specific architecture
2021-10-03 00:59:24 +03:00
6be6614e6d add package request endpoint 2021-10-01 08:58:50 +03:00
d60c553ea2 add debugtoolbar support 2021-09-26 12:31:12 +03:00
427ba0f0ea add ability to specify package source explicitly during the addition 2021-09-26 09:55:14 +03:00
266d2bd77d define permissions in views directly 2021-09-25 17:03:46 +03:00
f333e89bd1 only set file rights if requested 2021-09-23 20:36:22 +03:00
57f45fdc89 better reload 2021-09-17 19:15:53 +03:00
6d1f641e50 add ability to reload authentication module 2021-09-17 16:05:38 +03:00
0cf7756ec4 add ability to remove an user
also replace old user by new one before creation
2021-09-16 02:41:56 +03:00
b561bcc25d remove own implementations of getlist and getpath method in order to use
converters feature
2021-09-14 03:57:20 +03:00
25d76b0482 add ability to filter status response by package status 2021-09-13 23:27:36 +03:00
d2fe4b044f raise InvalidCommand exception in case if remove option supplied without
package
2021-09-13 22:38:38 +03:00
22272167e6 change spelling for distro name 2021-09-13 02:49:22 +03:00
68aa327670 expiration on server side support (#33) 2021-09-13 01:18:04 +03:00
41c8f4c35f return description from the search 2021-09-12 22:31:34 +03:00
168b2f6880 OAuth2 (#32)
* make auth method asyncs

* oauth2 demo support

* full coverage

* update docs
2021-09-12 21:41:38 +03:00
5bb244cbe8 add static files support and cookie expiration settings 2021-09-11 16:34:43 +03:00
11c03a9041 Add ability to trigger updates from the web (#31)
* add external process spawner and update test cases

* pass no_report to handlers

* provide service api endpoints

* do not spawn process for single architecture run

* pass no report to handlers

* make _call method of handlers public and also simplify process spawn

* move update under add

* implement actions from web page

* clear logging & improve l&f
2021-09-10 00:33:35 +03:00
214f319123 add option to set user-password for service when updating its hash 2021-09-05 15:40:03 +03:00
cfbc9d3267 move api endpoints to status-api 2021-09-05 06:46:14 +03:00
19d1e17727 allow read only pages to be requested without authorization 2021-09-05 06:09:43 +03:00
19ba2ba8e5 migration of jinja tempaltes to bootstrap (#30) 2021-09-05 05:27:58 +03:00
f49894107a add remove uknown method (#29) 2021-09-03 02:28:27 +03:00
b7cff0a19d Auth support (#25)
* initial auth implementation

* add create user parser

* add tests

* update dependencies list

* add login annd logout to index also improve auth

* realworld fixes

* add method set_option to Configuration and also use it everywhere
* split CreateUser handler to additional read method
* check user duplicate on auth mapping read
* generate salt by using passlib instead of random.choice
* case-insensetive usernames
* update dependencies
* update configuration reference
* improve tests

* fix codefactor errors

* hide fields if authorization is enabled, but no auth supplied

* add settings object for auth provider

* readme update
2021-09-02 23:36:00 +03:00
284f449225 Release 1.2.6 2021-08-21 16:19:21 +03:00
2842e40bf7 split S3.sync to different methods 2021-08-18 23:59:18 +03:00
343768e015 guess mime type for local files 2021-08-18 05:04:26 +03:00
8f55cc600e add manpage generator 2021-08-17 04:05:18 +03:00
2e89b9483f logging rethink
* well lets replace f-strings by %s as it is originally recommended
* use syslog handler by default
2021-08-17 00:23:34 +03:00
d3a00e1937 use asyncmock from unittest library 2021-08-11 21:09:10 +03:00
0129ec39ec use nosec instead of disabling mktemp rule 2021-08-11 02:51:29 +03:00
78636c2035 web templates improvements
* enable jinja autoescape by default for jinja raw generator
* allow to search by multiple strings (OR)
* replace test templates by symlink
2021-08-11 02:12:14 +03:00
47c4e5bb42 move web server to loopback by default 2021-08-11 02:02:18 +03:00
d548fa02f0 add docstrings for every fixture and test methods
also add tests for missing components
2021-08-11 01:55:27 +03:00
b5c6c286f7 skip update process if no update supplied 2021-08-10 23:25:12 +03:00
aad599be67 Native s3 sync (#23)
* Native S3 sync implementation

* fix imports

* fix paths reading

* install s3 components duriing test stage
2021-08-10 23:18:56 +03:00
dad3f418ad handle provides list 2021-07-05 22:08:04 +03:00
33cfa8fef8 Feature/all archs (#21)
* add init subcommand

* add also init command to repository object

* add ability to generate list of architectures

* check if architecture list is not empty
2021-05-23 16:40:40 +03:00
9d37e3ae23 imply no-log for every unsafe parser (#20) 2021-05-19 23:30:59 +03:00
2eb2d3a70a allow to specify list of package dependencies in rebuild target
also replace nargs= by action=append in non-positional args. It is
required to make arguments parsing result more predictable and
consistent
2021-04-18 13:34:27 +03:00
856a94ff00 import pgp key implementation (#17)
* import pgp key implementation

* do not ask confirmation for local sign. Also add argparser test

* superseed requests by python-aur package

* ...and drop --skippgpcheck makgepkg flag by default
2021-04-10 01:37:45 +03:00
f2b39fe439 add search subparser (#15) 2021-04-09 11:57:06 +03:00
213b2c65a0 Add web status route (#13)
* add status route

* typed status and get status at the start of application
2021-04-08 01:48:53 +03:00
fbccb1394d format long line 2021-04-06 05:53:38 +03:00
01887ec44b add ability to skip email report generation for empty update list 2021-04-06 05:51:50 +03:00
4729c0ea5f Email report (#11)
* Demo email report implementation

* improved ssl mode

* correct default option spelling and more fields to be hidden for not
extended reports
2021-04-06 05:45:17 +03:00
fcb892f39f implicit type conversion from command line 2021-04-04 23:53:30 +03:00
1785b0470f more options in setup command 2021-04-04 15:42:06 +03:00
c04f940ad3 fix check errors 2021-04-04 14:00:42 +03:00
207da4caa7 100% coverage 2021-04-03 21:30:57 +03:00
cad85b0f20 add ability to fitler by dependency list 2021-04-02 04:20:39 +03:00
3e8e54857f add ability to remove package from status page 2021-04-02 01:26:46 +03:00
6470b13511 Release 0.20.0 2021-04-01 02:38:59 +03:00
4e08297311 constistent classmethod and staticmethod usage
General idea is to use classmethod for every constructor and
statismethod otherwise.
Also use self and cls whenever it's possible to call static and class
methods
2021-03-31 04:29:08 +03:00
32c3c52874 change arch specific section naming from section_arch to section:arch
Some archs can have _ in their name. Also in future we can use sections
with similar names
2021-03-31 02:31:14 +03:00
e21b990777 add now argument to add command 2021-03-31 02:19:13 +03:00
cf276f2398 make configuration object arch-specific 2021-03-31 00:04:13 +03:00
4ca2348f0d more verbose variables 2021-03-30 05:29:13 +03:00
eb02e1e62b merge settings groups instead of using whole group 2021-03-30 04:58:15 +03:00
8d7d508bb2 move rsync and s3 options to configuration 2021-03-30 02:38:18 +03:00
eb7f3b2484 rename gpg test to correct naming 2021-03-30 01:53:23 +03:00
e87402fdde add more tests 2021-03-30 01:42:01 +03:00
74fc283078 more properties to be shown in status pages 2021-03-29 11:48:32 +03:00
63529633ec add status update subcommand
also satisfy pylint with too big method with too much variables
2021-03-29 04:17:10 +03:00
a89888769f Setup command (#9)
* block issues without templates

* add setup subcommand

* handle devtools config correctly
2021-03-29 03:24:58 +03:00
d744fe1647 remove unused import 2021-03-28 16:24:51 +03:00
6f1b37c4cb add sign command (#7) (#8) 2021-03-28 16:24:00 +03:00
60b8477cde Add tests (#1) (#5)
* add models tests (#1)

also replace single quote to double one to confort PEP docstring
+ move _check_output to class properties to make it available for
mocking

* alpm tests implementation

* try to replace os with pathlib

* update tests for pathlib

* fix includes glob and trim version from dependencies

* build_tools package tests

* repository component tests

* add sign tests

* complete status tests

* handle exceptions in actual_version calls

* complete core tests

* move configuration to root conftest

* application tests

* complete application tests

* change copyright to more generic one

* base web tests

* complete web tests

* complete testkit

also add argument parsers test
2021-03-28 15:30:51 +03:00