docs: url spelling

Also add recipe for OAuth with GitHub setup

docs/configuration.rst

@@ -64,7 +64,7 @@ Base authorization settings. ``OAuth`` provider requires ``aioauth-client`` libr
 * ``allow_read_only`` - allow requesting status APIs without authorization, boolean, required.
 * ``client_id`` - OAuth2 application client ID, string, required in case if ``oauth`` is used.
 * ``client_secret`` - OAuth2 application client secret key, string, required in case if ``oauth`` is used.
-* ``cookie_secret_key`` - secret key which will be used for cookies encryption, string, optional. It must be 32 url-safe base64-encoded bytes and can be generated as following ``base64.urlsafe_b64encode(os.urandom(32)).decode("utf8")``. If not set, it will be generated automatically; note, however, that in this case, all sessions will be automatically invalidated during the service restart.
+* ``cookie_secret_key`` - secret key which will be used for cookies encryption, string, optional. It must be 32 URL-safe base64-encoded bytes and can be generated as following ``base64.urlsafe_b64encode(os.urandom(32)).decode("utf8")``. If not set, it will be generated automatically; note, however, that in this case, all sessions will be automatically invalidated during the service restart.
 * ``max_age`` - parameter which controls both cookie expiration and token expiration inside the service in seconds, integer, optional, default is 7 days.
 * ``oauth_icon`` - OAuth2 login button icon, string, optional, default is ``google``. Must be valid `Bootstrap icon <https://icons.getbootstrap.com/>`__ name.
 * ``oauth_provider`` - OAuth2 provider class name as is in ``aioauth-client`` (e.g. ``GoogleClient``, ``GithubClient`` etc), string, required in case if ``oauth`` is used.
@@ -86,7 +86,7 @@ Build related configuration. Group name can refer to architecture, e.g. ``build:
 * ``triggers`` - list of ``ahriman.core.triggers.Trigger`` class implementation (e.g. ``ahriman.core.report.ReportTrigger ahriman.core.upload.UploadTrigger``) which will be loaded and run at the end of processing, space separated list of strings, optional. You can also specify triggers by their paths, e.g. ``/usr/lib/python3.10/site-packages/ahriman/core/report/report.py.ReportTrigger``. Triggers are run in the order of definition.
 * ``triggers_known`` - optional list of ``ahriman.core.triggers.Trigger`` class implementations which are not run automatically and used only for trigger discovery and configuration validation.
 * ``vcs_allowed_age`` - maximal age in seconds of the VCS packages before their version will be updated with its remote source, integer, optional, default is 7 days.
-* ``workers`` - list of worker nodes addresses used for build process, space separated list of strings, optional. Each worker address must be valid and reachable url, e.g. ``https://10.0.0.1:8080``. If none set, the build process will be run on the current node. There is also special trigger which loads this value based on the list of the discovered nodes.
+* ``workers`` - list of worker nodes addresses used for build process, space separated list of strings, optional. Each worker address must be valid and reachable URL, e.g. ``https://10.0.0.1:8080``. If none set, the build process will be run on the current node. There is also special trigger which loads this value based on the list of the discovered nodes.
 
 ``repository`` group
 --------------------
@@ -109,7 +109,7 @@ Settings for signing packages or repository. Group name can refer to architectur
 Reporting to web service related settings. In most cases there is fallback to web section settings.
 
 * ``enabled`` - enable reporting to web service, boolean, optional, default ``yes`` for backward compatibility.
-* ``address`` - remote web service address with protocol, string, optional. In case of websocket, the ``http+unix`` scheme and url encoded address (e.g. ``%2Fvar%2Flib%2Fahriman`` for ``/var/lib/ahriman``) must be used, e.g. ``http+unix://%2Fvar%2Flib%2Fahriman%2Fsocket``. In case if none set, it will be guessed from ``web`` section.
+* ``address`` - remote web service address with protocol, string, optional. In case of websocket, the ``http+unix`` scheme and URL encoded address (e.g. ``%2Fvar%2Flib%2Fahriman`` for ``/var/lib/ahriman``) must be used, e.g. ``http+unix://%2Fvar%2Flib%2Fahriman%2Fsocket``. In case if none set, it will be guessed from ``web`` section.
 * ``password`` - password to authorize in web service in order to update service status, string, required in case if authorization enabled.
 * ``suppress_http_log_errors`` - suppress HTTP log errors, boolean, optional, default ``no``. If set to ``yes``, any HTTP log errors (e.g. if web server is not available, but HTTP logging is enabled) will be suppressed.
 * ``timeout`` - HTTP request timeout in seconds, integer, optional, default is ``30``.
@@ -123,7 +123,7 @@ Web server settings. This feature requires ``aiohttp`` libraries to be installed
 * ``address`` - optional address in form ``proto://host:port`` (``port`` can be omitted in case of default ``proto`` ports), will be used instead of ``http://{host}:{port}`` in case if set, string, optional. This option is required in case if ``OAuth`` provider is used.
 * ``enable_archive_upload`` - allow to upload packages via HTTP (i.e. call of ``/api/v1/service/upload`` uri), boolean, optional, default ``no``.
 * ``host`` - host to bind, string, optional.
-* ``index_url`` - full url of the repository index page, string, optional.
+* ``index_url`` - full URL of the repository index page, string, optional.
 * ``max_body_size`` - max body size in bytes to be validated for archive upload, integer, optional. If not set, validation will be disabled.
 * ``port`` - port to bind, integer, optional.
 * ``service_only`` - disable status routes (including logs), boolean, optional, default ``no``.
@@ -145,7 +145,7 @@ Keyring generator plugin
 
 * ``type`` - type of the generator, string, optional, must be set to ``keyring-generator`` if exists.
 * ``description`` - keyring package description, string, optional, default is ``repo PGP keyring``, where ``repo`` is the repository name.
-* ``homepage`` - url to homepage location if any, string, optional.
+* ``homepage`` - URL to homepage location if any, string, optional.
 * ``license`` - list of licenses which are applied to this package, space separated list of strings, optional, default is ``Unlicense``.
 * ``package`` - keyring package name, string, optional, default is ``repo-keyring``, where ``repo`` is the repository name.
 * ``packagers`` - list of packagers keys, space separated list of strings, optional, if not set, the ``key_*`` options from ``sign`` group will be used.
@@ -164,7 +164,7 @@ Mirrorlist generator plugin
 
 * ``type`` - type of the generator, string, optional, must be set to ``mirrorlist-generator`` if exists.
 * ``description`` - mirrorlist package description, string, optional, default is ``repo mirror list for use by pacman``, where ``repo`` is the repository name.
-* ``homepage`` - url to homepage location if any, string, optional.
+* ``homepage`` - URL to homepage location if any, string, optional.
 * ``license`` - list of licenses which are applied to this package, space separated list of strings, optional, default is ``Unlicense``.
 * ``package`` - mirrorlist package name, string, optional, default is ``repo-mirrorlist``, where ``repo`` is the repository name.
 * ``path`` - absolute path to generated mirrorlist file, string, optional, default is ``/etc/pacman.d/repo-mirrorlist``, where ``repo`` is the repository name.
@@ -175,7 +175,7 @@ Mirrorlist generator plugin
 
 Remote git source synchronization settings. Unlike ``Upload`` triggers those triggers are used for PKGBUILD synchronization - fetch from remote repository PKGBUILDs before updating process.
 
-It supports authorization; to do so you'd need to prefix the url with authorization part, e.g. ``https://key:token@github.com/arcan1s/ahriman.git``. It is highly recommended to use application tokens instead of your user authorization details. Alternatively, you can use any other option supported by git, e.g.:
+It supports authorization; to do so you'd need to prefix the URL with authorization part, e.g. ``https://key:token@github.com/arcan1s/ahriman.git``. It is highly recommended to use application tokens instead of your user authorization details. Alternatively, you can use any other option supported by git, e.g.:
 
 * by SSH key: generate SSH key as ``ahriman`` user and put public part of it to the repository keys.
 * by git credentials helper: consult with the `related man page <https://git-scm.com/docs/gitcredentials>`__.
@@ -187,7 +187,7 @@ Available options are:
 Remote pull trigger
 ^^^^^^^^^^^^^^^^^^^
 
-* ``pull_url`` - url of the remote repository from which PKGBUILDs can be pulled before build process, string, required.
+* ``pull_url`` - URL of the remote repository from which PKGBUILDs can be pulled before build process, string, required.
 * ``pull_branch`` - branch of the remote repository from which PKGBUILDs can be pulled before build process, string, optional, default is ``master``.
 
 ``remote-push`` group
@@ -195,7 +195,7 @@ Remote pull trigger
 
 Remote git source synchronization settings. Same as remote pull triggers those triggers are used for PKGBUILD synchronization - push updated PKGBUILDs to the remote repository after build process.
 
-It supports authorization; to do so you'd need to prefix the url with authorization part, e.g. ``https://key:token@github.com/arcan1s/ahriman.git``. It is highly recommended to use application tokens instead of your user authorization details. Alternatively, you can use any other option supported by git, e.g.:
+It supports authorization; to do so you'd need to prefix the URL with authorization part, e.g. ``https://key:token@github.com/arcan1s/ahriman.git``. It is highly recommended to use application tokens instead of your user authorization details. Alternatively, you can use any other option supported by git, e.g.:
 
 * by SSH key: generate SSH key as ``ahriman`` user and put public part of it to the repository keys.
 * by git credentials helper: consult with the `related man page <https://git-scm.com/docs/gitcredentials>`__.
@@ -209,7 +209,7 @@ Remote push trigger
 
 * ``commit_email`` - git commit email, string, optional, default is ``ahriman@localhost``.
 * ``commit_user`` - git commit user, string, optional, default is ``ahriman``.
-* ``push_url`` - url of the remote repository to which PKGBUILDs should be pushed after build process, string, required.
+* ``push_url`` - URL of the remote repository to which PKGBUILDs should be pushed after build process, string, required.
 * ``push_branch`` - branch of the remote repository to which PKGBUILDs should be pushed after build process, string, optional, default is ``master``.
 
 ``report`` group

docs/faq.rst

@@ -174,7 +174,7 @@ For that purpose you could use ``RemotePullTrigger`` trigger. To do so you will
    [gitremote]
    pull_url = https://github.com/username/repository
 
-During the next application run it will fetch repository from the specified url and will try to find packages there which can be used as local sources.
+During the next application run it will fetch repository from the specified URL and will try to find packages there which can be used as local sources.
 
 This feature can be also used to build packages which are not listed in AUR, the example of the feature use can be found `here <https://github.com/arcan1s/ahriman/tree/master/recipes/pull>`__.
 
@@ -191,7 +191,7 @@ For that purpose you'd need to use another trigger called ``RemotePushTrigger``.
    [gitremote]
    push_url = https://github.com/username/repository
 
-Unlike ``RemotePullTrigger`` trigger, the ``RemotePushTrigger`` more likely will require authorization. It is highly recommended to use application tokens for that instead of using your password (e.g. for GitHub you can generate tokens `here <https://github.com/settings/tokens>`__ with scope ``public_repo``). Authorization can be supplied by using authorization part of the url, e.g. ``https://key:token@github.com/username/repository``.
+Unlike ``RemotePullTrigger`` trigger, the ``RemotePushTrigger`` more likely will require authorization. It is highly recommended to use application tokens for that instead of using your password (e.g. for GitHub you can generate tokens `here <https://github.com/settings/tokens>`__ with scope ``public_repo``). Authorization can be supplied by using authorization part of the URL, e.g. ``https://key:token@github.com/username/repository``.
 
 How to change PKGBUILDs before build
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -461,7 +461,7 @@ The following environment variables are supported:
 * ``AHRIMAN_POSTSETUP_COMMAND`` - if set, the command which will be called (as root) after the setup command, but before any other actions.
 * ``AHRIMAN_PRESETUP_COMMAND`` - if set, the command which will be called (as root) right before the setup command.
 * ``AHRIMAN_REPOSITORY`` - repository name, default is ``aur-clone``.
-* ``AHRIMAN_REPOSITORY_SERVER`` - optional override for the repository url. Useful if you would like to download packages from remote instead of local filesystem.
+* ``AHRIMAN_REPOSITORY_SERVER`` - optional override for the repository URL. Useful if you would like to download packages from remote instead of local filesystem.
 * ``AHRIMAN_REPOSITORY_ROOT`` - repository root. Because of filesystem rights it is required to override default repository root. By default, it uses ``ahriman`` directory inside ahriman's home, which can be passed as mount volume.
 * ``AHRIMAN_UNIX_SOCKET`` - full path to unix socket which is used by web server, default is empty. Note that more likely you would like to put it inside ``AHRIMAN_REPOSITORY_ROOT`` directory (e.g. ``/var/lib/ahriman/ahriman/ahriman-web.sock``) or to ``/tmp``.
 * ``AHRIMAN_USER`` - ahriman user, usually must not be overwritten, default is ``ahriman``.

docs/triggers.rst

@@ -49,7 +49,7 @@ Generator for keyring package. This trigger will extract keys from local keychai
 ``ahriman.core.support.MirrorlistTrigger``
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-Simple generator for mirrorlist package, based on the urls which were set by configuration. This trigger will generate sources including PKGBUILD, which can be used later for package building.
+Simple generator for mirrorlist package, based on the URLs which were set by configuration. This trigger will generate sources including PKGBUILD, which can be used later for package building.
 
 ``ahriman.core.upload.UploadTrigger``
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -95,7 +95,7 @@ Trigger example
 
 Lets consider example of reporting trigger (e.g. `slack <https://slack.com/>`__, which provides easy HTTP API for integration triggers).
 
-In order to post message to slack we will need a specific trigger url (something like ``https://hooks.slack.com/services/company_id/trigger_id``), channel (e.g. ``#archrepo``) and username (``repo-bot``).
+In order to post message to slack we will need a specific trigger URL (something like ``https://hooks.slack.com/services/company_id/trigger_id``), channel (e.g. ``#archrepo``) and username (``repo-bot``).
 
 As it has been mentioned, our trigger must derive from specific class:
 

recipes/README.md

@@ -9,7 +9,9 @@ Collection of the examples of docker compose configuration files, which covers s
 * [Distributed](distributed): cluster of three nodes, one with web interface and two workers which are responsible for build process.
 * [Distributed manual](distributed-manual): same as [distributed](distributed), but two nodes and update process must be run on worker node manually.
 * [i686](i686): non-x86_64 architecture setup.
+* [Index](index): repository with index page generator enabled.
 * [Multi repo](multirepo): run web service with two separated repositories.
+* [OAuth](oauth): web service with OAuth (GitHub provider) authentication enabled.
 * [Pull](pull): normal service, but in addition with pulling packages from another source (e.g. GitHub repository).
 * [Sign](sign): create repository with database signing.
 * [Web](web): simple web service with authentication enabled.

recipes/index/README.md

@@ -0,0 +1,6 @@
+# Index
+
+1. Setup repository named `ahriman-demo` with architecture `x86_64`.
+2. Generate index page. 
+3. Repository is available at `http://localhost:8080/repo`.
+4. Index page is available at `http://localhost:8080/repo/ahriman-demo/x86_64/index.html`

recipes/index/compose.yml

@@ -0,0 +1,48 @@
+services:
+  backend:
+    image: arcan1s/ahriman:edge
+    privileged: true
+
+    environment:
+      AHRIMAN_DEBUG: yes
+      AHRIMAN_OUTPUT: console
+      AHRIMAN_REPOSITORY: ahriman-demo
+
+    configs:
+      - source: service
+        target: /etc/ahriman.ini.d/99-settings.ini
+
+    volumes:
+      - type: volume
+        source: repository
+        target: /var/lib/ahriman
+        volume:
+          nocopy: true
+
+    command: repo-report
+
+  frontend:
+    image: nginx
+    ports:
+      - 8080:80
+
+    configs:
+      - source: nginx
+        target: /etc/nginx/conf.d/default.conf
+
+    volumes:
+      - type: volume
+        source: repository
+        target: /srv
+        read_only: true
+        volume:
+          nocopy: true
+
+configs:
+  nginx:
+    file: nginx.conf
+  service:
+    file: service.ini
+
+volumes:
+  repository:

recipes/index/nginx.conf

@@ -0,0 +1,9 @@
+server {
+    listen 80;
+
+    location /repo {
+        rewrite ^/repo/(.*) /$1 break;
+        autoindex on;
+        root /srv/ahriman/repository;
+    }
+}

recipes/index/service.ini

@@ -0,0 +1,6 @@
+[report]
+target = html
+
+[html]
+path = /var/lib/ahriman/ahriman/repository/ahriman-demo/x86_64/index.html
+link_path = http://localhost:8080/repo/ahriman-demo/x86_64

recipes/oauth/README.md

@@ -0,0 +1,15 @@
+# OAuth
+
+1. Create user from `AHRIMAN_OAUTH_USER` environment variable (same as GitHub user).
+2. Configure OAuth to use GitHub provider with client ID and secret specified in variables `AHRIMAN_OAUTH_CLIENT_ID` and `AHRIMAN_OAUTH_CLIENT_SECRET` variables respectively.   
+3. Setup repository named `ahriman-demo` with architecture `x86_64`.
+4. Start web server at port `8080`.
+5. Repository is available at `http://localhost:8080/repo`.
+
+Before you start, you need to create an application. It can be done by:
+
+1. Go to `https://github.com/settings/applications/new`
+2. Set application name and its homepage.
+3. Set callback url to `http://localhost:8080/api/v1/login`
+4. Copy Client ID.
+5. Generate new client secret and copy it.

recipes/oauth/compose.yml

@@ -0,0 +1,58 @@
+services:
+  backend:
+    image: arcan1s/ahriman:edge
+    privileged: true
+
+    environment:
+      AHRIMAN_DEBUG: yes
+      AHRIMAN_OAUTH_CLIENT_ID: ${AHRIMAN_OAUTH_CLIENT_ID}
+      AHRIMAN_OAUTH_CLIENT_SECRET: ${AHRIMAN_OAUTH_CLIENT_SECRET}
+      AHRIMAN_OUTPUT: console
+      AHRIMAN_PORT: 8080
+      AHRIMAN_PRESETUP_COMMAND: sudo -u ahriman ahriman user-add ${AHRIMAN_OAUTH_USER} -R full -p ""
+      AHRIMAN_REPOSITORY: ahriman-demo
+      AHRIMAN_UNIX_SOCKET: /var/lib/ahriman/ahriman/ahriman.sock
+
+    configs:
+      - source: service
+        target: /etc/ahriman.ini.d/99-settings.ini
+
+    volumes:
+      - type: volume
+        source: repository
+        target: /var/lib/ahriman
+        volume:
+          nocopy: true
+
+    healthcheck:
+      test: curl --fail --silent --output /dev/null http://backend:8080/api/v1/info
+      interval: 10s
+      start_period: 30s
+
+    command: web
+
+  frontend:
+    image: nginx
+    ports:
+      - 8080:80
+
+    configs:
+      - source: nginx
+        target: /etc/nginx/conf.d/default.conf
+
+    volumes:
+      - type: volume
+        source: repository
+        target: /srv
+        read_only: true
+        volume:
+          nocopy: true
+
+configs:
+  nginx:
+    file: nginx.conf
+  service:
+    file: service.ini
+
+volumes:
+  repository:

recipes/oauth/nginx.conf

@@ -0,0 +1,18 @@
+server {
+    listen 80;
+
+    location /repo {
+        rewrite ^/repo/(.*) /$1 break;
+        autoindex on;
+        root /srv/ahriman/repository;
+    }
+
+    location / {
+        proxy_set_header Host $host;
+        proxy_set_header X-Real-IP $remote_addr;
+        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarder-Proto $scheme;
+
+        proxy_pass http://backend:8080;
+    }
+}

recipes/oauth/service.ini

@@ -0,0 +1,11 @@
+[auth]
+target = oauth
+client_id = $AHRIMAN_OAUTH_CLIENT_ID
+client_secret = $AHRIMAN_OAUTH_CLIENT_SECRET
+
+oauth_icon = github
+oauth_provider = GithubClient
+oauth_scopes = read:user
+
+[web]
+address = http://localhost:8080

src/ahriman/core/auth/oauth.py

@@ -130,7 +130,7 @@ class OAuth(Mapping):
             client.access_token = access_token
 
             user, _ = await client.user_info()
-            username: str = user.email  # type: ignore[attr-defined]
+            username: str = user.email or user.username  # type: ignore[attr-defined]
             return username
         except Exception:
             self.logger.exception("got exception while performing request")

src/ahriman/core/http/sync_http_client.py

@@ -77,12 +77,12 @@ class SyncHttpClient(LazyLogging):
         return session
 
     @staticmethod
-    def exception_response_text(exception: requests.exceptions.RequestException) -> str:
+    def exception_response_text(exception: requests.RequestException) -> str:
         """
         safe response exception text generation
 
         Args:
-            exception(requests.exceptions.RequestException): exception raised
+            exception(requests.RequestException): exception raised
 
         Returns:
             str: text of the response if it is not None and empty string otherwise

tests/ahriman/core/auth/test_oauth.py

@@ -75,6 +75,17 @@ async def test_get_oauth_username(oauth: OAuth, mocker: MockerFixture) -> None:
     assert email == "email"
 
 
+async def test_get_oauth_username_empty_email(oauth: OAuth, mocker: MockerFixture) -> None:
+    """
+    must read username if email is not available
+    """
+    mocker.patch("aioauth_client.GoogleClient.get_access_token", return_value=("token", ""))
+    mocker.patch("aioauth_client.GoogleClient.user_info", return_value=(aioauth_client.User(username="username"), ""))
+
+    username = await oauth.get_oauth_username("code")
+    assert username == "username"
+
+
 async def test_get_oauth_username_exception_1(oauth: OAuth, mocker: MockerFixture) -> None:
     """
     must return None in case of OAuth request error (get_access_token)