refactor: update GitHub workflows for release management and image build #97
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ilding - Added new workflows for creating releases, preparing releases, and publishing libraries on merge. - Implemented logic to derive version from pull request titles and create Git tags/releases accordingly. - Enhanced image building workflows to include digest capturing and improved error handling. - Refactored existing workflows to streamline the process of bumping versions for internal libraries and services. - Introduced scripts for bumping chart versions and updating pyproject dependencies. - Removed obsolete scripts and workflows to clean up the repository.
… management and development workflow - Updated Dockerfiles for services and libs to use Python 3.13 and 3.11.7. - Introduced separate Dockerfile.dev for development environments with optimized caching. - Enhanced dependency installation logic in Dockerfiles to streamline dev and prod setups. - Updated Poetry version to 2.1.3 across all services and libs. - Refined pyproject.toml files to include TestPyPI sources and group dependencies for better management. - Improved logging setup in Dockerfiles for better runtime diagnostics.
- Moved test dependencies to a dedicated group and added pytest and coverage. - Created a separate lint group with flake8 and related plugins. - Added debugpy and rag-core libraries to the dev group. - Updated production dependencies for rag-core libraries to version 2.2.1.
- Updated `pyproject.toml` and `poetry.lock` files to enforce `setuptools` version to be less than 80.9 in multiple services. - Adjusted per-file ignores for linting in `admin-backend` and `document-extractor`. - Streamlined Dockerfiles for `document-extractor`, `rag-backend`, and `mcp-server` to improve build efficiency and user permissions. - Ensured consistent formatting in dependency declarations across services. - Updated versioning for `pdfextractor_server` to 2.2.1. - Added minimal library structures in Dockerfiles to facilitate local package recognition by Poetry.
- Moved `pytest` and `pytest-asyncio` to a new test dependency group in `rag-core-lib`. - Added `coverage` to the test dependencies in `rag-core-lib`. - Created a separate lint dependency group in `rag-core-lib` and included `flake8` and related packages. - Updated markers in `poetry.lock` for better platform compatibility. - Removed `pytest-asyncio` from the `rag-backend` dependencies and adjusted its group to only include `test`.
… prod-local group for development
… Dockerfile cleanup and user permissions; adjust pyproject.toml for local dependencies
Contributor
There was a problem hiding this comment.
Pull Request Overview
This pull request refactors the release automation from a single monolithic workflow to a modular approach with label-based gating. It removes the legacy semantic-release workflow and introduces specialized workflows for each stage of the release process, along with new Python utilities for dependency and version management.
- Split release automation into 5 specialized workflows with label-based gating
- Added Python utilities for version bumping, dependency management, and chart updates
- Updated service names and versions to align with new release structure
Reviewed Changes
Copilot reviewed 14 out of 14 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| tools/update-helm-values.py | Removed legacy Helm values update script |
| tools/bump_pyproject_deps.py | Added utility for managing Python library versions and service dependency pins |
| tools/bump_chart_versions.py | Added utility for converting app versions to Helm chart versions |
| services/*/pyproject.toml | Updated service names and versions for consistency |
| .github/workflows/semantic-release.yml | Removed monolithic release workflow |
| .github/workflows/*.yml | Added modular workflows for prepare-release, publish-libs, build-images, create-release, and publish-chart |
| .github/workflows/lint-and-test.yml | Added label-based gating to skip CI during release automation |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
.github/workflows/publish-chart.yml
Outdated
| uses: azure/setup-helm@v4 | ||
|
|
||
| - name: Login to GHCR for Helm OCI | ||
| run: echo ${{ secrets.GHCR_PAT }} | helm registry login ghcr.io -u ${{ github.actor }} --password-stdin |
There was a problem hiding this comment.
The GHCR_PAT secret is being echoed directly in the pipeline output, which could expose the token in logs. Use the helm registry login command with the --password-stdin flag correctly by piping from a secure environment variable instead.
Suggested change
| run: echo ${{ secrets.GHCR_PAT }} | helm registry login ghcr.io -u ${{ github.actor }} --password-stdin | |
| run: printf "%s" "${{ secrets.GHCR_PAT }}" | helm registry login ghcr.io -u ${{ github.actor }} --password-stdin |
tools/bump_pyproject_deps.py
Outdated
Comment on lines
39
to
51
| for i, line in enumerate(lines): | ||
| stripped = line.strip() | ||
| if stripped.startswith('[tool.poetry]'): | ||
| in_tool_poetry = True | ||
| continue | ||
| if in_tool_poetry and stripped.startswith('[') and not stripped.startswith('[tool.poetry]'): | ||
| # left the section without finding version; stop scanning section | ||
| break | ||
| if in_tool_poetry and stripped.startswith('version'): | ||
| # Replace only the version value, keep indentation and spacing | ||
| lines[i] = re.sub(r'version\s*=\s*"[^"]*"', f'version = "{new_version}"', line) | ||
| return ''.join(lines) | ||
| # If no version line found, append it to the [tool.poetry] section |
There was a problem hiding this comment.
This fallback adds a duplicate [tool.poetry] section when no version line is found, which would create invalid TOML. The logic should append the version line to an existing [tool.poetry] section or handle the case where the section doesn't exist properly.
Suggested change
| for i, line in enumerate(lines): | |
| stripped = line.strip() | |
| if stripped.startswith('[tool.poetry]'): | |
| in_tool_poetry = True | |
| continue | |
| if in_tool_poetry and stripped.startswith('[') and not stripped.startswith('[tool.poetry]'): | |
| # left the section without finding version; stop scanning section | |
| break | |
| if in_tool_poetry and stripped.startswith('version'): | |
| # Replace only the version value, keep indentation and spacing | |
| lines[i] = re.sub(r'version\s*=\s*"[^"]*"', f'version = "{new_version}"', line) | |
| return ''.join(lines) | |
| # If no version line found, append it to the [tool.poetry] section | |
| tool_poetry_start = None | |
| tool_poetry_end = None | |
| version_found = False | |
| for i, line in enumerate(lines): | |
| stripped = line.strip() | |
| if stripped.startswith('[tool.poetry]'): | |
| in_tool_poetry = True | |
| tool_poetry_start = i | |
| continue | |
| if in_tool_poetry and stripped.startswith('[') and not stripped.startswith('[tool.poetry]'): | |
| # left the section without finding version; stop scanning section | |
| tool_poetry_end = i | |
| in_tool_poetry = False | |
| break | |
| if in_tool_poetry and stripped.startswith('version'): | |
| # Replace only the version value, keep indentation and spacing | |
| lines[i] = re.sub(r'version\s*=\s*"[^"]*"', f'version = "{new_version}"', line) | |
| version_found = True | |
| return ''.join(lines) | |
| # If [tool.poetry] section exists but no version line found, insert it | |
| if tool_poetry_start is not None: | |
| insert_at = tool_poetry_start + 1 | |
| # If we found the end of the section, insert before it | |
| if tool_poetry_end is not None: | |
| insert_at = tool_poetry_end | |
| # Determine indentation (match other lines in section, or default to 4 spaces) | |
| indent = "" | |
| for j in range(tool_poetry_start + 1, (tool_poetry_end or len(lines))): | |
| content = lines[j].lstrip('\r\n') | |
| if content and not content.strip().startswith('['): | |
| indent = lines[j][:len(lines[j]) - len(lines[j].lstrip())] | |
| break | |
| lines.insert(insert_at, f"{indent}version = \"{new_version}\"\n") | |
| return ''.join(lines) | |
| # If no [tool.poetry] section found, append it at the end |
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
…tcloud/rag-template into refactor/prod-and-dev-Dockerfiles
…xception and removing traceback usage
…oss extractor modules
…d dummy test module with basic functionality
- Updated version to v3.4.0 in pyproject.toml for mcp-server and rag-backend. - Enhanced author and maintainer information in pyproject.toml files. - Updated fastmcp dependency from ^2.8.0 to ^2.13.1 in mcp-server. - Updated various dependencies in rag-backend, including fastapi to ^0.121.2 and langchain to ^1.0.8. - Improved docstrings for chat_endpoint.py, container.py, and main.py for better clarity. - Added new dependencies in poetry.lock for langchain-classic and annotated-doc. - Updated test file dummy3_test.py with module-level docstring and improved test function docstring. - Adjusted per-file ignores in pyproject.toml for better linting.
…e-clean-semver workflow
…or dependency management
…low_dispatch for main branch
…ump script for PR label checks
…and API request handling
…eplace add_testpypi_source script
…rride and improve change detection logic
…api-lib, extractor-api-lib, and rag-core-api
Comment on lines
13
to
62
| def ensure(mapping: Dict[str, Any], key: str) -> Dict[str, Any]: | ||
| """Ensure key exists and is a dict.""" | ||
| if key not in mapping or mapping[key] is None: | ||
| mapping[key] = {} | ||
| if not isinstance(mapping[key], dict): | ||
| raise TypeError(f"Expected dict at {key}, got {type(mapping[key])}") | ||
| return mapping[key] | ||
|
|
||
|
|
||
| def update_values(values_path: Path, image_registry: str, image_tag: str) -> None: | ||
| if values_path.exists(): | ||
| data = yaml.safe_load(values_path.read_text(encoding="utf-8")) or {} | ||
| else: | ||
| data = {} | ||
|
|
||
| components = { | ||
| "backend": "rag-backend", | ||
| "adminBackend": "admin-backend", | ||
| "extractor": "document-extractor", | ||
| "frontend": "frontend", | ||
| "adminFrontend": "admin-frontend", | ||
| } | ||
|
|
||
| for key, image_name in components.items(): | ||
| comp = ensure(data, key) | ||
| image_block = ensure(comp, "image") | ||
| image_block["repository"] = f"{image_registry}/{image_name}" | ||
| image_block["tag"] = image_tag | ||
|
|
||
| backend = ensure(data, "backend") | ||
| mcp = ensure(backend, "mcp") | ||
| mcp_image = ensure(mcp, "image") | ||
| mcp_image["repository"] = f"{image_registry}/mcp-server" | ||
| mcp_image["tag"] = image_tag | ||
|
|
||
| values_path.write_text(yaml.safe_dump(data, sort_keys=False), encoding="utf-8") | ||
|
|
||
|
|
||
| def main() -> None: | ||
| parser = argparse.ArgumentParser(description="Update image overrides in a values file.") | ||
| parser.add_argument("--values-file", required=True, help="Path to values-qa.yaml in deployment repo") | ||
| parser.add_argument("--image-registry", required=True, help="Image registry base (e.g. registry.onstackit.cloud/qa-rag-template)") | ||
| parser.add_argument("--image-tag", required=True, help="Image tag/version to set") | ||
| args = parser.parse_args() | ||
|
|
||
| update_values(Path(args.values_file), args.image_registry, args.image_tag) | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| main() |
There was a problem hiding this comment.
Missing type hints for function return values. The functions 'ensure', 'update_values', and 'main' should specify return types for better type safety and code clarity. Following the repository's Python conventions, all functions should have complete type annotations.
tools/publish_libs.sh
Outdated
| echo "$name==$version visible on $label" | ||
| return 0 | ||
| fi | ||
| sleep 5 |
There was a problem hiding this comment.
The hardcoded 5-second sleep on line 59 in the wait loop is a magic number. This should be extracted as a constant (e.g., 'POLL_INTERVAL_SECONDS = 5') at the top of the script for better maintainability and clarity.
tools/allow_chart_bump.py
Outdated
| req.add_header("Authorization", f"Bearer {token}") | ||
| req.add_header("Accept", accept or "application/vnd.github+json") | ||
| req.add_header("X-GitHub-Api-Version", API_VERSION) | ||
| with urllib.request.urlopen(req) as resp: |
There was a problem hiding this comment.
The workflow uses 'urllib.request.urlopen' without timeout configuration, which can cause the workflow to hang indefinitely if the GitHub API is slow or unresponsive. Add a timeout parameter to prevent hanging, e.g., 'urllib.request.urlopen(req, timeout=30)'.
tools/bump_chart_versions.py
Outdated
…rsion management, and testing
stackit-sven
approved these changes
Jan 21, 2026
a-klos
changed the base branch from
refactor/prod-and-dev-Dockerfiles
to
release/v4
a-klos
changed the base branch from
release/v4
to
refactor/prod-and-dev-Dockerfiles
a-klos
changed the base branch from
refactor/prod-and-dev-Dockerfiles
to
release/v4
- Added optional fields for username and password for key value store authentication. - Introduced SSL configuration options including use_ssl, ssl_cert_reqs, ssl_ca_certs, ssl_certfile, ssl_keyfile, and ssl_check_hostname. chore(poetry.lock): update package groups for consistency - Ensured all packages have appropriate group assignments for main, lint, and test categories. - Removed duplicate group entries and ensured proper formatting across the file.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request restructures and improves the release automation for the repository by splitting the previous monolithic workflow into several specialized GitHub Actions workflows. These new workflows automate versioning, library publishing, Docker image building, Helm chart packaging, and dependency lockfile refreshing. Additionally, the workflows now use label-based gating to control releases and dependency updates, and they improve version extraction and artifact handling. The previous workflow file
.github/workflows/semantic-release.ymlhas been removed and replaced with more maintainable, modular workflows.Release and Versioning Automation
.github/workflows/prepare-release.ymlto automate semantic version calculation and bump internal library versions, opening a PR with the new versions and gating future steps with theprepare-releaselabel..github/workflows/create-release.ymlto create a Git tag and GitHub Release when a PR with therefresh-lockslabel is merged tomain, extracting the version from the PR title.Library Publishing and Dependency Management
.github/workflows/publish-libs-on-merge.ymlto publish Python libraries to TestPyPI after aprepare-releasePR is merged, update service dependency pins, refresh lockfiles, and open a PR with updated lockfiles using therefresh-lockslabel.Docker Image and Helm Chart Automation
.github/workflows/build-images.ymlto build and publish Docker images for all services when a release is published, capturing image digests as artifacts..github/workflows/publish-chart.ymlto package and publish the Helm chart after images are built, bumping chart versions and opening a PR for chart version updates using thechart-bumplabel.Workflow Gating and Cleanup
.github/workflows/lint-and-test.ymlto skip jobs if any of the release-related labels (prepare-release,refresh-locks,chart-bump) are present, preventing unnecessary CI runs during release automation..github/workflows/semantic-release.ymlworkflow, which previously handled all release steps in one file, in favor of the new modular approach.These changes collectively make the release process more robust, modular, and maintainable, while ensuring that versioning, publishing, and dependency updates are tightly controlled and automated.