mirror of
https://seed.radicle.dev/rad:z4V1sjrXqjvFdnCUbxPFqd5p4DtH5.git
synced 2026-11-21 14:20:42 +08:00
httpd: Build release archives in CI
- Add build/ci to build the archives of an annotated release tag - Run it on the large_linux_arm Buildkite queue - Keep the archives as artifacts and show checksums in an annotation - Pass --load to docker, as a remote builder keeps the image in its cache - Add build/fetch to download the archives of the CI release job - Pick the job by its step key, so a failed test elsewhere does not hide it - Add build/compare to check a build against another - Document the CI build, the download and the comparison
This commit is contained in:
@@ -0,0 +1,24 @@
|
||||
# Builds the radicle-httpd and radicle-search release archives from a
|
||||
# `releases/httpd/<version>` tag, and keeps them as Buildkite artifacts. A
|
||||
# delegate downloads them with explorer/crates/radicle-httpd/build/fetch, then
|
||||
# signs and uploads them, see explorer/RELEASE.md.
|
||||
#
|
||||
# garden-broker builds a tag push like a branch push: BUILDKITE_BRANCH is the
|
||||
# tag name and BUILDKITE_COMMIT the tag object. build/ci checks that the build
|
||||
# really is of an annotated tag with that name.
|
||||
#
|
||||
# The step runs on the agent rather than in a container, since the build itself
|
||||
# runs in Docker.
|
||||
|
||||
steps:
|
||||
- label: "📦 httpd release build"
|
||||
key: httpd-release-build
|
||||
if: build.branch =~ /^releases\/httpd\//
|
||||
# The build runs on linux/arm64, see explorer/crates/radicle-httpd/build/build.
|
||||
agents:
|
||||
queue: large_linux_arm
|
||||
timeout_in_minutes: 90
|
||||
artifact_paths:
|
||||
- "explorer/crates/radicle-httpd/build/artifacts/*"
|
||||
commands:
|
||||
- explorer/crates/radicle-httpd/build/ci
|
||||
@@ -13,6 +13,10 @@
|
||||
# on the `explorer-check` key defined there, and Buildkite rejects a step whose
|
||||
# dependency is not yet part of the build.
|
||||
#
|
||||
# A push of a `releases/httpd/<version>` tag also starts a build, with the tag
|
||||
# name as the branch. explorer-release.yml only runs then; the other suites run
|
||||
# too, against the tagged commit.
|
||||
#
|
||||
# Superseded builds are not cancelled. Buildkite has no YAML equivalent of the
|
||||
# GitHub Actions cancel-in-progress.
|
||||
|
||||
@@ -23,3 +27,4 @@ steps:
|
||||
- buildkite-agent pipeline upload .buildkite/explorer-e2e.yml
|
||||
- buildkite-agent pipeline upload .buildkite/explorer-visual.yml
|
||||
- buildkite-agent pipeline upload .buildkite/explorer-rust.yml
|
||||
- buildkite-agent pipeline upload .buildkite/explorer-release.yml
|
||||
|
||||
@@ -5,6 +5,7 @@ node_modules/
|
||||
NOTES
|
||||
config/local*
|
||||
crates/radicle-httpd/build/artifacts
|
||||
crates/radicle-httpd/build/artifacts-ci
|
||||
target
|
||||
target-docker
|
||||
|
||||
|
||||
+46
-15
@@ -34,8 +34,11 @@ for which releases explorer supports.
|
||||
git config set gpg.format ssh
|
||||
git config set user.signingKey "$(rad self --ssh-key)"
|
||||
```
|
||||
- Working `podman` setup (see [Podman Setup](#podman-setup) below)
|
||||
- Required tools: `podman`, `rad`, `sha256sum`
|
||||
- Working `podman` setup (see [Podman Setup](#podman-setup) below), or Docker
|
||||
- Required tools: `podman` or `docker`, `rad`, `sha256sum`, `curl`, `jq`,
|
||||
`ssh`, `scp`, `ssh-keygen`
|
||||
- To download the CI build: a Buildkite API token with `read_builds` and
|
||||
`read_artifacts`
|
||||
|
||||
## Prepare
|
||||
|
||||
@@ -83,38 +86,65 @@ Normal patches add their highlights to `## Unreleased` in
|
||||
rad sync --announce
|
||||
```
|
||||
|
||||
## Build and upload
|
||||
## Build
|
||||
|
||||
Do these steps after the tag is canonical.
|
||||
Pushing the tag starts the `📦 httpd release build` step in Buildkite. It builds
|
||||
the archives in Docker and keeps them as build artifacts. You can also build
|
||||
them locally. Either way, one build is enough to publish, but a second build in
|
||||
another environment is recommended: it shows that neither environment changed
|
||||
the archives.
|
||||
|
||||
1. Check out the tag. The build reads the version from it with `git describe`:
|
||||
1. Check out the tag. The scripts read the version from it with `git describe`:
|
||||
|
||||
```sh
|
||||
git switch --detach releases/httpd/<version>
|
||||
```
|
||||
|
||||
2. Run the reproducible build. A single run compiles the workspace and packages
|
||||
**both** binaries, one archive per binary per target. It uses podman; set
|
||||
`CONTAINER=docker` to use Docker:
|
||||
2. Get the archives in one of these ways:
|
||||
- Download the CI build. This needs a Buildkite API token with `read_builds`
|
||||
and `read_artifacts`:
|
||||
|
||||
```sh
|
||||
crates/radicle-httpd/build/build
|
||||
```
|
||||
```sh
|
||||
export BUILDKITE_API_TOKEN=<token> BUILDKITE_ORG=<org> BUILDKITE_PIPELINE=<pipeline>
|
||||
crates/radicle-httpd/build/fetch <version>
|
||||
```
|
||||
|
||||
This writes to `crates/radicle-httpd/build/artifacts/`:
|
||||
- Build locally. A single run compiles the workspace and packages **both**
|
||||
binaries, one archive per binary per target. It uses podman; set
|
||||
`CONTAINER=docker` to use Docker:
|
||||
|
||||
```sh
|
||||
crates/radicle-httpd/build/build
|
||||
```
|
||||
|
||||
Both write to `crates/radicle-httpd/build/artifacts/`:
|
||||
- `radicle-httpd-<version>-<target>.tar.xz` (binary + man page)
|
||||
- `radicle-search-<version>-<target>.tar.xz` (binary)
|
||||
|
||||
each with a `.sha256` alongside.
|
||||
|
||||
3. Sign the archives with your Radicle key. This checks each archive against its
|
||||
3. Recommended: compare a local build with the CI build. `build` overwrites
|
||||
`artifacts/`, so build first, then fetch the CI build into another folder:
|
||||
|
||||
```sh
|
||||
crates/radicle-httpd/build/build
|
||||
crates/radicle-httpd/build/fetch -o crates/radicle-httpd/build/artifacts-ci <version>
|
||||
crates/radicle-httpd/build/compare crates/radicle-httpd/build/artifacts-ci
|
||||
```
|
||||
|
||||
Do not publish if the archives differ. `compare` takes any folder of archives
|
||||
with their `.sha256`, so the build of another delegate works too.
|
||||
|
||||
## Publish
|
||||
|
||||
1. Sign the archives with your Radicle key. This checks each archive against its
|
||||
`.sha256` and writes a `.sig` alongside:
|
||||
|
||||
```sh
|
||||
crates/radicle-httpd/build/sign
|
||||
```
|
||||
|
||||
4. Upload. This publishes each binary into its own sibling folder on
|
||||
2. Upload. This publishes each binary into its own sibling folder on
|
||||
[files.radicle.dev][files] and creates the version-less and `latest` symlinks
|
||||
for both:
|
||||
|
||||
@@ -171,7 +201,8 @@ Do these steps after the tag is canonical.
|
||||
> - macOS binaries are unsigned and not notarized; download via CLI only
|
||||
> - The build always runs in a `linux/arm64` container, with the base images and
|
||||
> zig pinned, so that builds on different hosts give the same bytes. It runs
|
||||
> natively on Apple silicon, and under emulation on amd64 hosts
|
||||
> natively on Apple silicon and on the `large_linux_arm` Buildkite queue, and
|
||||
> under emulation on amd64 hosts
|
||||
|
||||
## Podman Setup
|
||||
|
||||
|
||||
@@ -24,6 +24,9 @@ main() {
|
||||
;;
|
||||
docker)
|
||||
engine=docker
|
||||
# A remote builder, as in Buildkite, only keeps the image in its cache
|
||||
# unless asked to load it into Docker.
|
||||
load=--load
|
||||
;;
|
||||
*)
|
||||
echo "fatal: CONTAINER must be podman or docker, not '$CONTAINER'" >&2
|
||||
@@ -62,9 +65,9 @@ main() {
|
||||
echo "Building image ($image) with $CONTAINER.."
|
||||
|
||||
# Build on linux/arm64 on every host, so that builds on different hosts
|
||||
# give the same bytes. Apple silicon runs it natively; amd64 hosts run it
|
||||
# under emulation.
|
||||
$engine build \
|
||||
# give the same bytes. Apple silicon and the Buildkite release queue run it
|
||||
# natively; amd64 hosts run it under emulation.
|
||||
$engine build $load \
|
||||
--build-arg "RUST_VERSION=$rust_version" \
|
||||
--build-arg SOURCE_DATE_EPOCH \
|
||||
--build-arg TZ \
|
||||
|
||||
Executable
+45
@@ -0,0 +1,45 @@
|
||||
#!/bin/sh
|
||||
set -e
|
||||
|
||||
# Builds the release archives in Buildkite, from a `releases/httpd/<version>`
|
||||
# tag. garden-broker passes the tag name as BUILDKITE_BRANCH and the tag object
|
||||
# as BUILDKITE_COMMIT; Buildkite sets no BUILDKITE_TAG.
|
||||
|
||||
tag="$BUILDKITE_BRANCH"
|
||||
|
||||
case "$tag" in
|
||||
releases/httpd/*) ;;
|
||||
*)
|
||||
echo "fatal: '$tag' is not a releases/httpd/<version> tag" >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
# Reject a branch that only has the name of a release tag.
|
||||
if [ "$(git cat-file -t "$BUILDKITE_COMMIT")" != tag ]; then
|
||||
echo "fatal: $BUILDKITE_COMMIT is not an annotated tag" >&2
|
||||
exit 1
|
||||
fi
|
||||
name="$(git cat-file tag "$BUILDKITE_COMMIT" | sed -n 's/^tag //p')"
|
||||
if [ "$name" != "$tag" ]; then
|
||||
echo "fatal: tag object $BUILDKITE_COMMIT is named '$name', not '$tag'" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
version="${tag#releases/httpd/}"
|
||||
if ! echo "$version" | grep -Eq '^[0-9]+\.[0-9]+\.[0-9]+$'; then
|
||||
echo "fatal: '$version' is not a MAJOR.MINOR.PATCH version" >&2
|
||||
exit 1
|
||||
fi
|
||||
export RADICLE_HTTPD_VERSION="$version"
|
||||
|
||||
CONTAINER=docker "$(dirname "$0")/build"
|
||||
|
||||
checksums="$("$(dirname "$0")/checksums" | sort -k2)"
|
||||
buildkite-agent annotate --style info --context httpd-release <<EOF
|
||||
radicle-httpd + radicle-search $version
|
||||
|
||||
\`\`\`
|
||||
$checksums
|
||||
\`\`\`
|
||||
EOF
|
||||
Executable
+38
@@ -0,0 +1,38 @@
|
||||
#!/bin/sh
|
||||
set -e
|
||||
|
||||
# Compares the checksums of the archives in `artifacts/` with the ones in
|
||||
# another folder, for example a CI build from `fetch -o`, or a build of
|
||||
# another delegate.
|
||||
|
||||
if [ $# -ne 1 ]; then
|
||||
echo "Usage: $0 <dir>" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
ours="$(dirname "$0")/artifacts"
|
||||
theirs="$1"
|
||||
|
||||
sums() {
|
||||
if ! ls "$1"/*.tar.xz.sha256 >/dev/null 2>&1; then
|
||||
echo "fatal: no checksums in $1" >&2
|
||||
exit 1
|
||||
fi
|
||||
cat "$1"/*.tar.xz.sha256 | sort -k2
|
||||
}
|
||||
|
||||
a="$(sums "$ours")"
|
||||
b="$(sums "$theirs")"
|
||||
|
||||
if [ "$a" = "$b" ]; then
|
||||
echo "$a"
|
||||
echo
|
||||
echo "The archives in $ours and $theirs are the same."
|
||||
else
|
||||
echo "fatal: the archives differ ('<' $ours, '>' $theirs):" >&2
|
||||
printf '%s\n' "$a" >"${TMPDIR:-/tmp}/compare.$$.a"
|
||||
printf '%s\n' "$b" >"${TMPDIR:-/tmp}/compare.$$.b"
|
||||
diff "${TMPDIR:-/tmp}/compare.$$.a" "${TMPDIR:-/tmp}/compare.$$.b" >&2 || true
|
||||
rm -f "${TMPDIR:-/tmp}/compare.$$.a" "${TMPDIR:-/tmp}/compare.$$.b"
|
||||
exit 1
|
||||
fi
|
||||
Executable
+81
@@ -0,0 +1,81 @@
|
||||
#!/bin/sh
|
||||
set -e
|
||||
|
||||
# Downloads the release archives that Buildkite built from the tag
|
||||
# `releases/httpd/<version>`, see .buildkite/explorer-release.yml.
|
||||
|
||||
usage() {
|
||||
echo "Usage: $0 [-o <dir>] <version>" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
outdir="$(dirname "$0")/artifacts"
|
||||
while getopts o: opt; do
|
||||
case "$opt" in
|
||||
o) outdir="$OPTARG" ;;
|
||||
*) usage ;;
|
||||
esac
|
||||
done
|
||||
shift $((OPTIND - 1))
|
||||
[ $# -eq 1 ] || usage
|
||||
version="$1"
|
||||
|
||||
for tool in curl jq sha256sum; do
|
||||
if ! command -v "$tool" >/dev/null; then
|
||||
echo "fatal: $tool is not installed" >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
|
||||
for var in BUILDKITE_API_TOKEN BUILDKITE_ORG BUILDKITE_PIPELINE; do
|
||||
if eval "[ -z \"\$$var\" ]"; then
|
||||
echo "fatal: set $var; the token needs read_builds and read_artifacts" >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
|
||||
api="https://api.buildkite.com/v2/organizations/$BUILDKITE_ORG/pipelines/$BUILDKITE_PIPELINE"
|
||||
|
||||
get() {
|
||||
curl -sSf -H "Authorization: Bearer $BUILDKITE_API_TOKEN" "$@"
|
||||
}
|
||||
|
||||
# The other suites run on the tag too, so look for a passed release job rather
|
||||
# than a passed build: a flaky test elsewhere must not hide the archives.
|
||||
match="$(get "$api/builds?branch=releases/httpd/$version" | jq -c '
|
||||
first(.[] | . as $build | .jobs[]?
|
||||
| select(.step_key == "httpd-release-build" and .state == "passed")
|
||||
| {number: $build.number, web_url: $build.web_url, job: .id}) // empty')"
|
||||
if [ -z "$match" ]; then
|
||||
echo "fatal: no passed httpd-release-build job for releases/httpd/$version" >&2
|
||||
exit 1
|
||||
fi
|
||||
number="$(echo "$match" | jq -r .number)"
|
||||
web_url="$(echo "$match" | jq -r .web_url)"
|
||||
job="$(echo "$match" | jq -r .job)"
|
||||
|
||||
echo "Downloading the archives of $web_url into $outdir.."
|
||||
mkdir -p "$outdir"
|
||||
|
||||
get "$api/builds/$number/jobs/$job/artifacts?per_page=100" |
|
||||
jq -r '.[] | select(.path | startswith("explorer/crates/radicle-httpd/build/artifacts/")) | "\(.filename) \(.download_url)"' |
|
||||
while read -r filename url; do
|
||||
echo "$filename"
|
||||
# The API redirects to storage that rejects the token, so follow the
|
||||
# redirect without it.
|
||||
location="$(get -o /dev/null -w '%{redirect_url}' "$url")"
|
||||
curl -sSf -o "$outdir/$filename" "$location"
|
||||
done
|
||||
|
||||
for target in $(cat "$(dirname "$0")/TARGETS"); do
|
||||
for bin in radicle-httpd radicle-search; do
|
||||
filename="$bin-$version-$target.tar.xz"
|
||||
if [ ! -f "$outdir/$filename" ]; then
|
||||
echo "fatal: the build has no $filename" >&2
|
||||
exit 1
|
||||
fi
|
||||
(cd "$outdir" && sha256sum --check --quiet "$filename.sha256")
|
||||
done
|
||||
done
|
||||
|
||||
echo "Fetch successful."
|
||||
Reference in New Issue
Block a user