Running benchmarks on AWS (#28)

.github/workflows/build.yaml

@@ -4,6 +4,7 @@ on:
   pull_request:
     paths:
       - 'binder/**.yml'
+      - 'binder/Dockerfile'
       - 'binder/apt.txt'
 
 concurrency:

.github/workflows/main.yml

@@ -49,6 +49,10 @@ jobs:
         run: |
           playwright install
 
+      - name: Run tests
+        run: |
+          pytest -v -s
+
       - name: Run test benchmarks
         if: github.event_name != 'workflow_dispatch'
         run: |

README.md

@@ -17,6 +17,32 @@
 
 ## Running the benchmarks
 
+The repository contains a set of benchmarks that can be run locally or remotely using `coiled`. The benchmarks are run using the `main.py` script. The script takes the following arguments:
+
+```bash
+$ python main.py --help
+usage: main.py [-h] [--runs RUNS] [--detect-provider] [--approach APPROACH] [--dataset DATASET]
+               [--zarr-version ZARR_VERSION] [--non-headless] [--s3-bucket S3_BUCKET] [--action ACTION]
+               [--zoom-level ZOOM_LEVEL]
+
+options:
+  -h, --help            show this help message and exit
+  --runs RUNS           Number of runs to perform
+  --detect-provider     Detect provider
+  --approach APPROACH   Approach to use. Must be one of: ['direct-client']
+  --dataset DATASET     dataset name. Must be one of: ['1MB-chunks', '5MB-chunks', '10MB-chunks', '25MB-chunks']
+  --zarr-version ZARR_VERSION
+                        Zarr version. Must be one of: ['v2', 'v3']
+  --non-headless        Run in non-headless mode
+  --s3-bucket S3_BUCKET
+                        S3 bucket name
+  --action ACTION       Action to perform. Must be one of: ['zoom_in', 'zoom_out']
+  --zoom-level ZOOM_LEVEL
+                        Zoom level
+```
+
+### Local
+
 To run the benchmarks, you will need to install `playwright` and software packages specified in `binder/environment.yml`. This can be done by running the following commands:
 
 ```bash
@@ -31,13 +57,15 @@ playwright install
 Once the environment is set up, you can run the benchmarks by running the following command:
 
 ```bash
-python main.py
+python main.py --dataset 1MB-chunks --zarr-version v2 --action zoom_in --zoom-level 4 --s3-bucket s3://carbonplan-benchmarks
 ```
 
+### Remote via Coiled
+
 To run the benchmark using `coiled`, you can run the following command:
 
 ```bash
-coiled run --container quay.io/carbonplan/benchmark-maps --file main.py bash main.sh
+coiled run --gpu --container quay.io/carbonplan/benchmark-maps --file main.py bash main.sh
 ```
 
 ## license

binder/Dockerfile

@@ -1,12 +1,22 @@
 # Inherit from an upstream image
 FROM jupyter/base-notebook:2023-05-15
 
+# Copy necessary files
 COPY binder/environment.yml /tmp/environment.yml
 COPY binder/apt.txt /tmp/apt.txt
 
-RUN mamba env update --prefix ${CONDA_DIR} --file /tmp/environment.yml
+# Switch to root user to install apt packages
+USER root
 
+# Install apt packages and clean up
+RUN apt-get update && \
+    xargs -a /tmp/apt.txt apt install -y && \
+    apt-get autoremove -y && \
+    apt-get autoclean && \
+    rm -rf /var/lib/apt/lists/* && \
+    rm /tmp/apt.txt
 
-USER root
-RUN apt-get update && xargs -a /tmp/apt.txt apt install -y && rm -r /var/lib/apt/lists/*
-USER 1001
+# Install conda packages and clean up
+USER jovyan
+RUN mamba env update --prefix ${CONDA_DIR} --file /tmp/environment.yml && \
+    mamba clean --all -f -y

binder/apt.txt

@@ -16,3 +16,8 @@ libxdamage1
 libxfixes3
 libxkbcommon0
 libxrandr2
+xorg
+xserver-xorg
+xvfb
+libx11-dev
+libxext-dev

main.py

@@ -78,6 +78,7 @@ async def run(
     headless: bool = False,
 ):
     # Launch browser and create new page
+    # https://chromium.googlesource.com/chromium/src/+/master/ui/gl/gl_switches.cc
     chrome_args = [
         '--enable-features=Vulkan,UseSkiaRenderer',
         '--enable-unsafe-webgpu',

main.sh

@@ -1,4 +1,6 @@
 #!/bin/bash
 set -e
+
+
 playwright install
-python main.py
+pytest -v -s

tests/test_gpu.py

@@ -0,0 +1,28 @@
+import platform
+
+from playwright.sync_api import sync_playwright
+
+# get the OS name
+os_name = platform.system()
+chrome_args = ['--enable-unsafe-webgpu', '--ignore-gpu-blocklist', '--disable-software-rasterizer']
+
+if os_name == 'Linux':
+    chrome_args.extend(
+        ['--enable-features=Vulkan,UseSkiaRenderer', '--use-angle=vulkan', '--enable-gpu']
+    )
+
+
+def test_gpu_hardware_acceleration():
+    with sync_playwright() as p:
+        browser = p.chromium.launch(args=chrome_args)
+        page = browser.new_page()
+        page.goto('chrome://gpu')
+        feature_status_list = page.query_selector('.feature-status-list')
+        assert 'Hardware accelerated' in feature_status_list.inner_text()
+        # Check if webGL is enabled
+        print(feature_status_list.inner_text())
+        assert 'OpenGL: Enabled' in feature_status_list.inner_text()
+        assert 'WebGL: Hardware accelerated' in feature_status_list.inner_text()
+        assert 'WebGL2: Hardware accelerated' in feature_status_list.inner_text()
+        assert 'WebGPU: Hardware accelerated' in feature_status_list.inner_text()
+        browser.close()