feat(ext/canvas): enhance `createImageBitmap` specification compliance #25517

Hajime-san · 2024-09-08T22:17:24Z

Feature

resolves #26032

basic support for decoding image format

jpeg, png, gif, bmp, ico, webp
- animated image
- 16-bit image
- Grayscale image

issues for supporting AVIF

The AVIF image format is supported by all browsers today.
However, the standardization of sniffing mime type seems to have hard going.

Missing image codecs whatwg/mimesniff#143

support `colorSpace` for `ImageData`

~~There can convert color space from srgb to display-p3 with createImageBitmap.~~

This was a misunderstanding on my part, so I reverted it.
The conversion using the ImageData colorSpace option was probably the responsibility of CanvasRenderingContext2D.
I've left the implementation commented out as it may be useful as a reference for implementing CanvasRenderingContext2D or OffscreenCanvasRenderingContext2D.

support `colorSpaceConversion` option

The specifications are not particularly clear about the implementation details, and I had a hard time understanding its behavior, but I eventually do reverse engineering the results of wpt and implemented it to be consistent with those results.

createImageBitmap: Clarify treatment of colorSpaceConversion:"none" ImageBitmaps whatwg/html#10578

port to Rust

To simplify code, almost logic moved to Rust side.

added a simple architecture document for README

It is basically based on the image implementation, and I wrote roughly how you can handle data effectively.

performance impact

I was curious to see if the increased amount of Rust code, especially static dispatch, was affecting the bootstrap time of the runtime.
I'm not very confident about the suitability of this benchmark.

bootstrap

console.log('Hello, world!');

this pr fb6b87d

% hyperfine --warmup 5 'target/release/deno run bootstrap.ts'
Benchmark 1: target/release/deno run bootstrap.ts
  Time (mean ± σ):      21.5 ms ±   3.6 ms    [User: 14.2 ms, System: 5.5 ms]
  Range (min … max):    18.7 ms …  54.5 ms    102 runs
 
  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

main b1b72a8

% hyperfine --warmup 5 'target/release/deno run bootstrap.ts'
Benchmark 1: target/release/deno run bootstrap.ts
  Time (mean ± σ):      22.2 ms ±   4.7 ms    [User: 14.8 ms, System: 5.4 ms]
  Range (min … max):    18.8 ms …  66.8 ms    97 runs
 
  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

API benchmark

const imageData = new ImageData(new Uint8ClampedArray([255, 0, 0, 255]), 1, 1);

Deno.bench("createImageBitmap", async () => {
  await createImageBitmap(imageData);
})

this pr acc4119

% target/release/deno bench bench.ts
    CPU | Apple M1
Runtime | Deno 2.0.0-rc.2 (aarch64-apple-darwin)

file:///path/to/deno/bench.ts

benchmark           time/iter (avg)        iter/s      (min … max)           p75      p99     p995
------------------- ----------------------------- --------------------- --------------------------
createImageBitmap            1.6 µs       615,000 (  1.6 µs …   1.7 µs)   1.6 µs   1.7 µs   1.7 µs

main b1b72a8

% target/release/deno bench bench.ts
    CPU | Apple M1
Runtime | Deno 1.46.1 (aarch64-apple-darwin)

file:///path/to/deno/bench.ts

benchmark           time/iter (avg)        iter/s      (min … max)           p75      p99     p995
------------------- ----------------------------- --------------------- --------------------------
createImageBitmap            3.5 µs       287,200 (  3.3 µs …   3.6 µs)   3.5 µs   3.6 µs   3.6 µs

extra

The benchmarks below are for when the argument to op_create_image_bitmap is a struct except for the first argument of buf.
It seems that serializing the argument to op as a struct has a fairly large impact on performance.
This may apply to things other than op_create_image_bitmap as well.

% target/release/deno bench bench.ts
    CPU | Apple M1
Runtime | Deno 2.0.0-rc.2 (aarch64-apple-darwin)

file:///path/to/deno/bench.ts

benchmark           time/iter (avg)        iter/s      (min … max)           p75      p99     p995
------------------- ----------------------------- --------------------- --------------------------
createImageBitmap            4.2 µs       235,900 (  4.1 µs …   4.5 µs)   4.3 µs   4.5 µs   4.5 µs

plans for enhancement in another PRs

improve performance
There is known to that processing image is a cpu-bound task. So, op_create_image_bitmap will be async fn and consider to be data-parallelism with using rayon.
align error message
- Normalize error message grammar (TS/JS) #25269
- refactor(ext): align error messages #25310

…xcept for JPG and animated image - https://mimesniff.spec.whatwg.org/#matching-an-image-type-pattern

- adding `DOMExceptionInvalidStateError` in global due to handling not supported image format - animation image is not supported currently

petamoriken · 2024-09-29T20:05:12Z

ext/canvas/image_ops.rs

+#[cfg(test)]
+mod tests {


It would be better to add a test for EXIF orientation

Thanks!
I added some unit tests that based on wpt.
It seems that image crate will soon release an API for image orientation, so we will be able to pass tests correctly with using it.

https://github.com/image-rs/image/blob/bd0f7451a367de7ae3d898dcf1e96e9d0a1c4fa1/CHANGES.md#version-0253

Fix for orientation issue of JPEG images image-rs/image#1958

BTW, it seems that there are various difficulties with imageOrientation.

Add createImageBitMaps's imageOrientation "none" whatwg/html#8085

Clarify how ImageBitmap interacts with EXIF orientation and image-orientation property whatwg/html#7210

Behavior of createImageBitmap with { imageOrientation: "flipY" } is unclear whatwg/html#8118

crowlKats · 2024-10-02T13:00:00Z

@Hajime-san FYI I'll review this PR after we release 2.0; not enough time to do it beforehand, especially since its such a big PR

Hajime-san · 2024-10-03T02:12:01Z

ext/web/01_mimesniff.js

+ * NOTE: Some browsers have implementation-defined image formats.
+ * For example, The AVIF image format is supported by all browsers today.
+ * However, the standardization seems to have hard going.
+ * See: https://github.com/whatwg/mimesniff/issues/143
 * @param {Uint8Array} input
 * @returns {string | undefined}
 */


This question is unrelated to quoted the line.
I think that the sniffImage function should receives another argument which is a byte sequence as Uint8Array and passes to imageTypePatternMatchingAlgorithm.
It seems that the imageTypeMatched variable will always become undefined with the current implementation.

deno/ext/web/01_mimesniff.js

Lines 418 to 441 in c7cba4e

/**

* Ref: https://mimesniff.spec.whatwg.org/#rules-for-sniffing-images-specifically

* @param {string} mimeTypeString

* @returns {string}

*/

function sniffImage(mimeTypeString) {

const mimeType = parseMimeType(mimeTypeString);

if (mimeType === null) {

return mimeTypeString;

}

if (isXML(mimeType)) {

return mimeTypeString;

}

const imageTypeMatched = imageTypePatternMatchingAlgorithm(

new TextEncoder().encode(mimeTypeString),

);

if (imageTypeMatched !== undefined) {

return imageTypeMatched;

}

return mimeTypeString;

}

Hajime-san · 2024-10-04T00:18:19Z

ext/canvas/01_image.js

+    // TODO(Hajime-san): this should be real async
+    const processedImage = op_create_image_bitmap(


Initially, I thought it would be necessary to make it an async op, but when I actually implemented it that seemed to reduce performance compared to fb6b87d #25517 (comment).
In this case, is it not necessary to make it an async op?🤔

% target/release/deno bench bench.ts CPU | Apple M1 Runtime | Deno 2.0.0-rc.9 (aarch64-apple-darwin) file:///path/to/deno/bench.ts benchmark time/iter (avg) iter/s (min … max) p75 p99 p995 ------------------- ----------------------------- --------------------- -------------------------- createImageBitmap 4.8 µs 208,800 ( 4.7 µs … 5.1 µs) 4.8 µs 5.1 µs 5.1 µs

Hajime-san · 2024-10-06T08:09:54Z

Note for supporting AVIF: I've implemented AVIF support using the AvifDecoder here 111f00b, although it only supports 8-bit currently.
As mentioned in the overview of this PR, sniffing MIME about AVIF is an ongoing issue, but I believe this implementation could support it without deviating from the spec.
The AvifDecoder depends on dav1d and seems to require additional tools to build locally and in CI.

Build Failure log

% cargo b   
   ...
   Compiling dav1d-sys v0.8.2
error: failed to run custom build command for `dav1d-sys v0.8.2`

Caused by:
  process didn't exit successfully: `/path/to/deno/target/debug/build/dav1d-sys-7acb1adeb047a692/build-script-build` (exit status: 101)
  --- stdout
  cargo:rerun-if-env-changed=DAV1D_NO_PKG_CONFIG
  cargo:rerun-if-env-changed=PKG_CONFIG_aarch64-apple-darwin
  cargo:rerun-if-env-changed=PKG_CONFIG_aarch64_apple_darwin
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG
  cargo:rerun-if-env-changed=PKG_CONFIG
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH_aarch64-apple-darwin
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH_aarch64_apple_darwin
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_PATH
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR_aarch64-apple-darwin
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR_aarch64_apple_darwin
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_LIBDIR
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_aarch64-apple-darwin
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_aarch64_apple_darwin
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_SYSROOT_DIR
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR

  --- stderr
  thread 'main' panicked at /path/to/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dav1d-sys-0.8.2/build.rs:82:10:
  called `Result::unwrap()` on an `Err` value: PkgConfig(
  pkg-config exited with status code 1
  > PKG_CONFIG_ALLOW_SYSTEM_CFLAGS=1 pkg-config --libs --cflags dav1d dav1d >= 1.3.0

  The system library `dav1d` required by crate `dav1d-sys` was not found.
  The file `dav1d.pc` needs to be installed and the PKG_CONFIG_PATH environment variable must contain its parent directory.
  The PKG_CONFIG_PATH environment variable is not set.

  HINT: if you have installed the library, try setting PKG_CONFIG_PATH to the directory containing `dav1d.pc`.
  )
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

% uname -a
Darwin user.local 23.5.0 Darwin Kernel Version 23.5.0: Wed May  1 20:16:51 PDT 2024; root:xnu-10063.121.3~5/RELEASE_ARM64_T8103 arm64

% brew install dav1d
% cargo b

Hajime-san added 21 commits August 29, 2024 17:35

basic support all image type pattern of unsigned 8bit when its blob e…

198c90a

…xcept for JPG and animated image - https://mimesniff.spec.whatwg.org/#matching-an-image-type-pattern

support 8bit JPG

b9fe5e8

handling unsuportted situation

77703e8

- adding `DOMExceptionInvalidStateError` in global due to handling not supported image format - animation image is not supported currently

fmt

22cf2f9

cleanup premultiplyAlpha and adding test

32d71f5

handling unsupported bit depth

827e7b1

port to rust

0de8a9b

support animated PNG

d0da79e

support animated WebP

d255518

support animated GIF

396c09c

support colorspaceConversion for ImageData

7d3e39d

support colorspaceConversion for Blob

a6604c1

clean up unit test

1e21467

clean up

1f7524a

add unit test

98e9304

update README

4bc517e

Merge remote-tracking branch 'upstream/main' into update-imagebitmap

dbac50d

fix error handling

f0895c6

update README

86f0000

revert "support colorspaceConversion for ImageData"

75d26df

switch to static link for lcms2

1de527f

Hajime-san force-pushed the update-imagebitmap branch from a234586 to 1de527f Compare September 17, 2024 01:56

Hajime-san added 2 commits September 17, 2024 12:47

Merge branch 'main' into update-imagebitmap

825515e

add comments for the reference of image mimesniff

bd77fb9

Hajime-san force-pushed the update-imagebitmap branch from 055395a to bd77fb9 Compare September 21, 2024 07:04

marvinhagemeister requested a review from littledivy September 23, 2024 13:37

Hajime-san added 4 commits September 29, 2024 10:19

clean up process_icc_profile_conversion

2d58fd1

clean up

d258914

fmt

fa01320

avoid copying vector

8d8d726

Hajime-san added 3 commits September 29, 2024 13:53

image_ops takes responsibility of treat alpha channel

fc4064e

avoid copying string

fb6b87d

perf: return the value not a struct but a tuple

3beb65b

petamoriken reviewed Sep 29, 2024

View reviewed changes

crowlKats self-requested a review September 29, 2024 20:56

Hajime-san added 3 commits October 2, 2024 18:55

add unit test for imageOrientation

829f523

add comments for the reference of issues about imageOrientation

9a9664e

Merge branch 'main' into update-imagebitmap

f3aa376

Hajime-san mentioned this pull request Oct 2, 2024

Blob type set to empty string causing InvalidStateError #25971

Closed

update error message

5e52b86

Hajime-san commented Oct 3, 2024

View reviewed changes

handle 32-bit depth

1b545f9

Hajime-san force-pushed the update-imagebitmap branch from d96a794 to 1b545f9 Compare October 3, 2024 22:11

Hajime-san commented Oct 4, 2024

View reviewed changes

clean up

977026c

petamoriken mentioned this pull request Oct 5, 2024

createImageBitmap from blob [v.2.0.0-rc10] #26032

Open

fix

69addc8

Hajime-san force-pushed the update-imagebitmap branch from 111f00b to 69addc8 Compare October 6, 2024 07:35

perf: pass u8 to op

acc4119

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(ext/canvas): enhance `createImageBitmap` specification compliance #25517

feat(ext/canvas): enhance `createImageBitmap` specification compliance #25517

Hajime-san commented Sep 8, 2024 •

edited

Loading

petamoriken Sep 29, 2024

Hajime-san Oct 2, 2024

Hajime-san Oct 2, 2024

crowlKats commented Oct 2, 2024

Hajime-san Oct 3, 2024

Hajime-san Oct 4, 2024 •

edited

Loading

Hajime-san commented Oct 6, 2024

	/**
	* Ref: https://mimesniff.spec.whatwg.org/#rules-for-sniffing-images-specifically
	* @param {string} mimeTypeString
	* @returns {string}
	*/
	function sniffImage(mimeTypeString) {
	const mimeType = parseMimeType(mimeTypeString);
	if (mimeType === null) {
	return mimeTypeString;
	}

	if (isXML(mimeType)) {
	return mimeTypeString;
	}

	const imageTypeMatched = imageTypePatternMatchingAlgorithm(
	new TextEncoder().encode(mimeTypeString),
	);
	if (imageTypeMatched !== undefined) {
	return imageTypeMatched;
	}

	return mimeTypeString;
	}

		// TODO(Hajime-san): this should be real async
		const processedImage = op_create_image_bitmap(

		#[cfg(test)]
		mod tests {

feat(ext/canvas): enhance createImageBitmap specification compliance #25517

Are you sure you want to change the base?

feat(ext/canvas): enhance createImageBitmap specification compliance #25517

Conversation

Hajime-san commented Sep 8, 2024 • edited Loading

Feature

basic support for decoding image format

issues for supporting AVIF

support colorSpace for ImageData

support colorSpaceConversion option

port to Rust

added a simple architecture document for README

performance impact

bootstrap

API benchmark

extra

plans for enhancement in another PRs

petamoriken Sep 29, 2024

Choose a reason for hiding this comment

Hajime-san Oct 2, 2024

Choose a reason for hiding this comment

Hajime-san Oct 2, 2024

Choose a reason for hiding this comment

crowlKats commented Oct 2, 2024

Hajime-san Oct 3, 2024

Choose a reason for hiding this comment

Hajime-san Oct 4, 2024 • edited Loading

Choose a reason for hiding this comment

Hajime-san commented Oct 6, 2024

feat(ext/canvas): enhance `createImageBitmap` specification compliance #25517

feat(ext/canvas): enhance `createImageBitmap` specification compliance #25517

Hajime-san commented Sep 8, 2024 •

edited

Loading

support `colorSpace` for `ImageData`

support `colorSpaceConversion` option

Hajime-san Oct 4, 2024 •

edited

Loading