Repository navigation
fix(ui): capture TextureView and SurfaceView pixels in Android screenshots - #257
YevheniiKotyrlo wants to merge 2 commits into
Conversation
|
@YevheniiKotyrlo is attempting to deploy a commit to the Callstack Team on Vercel. A member of the Team first needs to authorize it. |
EvidenceBefore / after.
On Effects only the GPU renderer draws. A red 80 dp view in a white 160 dp area, the area captured as an element, on a nubia NX809J (Android 16), the playground's arm64 debug build with
Capture cost.
Galaxy S23 (SM-S911B, Android 14), the playground's arm64 debug build, at thermal status 0 before and after the series. The three sides run the same JavaScript, and only the APK's |
…shots The capture drew the view tree into a software bitmap, where a TextureView draws nothing and a SurfaceView leaves the hole its window keeps over it. From Android 8 an on-screen region is copied with PixelCopy: the window's surface, and behind it the frame of each visible SurfaceView in the region, scaled into the box the screen shows it in. A copy that times out acquiring its frame is requested again, up to four times; any other failed copy, and a capture while the window is not shown, resolves null. A window with elevation on Android 8.0 and 8.1 is drawn from its view tree with its SurfaceViews behind it, and below Android 8 the view tree is drawn as before, both where the window shows it. An element partly off the screen, scaled to nothing or detached from the window is drawn by itself, at its scroll offset. A view's box in the window follows its own and its ancestors' transforms. The playground gets a TextureView, SurfaceViews painted from the CPU and one playing a video, a spec that screenshots them, and Android baselines at the E2E emulator's geometry.
5a517c2 to
9327773
Compare
# Conflicts: # packages/ui/android/src/main/java/com/harnessui/UIHelperImpl.kt
What is this?
On Android,
screen.screenshot()now shows what a TextureView or a SurfaceView displays, such as a WebGPU or Skia canvas, a video or a camera preview, instead of transparent pixels (#197). Before and after on the E2E workflow's emulator, and what the copy costs (on a Galaxy S23, 0.4 ms more per element and 16 ms more per whole-window capture), in this comment.How does it work?
The capture drew the view tree into a software bitmap, where a TextureView draws nothing and a SurfaceView leaves the hole its window keeps over it. From Android 8 an on-screen region is copied with
PixelCopy: the window's surface, which holds what TextureViews render, and the frame of each visible SurfaceView in the region, scaled into the box the screen shows it in whatever size its producer queues it at, as a video decoder does, and drawn behind the window's copy. Only the part of the region on the window is copied, at its offset. A SurfaceView with no frame yet, or a protected one, adds nothing; a copy that times out acquiring its frame is requested again, up to four times; and any other copy that fails resolvesnullrather than a different image, as does a capture while the window is not shown, such as with the screen off. A window with elevation on Android 8.0 and 8.1, wherePixelCopydoes not offset the rect by its surface insets, is drawn from its view tree with the SurfaceViews behind it, and below Android 8 the capture draws the view tree as onmain; both software draws place it where the window shows it, through the root view's transform and the window's pan. An element partly off the screen, scaled to nothing or detached from the window is still drawn by itself, at its scroll offset, so a TextureView or a SurfaceView in it still comes out transparent.A view's box in the window follows its own and its ancestors' transforms, as
convertRect:toView:gives it on iOS, and is rounded as the renderer places a SurfaceView, so an element under a transform is captured in the box it covers on the screen. A capture now shows what the window shows, so effects only the GPU renderer draws, such as elevation shadows andfilterblurs, appear in it, and a baseline recorded without them needs recording again. Two cases stay out of reach. A SurfaceView set above its window withsetZOrderOnTopis left out, and overlapping SurfaceViews are stacked in the order of the view tree, where the screen puts one set withsetZOrderMediaOverlayon top and the rest in the order their surfaces were created: before Android 16 only hidden API tells the first two apart, the module compiles against an earlier SDK by default, and no API reports the creation order. The host-sideadb exec-out screencapthat #197 also names has neither limit, at the cost of a round trip to the host per screenshot.The playground gets a TextureView and three SurfaceViews: two painted from the CPU, one in a single colour and one split in two, and one playing a short video split the same way, whose frames are larger than the view. Its spec screenshots each view; a row where the SurfaceView sits at an offset under a view drawn over part of it; the split SurfaceView under scales of 0.5, 0.35 and 1.5, under a 4 dp translation, and its right half alone; the video; a region running past the window's edge; and elements scaled to nothing, under a moved root view, under nested rotations, clipped out of the view tree and scrolled while partly off the screen. It also checks that the window is still captured while a SurfaceView is scaled to nothing. The Android baselines are at the E2E emulator's geometry, and on my fork E2E Android passes against them, merged with the current
main(run).Why is this useful?
Screenshot tests of GPU-rendered views on Android compare real pixels: a canvas, a video or a map that renders wrong now fails its test, where before both the baseline and the capture held transparent pixels.
Fixes #197